rgw: add cls _bucket_resharding_set and cls_bucket_resharding_clear

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add resharding flag and the new bucket instance id to RGWBucketInfo

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add reshard add, list, execute and cancel commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: move resharding into a separate function

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add reshard_pool

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add cls_rgw_reshard ops

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15119 from ceph/wip-rgw-config-docs

doc: mention certain conf vars should be in global

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14351 from yehudasa/wip-rgw-mdsearch

rgw: metadata search part 2

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15427 from tchaikov/wip-vstart-osd-id

vstart: print "start osd.$id" instead of "start osd$id"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15217 from badone/wip-async-sleep-timer-fix

osd: Move scrub sleep timer to osdservice

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15396 from jcsp/wip-ceph-status

mon: revise "ceph status" output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15011 from xiexingguo/wip-crush-tool

tools/crushtool: replicated-rule API support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15176 from SUSE/wip-mgr-fix-help

mgr,osd:  ceph-mgr --help, unify usage text of other daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15395 from liewegas/wip-log-entity

common/LogEntry: include EntityName in log entries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15398 from liewegas/wip-bluestore-leak

os/bluestore: fix buffers pinned by indefinitely deferred writes

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15402 from liewegas/wip-bluestore-mempools

os/bluestore: improve mempool usage

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15403 from liewegas/wip-mempool-dump

mempool: improve dump; fix buffer accounting bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15431 from rzarzynski/wip-rgw-vim-modeline-for-orhpantool

rgw: add the Vim's modeline into rgw_orphan.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15425 from yaozongyou/fix-build-error

cmake: link against fcgi only if enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add the Vim's modeline into rgw_orphan.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

vstart: print "start osd.$id" instead of "start osd$id"

"osd.9" is more consistent with other places where an osd is referenced.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15408 from yonghengdexin735/wip-zzz-add_const

include/lru.h: add const to member functions

Reviewed-by: Jos Collin <jcollin@redhat.com>

Fix cmake build error.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #15420 from yuriw/wip_remove_default_idle_timeout

qa: Removed all 'default_idle_timeout' due to chnage in rwg task

Merge pull request #15397 from dzafman/wip-20089

osd: Object level shard errors are tracked and used if no auth available

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/cephtool/test.sh: adjust watch test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15417 from smithfarm/wip-cond-epoch-2

build/ops: rpm: move _epoch_prefix below Epoch definition

include/lru: add const to member function

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Removed all 'default_idle_timeout' due to chnage in rwg task
https://github.com/ceph/ceph/commit/8c74c8a639cd3dead52e1942b36f6eb3f1ceab2c#diff-995b04809fcabacc3e3ecfaea903a41aL539

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tasks/rgw_multisite.py: adjust zone init

zone is now a ZoneConn object. Also, change import to make it relative
so that qa task can locate it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

build/ops: rpm: move _epoch_prefix below Epoch definition

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15400 from yehudasa/wip-20128

qa/rgw_snaps: move default_idle_timeout config under the client

osd: Object level shard errors are tracked and used if no auth available

Shards with object mismatch are tracked to mark them inconsistent
Fix test because storing omap_digest in object_info not behaving as before

Fixes: http://tracker.ceph.com/issues/20089
Signed-off-by: David Zafman <dzafman@redhat.com>

osd-scrub-repair: Fix broken test and correctly handle ec overwrite testing

Caused by: 3ca750d41dfe33c6efea4abc96d2bd426a9742b9

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15212 from neha-ojha/wip-fix-recovery-sleep-configurable

osd: Implement asynchronous recovery sleep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15262 from Adirl/revert_cm

msg/async: Revert RDMA-CM

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15307 from ovh/bp-64bit-mem-is-zero

inline_memory: optimized mem_is_zero for non-x64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15340 from wjwithagen/wip-wjw-libcommon

cmake: do not link libcommon against some libs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15342 from chardan/jfw-wip-add_VLA_warning

build: Adds C++ warning flag for C Variable-Length Arrays.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15364 from joscollin/wip-cleanup-putto-operator-const

src: put-to operator function - const input cleanup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15372 from ganeshmaharaj/reuse-isal

erasure-code: Remove duplicate of isa-l files

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: release aios and pinned buffers on io complete

Once we're done with our IO, clear the aio list so that the pinned buffers
are unpinned.  This ensures we release memory quickly, even if the
TransContext sticks around for a while (e.g., in the osr q).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: submit deferred if txc cleanup is blocked

If we have a single deferred write, and then a uniform workload with *no*
deferred writes, we will never actually submit it.  Meanwhile, the txc is
stuck on the osr q and nothing ever gets retired.

Simple fix is to submit any deferred ops if the osr queue is blocked by
a queued deferred write and the osr queue length is above some
threshold.  This prevents memory from being pinned indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15209 from liewegas/wip-sortbitwise-notes

doc/release-notes: update which jewel version does sortbitwise warning

Merge pull request #15274 from liewegas/wip-auto-ccache

do_cmake.sh: enable ccache if installed

Reviewed-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: enable ccache if installed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15391 from liewegas/wip-authors

AUTHORS: update with release manager, backport team

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15410 from dachary/wip-19941-ceph-disk

ceph-disk: do not setup_statedir on trigger

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15412 from dachary/wip-choose-args

crush: add missing tunable in tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

crush: add missing tunable in tests

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15286 from smithfarm/wip-cond-epoch

build/ops: rpm: apply epoch only if %epoch macro is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15240 from ukernel/wip-20055

osdc/Journaler: avoid executing on_safe contexts prematurely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15282 from david-z/wip-fix-mantle

mds: Pass empty string to clear mantle balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15314 from renhwztetecs/renhw-wip-client-cleanup

client: fix display ino in the ldout

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15300 from ukernel/wip-20083

mds: properly create aux subtrees for pinned directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15278 from viCEnTECheNg/wip-data-pools-type

mds: change the type of data_pools

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15205 from batrick/i20039

mds: check export pin during replay

Reviewed-by: Yan, Zheng <zyan@redhat.com>

ceph-disk: do not setup_statedir on trigger

trigger may run when statedir is unavailable and does not use it.

Fixes: http://tracker.ceph.com/issues/19941
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15406 from tchaikov/wip-setuptools-vendors-six-or-not

install-deps.sh: workaround setuptools' dependency on six

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps.sh: workaround setuptools' dependency on six

see also https://github.com/pypa/setuptools/issues/1042

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: mempools for writing and writing_deferred

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_txc mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename cache mempools to bluestore_cache_*

These map directly onto the bluestore_cache_size setting.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: maybe_rebuild before adding to cache too

Signed-off-by: Sage Weil <sage@redhat.com>

AUTHORS: update with release manager, backport team

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15370 from liewegas/wip-jewel-tunables

crush, mon: make jewel the lower bound for client/crush compat for new clusters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/cli/osdmaptool/: fix tests

CRUSH tunables now default to jewel; outputs vary and
upmap results are a bit different.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix some append_buffer vs mempool assignment; add tests

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: make dump more concise

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool: adjust tests

Specify chooseleaf_stable for --build tests to match prior output.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: set mon_osd_initial_require_min_compat_client = jewel

IMO a jewel lower bound is reasonable for new clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: set default tunables to jewel

For new clusters.

This is the last "disruptive" tunable change we've made (where adjusting/
fixing it requires lots of data movement).  For new clusters a lower bound
of jewel seems pretty reasonable, and this has been upstream in Linux
for a while now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15267 from joscollin/wip-cleanup-redundant-headers-5

common: Remove redundant includes - 5

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa/rgw_snaps: move default_idle_timeout config under the client

Fixes: http://tracker.ceph.com/issues/20128
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15311 from dachary/wip-crush-add-remove

crush: update choose_args when items are added/removed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15318 from tchaikov/wip-20067

mon: trim the creating_pgs after updating it with pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15352 from liewegas/wip-buffer-mempools

buffer: allow buffers to be accounted in arbitrary mempools

Merge pull request #15357 from ceph/typo_bluestore

os/bluestore: fix a typo about bleustore

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15385 from myoungwon/wip-20114

test/osd/TestRados.cc: run set-redirect test after finishing setup

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "osd: When scrub finds an attr error mark shard inconsistent"

This reverts commit 44736aac74493b5532c9e3574a37a3642766f212.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14457 from ceph/wip-rest-flask

ceph-mgr: Implement new pecan-based rest api

rgw: fix import

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon: revise "ceph status" output

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15222 from guihecheng/rgw_file-fix-unlink

rgw_file: fix flags set on unsuccessful unlink

Merge pull request #15329 from guihecheng/rgw_file-readdir

rgw_file: add lock protection for readdir against gc

Merge pull request #14871 from ceph/wip-selinux-optimize

selinux: Do parallel relabel on package install

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

mempool: dump total, too

Signed-off-by: Sage Weil <sage@redhat.com>

osd: (loosely) mark cached map bl buffers as osd_mapbl

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: mark buffer caches buffers as bluestore_data

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: remove buffer_data mempool

We are explicitly accounting for buffers via arbitrary pools, defaulting
to buffer_anon, so we don't need buffer_data anymore.  It leads to
double-counting.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: allow buffers to map into arbitrary mempools

- default is buffer_anon
- buffer_data includes *all* buffers across all mempools, and is the
*actual* allocation (includes raw_combined).
- buffer_meta is other buffer-related overhead.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: allow pool counts to be adjusted explicitly

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix mempool accounting for claimed chars

These are allocated by the caller, outside of the mempool; do not
deallocate via the mempool.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: kill unittest_? mempools

These were just annoying.  Use some real ones that will be empty
for the unit tests.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15291 from liewegas/wip-osdmap-pgtemp

osd/OSDMap: make pg_temp more efficient

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15321 from liewegas/wip-failed-msg

mon/OSDMonitor: change info in 'osd failed' messages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/rados: add monitor_log2

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add rados_monitor_log2 that includes EntityName

Signed-off-by: Sage Weil <sage@redhat.com>

librados: use EntityName in monitor_log line

Signed-off-by: Sage Weil <sage@redhat.com>