rgw: fix update_bucket_id, reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: add rgw-admin reshard status command and initial blocking logic

A command that returns the reshard status field on the bucket index objects (all
shards).
Also, initial implementation of identifying and blocking while reshard in
progress.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more detailed resharding status kept in bucket header

So that we can differentiate between in-progress and complete status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mark source bucket shards when starting reshard

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move resharding code to BucketReshard class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: split per-bucket resharding logic out from general resharding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revert some reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: create an objclass method to guard in-progress resharding

This method will be called on the bucket index shard.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWReshard update to new cls API

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: adjust types and ops

 - don't keep data bufferlist in struct cls_rgw_bucket_instance_entry
 - remove rgw_get_bucket_instance

We need to keep the reshard info as part of the bucket index header (for
each source bucket shard), and not in a separate object. Otherwise it'll
be impossible to have the resharding check atomic as part of the regular
bucket index updates.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: minor changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: generate_reshard_key, change output param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: return default header if not initialized

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cls_reshard_get remove unneeded check for bufferlist length

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: get_bucket_header will return -ENOENT when there is no header

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add guard class for bucket index lock

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add reshard_pool_ctx

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for exisiting entries in cls_rgw_reshard_add

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use pointer for is_truncated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cls_rgw_reshard use helper function the generate key

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: limit op.max and add extra entry to handle tuncation in cls_rgw_reshard_list

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use .rgw.log:reshard pool for resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove RGWReshard::get_head

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: replace dout with ldout

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove cls_reshard_get_head

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: store resharding information in bucket header

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: handle correctly when there is no bucket or it doesn't have resharding flag

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: block on bucket index operation when resharding the index

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard object to RGWRados

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add lock/unlock bucket instance and block_while_resharding methods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add cls_rgw_get_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: implement reshard execute command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radsogw-admin add create_new_bucket_instance

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use const string for oid in get_bucket_instance_from_oid

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard::clear_bucket_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard::set_bucket_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add missing handles for set_resharding, clear_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add cls _bucket_resharding_set and cls_bucket_resharding_clear

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add resharding flag and the new bucket instance id to RGWBucketInfo

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: Add reshard add, list, execute and cancel commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: move resharding into a separate function

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add reshard_pool

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add cls_rgw_reshard ops

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #15119 from ceph/wip-rgw-config-docs

doc: mention certain conf vars should be in global

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14351 from yehudasa/wip-rgw-mdsearch

rgw: metadata search part 2

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #15427 from tchaikov/wip-vstart-osd-id

vstart: print "start osd.$id" instead of "start osd$id"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15217 from badone/wip-async-sleep-timer-fix

osd: Move scrub sleep timer to osdservice

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15396 from jcsp/wip-ceph-status

mon: revise "ceph status" output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15011 from xiexingguo/wip-crush-tool

tools/crushtool: replicated-rule API support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15176 from SUSE/wip-mgr-fix-help

mgr,osd:  ceph-mgr --help, unify usage text of other daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15395 from liewegas/wip-log-entity

common/LogEntry: include EntityName in log entries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15398 from liewegas/wip-bluestore-leak

os/bluestore: fix buffers pinned by indefinitely deferred writes

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15402 from liewegas/wip-bluestore-mempools

os/bluestore: improve mempool usage

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15403 from liewegas/wip-mempool-dump

mempool: improve dump; fix buffer accounting bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15431 from rzarzynski/wip-rgw-vim-modeline-for-orhpantool

rgw: add the Vim's modeline into rgw_orphan.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15425 from yaozongyou/fix-build-error

cmake: link against fcgi only if enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add the Vim's modeline into rgw_orphan.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

vstart: print "start osd.$id" instead of "start osd$id"

"osd.9" is more consistent with other places where an osd is referenced.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15408 from yonghengdexin735/wip-zzz-add_const

include/lru.h: add const to member functions

Reviewed-by: Jos Collin <jcollin@redhat.com>

Fix cmake build error.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #15420 from yuriw/wip_remove_default_idle_timeout

qa: Removed all 'default_idle_timeout' due to chnage in rwg task

Merge pull request #15397 from dzafman/wip-20089

osd: Object level shard errors are tracked and used if no auth available

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/cephtool/test.sh: adjust watch test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15417 from smithfarm/wip-cond-epoch-2

build/ops: rpm: move _epoch_prefix below Epoch definition

include/lru: add const to member function

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Removed all 'default_idle_timeout' due to chnage in rwg task
https://github.com/ceph/ceph/commit/8c74c8a639cd3dead52e1942b36f6eb3f1ceab2c#diff-995b04809fcabacc3e3ecfaea903a41aL539

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tasks/rgw_multisite.py: adjust zone init

zone is now a ZoneConn object. Also, change import to make it relative
so that qa task can locate it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

build/ops: rpm: move _epoch_prefix below Epoch definition

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15400 from yehudasa/wip-20128

qa/rgw_snaps: move default_idle_timeout config under the client

osd: Object level shard errors are tracked and used if no auth available

Shards with object mismatch are tracked to mark them inconsistent
Fix test because storing omap_digest in object_info not behaving as before

Fixes: http://tracker.ceph.com/issues/20089
Signed-off-by: David Zafman <dzafman@redhat.com>

osd-scrub-repair: Fix broken test and correctly handle ec overwrite testing

Caused by: 3ca750d41dfe33c6efea4abc96d2bd426a9742b9

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15212 from neha-ojha/wip-fix-recovery-sleep-configurable

osd: Implement asynchronous recovery sleep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15262 from Adirl/revert_cm

msg/async: Revert RDMA-CM

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15307 from ovh/bp-64bit-mem-is-zero

inline_memory: optimized mem_is_zero for non-x64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15340 from wjwithagen/wip-wjw-libcommon

cmake: do not link libcommon against some libs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15342 from chardan/jfw-wip-add_VLA_warning

build: Adds C++ warning flag for C Variable-Length Arrays.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15364 from joscollin/wip-cleanup-putto-operator-const

src: put-to operator function - const input cleanup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15372 from ganeshmaharaj/reuse-isal

erasure-code: Remove duplicate of isa-l files

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: release aios and pinned buffers on io complete

Once we're done with our IO, clear the aio list so that the pinned buffers
are unpinned.  This ensures we release memory quickly, even if the
TransContext sticks around for a while (e.g., in the osr q).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: submit deferred if txc cleanup is blocked

If we have a single deferred write, and then a uniform workload with *no*
deferred writes, we will never actually submit it.  Meanwhile, the txc is
stuck on the osr q and nothing ever gets retired.

Simple fix is to submit any deferred ops if the osr queue is blocked by
a queued deferred write and the osr queue length is above some
threshold.  This prevents memory from being pinned indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15209 from liewegas/wip-sortbitwise-notes

doc/release-notes: update which jewel version does sortbitwise warning

Merge pull request #15274 from liewegas/wip-auto-ccache

do_cmake.sh: enable ccache if installed

Reviewed-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: enable ccache if installed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15391 from liewegas/wip-authors

AUTHORS: update with release manager, backport team

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15410 from dachary/wip-19941-ceph-disk

ceph-disk: do not setup_statedir on trigger

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15412 from dachary/wip-choose-args

crush: add missing tunable in tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

crush: add missing tunable in tests

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15286 from smithfarm/wip-cond-epoch

build/ops: rpm: apply epoch only if %epoch macro is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15240 from ukernel/wip-20055

osdc/Journaler: avoid executing on_safe contexts prematurely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15282 from david-z/wip-fix-mantle

mds: Pass empty string to clear mantle balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15314 from renhwztetecs/renhw-wip-client-cleanup

client: fix display ino in the ldout

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15300 from ukernel/wip-20083

mds: properly create aux subtrees for pinned directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15278 from viCEnTECheNg/wip-data-pools-type

mds: change the type of data_pools

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15205 from batrick/i20039

mds: check export pin during replay

Reviewed-by: Yan, Zheng <zyan@redhat.com>

ceph-disk: do not setup_statedir on trigger

trigger may run when statedir is unavailable and does not use it.

Fixes: http://tracker.ceph.com/issues/19941
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15406 from tchaikov/wip-setuptools-vendors-six-or-not

install-deps.sh: workaround setuptools' dependency on six

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps.sh: workaround setuptools' dependency on six

see also https://github.com/pypa/setuptools/issues/1042

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: mempools for writing and writing_deferred

Signed-off-by: Sage Weil <sage@redhat.com>