os/bluestore: reduce overhead when trying to unshard blobs

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: atomic deferred_batch_ops

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: avoid overloading of extents during reshard

This is potentially big stuff...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15493 from yehudasa/wip-rgw-dynamic-sharding

rgw: dynamic resharding

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: add missing members initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14239 from liewegas/wip-bluestore-ec-clone

os/bluestore: try to unshare blobs for EC overwrite workload

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15299 from joscollin/wip-cleanup-redundant-headers-6

common: Remove redundant includes - 6

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15429 from jcsp/wip-nuke-rest

pybind/mgr: Delete `rest` module

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix a race

We referred to complete_op_data after aio call that could have released
it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/cli/radosgw-admin: update reference help.t

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix index completion

Was missing bucket complete op, only had the guard (bad merge?).
Pass zones_trace as pointer and treat it as optional param (which
it is).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: increase max shard prime, consolidate logic

Handle larger num of shards (max ~64k now). Use the same hash logic for shard ids
everywhere.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new num shards proportional to number of objects per shard

and also limit number of shards to not exceed the max.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle zones_trace in bi async completion handler

needed following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove debug helper code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ignore -ENOENT on bi_list()

just treat it as success with empty result

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: run resharding thread only on master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add check to see if out is null

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw:  update entry in resharding log when resharding has started

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: start resharding theard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw-admin: radosgw-admin reshard execute -> reshard process

to be in line with other radosgw-admin commands.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket instance with resharding details

can be useful info for later in a cross zone sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: reshard execute output adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: remove unused field in reshard entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard: renew lease and handle marker when listing reshard repo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix reshard_list truncation identification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: expose api to generate reshard indexing key

So that we could use it as a marker when doing reshard_list.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded and duplicate code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize bucket structure

after reading bucket info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: revive reshard execute command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket_id does not include bucket name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bi_list() initialize is_truncated if -ENOENT

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: internal api cleanup

rename method, minor change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: reshard add command doesn't read existing status

No need to read existing status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix remove_reshard objclass op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: modify logshard key

The logshard key shouldn't include bucket id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes and adjustments related to resharding scheduler

still wip, but:
 - get rid of some unneeded rados locking
 - rename shards to logshards where relevant to avoid confusion
 - remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: actually reshard an entry from the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check if buckets needs resharding on all ops that add new objects

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for bucket resharding  when adding an object

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a function to add a bucket to the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a check_bucket_shards to the quota

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: moved reshard_bucket into RGWReshard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove cct from RGWReshard and BucketIndexLockGuard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add resharding thread

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: store new num shards in bucket index reshard entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix deletion of index completion object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard complete handling, use multiple locks

to reduce contention. Also, fix completions cleanup.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clean up index completions on shutdown

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update data log after index completion reshard handling

Also rearrange code a bit.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard resharding in async completions

Catch return value from async index completions, resend operations
synchronously on a separate worker thread if needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard guard for versioned operations

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: changes to support reshard guard on olh bi ops

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle reshard wait logic in cancel() too

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix update_bucket_id, reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: add rgw-admin reshard status command and initial blocking logic

A command that returns the reshard status field on the bucket index objects (all
shards).
Also, initial implementation of identifying and blocking while reshard in
progress.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more detailed resharding status kept in bucket header

So that we can differentiate between in-progress and complete status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mark source bucket shards when starting reshard

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move resharding code to BucketReshard class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: split per-bucket resharding logic out from general resharding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revert some reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: create an objclass method to guard in-progress resharding

This method will be called on the bucket index shard.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWReshard update to new cls API

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: adjust types and ops

 - don't keep data bufferlist in struct cls_rgw_bucket_instance_entry
 - remove rgw_get_bucket_instance

We need to keep the reshard info as part of the bucket index header (for
each source bucket shard), and not in a separate object. Otherwise it'll
be impossible to have the resharding check atomic as part of the regular
bucket index updates.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: minor changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: generate_reshard_key, change output param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: return default header if not initialized

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cls_reshard_get remove unneeded check for bufferlist length

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: get_bucket_header will return -ENOENT when there is no header

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add guard class for bucket index lock

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add reshard_pool_ctx

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for exisiting entries in cls_rgw_reshard_add

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use pointer for is_truncated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cls_rgw_reshard use helper function the generate key

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: limit op.max and add extra entry to handle tuncation in cls_rgw_reshard_list

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use .rgw.log:reshard pool for resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove RGWReshard::get_head

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: replace dout with ldout

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove cls_reshard_get_head

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: store resharding information in bucket header

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: handle correctly when there is no bucket or it doesn't have resharding flag

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: block on bucket index operation when resharding the index

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard object to RGWRados

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add lock/unlock bucket instance and block_while_resharding methods

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add cls_rgw_get_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radosgw-admin: implement reshard execute command

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

radsogw-admin add create_new_bucket_instance

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use const string for oid in get_bucket_instance_from_oid

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard::clear_bucket_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add RGWReshard::set_bucket_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add missing handles for set_resharding, clear_resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #14657 from chardan/jfw-wip-halflife_atomic_t-mothra

messenger,client,compressor: migrate atomic_t to std::atomic

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15380 from xiexingguo/wip-cache-trim

os/bluestore: move cache_trim into MempoolThread

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15409 from Liuchang0812/wip-support-mon-target-in-pybind

pybind: support mon target in pybind

Reviewed-by: Kefu Chai <kchai@redhat.com>