Merge pull request #15502 from xiexingguo/wip-fix-overload-extents

os/bluestore: avoid overloading extents during reshard; atomic deferred_batch_ops

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15390 from liewegas/wip-osdmap-state

osd/OSDMap: make osd_state 32 bits wide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15515 from tchaikov/wip-crc32-aarch64

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

Reviewed-by: Dan Mick <dmick@redhat.com>

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

HAVE_ARMV8_CRC is defined by "acconfig.h". so we need to include it for
the definition.

it's a regresssion introduced by 2a3382f. in that change,
crc32c_intel_baseline.h was removed from crc32.cc, and that header
includes accconfig.h indirectly. that's why we will have a dummy
ceph_crc32c_aarch64() which always returns 0. the dummy implementation
was intended to please the compiler. but we can probably just drop it in
future as a cleanup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15511 from tchaikov/wip-doc-cartesian-product

doc: update sample explaning "%" operator in test suites

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #14074 from jecluis/wip-mon-osd-replacement

mon: replace osds with `osd destroy` and `osd new`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15487 from liewegas/wip-no-more-stat-spam

mgr/DaemonServer: stop spamming log with pg stats

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15494 from liewegas/wip-20187

qa/workunits/cephtool/test.sh: fix osd full health detail grep

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update sample explaning "%" operator in tes suites

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15499 from jcsp/wip-doc-whitespace

doc: fix syntax on code snippets in cephfs/multimds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15492 from dillaman/wip-20185

rbd: properly decode features when using image name optional

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15483 from dillaman/wip-20182

librbd: filter expected error codes from is_exclusive_lock_owner

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15490 from ceph/wip-fix-selinux

selinux: Install ceph-base before ceph-selinux

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15415 from dillaman/wip-rbd-valgrind

librbd: fix valgrind errors and ensure tests detect future leaks

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15140 from dillaman/wip-18963

rbd-mirror: permit release of local image exclusive lock after force promotion

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: reduce overhead when trying to unshard blobs

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: atomic deferred_batch_ops

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: avoid overloading of extents during reshard

This is potentially big stuff...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15484 from Adirl/wip

msg/async/rdma: Add DSCP support

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15493 from yehudasa/wip-rgw-dynamic-sharding

rgw: dynamic resharding

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: add missing members initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14239 from liewegas/wip-bluestore-ec-clone

os/bluestore: try to unshare blobs for EC overwrite workload

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15299 from joscollin/wip-cleanup-redundant-headers-6

common: Remove redundant includes - 6

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15429 from jcsp/wip-nuke-rest

pybind/mgr: Delete `rest` module

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix a race

We referred to complete_op_data after aio call that could have released
it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/cli/radosgw-admin: update reference help.t

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix index completion

Was missing bucket complete op, only had the guard (bad merge?).
Pass zones_trace as pointer and treat it as optional param (which
it is).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: increase max shard prime, consolidate logic

Handle larger num of shards (max ~64k now). Use the same hash logic for shard ids
everywhere.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new num shards proportional to number of objects per shard

and also limit number of shards to not exceed the max.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle zones_trace in bi async completion handler

needed following a rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove debug helper code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ignore -ENOENT on bi_list()

just treat it as success with empty result

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: run resharding thread only on master zone

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add check to see if out is null

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw:  update entry in resharding log when resharding has started

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: start resharding theard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw-admin: radosgw-admin reshard execute -> reshard process

to be in line with other radosgw-admin commands.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update bucket instance with resharding details

can be useful info for later in a cross zone sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: reshard execute output adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: remove unused field in reshard entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard: renew lease and handle marker when listing reshard repo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix reshard_list truncation identification

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: expose api to generate reshard indexing key

So that we could use it as a marker when doing reshard_list.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded and duplicate code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize bucket structure

after reading bucket info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: revive reshard execute command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket_id does not include bucket name

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bi_list() initialize is_truncated if -ENOENT

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: internal api cleanup

rename method, minor change

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: reshard add command doesn't read existing status

No need to read existing status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: fix remove_reshard objclass op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: modify logshard key

The logshard key shouldn't include bucket id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: multiple fixes and adjustments related to resharding scheduler

still wip, but:
 - get rid of some unneeded rados locking
 - rename shards to logshards where relevant to avoid confusion
 - remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix compilation following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: replace reshard blocking sleep with interruptible condition wait

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: actually reshard an entry from the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check if buckets needs resharding on all ops that add new objects

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add bucket to resharding queue if needs resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for bucket resharding  when adding an object

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a function to add a bucket to the resharding queue

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add a check_bucket_shards to the quota

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: moved reshard_bucket into RGWReshard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove cct from RGWReshard and BucketIndexLockGuard

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add resharding thread

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: store new num shards in bucket index reshard entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix deletion of index completion object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard complete handling, use multiple locks

to reduce contention. Also, fix completions cleanup.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clean up index completions on shutdown

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update data log after index completion reshard handling

Also rearrange code a bit.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: guard resharding in async completions

Catch return value from async index completions, resend operations
synchronously on a separate worker thread if needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reshard guard for versioned operations

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: changes to support reshard guard on olh bi ops

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle reshard wait logic in cancel() too

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix update_bucket_id, reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: add rgw-admin reshard status command and initial blocking logic

A command that returns the reshard status field on the bucket index objects (all
shards).
Also, initial implementation of identifying and blocking while reshard in
progress.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more detailed resharding status kept in bucket header

So that we can differentiate between in-progress and complete status.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mark source bucket shards when starting reshard

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move resharding code to BucketReshard class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: split per-bucket resharding logic out from general resharding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revert some reshard blocking logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: create an objclass method to guard in-progress resharding

This method will be called on the bucket index shard.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWReshard update to new cls API

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cls/rgw: adjust types and ops

 - don't keep data bufferlist in struct cls_rgw_bucket_instance_entry
 - remove rgw_get_bucket_instance

We need to keep the reshard info as part of the bucket index header (for
each source bucket shard), and not in a separate object. Otherwise it'll
be impossible to have the resharding check atomic as part of the regular
bucket index updates.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: minor changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: generate_reshard_key, change output param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/rgw: return default header if not initialized

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: cls_reshard_get remove unneeded check for bufferlist length

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: get_bucket_header will return -ENOENT when there is no header

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add guard class for bucket index lock

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add reshard_pool_ctx

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: check for exisiting entries in cls_rgw_reshard_add

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use pointer for is_truncated

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: cls_rgw_reshard use helper function the generate key

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: limit op.max and add extra entry to handle tuncation in cls_rgw_reshard_list

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use .rgw.log:reshard pool for resharding

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: remove RGWReshard::get_head

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: replace dout with ldout

Signed-off-by: Orit Wasserman <owasserm@redhat.com>