build/ops: rpm: selinux-policy fixes

Requiring the file /usr/share/selinux/devel/policyhelp breaks fc27 and
Rawhide builds as they do not have that file. This exposed the fact this
code had not worked in some time due to changes in selinux policy
modules packaging. See
https://bugzilla.redhat.com/show_bug.cgi?id=999584

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17977 from cbodley/wip-rgw-beast-qa

qa/rgw: add beast frontend to some rgw suites

Merge PR #18600 into master

* refs/pull/18600/head:
MDSMonitor: only clog changes to active

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

githubmap: update contributors

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #18774 into master

* refs/pull/18774/head:
mds: fix CDir::log_mark_dirty()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18887 from dillaman/wip-20789

librbd: compare and write against a clone can result in failure

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18924 from ceph/wip-cv-filestore-devices

ceph-volume: allow using a device or partition for `lvm --data`

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18700 from tianshan/fix_cls_bucket_head

rgw: fix cls_bucket_head result order consistency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18926 from theanalyst/wip-mp-jewel-fix

rgw: set sync_from_all as true when no value is seen

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume tests.functional create a common playbook directory for setups

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume lvm prepare can accept devices for --data

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add data device and journal to xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional tox.ini addition of setup playbook for partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw data device to centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add setup playbook for centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add raw device to centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common update --data flag to reflect device acceptance

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18969 from vshankar/rbd-mirror-image-map-state-transition

rbd-mirror: use next transition state to check transition completeness

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18960 from Liuchang0812/mailmap-update

.mailmap, .organizationmap: update JingChen, ZongyouYao, ShanchunLv's…

test/librbd: disable compare and write if skipping partial discard

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor object write request state machines

Fixes: http://tracker.ceph.com/issues/20789
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: consolidate all object discard-related logic into single state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify interface between object IO and copyup state machines

The initial copyup was not receiving a write hint and the
code for hints was duplicated multiple times. Additionally,
the object map state should match the last executed write op.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: copyup state machine needs to handle empty write ops

The compare-and-write object operation cannot be executed
concurrently within a copyup operation since the object might not
exist yet (if not performing a deep-copy).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: object cacher should re-use read state machine

This adds support for sparse-reads and ensures all object reads
utilize a single, tested code path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refactor io::ObjectReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: default template parameter for C_SparseReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce lock scope when assembling read results

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ObjectReadRequest does not require buffer extents

The ReadResult wrapper requires the buffer extents to know
how to properly reconstruct the out buffer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: templatize IO object request state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: mocked remainder of IO ops used by librbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: simulate set_alloc_hint creating a non-existent file

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: use next transition state to check transition completeness

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #18952 from dillaman/wip-test-librbd

test/librbd: clean up for several mock function tests

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18947 from dillaman/wip-22131

rbd-nbd: certain kernels may not discover resized block devices

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18909 from dillaman/wip-22120

librbd: possible deadlock with synchronous maintenance operations

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18923 from dillaman/wip-21966

librbd: prevent overflow of discard API result code

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

Merge pull request #18918 from ZVampirEM77/wip-em-rewrite-cmd

rgw: add rewrite cmd and options into radosgw-admin usage and doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

.mailmap, .organizationmap: update JingChen, ZongyouYao, ShanchunLv's mailmap and affiliation

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: add rewrite related cmd and parameter into radosgw-admin help info

+ radosgw-admin object rewrite
+ radosgw-admin bucket rewrite
+ --min-rewrite-size
+ --max-rewrite-size
+ --min-rewrite-stripe-size

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Merge pull request #18808 from tchaikov/wip-qa-max-pg-per-osd

qa: do not wait for down/out osd for pg convergence

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15788 from vshankar/mirror-ha-poolwatcher-policy

rbd-mirror: track images in policy map in support of A/A

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18953 from cbodley/wip-22139

common: fix BoundedKeyCounter const_pointer_iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

common: fix BoundedKeyCounter const_pointer_iterator

with libc++, clang fails to compile a call to:

  vector::assign(const_pointer_iterator, const_pointer_iterator)

because const_pointer_iterator does not satisfy the InputIterator
concept. added the necessary typedefs for value_type and reference
to reflect the pointer type

Fixes: http://tracker.ceph.com/issues/22139
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/librbd: fixed gmock uninteresting function call warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: eliminate sleeps within deep-copy mock tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18883 from cbodley/wip-22083+

rgw: set num_shards on 'radosgw-admin data sync init'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: set num_shards on 'radosgw-admin data sync init'

Fixes: http://tracker.ceph.com/issues/22083
Signed-off-by: Casey Bodley <cbodley@redhat.com>

dencoder/rgw: expose rgw sync status types

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: added missing locks for snap rollback and rename

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18940 from trociny/wip-valgrind-DeadLockOwner

test/librbd: valgrind warning in TestMockManagedLockBreakRequest.DeadLockOwner

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17761 from cbodley/wip-rgw-bilog-trim

rgw multisite: automated trimming for bucket index logs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #18811 from oritwas/wip-rgw-22046

tenant fixes for dynamic resharding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18920 from yaozongyou/wip-ceph-dencoder-add-types

ceph-dencoder: add support for cls_rgw_lc_obj_head

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18946 from joscollin/wip-add-cbodley

.githubmap: Add cbodley

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18900 from qrGitHub/master

rgw.iam: change '1' to '1ULL' in function print_actions

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

librbd: invalidating the cache shouldn't hold write lock

This can cause deadlock when readahead is in-progress since neither
can make forward progress.

Fixes: http://tracker.ceph.com/issues/22131
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: rescan partition table after image resize event

Fixes: http://tracker.ceph.com/issues/22131
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

.githubmap: Add cbodley

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18921 from Leeshine/wip-data-sync-cmd

rgw: add metadata and data sync related cmd into radosgw-admin usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

rgw: add data sync related cmd into radosgw-admin help info

+ radosgw-admin data sync status
+ radosgw-admin data sync init
+ radosgw-admin data sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

rgw: add metadata sync related cmd into radosgw-admin help info

+ radosgw-admin metadata sync status
+ radosgw-admin metadata sync init
+ radosgw-admin metadata sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

librbd: prevent overflow of discard API result code

Prevent discard/writesame lengths larger than 2GB.

Fixes: http://tracker.ceph.com/issues/21966
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: valgrind warning in TestMockManagedLockBreakRequest.DeadLockOwner

When running with 'debug_rbd >= 20':

  Conditional jump or move depends on uninitialised value(s)
    by 0x94277B: operator<< (ostream:167)
    by 0x94277B: librbd::managed_lock::BreakRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_watchers(int) (BreakRequest.cc:101)

Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>

rgw: fix cls_bucket_head result order consistency

headers's key is string but bucket_instance_ids's key is int
looked around and found no one use header's key, so change
header from map to vector

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #18905 from ifed01/wip-bluestore-recalc-alloc

os/bluestore: remove ineffective BlueFS fnode extent calculation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18903 from jan--f/jan-mgr-prometheus-unkown-pgstate

pybind/mgr/prometheus: don't crash when encountering an unknown PG state

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18885 from dzafman/wip-22112

ceph-objectstore-tool: Fix set-size to clear data_digest if changing …

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18622 from majianpeng/bluestore-opseq

os/bluestore: OpSequencer: reduce kv_submitted_waiters if _is_all_kv_submitted() return true.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #17032 from yangdongsheng/rbd_qos

rbd: implement image qos in tokenbucket algorithm

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18483 from Songweibin/wip-list-children2

 rbd: fix rbd children listing when child is in trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18007 from yehudasa/wip-rgw-config-desc

options.cc: document rgw config options

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Abhishek Lekshmanan <alekshmanan@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: set sync_from_all as true when no value is seen

In order to support jewel-luminous multisite scenarios where when a
jewel cluster is the master, the sync-from-all json field wouldn't be
set leading to the secondary not syncing data as this value can't be
overriden to true, so defaulting the value of sync_from_all to true when
we don't encounter it in the json

Fixes: http://tracker.ceph.com/issues/22062
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #18894 from liewegas/wip-fix-balancer

mgr/balancer: skip CRUSH_ITEM_NONE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18763 from cfanz/master

rgw: don't write bucket_header when it is not changed in rgw_bucket_prepare_op

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18852 from theanalyst/wip-22083

rgw: data sync: set num_shards when building full maps

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #18866 from cbodley/wip-22101

rgw: RGWEnv::set() takes std::string

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #18875 from yaozongyou/rgw-cleanup

rgw: remove unused disable_signal_fd

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18895 from cooboos/wip-opt-maperase

rgw: optimize the rgw-attr del code logic

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18845 from yaozongyou/wip-lifecycle-log-level

rgw: update life cycle related log level

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: possible deadlock with synchronous maintenance operations

Fixes: http://tracker.ceph.com/issues/22120
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-dencoder: adding support for cls_rgw_lc_obj_head

After adding cls_rgw_lc_obj_head, we can using the following command
to look into lc processing related information:
rados -p default.rgw.log getomapheader lc.0 ./header.bin --namespace=lc
ceph-dencoder type cls_rgw_lc_obj_head import ./header.bin decode dump_json

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #18873 from tchaikov/wip-bluestore-aio-log

os/bluestore: print aio in batch

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #18888 from tchaikov/wip-osx

common,test,cmake: various changes to re-enable build on osx

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #16369 from bspark8/wip_dmc2_client_tracker

dmclock: Delivery of the dmclock delta, rho and phase parameter + Enabling the client service tracker

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18798 from cbodley/wip-str-list-view

 common: add for_each_substr() for cheap string split

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: add new API methods and expand the rbd CLI to list trashed images

implement librbd::RBD::list_children2 and rbd_list_children2
methods and expand the rbd CLI to list trashed images

Signed-off-by: songweibin <song.weibin@zte.com.cn>

librbd: limit IO per second by TokenBucketThrottle

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #18663 from dragonylffly/wip-rbd-nbd-fix

rbd-nbd: bug fix when running in container

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.prepare add example of raw device or partition in help menu

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18713 from liewegas/wip-snapset-snaps

osd/PrimaryLogPG: do not pull-up snapc to snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.prepare allow partition or raw device as input for --data in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #18882 from ceph/wip-cv-simple-testing

ceph-volume add functional tests for simple, rearrange lvm tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

pybind/mgr/prometheus: don't crash when encountering an unknown PG state

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #18617 from tchaikov/wip-mon-cleanup

mon: remove unused waiting_for_commit

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove ineffective BlueFS fnode extent calculation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>