mgr/dashboard: fix create snapshot is disabled for rbd images

also the rename and rollback option in the snapshot list is enabled even if there isn't anything in the list

also disabled the Mirror Image Snapshot option in the Create RBD
Snapshot form (only shown for images configured with Snapshot mirroring)
with the helpe

Fixes: https://tracker.ceph.com/issues/61296
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51555 from rhcs-dashboard/rearrange-embedded-graphana-buttons

mgr/dashboard: fix spacing of embedded grafana buttons

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51533 from rhcs-dashboard/fix-tooltip-silence-form

mgr/dashboard: fix tooltip for alerts silence matchers

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51556 from rhcs-dashboard/disable-create-osd

mgr/dashboard: add warning message on why osd creation disabled

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51541 from rhcs-dashboard/filter-hosts

mgr/dashboard: filte hosts by services

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #51532 from rhcs-dashboard/reorder-daemon-page

mgr/dashboard: reorder rgw daemons list items

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>

Merge pull request #51567 from rhcs-dashboard/remove-card-title

mgr/dashboard: remove title tooltip of cards landing page

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51362 from Svelar/filestore-drop

vstart: remove filestore options and config snippets

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51580 from zdover23/wip-doc-2023-04-19-radosgw-dynamicresharding

doc/radosgw: explain multisite dynamic sharding

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: explain multisite dynamic sharding

Add a note to doc/radosgw/dynamicresharding.rst and a note to
doc/radosgw/multisite.rst that explains that dynamic resharding is not
supported in releases prior to Reef.

This commit is made in response to a request from Mathias Chapelain.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51406 from votdev/issue_59689_sso_str_decode

mgr/dashboard: SSO error: AttributeError: 'str' object has no attribute 'decode'

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/dashboard: fix spacing of embedded grafana buttons

Fixes: https://tracker.ceph.com/issues/61220
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #51564 from cloudbehl/rgeDeamonsGateway

mgr/dashboard: Change Daemons in Object Gateway section to Gateways

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51563 from rhcs-dashboard/encryption-defaults

mgr/dashboard: put some default values on rgw encryption config form

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>

Merge pull request #51554 from rhcs-dashboard/roles-breadcrumbs

mgr/dashboard: fix rgw roles breadcrumbs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #51550 from rhcs-dashboard/hide-selected-gw-dropdown

mgr/dashboard: remove Selected Object Gateway dropdown

Reviewed-by: Pegonzal <NOT@FOUND>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #51546 from rhcs-dashboard/add-scroll-inside-tables

mgr/dashboard: add-scroll-in-key-value-tables

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51529 from rhcs-dashboard/encyrption-form-default

mgr/dashboard: default encryption type to sse-s3 in rgw

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: cloudbehl <NOT@FOUND>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #50691 from amathuria/wip-amat-out-of-order

mClockScheduler: Set priority cutoff in the mClock Scheduler

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #51572 from zdover23/wip-doc-2023-04-19-rados-operations-devices

doc/rados: line-edit devices.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: line-edit devices.rst

Edit doc/rados/operations/devices.rst.

Co-authored-by: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: add warning message on why osd creation disabled

When no options are present when creating OSDs, you can see the form
with lots of things disabled without having a clear explanation. This
PR introduces a warning banner with a description on why it is
happening.

Fixes: https://tracker.ceph.com/issues/61223
Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

mgr/dashboard: remove title tooltip of cards landing page

Fixes: https://tracker.ceph.com/issues/61250
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #51530 from rhcs-dashboard/fix-61213-main

mgr/dashboard: Rename Cluster > Monitoring to Alerts and change Cluster>Users to Cluster>Ceph Users

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard:Change Daemons > Object Gateway section to Gateways
- Change the Daemons to Gateways
- Changing the breadcrums to call it Gateways
- In list page Calling Daemons List as Gateways List

Fixes: https://tracker.ceph.com/issues/61245
Signed-off-by: Ankush Behl <cloudbehl@gmail.com>

mgr/dashboard: put some default values on rgw encryption config form

Fixes: https://tracker.ceph.com/issues/61246
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: filte hosts by services

Fixes: https://tracker.ceph.com/issues/61242
Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

mgr/dashboard: fix rgw roles breadcrumbs

Fixes: https://tracker.ceph.com/issues/61241
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #51547 from mitsu-ko/patch2

doc: fix underbar to hyphen in stripe-unit and stripe-count

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: add-scroll-in-key-value-tables

Fixes: https://tracker.ceph.com/issues/61220
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

mgr/dashboard: reorder rgw daemons list items

Fixes: https://tracker.ceph.com/issues/61212
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Rename Cluster > Monitoring to Alerts and change Cluster>Users to Cluster>Ceph Users

Fixes: https://tracker.ceph.com/issues/61213
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #51531 from rhcs-dashboard/rename-service-details-to-daemons

mgr/dashboard: rename Service>Details to Service>Daemons

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: default encryption type to sse-s3 in rgw

by default the form doesn't select any values, so i am defaulting it to
sse-s3

Fixes: https://tracker.ceph.com/issues/61211
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix tooltip for alerts silence matchers

Fixes: https://tracker.ceph.com/issues/61208
Signed-off-by: avanthakkar <avanjohn@gmail.com>

doc: fix under bar to hyphen in stripe-unit and stripe-count

Signed-off-by: Mitsumasa KONDO <kondo.mitsumasa@gmail.com>

Merge pull request #51449 from amathuria/wip-rbd-suite-change-mclock-profile

qa/tasks: Changing default mClock profile to high_recovery_ops

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #49691 into main

* refs/pull/49691/head:
qa: add test for opening a file via a hard link that is not in the same mds as the inode
mds: rdlock_path_xlock_dentry supports returning auth target inode

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: remove Selected Object Gateway dropdown

Only from Daemons and Multisite page because its not needed there.

Fixes: https://tracker.ceph.com/issues/61230
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/tasks: Change default mClock profile to high_recovery_ops

With the new mClock default profile, tests were failing with "Exiting scrub checking -- not all pgs scrubbed" due to slower scrubs.
Changing the default profile to high_recovery_ops for testing purposes will fix this issue.

Fixes: https://tracker.ceph.com/issues/61228
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #46985 from fengyueqiao/fix-the-comment-of-RecoveryState-hierarchical

osd: fix the comment of RecoveryState hierarchical structure

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51534 from zdover23/wip-doc-2023-05-17-cephfs-mirroring-cleanup

doc/cephfs: line-edit "Mirroring Module"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51444 from jianwei1216/handle_osd_map_use_deep_copy

osd/OSDMap: improve efficiency for OSD::handle_osd_map

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #50531 from YiteGu/op-delayed-detail

osd/OpRequest: Add detail description for delayed op in osd log file

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: rename Service>Details to Service>Daemons

Daemons and services are not interchangeable in cephadm terms. As stated
in the docs: `A daemon is a systemd unit that is running and part of a
service.` https://docs.ceph.com/en/latest/cephadm/services/#daemon-status

Fixes: https://tracker.ceph.com/issues/61214
Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

doc/cephfs: line-edit "Mirroring Module"

Line-edit the "Mirroring Module" section of
doc/cephfs/cephfs-mirroring.rst. Add prompts and formatting where such
things contribute to the realization of adequate sentences.

This commit is a follow-up to https://github.com/ceph/ceph/pull/51505.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51513 from cbodley/wip-59490

rgw: use string_view to parse Accept header

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #51279 from pkalever/cleanup-logs

librbd: clean up a few log messages

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #50594 into main

* refs/pull/50594/head:
mds: remove inappropriate initialization of num_imported

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51505 from vshankar/wip-cephfs-mirror-add-explanation

doc: explain cephfs mirroring `peer_add` step in detail

Reviewed-by: Zac Dover <zac.dover.proton.me>

Merge pull request #51297 from Matan-B/wip-matanb-locker-request-str

librbd/managed_lock/GetLockerRequest: Fix no valid lockers case

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51454 from ajarr/wip-59713

rbd_support: recover from "double blocklisting"

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd: cleanup few log lines

Currently:
2023-04-28T12:20:17.135+0530 7f1a7ccd2640 20 librbd::ExclusiveLock: 0x7f1a4c000b60 block_requestsr=0
[...]
2023-04-28T12:20:17.135+0530 7f1a7ccd2640 20 librbd::ManagedLock: 0x7f1a4c000b78 is_lock_owner: =1
[...]
2023-04-18T21:18:50.695+0530 7f0a1ffff640 20 librbd::ImageWatcher: 0x7f0a10007840 remote async request finished: [4305,140018014309120,1] = -110

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

rgw: use string_view to parse Accept header

avoid copying the header into a separate buffer to do comparisons

Fixes: https://tracker.ceph.com/issues/59490
Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart: modify FreeBSD objectstore to memstore.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

osd/OSDMap: improve efficiency for OSD::handle_osd_map

- improve OSDMap::decode performance by page-aligning bufferlist
  - align osdmap buffer memory pages in OSDService::_get_map_bl()
  - align osdmap bluffer memory pages in OSDService::get_inc_map_bl
- use OSDMap::deepish_copy_from in OSD::handle_osd_map instead of OSDMap::decode

co-author: yanghonggang <yanghonggang_yewu@cmss.chinamobile.com>
Signed-off-by: zhangjianwei <zhangjianwei2_yewu@cmss.chinamobile.com>

doc: explain cephfs mirroring `peer_add` step in detail

@zdover23 reached out regarding missing explanation for `peer_add`
step in cephfs mirroring documentation. Add some explanation and
and example to make the step clear.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51492 from zdover23/wip-doc-2023-05-16-start-os-recommendations

doc/start: KRBD feature flag support note

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #51385 from rhcs-dashboard/fix-cephPgImbalance-regression

mgr/dashboard: fix regression caused by cephPgImabalance alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50537 from mrstarke/main

rgw/swift: check position of first slash in slo manifest files

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49767 from cbodley/wip-rgw-replicated-from

rgw: fetch_remote_obj() adds x-rgw-replicated-from headers

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

rbd_support: recover from "double blocklisting"

Recover from being blocklisted while recovering from blocklisting.
When the rbd_support  module is being set up to recover from client
blocklisting, the module's new rados client connection can also get
blocklisted. Currently, this will cause the recovery to fail and
the module will remain inoperable. Instead, retry module recovery
when the new client gets blocklisted during the module setup in the
recovery thread.

Fixes: https://tracker.ceph.com/issues/59713
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #50536 from smanjara/wip-rgw-return-enoent

rgw/multisite: don't retry in error repo if bucket instance does not exist

Merge pull request #51085 from smanjara/wip-fix-num-shards-0

rgw/multisite: handle shard_id correctly in data sync when bucket num_shards is 0

doc/start: KRBD feature flag support note

Add KRBD feature flag support note to doc/start/os-recommendations.rst.

This change was suggested by Anthony D'Atri in https://github.com/ceph/ceph/pull/51485.

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #47752 into main

* refs/pull/47752/head:
test/libcephfs: add test case for revoking caps
mds: remove the cap directly when releasing the cap
mds: add the revoking caps back to _revokes list
mds: move confirm_receipt() to Capability.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51485 from idryomov/wip-os-recommendations-drop-4.14

doc/start/os-recommendations: drop 4.14 kernel and reword guidance

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51453 from yuvalif/wip-yuval-fix-59733

rgw/lua: add lib64 to the package search path

Merge pull request #45958 from ofriedma/wip-ofriedma-sync-fairness-bidding

rgw: sync fairness bidding

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/start/os-recommendations: drop 4.14 kernel and reword guidance

The 4.14 LTS kernel has less than a year left in terms of maintenance,
drop it.

Also, the current wording with an explicit list of kernels tends to go
stale: it's missing the latest 6.1 LTS kernel.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51415 from yuvalif/wip-yuval-fix-59698

rgw/lua: add more info on package install errors

Merge pull request #50471 from galsalomon66/json_s3tests_and_more

Trino/RGW alignments  & adding JSON queries to s3tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51343 from guits/tracker_59640

ceph-volume: fix a bug in `get_lvm_fast_allocs()` (batch)

Merge pull request #51410 from ceph/revert-50392-seastore-cephadm

Revert "ceph-volume: assign seastore as object store backend when deplying crimson-osd using LVM with cephadm"

Merge pull request #51473 from zdover23/wip-doc-2023-05-15-rados-operations-devices

doc/rados: edit devices.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com

mds: remove inappropriate initialization of num_imported

The variable num_imported is being passed by reference. Additionally,
the decode_import_dir() function is invoked from handle_export_dir(),
where num_imported is initialized and passed by reference.
Therefore, there is no need to initialize it again within
the decode_import_dir() function.

fixes: https://tracker.ceph.com/issues/59107

Signed-off-by: Yongseok Oh <yongseok.oh@linecorp.com>

doc/rados: edit devices.rst

Line-edit doc/rados/operations/devices.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #51386 into main

* refs/pull/51386/head:
qa: ignore cluster warning when fs flag refuse_client_session is set

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #50806 from rhcs-dashboard/dashboard-multisite-migrate

Dashboard multisite migrate

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Migrate from single site to multi-site

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #51463 from zdover23/wip-doc-2023-05-13-fs-volumes-1-of-x

doc/cephfs: edit fs-volumes.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit fs-volumes.rst (1 of x)

Edit the syntax of the English language in the file
doc/cephfs/fs-volumes.rst up to (but not including) the section called
"FS Subvolumes".

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51458 from zdover23/wip-doc-2023-05-12-cephfs-fs-volumes-prompt-rectification

doc/cephfs: rectify prompts in fs-volumes.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: rectify prompts in fs-volumes.rst

Make sure all prompts are unselectable. This PR is meant to be
backported to Reef, Quincy, and Pacific, to get all of the prompts into
a fit state so that a line-edit can be performed on the Englsh language
in this file.

Follows https://github.com/ceph/ceph/pull/51427.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/lua: add lib64 to the package search path

Fixes: https://tracker.ceph.com/issues/59733
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #51448 from Matan-B/wip-matanb-crimson-only-mclock-boot

crimson/osd/scheduler/mclock_scheduler: Fix OSD unable to start

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51388 from Matan-B/wip-matanb-c-enable-rbd-tests

qa/suites/crimson: Enhance rbd api testing

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd/scheduler/mclock_scheduler: Fix OSD unable to start

https://github.com/ceph/ceph/pull/49975 Introduced changes to
mclock conf value types which caused the osd to stall while booting.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "ceph-volume: assign seastore as object store backend when deplying crimson-osd using LVM with cephadm"

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge PR #51251 into main

* refs/pull/51251/head:
PendingReleaseNotes: add a note about deleting files from lost+found directory
qa: add checks that validate removal of entries from lost+found dir
mds: allow unlink operation under lost+found directory

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #51201 into main

* refs/pull/51201/head:
qa: run scrub post file system recovery

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #51188 into main

* refs/pull/51188/head:
client: use deep-copy when setting permission during make_request

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #51423 from bigjust/replace-go-example-mods

examples: replace example go modules with instructions to run

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #51427 from zdover23/wip-doc-2023-05-10-cephfs-fs-volumes-prompt-fix

doc/cephfs: fix prompts in fs-volumes.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

rgw/multisite: use intrusive_ptr instead of raw ptr for notify_stack

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: - add a lost_bid variable to handle is_highest_bidder() control flow
- check for is_highest_bidder() before even attempting to take the lock
- don't block on RGWMetaSyncShardNotifyCR()
- other minor fixes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: clean up

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: periodically check if we still have the highest bid.
If we are outbid, exit after updating sync status.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: add a timer coroutine to call notify_cr() at intervals

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: multisite metadata sync fairness

    multisite metadata sync fairness

    The approach of this commit is to allow multiple RGWs to participate in the multisite metadata sync.
    Before this commit only single RGW has caught the all the sync locks.
    This feature is using bidding algorithm.
    For each lock, RGW is randomizing a number from 0 to shard count and for each shard is picking randomally one number and giving it as the bid_amount.
    each one of those vectors each RGW handles are being sent using watch notify (based on RADOS watch notify).
    Each time the RGW tries to lock it will compare its bid for the lock and the bids of other rgws, if the current RGW has the highest bid it will try to acquire the lock.

    Important configs:

    rgw_sync_work_period - For how long the RGW will sync until it will send unlock (very important in the beggining, because in the beginning only single RGW holds the locks)
    rgw_sync_lease_period - not new to this commit but affecting it, For how many seconds the RGW will request from the RADOS to keep the lock, mainly important in case of failure, so automatically the RGW will lose a lock if it's down

Fixes: https://tracker.ceph.com/issues/41230
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Signed-off-by: Or Friedmann <ofriedma@ibm.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>