Merge pull request #51297 from Matan-B/wip-matanb-locker-request-str

librbd/managed_lock/GetLockerRequest: Fix no valid lockers case

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51454 from ajarr/wip-59713

rbd_support: recover from "double blocklisting"

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51492 from zdover23/wip-doc-2023-05-16-start-os-recommendations

doc/start: KRBD feature flag support note

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #51385 from rhcs-dashboard/fix-cephPgImbalance-regression

mgr/dashboard: fix regression caused by cephPgImabalance alert

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #50537 from mrstarke/main

rgw/swift: check position of first slash in slo manifest files

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49767 from cbodley/wip-rgw-replicated-from

rgw: fetch_remote_obj() adds x-rgw-replicated-from headers

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

rbd_support: recover from "double blocklisting"

Recover from being blocklisted while recovering from blocklisting.
When the rbd_support  module is being set up to recover from client
blocklisting, the module's new rados client connection can also get
blocklisted. Currently, this will cause the recovery to fail and
the module will remain inoperable. Instead, retry module recovery
when the new client gets blocklisted during the module setup in the
recovery thread.

Fixes: https://tracker.ceph.com/issues/59713
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #50536 from smanjara/wip-rgw-return-enoent

rgw/multisite: don't retry in error repo if bucket instance does not exist

Merge pull request #51085 from smanjara/wip-fix-num-shards-0

rgw/multisite: handle shard_id correctly in data sync when bucket num_shards is 0

doc/start: KRBD feature flag support note

Add KRBD feature flag support note to doc/start/os-recommendations.rst.

This change was suggested by Anthony D'Atri in https://github.com/ceph/ceph/pull/51485.

Co-authored-by: Ilya Dryomov <idryomov@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #47752 into main

* refs/pull/47752/head:
test/libcephfs: add test case for revoking caps
mds: remove the cap directly when releasing the cap
mds: add the revoking caps back to _revokes list
mds: move confirm_receipt() to Capability.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51485 from idryomov/wip-os-recommendations-drop-4.14

doc/start/os-recommendations: drop 4.14 kernel and reword guidance

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51453 from yuvalif/wip-yuval-fix-59733

rgw/lua: add lib64 to the package search path

Merge pull request #45958 from ofriedma/wip-ofriedma-sync-fairness-bidding

rgw: sync fairness bidding

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/start/os-recommendations: drop 4.14 kernel and reword guidance

The 4.14 LTS kernel has less than a year left in terms of maintenance,
drop it.

Also, the current wording with an explicit list of kernels tends to go
stale: it's missing the latest 6.1 LTS kernel.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51415 from yuvalif/wip-yuval-fix-59698

rgw/lua: add more info on package install errors

Merge pull request #50471 from galsalomon66/json_s3tests_and_more

Trino/RGW alignments  & adding JSON queries to s3tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51343 from guits/tracker_59640

ceph-volume: fix a bug in `get_lvm_fast_allocs()` (batch)

Merge pull request #51410 from ceph/revert-50392-seastore-cephadm

Revert "ceph-volume: assign seastore as object store backend when deplying crimson-osd using LVM with cephadm"

Merge pull request #51473 from zdover23/wip-doc-2023-05-15-rados-operations-devices

doc/rados: edit devices.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com

doc/rados: edit devices.rst

Line-edit doc/rados/operations/devices.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #51386 into main

* refs/pull/51386/head:
qa: ignore cluster warning when fs flag refuse_client_session is set

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #50806 from rhcs-dashboard/dashboard-multisite-migrate

Dashboard multisite migrate

Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: Migrate from single site to multi-site

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #51463 from zdover23/wip-doc-2023-05-13-fs-volumes-1-of-x

doc/cephfs: edit fs-volumes.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit fs-volumes.rst (1 of x)

Edit the syntax of the English language in the file
doc/cephfs/fs-volumes.rst up to (but not including) the section called
"FS Subvolumes".

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51458 from zdover23/wip-doc-2023-05-12-cephfs-fs-volumes-prompt-rectification

doc/cephfs: rectify prompts in fs-volumes.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: rectify prompts in fs-volumes.rst

Make sure all prompts are unselectable. This PR is meant to be
backported to Reef, Quincy, and Pacific, to get all of the prompts into
a fit state so that a line-edit can be performed on the Englsh language
in this file.

Follows https://github.com/ceph/ceph/pull/51427.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw/lua: add lib64 to the package search path

Fixes: https://tracker.ceph.com/issues/59733
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #51448 from Matan-B/wip-matanb-crimson-only-mclock-boot

crimson/osd/scheduler/mclock_scheduler: Fix OSD unable to start

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51388 from Matan-B/wip-matanb-c-enable-rbd-tests

qa/suites/crimson: Enhance rbd api testing

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

crimson/osd/scheduler/mclock_scheduler: Fix OSD unable to start

https://github.com/ceph/ceph/pull/49975 Introduced changes to
mclock conf value types which caused the osd to stall while booting.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "ceph-volume: assign seastore as object store backend when deplying crimson-osd using LVM with cephadm"

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge PR #51251 into main

* refs/pull/51251/head:
PendingReleaseNotes: add a note about deleting files from lost+found directory
qa: add checks that validate removal of entries from lost+found dir
mds: allow unlink operation under lost+found directory

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #51201 into main

* refs/pull/51201/head:
qa: run scrub post file system recovery

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge PR #51188 into main

* refs/pull/51188/head:
client: use deep-copy when setting permission during make_request

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #51423 from bigjust/replace-go-example-mods

examples: replace example go modules with instructions to run

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #51427 from zdover23/wip-doc-2023-05-10-cephfs-fs-volumes-prompt-fix

doc/cephfs: fix prompts in fs-volumes.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

rgw/multisite: use intrusive_ptr instead of raw ptr for notify_stack

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: - add a lost_bid variable to handle is_highest_bidder() control flow
- check for is_highest_bidder() before even attempting to take the lock
- don't block on RGWMetaSyncShardNotifyCR()
- other minor fixes

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: clean up

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: periodically check if we still have the highest bid.
If we are outbid, exit after updating sync status.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: add a timer coroutine to call notify_cr() at intervals

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: multisite metadata sync fairness

    multisite metadata sync fairness

    The approach of this commit is to allow multiple RGWs to participate in the multisite metadata sync.
    Before this commit only single RGW has caught the all the sync locks.
    This feature is using bidding algorithm.
    For each lock, RGW is randomizing a number from 0 to shard count and for each shard is picking randomally one number and giving it as the bid_amount.
    each one of those vectors each RGW handles are being sent using watch notify (based on RADOS watch notify).
    Each time the RGW tries to lock it will compare its bid for the lock and the bids of other rgws, if the current RGW has the highest bid it will try to acquire the lock.

    Important configs:

    rgw_sync_work_period - For how long the RGW will sync until it will send unlock (very important in the beggining, because in the beginning only single RGW holds the locks)
    rgw_sync_lease_period - not new to this commit but affecting it, For how many seconds the RGW will request from the RADOS to keep the lock, mainly important in case of failure, so automatically the RGW will lose a lock if it's down

Fixes: https://tracker.ceph.com/issues/41230
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Signed-off-by: Or Friedmann <ofriedma@ibm.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sync: add a BidManager to metadata sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BidManager for sync fairness via watch/notify

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/cephfs: fix prompts in fs-volumes.rst

Fixed a regression introduced in
e5355e3d66e1438d51de6b57eae79fab47cd0184 that broke the unselectable
prompts in the RST.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51345 from cbodley/wip-59639

rgw/dbstore: allow NULL RealmIDs in sqlite schema

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

examples: replace example go modules with instructions to run

Signed-off-by: Justin Caratzas <jcaratza@ibm.com>

Merge pull request #50627 from AliMasarweh/wip-ali-masa-multipart-populate-etag

RGW: Solving the issue of not populating etag in Multipart upload result
Reviewed-by: Daniel Gryniewicz <dang1@ibm.com>

rolling back

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #49742 from ajarr/fix-56724

mgr/rbd_support: recover from rados client blocklisting

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51166 from chrisphoffman/wip-rbd-59393

librbd: localize snap_remove op for mirror snapshots

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #43184 into main

* refs/pull/43184/head:
qa: fix journal flush failure issue due to the MDS daemon crashes
qa: add test support for the alloc ino failing
mds: do not take the ino which has been used

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: run scrub post file system recovery

Running file system scrub is recommended post running filesystem
data and metadata recovery. Running scrub isn't covered in tests.

Fixes: http://tracker.ceph.com/issues/59527
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51167 from liu-chunmei/teuthology-multicore

crimson/qa: make crimson run multicore in teuthology test

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51301 from ceph/wip-yuriw-release-16.2.13-main

doc: 16.2.13 Release Notes

doc: 16.2.13 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #50411 from xxhdx1985126/wip-58928

crimson/osd: start operations asynchrously

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

rgw/lua: add more info on package install errors

Fixes: https://tracker.ceph.com/issues/59698
Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #51403 from zdover23/wip-doc-2023-05-09-start-get-involved-planet-ceph

doc/start: fix "Planet Ceph" link

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

adding s3test albin/json-op-serial
modify json chunk processing function to handle offset/length as csv-processing
a fix valgrind :: Conditional jump or move depends on uninitialised value
upon using Trino the Trino-server issue multiple requests per single query,upon completion of all requests
the results are merged (by Trino).  these request splits the input into equal parts; the RGW side should be aligned with Trino expectations(for result).
fixing the main routine for shaping the chunk (range-scan) for Trino processing
upon removing the payload-TAG, it need to change the response element index
handling more use cases for "shaping" the processed chunk by s3select per Trino request
re-shape the processed chunk only upon Trino sent the request
bug-fix: the chunk offset was not handle correctly
bug-fix: progress-message calcualation
modifying the range-request boundaries only upon Trino request.

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

crimson/qa: enable multicore for crimson in teuthology test

Signed-off-by: chunmei <chunmei.liu@intel.com>

Merge pull request #47749 from xxhdx1985126/wip-intra-fixedkvbtree-pointers-2

crimson/os/seastore/btree: link fixedkvbtree's nodes and logical extents with forward and backward pointers, and drop the pin_set

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51308 from jzhu116-bloomberg/wip-59592

rgw/notification: remove non x-amz-meta-* attributes from bucket notifications

crimson/tools/store_nbd: read logical extents via
TransactionManager::read_pin()

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: add comment about backref_extent_entry_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: complement lba test with logical extents

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

test/crimson/seastore: check intra-fixedkv-btree parent->child trackers during unittests

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: drop btree_pin_set_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction_manager: follow leaf<->logical extent pointers to read extent

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/lba_manager: link lba leaf nodes with logical extents by pointers

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: "templatize" btree leaf node to distinguish leaf nodes with(out) children

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: link fixed-kv-btree and root_block with pointers

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: more debug logs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/backref_manager: retrieve live backref extents throught the backref tree

After involving intra-fixed-kv-btree parent-child pointers, we need to keep the
invariant that it's only when extents are not in transactions' read_set that
we can directly query cache with inspecting the transaction

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: avoid searching transactions' read_set when retrieving btree nodes

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: search fixed-kv-btree by parent<->child pointers

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: invalidate out-dated extent when initiating Cache

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cached_extent: improve the representation of "has_been_invalidated"

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: don't go to leaf nodes when updating internal mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/btree: introduce parent<->child pointers for fixed-kv-btree nodes

maintain correct parent<->child pointers when modifying the btree

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

doc/start: fix "Planet Ceph" link

Fix a link to Planet Ceph on the doc/start/get-involved.rst page.

Reported 2023 Apr 21, here:
https://pad.ceph.com/p/Report_Documentation_Bugs

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51355 from aravind-wdc/wip-crimson-zbd

crimson/os/seastore: enable SMR HDD

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #51392 from parth-gr/rgw-mutisite-ceph-doc

doc: update multisite doc

Reviewed-by: Jiffin Tony Thottan <jthottan@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50857 from kamoltat/wip-ksirivad-iswriteable

mon/Monitor.cc: exit function if !osdmon()->is_writeable()
Reviewd-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #51394 from rzarzynski/wip-doc-encode-stdoptional

doc/dev/encoding.txt: update per std::optional

Reviewed-by: Zac Dover <zac.dover@proton.me>

qa/workunits/rbd: Add tests for rbd_support module recovery

... after the module's RADOS client is blocklisted.

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/rbd_support: recover from rados client blocklisting

In certain scenarios the OSDs were slow to process RBD requests.
This lead to the rbd_support module's RBD client not being able to
gracefully handover a RBD exclusive lock to another RBD client.
After the condition persisted for some time, the other RBD client
forcefully acquired the lock by blocklisting the rbd_support module's
RBD client, and consequently blocklisted the module's RADOS client. The
rbd_support module stopped working. To recover the module, the entire
mgr service had to be restarted which reloaded other mgr modules.

Instead of recovering the rbd_support module from client blocklisting
by being disruptive to other mgr modules, recover the module
automatically without restarting the mgr serivce. On client getting
blocklisted, shutdown the module's handlers and blocklisted client,
create a new rados client for the module, and start the new handlers.

Fixes: https://tracker.ceph.com/issues/56724
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #51365 from nbalacha/fix-remove-unused-type

librbd: remove unused enum WriteOpType

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49975 from sseshasa/wip-fix-mclk-rec-backfill-cost

osd: mClock recovery/backfill cost fixes

Reviewed-by: Sam Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

pybind/rados: add ConnectionShutdown exception class

Signed-off-by: Ramana Raja <rraja@redhat.com>

mgr/rbd_support: notify the thread waiting on pending snapshot

... requests to be completed.

Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #51381 from Matan-B/wip-matanb-c-blocklist-fix

crimson/osd/osd_operations/client_request: Fix client blocklisting

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43245 from thiagoarrais/docs-java-examples

[rgw]: Update AWS SDK in Java examples

doc/dev/encoding.txt: update per std::optional

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: localize snap_remove op for mirror snapshots

A client may attempt a lock request not quickly enough to
obtain exclusive lock for operations when another competing
client responds quicker. This can happen when a peer site has
different performance characteristics or latency. Instead of
relying on this unpredictable behavior, localize operation to
primary cluster.

Fixes: https://tracker.ceph.com/issues/59393
Signed-off-by: Christopher Hoffman <choffman@redhat.com>

doc: update multisite doc

cmd for getting zone group was spelled incorrectly
Updated to rdosgw-admin

Signed-off-by: parth-gr <paarora@redhat.com>

librbd : remove unused enum type WriteOpType

This removes the unused enum WriteOpType from
the librbd deep_copy code.

Signed-off-by: N Balachandran <nibalach@redhat.com>

qa: ignore cluster warning when fs flag refuse_client_session is set

Fixes: https://tracker.ceph.com/issues/59667
Introduced-by: https://github.com/ceph/ceph/pull/48720
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>