cephfs-journal-tool: disambiguate usage of all keyword (in tool help).

        The fs:all for rank option description was confusing. It seemd
like the fs was optional, but it is mandatory. This change modifies the
help message to reflect the correct way to use all in the --rank option.

Fixes: https://tracker.ceph.com/issues/61753
Signed-off-by: Manish M Yathnalli <myathnal@redhat.com>

Merge pull request #51961 from Matan-B/wip-matanb-crimson-single-mosdmap-handling

Crimson/osd: Disable concurrent MOSDMap handling

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #52072 from NitzanMordhai/wip-nitzan-add-ec-clay-teuthology-tests

qa: adding clay test to thrash erasure code big

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #51667 from Matan-B/wip-matanb-share-osdmap

osd/OSD: cleanup OSDService::maybe_share_map()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51450 from batrick/i59716

pybind/rados: keep byte representation if decode fails

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #50206 from TRYTOBE8TME/wip-rgw-yield-work

rgw: replace null_yields with optional ones

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52123 from zdover23/wip-doc-2023-06-20-rados-operations-control-1-of-x

doc/rados: edit control.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit control.rst (1 of x)

Edit doc/rados/operations/control.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52109 from idryomov/wip-61616

librbd: avoid object map corruption in snapshots taken under I/O

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rbd: make continuous export-diff test actually work

The current version is pretty useless:

- "rbd bench" writes the same byte (0xff) over and over again, so
  almost all checksumming is in vain
- snapshots are taken in a steady state (i.e. not under I/O), so no
  race conditions can get exposed
- even with these caveats, it's not wired up into the suite

Redo this workunit to be a reliable reproducer for the issue fixed
in the previous commit and wire it up for both krbd and rbd-nbd.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson/osd/osd_operations/pg_advance_map: 'from' epoch may vary between PGS explanation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osdmap_gate: got_map() assert increase of current epoch

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #51827 from liu-chunmei/get_assert

crimson:  fix RefCountedObject::_get() assert

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd/shard_services: get_or_create_pg() remove unused variable

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/shard_services: Refactor OSDSingletonState::load_map

Added logs
If case reconstructed - epoch is unsigned.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_meta: Abort instead of throwing runtime_error

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: Add logs around handling of osdmaps

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/pg: Fix assert of osd_debug_verify_cached_snaps

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #52104 from anthonyeleven/anthonyeleven/fixfft

doc/rados/operations: Improve formatting in crush-map.rst

doc/rados/operations: Improve formatting in crush-map.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge PR #51500 into main

* refs/pull/51500/head:
test/libcephfs: add test case for revoking caps
client: issue a cap release immediately if no cap exists
mds: add the revoking caps back to _revokes list
mds: move confirm_receipt() to Capability.cc

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #51655 into main

* refs/pull/51655/head:
mds: display sane hex value (0x0) for empty feature bit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

src/rgw: Additional changes based on comments

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Changing null_yield in rgw_sal_rados.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Changing null_yield in rgw_reshard.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Passing optional_yield to rgw_rados.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rados: Changing null_yield to optional_yield

Passing optional_yield at all places in rgw_bucket.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

src/rgw: Changing null_yield to optional_yield

Focused on rgw_rados_operate() and passing null_yield in
rgw_lc.cc, rgw_gc.cc, rgw_object_expirer_core.cc

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw: pubsub: pass yield context

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rados: pass yield context in a few places

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #51681 from cbodley/wip-61349

librados: aio operate functions can set mtimes

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #51737 from cbodley/wip-rocksdb-44

bump rocksdb submodule to latest ceph-reef-v7.9.2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #51669 from ronen-fr/wip-rf-max-scrub-3

osd/scrub: increasing max_osd_scrubs to 3

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #48209 from kamoltat/wip-ksirivad-fix-tracker-57570

osd/OSDMap: Check for uneven weights & != 2 buckets post stretch mode

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: display sane hex value (0x0) for empty feature bit

Print a valid hex (0x0) during empty feature bit, so that the clients could recognize it.
When the _vec size becomes 0, print() function creates an invalid hex (0x) and 'perf stats'
crashes with the below error:
"
File "/opt/ceph/src/pybind/mgr/stats/fs/perf_stats.py", line 177, in notify_cmd
metric_features = int(metadata[CLIENT_METADATA_KEY]["metric_spec"]["metric_flags"]["feature_bits"], 16)
ValueError: invalid literal for int() with base 16: '0x'
"
This patch creates  a valid hex (0x0), when _vec size is 0.

Fixes: https://tracker.ceph.com/issues/59551
Signed-off-by: Jos Collin <jcollin@redhat.com>

librbd: stop passing IOContext to image dispatch write methods

This is a major footgun since any value passed e.g. at the API layer
may be stale by the time we get to object dispatch.  All callers are
passing the IOContext returned by get_data_io_context() for their
ImageCtx anyway, highlighting that the parameter is fictitious.

Only the read method can meaningfully take IOContext.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: use an up-to-date snap context when owning the exclusive lock

By effectively moving capturing of the snap context to the API layer,
commit 1d0a3b17f590 ("librbd: pass IOContext to image-extent IO
dispatch methods") introduced a nasty regression.  The snap context can
be captured only after exclusive lock is safely held for the duration
of dealing with the image request and even then must be refreshed if
a snapshot creation request is accepted from a peer.  This is needed to
ensure correctness of the object map in general and fast-diff states in
particular (OBJECT_EXISTS vs OBJECT_EXISTS_CLEAN) and object deltas
computed based off of them.  Otherwise the object map that is forked
for the snapshot isn't guaranteed to accurately reflect the contents of
the snapshot when the snapshot is taken under I/O (as in disabling the
object map may lead to different results being returned for reads).

The regression affects mainly differential backup and snapshot-based
mirroring use cases with object-map and/or fast-diff enabled: since
some object deltas may be incomplete, the destination image may get
corrupted.

This commit represents a reasonable minimal fix: IOContext passed
through to ImageDispatch is effected only for reads and just gets
ignored for writes.  The next commit cleans up further by undoing the
passing of IOContext through the image dispatch layers for writes.

Fixes: https://tracker.ceph.com/issues/61616
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #52105 from gsantella/patch-update-typo-on-dashboard

doc/mgr/dashboard: fix a typo

doc/mgr/dashboard: fix a typo

Signed-off-by: Guido Santella <gsantella@southhills.edu>

Merge pull request #52100 from zdover23/wip-doc-2023-06-17-rados-operations-crush-map-firefly-tunables

doc/rados: edit firefly tunables section

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: edit firefly tunables section

Edit the firefly tunables section in
/doc/rados/operations/crush-map.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52092 from zdover23/wip-doc-2023-06-16-rados-operations-crush-map-4-of-x

doc/rados: edit crush-map.rst (4 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit crush-map.rst (4 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50874 from rishabh-d-dave/mds-log-variable

mds: use variable g_ceph_context directly in MDSAuthCaps

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52095 from rhcs-dashboard/jsonschema-fix

mgr/dashboard: fix the jsonschema issue in install-deps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #51549 from vedanshbhartia/coverity_1517658

rgw: modify RGWRados::cls_bucket_list_ordered to store dirent key before move

Merge pull request #51719 from vedanshbhartia/coverity_1510555

rgw: Use a unique_ptr when creating a D3nCacheAioWriteRequest instance

Merge pull request #52076 from rhcs-dashboard/cephfs-no-mds-clients

mgr/dashboard: empty list of fs clients with no mds

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52089 from samarahu/d4n-cmake-fix

RGW: Place D4N files under cmake conditional

mds: use variable g_ceph_context directly in MDSAuthCaps

Variable g_ceph_context is global, therefore use it directly instead of
passing it as a parameter to method.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #52010 from pkalever/fix-ImageMap

rbd-mirror: don't refer to (remote) mirror_uuid as peer_uuid

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51132 from lxbsz/wip-59349

qa: wait for 100 seconds to make sure the quota to be enforced

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard: fix the jsonschema issue in install-deps

Fixes: https://tracker.ceph.com/issues/61690
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52067 from zdover23/wip-doc-2023-06-15-rados-operations-crush-map-3-of-x

doc/rados: edit crush-map.rst (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: edit crush-map.rst (3 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/dashboard: empty list of fs clients with no mds

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>
Fixes: https://tracker.ceph.com/issues/61708

qa: adding clay test to thrash erasure code big
Currently we don't have any clay test in the erasure code big tests
adding also clay tests.

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

rbd-mirror: don't refer to (remote) mirror_uuid as peer_uuid

peer_uuid and mirror_uuid are different identifiers, we should not use
peer_uuid naming for remote mirror_uuid.

Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com>

RGW: Place D4N files under cmake conditional

Signed-off-by: Samarah <samarah.uriarte@ibm.com>

Merge pull request #52064 from ajarr/wip-61688

mgr/rbd_support: add user-friendly stderr message when module is not ready

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51962 from vedanshbhartia/coverity_1511072

rgw: pass boost intrusive_ptr to DataLogBackends flat_map

Merge pull request #51676 from vedanshbhartia/coverity_move

rgw: Do not forward functor objects in rgw_iam_policy.h

Merge pull request #51965 from vedanshbhartia/coverity_1510687

rgw: Fix use after move in cls_fifo_legacy.cc

Merge pull request #50330 from mohamedawnallah/fix-RGWEnv.get()-nullptr-default-value-to-the-default-returned-value

src/rgw/rgw_env.cc: Fix potential null pointer dereference in RGWEnv::get()

osd/OSD: maybe_share_map()- edit log messages

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: maybe_share_map() use scope guard

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/Session: rename last_sent_epoch to projected_epoch

We always update last_sent_epoch to reflect the highest map
we think the peer will have, whether because we are sending
the incremental or due to the map_epoch on the MOSDPing.

last_epoch_sent was a confusing name since we also update it
when an MOSDPing from the peer reflects a higher epoch.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/OSD: maybe_share_map()- avoid locking `sent_epoch_lock` twice

Invoking `send_incremental_map()` doesn't wait until the replica
actually receives the map, so marking `last_epoch_sent` prior to
invoking it is just as correct.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/rbd_support: log number of images

... that have one snapshot request pending when the
mirror_snapshot_schedule handler is shutting down.

Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #52058 from guits/tracker_61673

ceph-volume: set lvm membership for mpath type devices

Merge PR #52029 into main

* refs/pull/52029/head:
debian: install cephfs-mirror systemd unit files and man page

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #52039 from zdover23/wip-doc-2023-06-14-operations-crush-map-2-of-x

doc/rados: edit crush-map.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

debian: install cephfs-mirror systemd unit files and man page

Fixes: https://tracker.ceph.com/issues/59682
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #50862 into main

* refs/pull/50862/head:
mon/MDSMonitor: batch last_metadata update with pending
mon/MDSMonitor: add debugging for update_metadata

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/rados: edit crush-map.rst (2 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52040 from zdover23/wip-doc-2023-06-14-operations-crush-map-ordered-list-fix

doc/rados: fix list in crush-map.rst

mgr/rbd_support: add user-friendly stderr message

... when the rbd_support module is not ready.

Fixes: https://tracker.ceph.com/issues/61688
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #51946 from phlogistonjohn/jjm-cephadm-write-new

cephadm: add write_new function for robust file writes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51924 from adk3798/limit-policy-log-level

python-common/drive_selection: lower log level of limit policy message

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51861 from adk3798/spec-no-spec-validate

python-common/drive_group: handle fields outside of 'spec' even when 'spec' is provided

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #51267 from phlogistonjohn/jjm-cephadm-versioning

Support cephadm "binary" versioning

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51832 from rkachach/fix_issue_61499

cephadm: adding support for ipv6 when deploying ceph-exporter

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51817 from rkachach/fix_issue_61494

mgr/cephadm: validate host label before removing

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51070 from adk3798/reconfig-firewall-ports

cephadm: allow ports to be opened in firewall during adoption, reconfig, redeploy

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

ceph-volume: set lvm membership for mpath type devices

If any lvs is present, self.lvs should be set accordingly for mpath
type devices as well.
Otherwise, when re-running a command like following:

```
ceph-volume lvm batch --bluestore --no-auto  /dev/mapper/mpatha /dev/mapper/mpathb
```

fails with error : "Device /dev/mapper/mpatha has a filesystem."

where the expected behavior is:

```
--> passed data devices: 2 physical, 0 LVM
--> relative data size: 1.0
--> All data devices are unavailable

Total OSDs: 0

  Type            Path                                                    LV Size         % of device
```

Fixes: https://tracker.ceph.com/issues/61673
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #52044 from vedanshbhartia/coverity_fix_tags

rgw: user correct tags for coverity annotations and add the SUPPRESS keyword

rgw: user correct tags for coverity annotations and add the SUPPRESS
keyword

Signed-off-by: Vedansh Bhartia <vedanshbhartia@gmail.com>

Merge pull request #52012 from cbodley/wip-61428

qa: suppress SyscallParam error during startup on jammy

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc/rados: fix list in crush-map.rst

Correct the numbering and formatting of an ordered list in
doc/rados/operations/crush-map.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #52014 from cbodley/wip-test-cls-refcount-rename

test/cls: rename cls_refcount tests

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #52009 from zdover23/wip-doc-2023-06-12-rados-operations-placement-groups-math-markup

doc/rados: add math markup to placement-groups.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52011 from samarahu/d4n-warning-cleanup

RGW: Cleanup D4N unit test warning

Merge pull request #52028 from zdover23/wip-doc-2023-06-13-dev-deploying-dev-cluster-link

doc/dev: fix "deploying dev cluster" link

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #51989 from samarahu/remove-trace-comment

RGW: Replace trace with d4n in yaml long desc

Merge pull request #52018 from zdover23/wip-doc-2023-06-13-operations-crush-map-1-of-x

doc/rados: edit crush-map.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/dev: fix "deploying dev cluster" link

Fix a link to "Deploying a Development Cluster" that had been broken
since the transition from master to main.

This is the first part of an initiative to improve the vstart
documentation.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: edit crush-map.rst (1 of x)

Edit doc/rados/operations/crush-map.rst.

https://tracker.ceph.com/issues/58485

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51966 from rhcs-dashboard/grafana-url-deprecated

mgr/dashboard: change deprecated grafana URL in daemon logs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #51953 from rhcs-dashboard/dashboard-status-popover

mgr/dashboard: add popover to cluster status card

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

crimson/osd/osd: handle_osd_map() remove unused ConnectionRef conn

Signed-off-by: Matan Breizman <mbreizma@redhat.com>