Merge pull request #52232 from batrick/wip-61410-reef

reef: mon/MDSMonitor: check fscid in pending exists in current

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52231 from batrick/wip-61759-reef

reef: pybind/rados: keep byte representation if decode fails

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52229 from batrick/wip-61693-reef

reef:  mon/MDSMonitor: batch last_metadata update with pending

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52126 from joscollin/wip-61735-reef

reef: mds: display sane hex value (0x0) for empty feature bit

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52073 from joscollin/wip-61694-reef

reef: debian: install cephfs-mirror systemd unit files and man page

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51684 from lxbsz/wip-61347

reef: mds: wait reintegrate to finish when unlinking

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51607 from vshankar/tr-59724

reef: mds: allow unlink from lost+found directory

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51506 from lxbsz/wip-59708

reef: mds: do not take the ino which has been used

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #51484 from vshankar/tr-59719

reef: client: use deep-copy when setting permission during make_request

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #52486 from zdover23/wip-doc-2023-07-17-backport-52483-to-reef

reef: doc/mgr/rgw.rst: add missing "ceph" command in cli specification

doc/mgr/rgw.rst: add missing "ceph" command in cli specification

The line begins with a prompt so I would
expect it to be a "complete" command,
including the `ceph` binary to run with all
the various listed arguments.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
(cherry picked from commit 37ea3e50f09be5490d7ec614798822aa6f86eaec)

Merge pull request #52478 from zdover23/wip-doc-2023-07-17-backport-52476-to-reef

reef: doc/mgr/rgw.rst: multisite typed wrong

Reviewed-by: Rongqi Sun (Svelar) <sunrongqi@huawei.co>

doc/mgr/rgw.rst: multisite typed wrong

Multisite is typed wrong as mutlisite so fix the spelling.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
(cherry picked from commit 33b8fdb20329342ef91a391a99065fdb88c16b23)

Merge pull request #52470 from zdover23/wip-doc-2023-07-16-backport-52465-to-reef

reef: doc/rados: edit ceph-conf.rst (2 of x)

doc/rados: edit ceph-conf.rst (2 of x)

Edit doc/rados/configuration/ceph-conf.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 69b9b20be14c7775e434b25753678533eb163a3f)

Merge pull request #52463 from zdover23/wip-doc-2023-07-15-backport-52356-to-reef

reef: doc/cephfs: add note to isolate metadata pool osds

doc/cephfs: add note to isolate metadata pool osds

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4e2e61f16438dcd5eb35854091114d18b6fe9a9e)

Merge pull request #52387 from rhcs-dashboard/cleanup-script-reef

reef: mgr/dashboard: move cephadm e2e cleanup to jenkins job config

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #52448 from zdover23/wip-doc-2023-07-14-backport-52423-to-reef

reef: doc/rados: edit ceph-conf.rst

Merge pull request #52450 from zdover23/wip-doc-2023-07-14-backport-52402-to-reef

reef: doc/rados: edit add-or-rm-osds (2 of x)

doc/rados: edit add-or-rm-osds (2 of x)

Edit doc/rados/operations/add-or-rm-osds.rst (2 of x).

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d83fa5352aa556c4e559e66dd461c200cc86c709)

doc/rados: edit ceph-conf.rst

Edit doc/rados/configuration/ceph-conf.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit dd8d91b0ded8be70bec3f40f499d7cf9432c9f23)

Merge pull request #52425 from zdover23/wip-doc-2023-07-13-backport-52419-to-reef

reef: docs: fix nfs cluster create syntax

docs: fix nfs cluster create syntax

The ceph prefix was missing from the command
example.

Signed-off-by: Paul Cuzner <pcuzner@ibm.com>
(cherry picked from commit 43b5f4b4f92341803753fcd3b67132546829e362)

Merge pull request #52071 from guits/wip-61339-reef

reef: ceph-volume: drop filestore support

Merge pull request #52061 from guits/wip-61152-reef

reef: ceph-volume: fix a bug in `get_lvm_fast_allocs()` (batch)

PendingReleaseNotes: add a note about deleting files from lost+found directory

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 85ce1607a36e642be320a2dea5866dd7f26c90b1)

qa: add checks that validate removal of entries from lost+found dir

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 0252313c8726c74f874436b5f003bb1ee934c571)

mds: allow unlink operation under lost+found directory

Deleting entries in lost+found directory is suggested once users
are done with those entries (possibly, by copying them), however,
unlinking is disallowed by the gating the operation via -EROFS.

Remove this constraint.

Fixes: http://tracker.ceph.com/issues/59569
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 241b9396dbc6d506cef297bdb89ab07fe3a7ce02)

Merge pull request #52344 from adk3798/reef-cephadm-fix-rgw-migration

reef: mgr/cephadm: fix rgw spec migration with simple specs

Reviewed-by: Redouane Kachach <rkachach@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #52343 from adk3798/reef-default-image

reef: mgr/cephadm: set default image for reef

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #52318 from mkogan1/wip-59739-reef

reef: rgw: D3N cache objects which oid contains slash

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #52316 from mkogan1/wip-61661-reef

reef: rgw: d3n: fix valgrind reported leak related to libaio ops

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #52297 from cbodley/wip-61888-reef

reef: rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #51910 from mkogan1/wip-58692-reef

reef: rgw: set bulk pool flag when creating data pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #50936 from ifed01/wip-ifed-report-min-alloc-size-ree

reef: os/bluestore: report min_alloc_size through "ceph osd metadata"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52329 from soumyakoduri/wip-skoduri-reef

[reef]rgw: Do not update mtime in `set_attrs`

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #52321 from batrick/wip-61899-reef

reef: pybind: drop GIL during library callouts

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

PendingReleaseNotes: note rgw's compress-encrypted zonegroup feature flag

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fd2ec27f538bceadec412a848e2d29b187ed04bb)

doc/rgw: document compress-encrypted zonegroup feature flag

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d85011a7fc5d3df42a2c046ec7a43fe29f1db08f)

rgw: require zonegroup feature to compress encrypted objects

Reef adds support for combining server-side encryption and compression,
but previous releases would not replicate such objects correctly. gate
the feature behind a zonegroup feature flag to require that all zones
upgrade to reef before enabling. this feature is not enabled by default
in new zonegroups

Fixes: https://tracker.ceph.com/issues/57905
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4bc52954ab7b38a8eac50c116196d6db728cee00)

Conflicts:
src/rgw/rgw_op.cc: SiteConfig -> sal::ZoneGroup::supports

rgw/sal: add ZoneGroup::supports()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rgw: don't capitalize zonegroup feature flags

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 39be699629df5c6a6b82fa956ed4c22fc4a38523)

rgw: don't enable all zone features by default

on zonegroup creation, all `supported` features get enabled by default.
however, some features should remain opt-in. add a separate list of
`enabled` features for the subset of features we want enabled by default
on zonegroup creation

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit cdb1be836cc1b8c2e8f178bc00bad24d2b8413bf)

Conflicts:
src/rgw/driver/rados/rgw_zone.cc no create_default_zonegroup()

mgr/dashboard: move cephadm e2e cleanup to jenkins job config

Follows: https://github.com/ceph/ceph-build/pull/2145

Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit a407f0b7834585c087001059d6cec613e524538b)

Merge pull request #52383 from zdover23/wip-doc-2023-07-11-backport-52366-to-reef

reef: doc/rados: edit add-or-rm-osds (1 of x)

doc/rados: edit add-or-rm-osds (1 of x)

Edit docs/rados/operations/add-or-rm-osds.

Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a3ab5a7ad8a44c702089294b107d09dca3a98139)

Merge pull request #52381 from zdover23/wip-doc-2023-07-11-backport-52371-to-reef

reef: doc/rados: introduce emdash

doc/rados: introduce emdash

Introduce the emdash character in
docs/rados/troubleshooting/troubleshooting-mon.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit a971a3a66a93ce52898f201a41a9aa13e0dd8bfc)

Merge pull request #52333 from rhcs-dashboard/wip-61911-reef

reef: mgr/dashboard: fix a bug where data would plot wrongly

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #52367 from zdover23/wip-doc-2023-07-09-backport-52327-to-reef

reef: doc: add information on expediting MDS recovery

doc: add information on expediting MDS recovery

Fixes: https://tracker.ceph.com/issues/61865
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0a15144c584d4445bd037340f053aa64b9c13f32)

Merge pull request #52361 from zdover23/wip-doc-2023-07-08-backport-52360-to-reef

reef: doc/radosgw: add Zonegroup policy explanation

Merge pull request #52226 from rzarzynski/wip-59102-reef

reef: msg/async: don't abort when public addrs mismatch bind addrs

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc/radosgw: add Zonegroup policy explanation

Add revised Zonegroup policy for "multi-zonegroups". This commit
includes changes that Casey Bodley made in
https://github.com/ceph/ceph/pull/52324#discussion_r1253482258 and that
I have integrated into the docs only now.

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit c6400ed79b1233a2238342e4931edf341f1c7b9f)

Merge pull request #52190 from ajarr/wip-61706-reef

reef: mgr/rbd_support: add user-friendly stderr message when module is not ready

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #52162 from alimaredia/wip-61773-reef

reef: Tempest tests Keystone Fix

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #52347 from zdover23/wip-doc-2023-07-07-backport-52319-to-reef

reef: doc/foundation: Updating foundation members for July 2023

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52348 from zdover23/wip-doc-2023-07-07-backport-52324-to-reef

reef: doc/radosgw: add Zonegroup purpose

doc/radosgw: add Zonegroup purpose

Add a note about the telos of zonegroups. The information added in this
commit was originally sent to ceph-users by Casey Bodley on 30 Jun 2023
in response to a question from Yixin Jin. Alexander Patrakov then
suggested that Casey's response should be added to
doc/radsogw/multisite.rst.

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 5c46a1a00962918c885e434986d0f846c3ee4730)

doc/foundation: Updating foundation members for July 2023

Signed-off-by: Mike Perez <thingee@gmail.com>
(cherry picked from commit 640390906f76da4bfd66bcd8006e964d6e73c98e)

mgr/cephadm: set default image for reef

This needs to be a v18 image instead of the
main branch ci build, otherwise those that
bootstrap without providing an image name
will get main instead of reef

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: fix rgw spec migration with simple specs

The rgw spec migration code, intended to formalize
the rgw_frontend_type spec option, doesn't work with
simple specs i.e.

service_type: rgw
service_id: rgw.1
service_name: rgw.rgw.1
placement:
  label: rgw

because the migration code assumes there will always be
a "spec" section inside the spec. This is the case for
more involved rgw specs such as

service_type: rgw
service_id: foo
placement:
  label: rgw
  count_per_host: 2
spec:
  rgw_realm: myrealm
  rgw_zone: myzone
  rgw_frontend_type: "beast"
  rgw_frontend_port: 5000

which is what the migration is actually concerned about
(verification of the rgw_frontend_type in these specs).

In the case where the spec is more simple, we should
just leave the spec alone and move on. Unfortunately
the current code assumes the field will always be
there and hits an unhandled KeyError when trying to
migrate the more simple specs. This causes the
cephadm module to crash shortly after starting an
upgrade to a version that includes this migration
and it's very difficult to find the root cause. This
can be worked around by adding fields to the rgw
spec before upgrade so the "spec" field exists in
the spec and the migration works as intended.

This commit fixes the migration in the simple
case as well as adding testing for that case to
both the unit tests and orch/cephadm teuthology
upgrade tests

Fixes: https://tracker.ceph.com/issues/61889
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 1860ef83877c76c51a20aac48036bd9590572cc2)

mgr/dashboard: fix a bug where data would plot on the graphs without converting to the data units of the graph

Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
(cherry picked from commit 7aa70a16a54c3387b2e021763ca1d118d6785835)

rgw: Do not update mtime in `set_attrs`

As per https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html,
the only way for users to modify object metadata is to make a copy of the object and
set the metadata. Hence retain old mtime for any other attr changes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit 31666d71aa5b58a9384938ba31892d48f2ac3f67)

pybind/rbd: drop GIL when calling into librbd

This was missing for rbd_mirror_peer_site_add() and
rbd_get_data_pool_id().

While at it, add a test for data_pool_id().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 1a60f66b0a13089c789e2e056b925ac6fef75021)

pybind/cephfs: drop gil during cephfs callouts

This has disastorous consequences including the possibility of deadlock.
In the best case, you have the rmdir holding the GIL until the MDS
responds!

Fixes: https://tracker.ceph.com/issues/61869
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5d249a4c978198f1e8974affac0297b71bebada5)

Merge pull request #52291 from zdover23/wip-doc-2023-07-03-backport-52280-to-reef-2nd-attempt

reef: doc/radosgw: refine "Zones" in multisite.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: refine "Zones" in multisite.rst

Describe in greater detail the function of Zones in serving S3 objects,
as requested by Alexander Patrakov here:
https://github.com/ceph/ceph/pull/49981#issuecomment-1615866374.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 8dab6c066c4dd3699c7b7c80f7a30a0e18326bba)

reef: rgw: D3N cache objects which oid contains slash

Fixes: https://tracker.ceph.com/issues/59739
Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit fcb91f3e2a6fd8066ac33950d421e0a9b9b4b490)

 Conflicts:
src/rgw/driver/rados/rgw_d3n_datacache.cc
src/rgw/rgw_aio.cc
src/rgw/rgw_d3n_cacherequest.h

reef: rgw: d3n: fix valgrind reported leak related to libaio ops

Fixes: https://tracker.ceph.com/issues/61661
The valgrind leak indication is a false positive in this case,
it is because the libaio internal thread have not timed out yet
when radosgw is terminated.

```
man aio_init
...
aio_idle_time
This field specifies the amount of time in seconds that a worker thread
should wait for further requests before terminating, after having
completed a previous request.   The
default value is 1.
...
```
for the sake of teuthology reducing the timeout

waiting for 2 minutes for example like below would also prevent the leak report
```
❯ env
LD_LIBRARY_PATH=/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/:$LD_LIBRARY_PATH
PYTHONPATH=$PYTHONPATH:/mnt/nvme5n1p1/src-git/ceph--up--master-clang/build/lib/cython_modules/lib.3
RAGWEED_CONF=$(realpath ./ragweed.conf) RAGWEED_STAGES=prepare,check tox
-- -v |& ccze -Aonolookups ;   sleep 2m | pv -t ; pkill radosgw
```

Signed-off-by: Mark Kogan <mkogan@redhat.com>
(cherry picked from commit f834e10cfe97c7df8fc5ead682975fe60b435bd7)

Merge pull request #52298 from zdover23/wip-doc-2023-07-04-backport-51422-to-reef

reef: doc/rgw/lua: add info uploading a script in cephadm deployment

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw/lua: add info uploading a script in cephadm deployment

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>
(cherry picked from commit 5cec4f719581e93760c22419e701c563a8688de9)

Merge pull request #52311 from zdover23/wip-doc-2023-07-05-backport-52207-to-reef

reef: doc/rados: edit crush-map-edits (2 of x)

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: edit crush-map-edits (2 of x)

Edit doc/rados/operations/crush-map-edits.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ae04debe9e2fa18c1cb8745b0573ff56576fd3b0)

Merge pull request #52309 from zdover23/wip-doc-2023-07-05-backport-52308-to-reef

reef: doc/dev: remove seqdiag assets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: remove seqdiag assets

Remove seqdiag assets to determine whether the docs can be built if they
are absent. (Currently they cannot be built when they are present.) If
this works, then these diagrams will be replaced, probably with .png
files.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit ce96ddd4c7da65351deaf14dc4f8b42f5148a352)

rgw: fetch_remote_obj() preserves RGW_ATTR_COMPRESSION of encrypted objects

if the source object was both compressed and encrypted, preserve its
original compression attribute so it can be decompressed the same way it
was originally compressed

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b91ff3fd86f52f7148daf7f9acc9864752186ce0)

rgw: fetch_remote_obj() will never verify etags of encrypted objects

fetch_remote_obj() transfers objects in their encrypted form, so does
not have access to the decrypted data for checksum verification

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ea45da747d5fa898b4aa4ec9223cd7b6983ca177)

rgw: rgwx-skip-decrypt also skips decompression of encrypted objects

compression is applied before encryption. so if we skip decryption, we
can't decompress either

Fixes: https://tracker.ceph.com/issues/57905
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8d35759eb8edcc7cd15e6fd67b91da637fc477ae)

Merge pull request #52278 from zdover23/wip-doc-2023-07-01-backport-52277-to-reef

reef: doc/radosgw: s/s3select/S3 Select/

doc/radosgw: s/s3select/S3 Select/

Correct several misspelling of "S3 Select". Hat tip to Anthony D'Atri,
who caught this in an earlier PR.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 71c327ef4f56a6ec900a8c5ea071aaf8005a60a5)

Merge pull request #52187 from zdover23/wip-doc-2023-06-26-backport-52181-2nd-attempt-to-reef

reef: doc/radosgw: remove pipes from s3select.rst

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

Merge pull request #52219 from zdover23/wip-doc-2023-06-28-backport-52194-to-reef

reef: doc/radosgw: edit "Overview" in s3select.rst

Merge pull request #52262 from zdover23/wip-doc-2023-06-30-backport-52250-to-reef

reef: doc/radosgw: edit "Basic Workflow" in s3select.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #52260 from ceph/reef-release

v18.1.2

doc/radosgw: edit "Basic Workflow" in s3select.rst

Edit the "Basic Workflow" section in doc/radosgw/s3select.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4d2c09b683421552cfb4df7f467f2d9a6c9c7c26)

Merge pull request #52136 from ljflores/wip-61603-reef

reef: qa/suites/rados: whitelist POOL_APP_NOT_ENABLED for cls tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #52135 from rzarzynski/wip-61742-reef

reef: mon/MonClient: before complete auth with error, reopen session

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge PR #50921 into reef

* refs/pull/50921/head:
qa: load file system info if not created

mon/MDSMonitor: check fscid in pending exists in current

A new file system that's not yet committed may be examined by ::tick.
Add a check before looking at the current fsmap.

Fixes: https://tracker.ceph.com/issues/59691
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 3b8b259854895046f19e518d566f713d4d847e79)

PendingReleaseNotes: add note that pyrados may have omap keys as bytes

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 2aa7becc554f1e8fa59132eb6c130ba864ff238b)

qa: test first-damage.py with binary file name

Specifically, one which cannot be decoded as utf8.

Fixes: https://tracker.ceph.com/issues/59716
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 1d3a549ec2dddf02129ff20cf50411294e64faaa)

tools/cephfs/first-damage: always use bytes representation of omap key

We cannot assume the encoding of a file name and we don't need it.

Fixes: https://tracker.ceph.com/issues/59716
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 56219bea697519ec96d7eed230c88648bf005cff)

test/pybind: add testing for rados omap keys as bytes

Fixes: https://tracker.ceph.com/issues/59716
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 08d3d9270cc080ed19184c45fbe7b3eec8d277a5)

pybind/rados: enable alternate types for key names

Specifically, allow the caller to select the Python bytes type as an
acceptable key type.

This is important when the omap keys contain data that cannot be decoded
as a UTF-8 string. Encountering such a key currently stops the iteration
and makes the key virtually inaccessible by pyrados.

Fixes: https://tracker.ceph.com/issues/59716
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit e796ea5c1489ad494dd28b68786a0dc25b62f9c3)

qa: add option to set python executable

To support systems like mine with multiple python installs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit b6867bb25534c4e10287f5ac6c44fd9c1a5f96ef)

mon/MDSMonitor: batch last_metadata update with pending

I believe the problem here is that the last_metadata change is lost in a
ECANCELED/EAGAIN transaction but the pending map change goes through in
the next one. I've been unable to find an exact way to reproduce this.
The problem seems to occur when upgrades are performed which would
indicate shuffling in the monitors where quorum would be lost repeatedly.

This seems to be the most likely explanation so let's go ahead and make
this change even without the reproducer. In any case, it has the added
benefit of batching the pending map update (to up:standby) with the
last_metadata update.

Fixes: https://tracker.ceph.com/issues/24403
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 6f69fe9739a23974a46a4e13e4c29b431d95acc4)

mon/MDSMonitor: add debugging for update_metadata

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 3faafba6e1bc223be071a194fdb10d6c6981c814)

msg/async: don't abort when public addrs mismatch bind addrs

Before the 69b47c805fdd2ecd4f58547d58c9f019fc62d447 (PR #50153)
a mismatch (in number or types of stored `entity_addr_t`) between
public addrs and bind addrs vectors was ignored and the former
was taking over anything else -- it was possible to e.g. bind to
both v1 and v2 addresses but expose v2 only. Unfortunately, that's
exactly how Rook configures ceph-mon:

```
debug 2023-03-16T21:01:48.389+0000 7f99822bf8c0  0 starting mon.a rank 0 at public addrs v2:172.30.122.144:3300/0 at bind addrs [v2:10.129.2.21:3300/0,v1:10.129.2.21:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid acc14d1b-fb2b-4f01-8b61-6e7cb26e9200
```

The consequnece is the following abort:

```
ceph version 17.2.5-1338.el9cp (5adce3015143c7c2cc135a71368be194744f5761) quincy (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd3) [0x7ff05392c1a6]
2: /usr/lib64/ceph/libceph-common.so.2(+0x165ac1) [0x7ff05394eac1]
3: (AsyncMessenger::bindv(entity_addrvec_t const&, std::optional<entity_addrvec_t>)+0x1fe) [0x7ff053baa0ce]
4: main()
5: /lib64/libc.so.6(+0x3feb0) [0x7ff053048eb0]
6: __libc_start_main()
7: _start()
debug *** Caught signal (Aborted) **  in thread 7ff052ab18c0 thread_name:ceph-mon 2023-03-16T09:56:35.995+0000 7ff052ab18c0 -1 /builddir/build/BUILD/ceph-17.2.5-1338-g484e8dbb/src/msg/msg_types.h: In function 'void entity_addr_t::set_port(int)' thread 7ff052ab18c0 time 2023-03-16T09:56:35.996339+0000 /builddir/build/BUILD/ceph-17.2.5-1338-g484e8dbb/src/msg/msg_types.h: 359: ceph_abort_msg("abort() called")
```

This commit brings the original logic back but in a way that
preserves the port numbers figured out by. e.g. `Processor::bind`.

Fixes: https://tracker.ceph.com/issues/59100
Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 68fbdcf9b0937e8102b86d0bf84815554b62328d)