Merge PR #56941 into main

* refs/pull/56941/head:
mds: find a new head for the batch ops when the head is dead

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57275 into main

* refs/pull/57275/head:
qa/fsx: use a specified sha1 to build the xfstest-dev

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge PR #57454 into main

* refs/pull/57454/head:
mds/quiesce-db: optimize peer updates
mds/quiesce-db: track db epoch separately from the membership epoch
mds/quiesce-db: test that a peer on a newer membership epoch can ack a root

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57274 into main

* refs/pull/57274/head:
mds: don't stall the asok thread for flush commands
qa/quiescer: relax some timing requirements in the quiescer

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57329 into main

* refs/pull/57329/head:
qa: unmount clients before damaging the fs

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57334 into main

* refs/pull/57334/head:
mds: remove erroneous debug message

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57476 from cyx1231st/wip-seastore-drop-unused

crimson/os/seastore/transaction_manager: drop unused code

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55985 from Svelar/make-dist

make-dist: remove old cruft recursively

Merge pull request #56349 from rhcs-dashboard/add-landing-page-upgrade-notification

mgr/dashboard: add upgrade notification

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #57478 into main

* refs/pull/57478/head:
.github: remove description for action

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57137 from sajibreadd/wip-65228

osd: CEPH_OSD_OP_FLAG_BYPASS_CLEAN_CACHE flag is passed from ECBackend

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56804 from sajibreadd/seg-fault-osd-shutdown-64373

osd: fix for segmentation fault on OSD fast shutdown

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #55298 from ifed01/wip-ifed-fsck-stats

os/bluestore: show some additional osd/pool stats on fsck completion.

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

.github: remove description for action

It is rejected by github's parser.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57099 from ceph/wip-lusov-quiesce-await-eperm

mds/quiesce: db: quiesce-await should EPERM if a set is past QS_QUIESCED
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #56882 from guits/cv-65477

ceph-volume: create LVs when using partitions

Merge pull request #57138 from guits/node-proxy-improvements

node-proxy: make the daemon discover endpoints

crimson/os/seastore/transaction_manager: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57164 from guits/bump-promtail

cephadm: change loki/promtail default image tags

Merge pull request #57000 from chenyuanrun/fix-mkdir-p

ceph-volume: use os.makedirs for mkdir_p

Merge PR #57453 into main

* refs/pull/57453/head:
doc: add status badge for backport creation
.github: use shorter name for backport tracker action
.github: document where runs/output can be examined

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #57391 from soumyakoduri/wip-skoduri-sync-policy

Add new sync-policy related params to boto3 extension

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: add upgrade notification

- Adds upgradable component to manage upgrades from outside upgrade component.
- Cleans a couple subscriptions that were not being destroyed

Fixes: https://tracker.ceph.com/issues/65012
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #53859 from rhcs-dashboard/add-pseudo-validation

mgr/dashboard: add absolute path validation for pseudo path of nfs export

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc: add status badge for backport creation

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

.github: use shorter name for backport tracker action

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

.github: document where runs/output can be examined

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57368 from myoungwon/wip-seastore-is-data-stable

crimson/os/seastore: add is_data_stable() to allow delta-overwrite on EXIST_CLEAN

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57444 from idryomov/wip-51845

qa/suites/krbd: drop pre-single-major and move "layering only" coverage

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #57434 from cyx1231st/wip-fix-seastore-capture

crimson/os/seastore/record_scanner: replace [=] capturing

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: add comments to add a note that seastore_obj_data_write_amplification needs to be reconsidered

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: replace is_left_stable/is_right_stable with is_left_fresh/is_right_fresh

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #55488 from cyx1231st/wip-crimson-load-obc

crimson/osd/osd_operations/client_request: make loading-obc concurrent

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mds/quiesce-db: optimize peer updates

Prevent sending of the same version to the same peer more than once a second

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-db: track db epoch separately from the membership epoch

Tracking the db epoch separately will make sure that replicas
only follow leader's epoch choice, even if they are already on
the new membership epoch. This eliminates races due to the
random order of mdsmap updates.

Fixes: https://tracker.ceph.com/issues/65977
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-db: test that a peer on a newer membership epoch can ack a root

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57356 from soumyakoduri/wip-skoduri-lc-notify

rgw/cloud-transition: fix the crash with publish_commit

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56574 from badone/wip-tracker-59196-lazy-omap-regex-segfault

test/lazy-omap-stats: Convert to boost::regex

Add new sync-policy related params to boto3 extension

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56979 from yuvalif/wip-yuval-65337

rgw/notification: start/stop endpoint managers in notification manager

Reviewed-By: cbodley@ibm.com , kchheda3@bloomberg.net

qa/suites/krbd: rename no-object-map to no-exclusive-lock

Exclusive lock has always been disabled by this facet, so it might as
well be reflected in its name.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: move "layering only" coverage to fsx

It makes much more sense there since it's where we actually create
clones and flatten them a lot.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: drop pre-single-major test

Single-major mapping scheme was introduced in 2014 and became the
default in 2017.  It's getting increasingly difficult to build and,
more importantly, to boot a 10 year old kernel with recent userspace
(systemd, etc).  If someone is still running such a kernel, it's
really unlikely that they would have the most recent rbd CLI tool
installed.

Fixes: https://tracker.ceph.com/issues/51845
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #56148 into main

* refs/pull/56148/head:
cephfs_mirror: check m_instance_watcher and m_mirror_watcher before using them

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56699 into main

* refs/pull/56699/head:
qa: ignore `Invalid tag char` warning
qa: ignore `object missing on disk` warning

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #56525 into main

* refs/pull/56525/head:
mgr/stats: initialize mx_last_updated in FSPerfStats

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56135 into main

* refs/pull/56135/head:
cephfs_mirror: increment sync_failures when sync_perms() and sync_snaps() fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/record_scanner: replace [=] capturing

Capturing with = is error prone and causes compile warning.

And it is wrong:
* cursor should be captured by reference
* mutable keyword is unnecessary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: change loki/promtail default image tags

This commit changes the default images for both loki and promtail
containers.
Also, to allow this update we need to update the configuration of loki
in order to add a new storage schema configuration:

```
    - from: 2024-05-03
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
```

This is because 3.0.0 uses new structured metadata.

See [1][2].

[1] https://grafana.com/docs/loki/latest/operations/storage/schema/
[2] https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/#when-to-use-structured-metadata

Fixes: https://tracker.ceph.com/issues/65784
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #55923 from NitzanMordhai/wip-nitzan-add-deprecate-cls-gather

objclass: deprecate cls_cxx_gather

Merge pull request #56983 from NitzanMordhai/wip-nitzan-thrash-erasure-code-crush-4-nodes-8-6

suites/ec-rados-plugin=jerasure-k=8-m=6-crush: roles set

test/rgw/notifications: make sure that http handler is running

fail test if not. to indicate this is a test issue
and not a product bug

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: prevent flooding the http server

in tests that require retries

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: avoid BlockingIOError when logging errors

for example. job: 7697397
in test: yuvalif-2024-05-08_09:55:02-rgw:notifications-wip-yuval-65337-distro-default-smithi

also reduce the side of the error log by sending less objects to the
test_ps_s3_persistent_topic_stats test

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notifications: fix lifecycle tests

* tests were passing only because they were not performings their asserts
* tests are now separated with their own attribute
* their topics are now marked "persistent" to workaround the issue in:
  https://tracker.ceph.com/issues/65645

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: start/stop endpoint managers in notification manager

Fixes: https://tracker.ceph.com/issues/65337
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rados/test: Remove cls_remote_reade since gather deprecated

https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

rgw/notifications: cleanup includes and unused parameters

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: enable notifications debug logs in vstart

as well as in:
- multisite tests (used for notification v2 migration tests)
- the qa suites running notifications

enable lifecycle logs in notification tests: for the lc notification test cases

this is needed after: 429967917b4f9b5f71214fa3c7718e780728249b

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57421 from zdover23/wip-doc-2024-05-12-dev-developer-guide-dash-devel

doc/dev: s/to asses/to assess/

Merge pull request #57420 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x-followup

doc/cephfs: edit fs-volumes.rst (1 of x) followup

doc/dev: s/to asses/to assess/

s/to asses/to assess/ in doc/deve/developer_guide/dash-devel.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: edit fs-volumes.rst (1 of x) followup

Include the suggestions for improving doc/cephfs/fs-volumes.rst made by
Anthony D'Atri here
https://github.com/ceph/ceph/pull/57415#discussion_r1597362110

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57415 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x

doc/cephfs: edit fs-volumes.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit fs-volumes.rst (1 of x)

Edit doc/cephfs/fs-volumes to the section "FS Subvolumes" (but not
including the section "FS Subvolumes").

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57214 from adk3798/cephadm-ignore-stray-on-upgrade

qa/cephadm: ignore CEPHADM_STRAY_DAEMON in upgrade tests

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57205 from SidGilad/bug/65739_support_cgroup_split_removal_for_adopt

mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56735 from samarahu/wip-rgw-d4n-long-descriptions

d4n: Add more in-depth config descriptions and improve naming

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Samarah Uriarte <samarah.uriarte@ibm.com>

Merge pull request #56585 from ivancich/wip-upgrade-arrow+flight-15.0.2

rgw: upgrade Apache Arrow submodule to 15.0.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57259 from pdvian/wip-fix-unfoundev-progress

mgr/cephadm: Fix unfound progress events

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57182 from adk3798/nvmeof-config-fix

mgr/cephadm:  update default NVMEoF container image version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57181 from adk3798/cleanup-iscsi-nvmeof-keyring

mgr/cephadm: cleanup iscsi and nvmeof keyrings upon daemon removal

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57096 from phlogistonjohn/jjm-smb-output-fmt

mgr/smb: disable JSON/YAML key sorting in favor of ordered dicts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57089 from adk3798/test-cephadm-timeout-ignore-refresh-failed

qa/cephadm: ignore CEPHADM_REFRESH_FAILED on timeout test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57080 from phlogistonjohn/jjm-teuth-pull-img

qa/tasks/cephadm: fix pulling containers from private registries

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57042 from adk3798/host-drain-test-ignore-stray-host

qa/cephadm: ignore stray warnings on host drain test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56970 from adk3798/cephadm-ignore-orch-paused

qa/cephadm: ignore CEPHADM_PAUSED on test_orch_cli test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56963 from adk3798/nfs-cluster-ls-json-pretty

mgr/pybind/object_format: fix json-pretty being marked invalid

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56957 from adk3798/no-stray-recent-removal

 mgr/cephadm: don't mark daemons created/removed in the last minute as stray

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56823 from adk3798/autotune-testing

mgr/cephadm: additional debug logging for autotuner

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56802 from clwluvw/rgw-mdsearch

rgw: allow disabling mdsearch APIs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57372 from ceph/wip-lusov-quiesce-docs

doc/cephfs: expand the role of quiesce in consistent backups

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore: add is_data_stable() to allow delta-overwrite on EXIST_CLEAN

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #57277 from tchaikov/test_election-free-tracker

test/mon/test_election: free ConnectionTracker if blocked

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #56493 from igomon-bloomberg/wip_kafka_sasl_username_password_attrs

rgw/s3-notifications: use user-name/password topic attributes for SASL authentication

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57075 from cbodley/wip-rgw-ldap-derr

rgw: don't print error when ldap isn't configured

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #57083 from ivancich/wip-optimize-bucket-check-mulitparts

rgw: make incomplete multipart upload part of bucket check efficient

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57098 from kchheda3/wip-dont-log-passwd

rgw/notification: Do not log endpoint as it could contain broker user-id & password.

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw: update s3select submodule addressing arrow 15.0.0 compilation issues

Update arrow submodule.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/cephfs: expand the role of quiesce in consistent backups

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>
Fixes: https://tracker.ceph.com/issues/65881

Merge pull request #57331 from ronen-fr/wip-rf-tier0

tests/librados: do not use '0' as a string

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57155 from mkogan1/wip-rgw-ssl-ec

rgw/beast: fix crash observed in SSL stream.async_shutdown()

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57360 from cbodley/wip-neorados-rm-completion.h

neorados: remove unused symlink to completion.h

Reviewed-by: Adam Emerson <aemerson@redhat.com>

objclass: deprecate cls_cxx_gather

cls_cxx_gather is not maintained and having issues with retry.
since there is no current use of it, we will deprecate it.

Fixes: https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #55773 from weirdwiz/main

exporter: handle exceptions gracefully

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw/cloud-transition: Use set_atomic() while updating HEAD

LC Cloud transition should use set_atomic() to prevent any overwrite
while updating the HEAD object.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/cloud-transition: fix the crash with accessing invalid attrs

The obj_state may not be valid anymore post LC operations (esp.,
cloud-transition). Hence read and store etag prior to them to be used
later by notification (publish_commit).

Fixes: https://tracker.ceph.com/issues/65862
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56865 from zmc/cmake-sccache-max-jobs

cmake: Intelligently set job limits for sccache

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/common/operation: distruct barrier after the wait future is resolved

Specifically, OrderedExclusivePhaseT::mutex must be unlocked only after
it is locked. Otherwise it can unlock other client requests
unexpectedly, causing incorrect order upon exit.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common/operation: cleanup, merge exit() into ~PipelineExitBarrierI()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>