qa/cephfs: block buggy tests in test_admin.py

Block test_idem_unaffected_root_squash temporarily and
test_multifs_single_path_rootsquash.

This test fails due to a known bug. Block it temporarily so that
test_admin.py can run fully and PRs under QA can be tested fully.
Otherwise, this test fails and that halts test_admin.py, which leaves
the PR partially untested.

This failure is then seen as an unrelated failure which lets the buggy
code get merged. This has happened recently.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #56640 from NitzanMordhai/wip-nitzan-public-addr-check-subnet-unreachable

common/pick_address: check if address in subnet all public address

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Prashant D <pdhange@redhat.com>

Merge pull request #57479 from ktdreyer/wip-66016-qatlib-arches

cmake: disable WITH_QATLIB/ZIP on non-x86

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56743 from NitzanMordhai/wip-nitzan-backword-forword-dencoder-tests

suites: adding dencoder test multi versions

Merge PR #56944 into main

* refs/pull/56944/head:
qa: add a YAML to ignore MGR_DOWN warning

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55735 from xxhdx1985126/wip-crimson-errorator-void-handler

crimson/common/errorator: disallow void-returning error handlers

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #57432 from cyx1231st/wip-seastore-cleanup-objdata-read

crimson/os/seastore/object_data_handler: cleanup read()

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge PR #56941 into main

* refs/pull/56941/head:
mds: find a new head for the batch ops when the head is dead

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57275 into main

* refs/pull/57275/head:
qa/fsx: use a specified sha1 to build the xfstest-dev

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

suites: adding dencoder test multi versions

We are currently conducting regular ceph-dencoder tests for backward compatibility.
However, we are omitting tests for forward compatibility.
This suite will introduce tests against the ceph-objects-corpus to address forward
compatibility issues that may arise.
the script will install N-2 version and run against the latest version corpus objects
that we have, then install N-1 to N version and check them as well.

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge PR #57454 into main

* refs/pull/57454/head:
mds/quiesce-db: optimize peer updates
mds/quiesce-db: track db epoch separately from the membership epoch
mds/quiesce-db: test that a peer on a newer membership epoch can ack a root

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57274 into main

* refs/pull/57274/head:
mds: don't stall the asok thread for flush commands
qa/quiescer: relax some timing requirements in the quiescer

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57329 into main

* refs/pull/57329/head:
qa: unmount clients before damaging the fs

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57334 into main

* refs/pull/57334/head:
mds: remove erroneous debug message

Reviewed-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57476 from cyx1231st/wip-seastore-drop-unused

crimson/os/seastore/transaction_manager: drop unused code

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #55985 from Svelar/make-dist

make-dist: remove old cruft recursively

Merge pull request #56349 from rhcs-dashboard/add-landing-page-upgrade-notification

mgr/dashboard: add upgrade notification

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge PR #57478 into main

* refs/pull/57478/head:
.github: remove description for action

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57137 from sajibreadd/wip-65228

osd: CEPH_OSD_OP_FLAG_BYPASS_CLEAN_CACHE flag is passed from ECBackend

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #56804 from sajibreadd/seg-fault-osd-shutdown-64373

osd: fix for segmentation fault on OSD fast shutdown

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #55298 from ifed01/wip-ifed-fsck-stats

os/bluestore: show some additional osd/pool stats on fsck completion.

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

.github: remove description for action

It is rejected by github's parser.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57099 from ceph/wip-lusov-quiesce-await-eperm

mds/quiesce: db: quiesce-await should EPERM if a set is past QS_QUIESCED
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #56882 from guits/cv-65477

ceph-volume: create LVs when using partitions

Merge pull request #57138 from guits/node-proxy-improvements

node-proxy: make the daemon discover endpoints

crimson/os/seastore/transaction_manager: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57164 from guits/bump-promtail

cephadm: change loki/promtail default image tags

Merge pull request #57000 from chenyuanrun/fix-mkdir-p

ceph-volume: use os.makedirs for mkdir_p

Merge PR #57453 into main

* refs/pull/57453/head:
doc: add status badge for backport creation
.github: use shorter name for backport tracker action
.github: document where runs/output can be examined

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #57391 from soumyakoduri/wip-skoduri-sync-policy

Add new sync-policy related params to boto3 extension

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: disable WITH_QATLIB/ZIP on non-x86

This feature is only relevant to x86 hosts.

Signed-off-by: Ken Dreyer <kdreyer@ibm.com>
Fixes: https://tracker.ceph.com/issues/66016
Signed-off-by: Ken Dreyer <kdreyer@ibm.com>

mgr/dashboard: add upgrade notification

- Adds upgradable component to manage upgrades from outside upgrade component.
- Cleans a couple subscriptions that were not being destroyed

Fixes: https://tracker.ceph.com/issues/65012
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

Merge pull request #53859 from rhcs-dashboard/add-pseudo-validation

mgr/dashboard: add absolute path validation for pseudo path of nfs export

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc: add status badge for backport creation

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

.github: use shorter name for backport tracker action

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

.github: document where runs/output can be examined

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57368 from myoungwon/wip-seastore-is-data-stable

crimson/os/seastore: add is_data_stable() to allow delta-overwrite on EXIST_CLEAN

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: adjust code indention of read()

To make it easier to read, make sure there are 2 spaces per nesting.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: cleanup read() and logs

Specifically, refactor based on read range, pin range and extent range.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: adjust l_current asserts for read()

l_current is only allowed to be larger than pin_key for the first pin.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: cleanup and renames

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #57444 from idryomov/wip-51845

qa/suites/krbd: drop pre-single-major and move "layering only" coverage

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #57434 from cyx1231st/wip-fix-seastore-capture

crimson/os/seastore/record_scanner: replace [=] capturing

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: add comments to add a note that seastore_obj_data_write_amplification needs to be reconsidered

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: replace is_left_stable/is_right_stable with is_left_fresh/is_right_fresh

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #55488 from cyx1231st/wip-crimson-load-obc

crimson/osd/osd_operations/client_request: make loading-obc concurrent

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mds/quiesce-db: optimize peer updates

Prevent sending of the same version to the same peer more than once a second

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-db: track db epoch separately from the membership epoch

Tracking the db epoch separately will make sure that replicas
only follow leader's epoch choice, even if they are already on
the new membership epoch. This eliminates races due to the
random order of mdsmap updates.

Fixes: https://tracker.ceph.com/issues/65977
Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

mds/quiesce-db: test that a peer on a newer membership epoch can ack a root

Signed-off-by: Leonid Usov <leonid.usov@ibm.com>

Merge pull request #57356 from soumyakoduri/wip-skoduri-lc-notify

rgw/cloud-transition: fix the crash with publish_commit

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #56574 from badone/wip-tracker-59196-lazy-omap-regex-segfault

test/lazy-omap-stats: Convert to boost::regex

Add new sync-policy related params to boto3 extension

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #56979 from yuvalif/wip-yuval-65337

rgw/notification: start/stop endpoint managers in notification manager

Reviewed-By: cbodley@ibm.com , kchheda3@bloomberg.net

qa/suites/krbd: rename no-object-map to no-exclusive-lock

Exclusive lock has always been disabled by this facet, so it might as
well be reflected in its name.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: move "layering only" coverage to fsx

It makes much more sense there since it's where we actually create
clones and flatten them a lot.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/krbd: drop pre-single-major test

Single-major mapping scheme was introduced in 2014 and became the
default in 2017.  It's getting increasingly difficult to build and,
more importantly, to boot a 10 year old kernel with recent userspace
(systemd, etc).  If someone is still running such a kernel, it's
really unlikely that they would have the most recent rbd CLI tool
installed.

Fixes: https://tracker.ceph.com/issues/51845
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #56148 into main

* refs/pull/56148/head:
cephfs_mirror: check m_instance_watcher and m_mirror_watcher before using them

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56699 into main

* refs/pull/56699/head:
qa: ignore `Invalid tag char` warning
qa: ignore `object missing on disk` warning

Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Leonid Usov <leonid.usov@ibm.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #56525 into main

* refs/pull/56525/head:
mgr/stats: initialize mx_last_updated in FSPerfStats

Reviewed-by: Neeraj Pratap Singh <neesingh@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56135 into main

* refs/pull/56135/head:
cephfs_mirror: increment sync_failures when sync_perms() and sync_snaps() fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/record_scanner: replace [=] capturing

Capturing with = is error prone and causes compile warning.

And it is wrong:
* cursor should be captured by reference
* mutable keyword is unnecessary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: change loki/promtail default image tags

This commit changes the default images for both loki and promtail
containers.
Also, to allow this update we need to update the configuration of loki
in order to add a new storage schema configuration:

```
    - from: 2024-05-03
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
```

This is because 3.0.0 uses new structured metadata.

See [1][2].

[1] https://grafana.com/docs/loki/latest/operations/storage/schema/
[2] https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/#when-to-use-structured-metadata

Fixes: https://tracker.ceph.com/issues/65784
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #55923 from NitzanMordhai/wip-nitzan-add-deprecate-cls-gather

objclass: deprecate cls_cxx_gather

Merge pull request #56983 from NitzanMordhai/wip-nitzan-thrash-erasure-code-crush-4-nodes-8-6

suites/ec-rados-plugin=jerasure-k=8-m=6-crush: roles set

test/rgw/notifications: make sure that http handler is running

fail test if not. to indicate this is a test issue
and not a product bug

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: prevent flooding the http server

in tests that require retries

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notification: avoid BlockingIOError when logging errors

for example. job: 7697397
in test: yuvalif-2024-05-08_09:55:02-rgw:notifications-wip-yuval-65337-distro-default-smithi

also reduce the side of the error log by sending less objects to the
test_ps_s3_persistent_topic_stats test

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test/rgw/notifications: fix lifecycle tests

* tests were passing only because they were not performings their asserts
* tests are now separated with their own attribute
* their topics are now marked "persistent" to workaround the issue in:
  https://tracker.ceph.com/issues/65645

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: start/stop endpoint managers in notification manager

Fixes: https://tracker.ceph.com/issues/65337
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rados/test: Remove cls_remote_reade since gather deprecated

https://tracker.ceph.com/issues/64258
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

rgw/notifications: cleanup includes and unused parameters

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/notifications: enable notifications debug logs in vstart

as well as in:
- multisite tests (used for notification v2 migration tests)
- the qa suites running notifications

enable lifecycle logs in notification tests: for the lc notification test cases

this is needed after: 429967917b4f9b5f71214fa3c7718e780728249b

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57421 from zdover23/wip-doc-2024-05-12-dev-developer-guide-dash-devel

doc/dev: s/to asses/to assess/

Merge pull request #57420 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x-followup

doc/cephfs: edit fs-volumes.rst (1 of x) followup

doc/dev: s/to asses/to assess/

s/to asses/to assess/ in doc/deve/developer_guide/dash-devel.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: edit fs-volumes.rst (1 of x) followup

Include the suggestions for improving doc/cephfs/fs-volumes.rst made by
Anthony D'Atri here
https://github.com/ceph/ceph/pull/57415#discussion_r1597362110

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57415 from zdover23/wip-doc-2024-05-11-cephfs-fs-volumes-1-of-x

doc/cephfs: edit fs-volumes.rst (1 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: edit fs-volumes.rst (1 of x)

Edit doc/cephfs/fs-volumes to the section "FS Subvolumes" (but not
including the section "FS Subvolumes").

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57214 from adk3798/cephadm-ignore-stray-on-upgrade

qa/cephadm: ignore CEPHADM_STRAY_DAEMON in upgrade tests

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57205 from SidGilad/bug/65739_support_cgroup_split_removal_for_adopt

mgr/cephadm: make setting --cgroups=split configurable for adopted daemons

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #56735 from samarahu/wip-rgw-d4n-long-descriptions

d4n: Add more in-depth config descriptions and improve naming

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Samarah Uriarte <samarah.uriarte@ibm.com>

Merge pull request #56585 from ivancich/wip-upgrade-arrow+flight-15.0.2

rgw: upgrade Apache Arrow submodule to 15.0.0

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57259 from pdvian/wip-fix-unfoundev-progress

mgr/cephadm: Fix unfound progress events

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57182 from adk3798/nvmeof-config-fix

mgr/cephadm:  update default NVMEoF container image version

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57181 from adk3798/cleanup-iscsi-nvmeof-keyring

mgr/cephadm: cleanup iscsi and nvmeof keyrings upon daemon removal

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57096 from phlogistonjohn/jjm-smb-output-fmt

mgr/smb: disable JSON/YAML key sorting in favor of ordered dicts

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57089 from adk3798/test-cephadm-timeout-ignore-refresh-failed

qa/cephadm: ignore CEPHADM_REFRESH_FAILED on timeout test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57080 from phlogistonjohn/jjm-teuth-pull-img

qa/tasks/cephadm: fix pulling containers from private registries

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57042 from adk3798/host-drain-test-ignore-stray-host

qa/cephadm: ignore stray warnings on host drain test

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56970 from adk3798/cephadm-ignore-orch-paused

qa/cephadm: ignore CEPHADM_PAUSED on test_orch_cli test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56963 from adk3798/nfs-cluster-ls-json-pretty

mgr/pybind/object_format: fix json-pretty being marked invalid

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #56957 from adk3798/no-stray-recent-removal

 mgr/cephadm: don't mark daemons created/removed in the last minute as stray

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56823 from adk3798/autotune-testing

mgr/cephadm: additional debug logging for autotuner

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #56802 from clwluvw/rgw-mdsearch

rgw: allow disabling mdsearch APIs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57372 from ceph/wip-lusov-quiesce-docs

doc/cephfs: expand the role of quiesce in consistent backups

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore: add is_data_stable() to allow delta-overwrite on EXIST_CLEAN

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #57277 from tchaikov/test_election-free-tracker

test/mon/test_election: free ConnectionTracker if blocked

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #56493 from igomon-bloomberg/wip_kafka_sasl_username_password_attrs

rgw/s3-notifications: use user-name/password topic attributes for SASL authentication

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57075 from cbodley/wip-rgw-ldap-derr

rgw: don't print error when ldap isn't configured

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #57083 from ivancich/wip-optimize-bucket-check-mulitparts

rgw: make incomplete multipart upload part of bucket check efficient

Reviewed-by: Casey Bodley <cbodley@redhat.com>