qa: ignore container checkpoint/restore related selinux denials for centos9

Fixes: http://tracker.ceph.com/issues/66640
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56150 from xxhdx1985126/wip-seastore-teuthology

qa/suites/crimson-rados: add seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58363 from zdover23/wip-doc-2024-07-01-start-hardware-recs

doc/start: s/overprovsioning/overprovisioning/

doc/start: s/overprovsioning/overprovisioning/

Restore a missing "i" in "overprovisioning" in
doc/start/hardware-recommendations.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58102 from yuvalif/wip-yuval-64690

rgw/amqp: fix valgrind errors on uninitialized memory

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/crimson-rados: disable seastore with default thrash

Currently we have 2 types of thrash tests: default and simple.
Seastore passes only the simple one.

The 2 types are seperated to diffrent dirs. `thrash` dir
will be tested with bluestore only.
`thrash_simple` will be tested with both backend.

Once Seastore passes default ones, we can merge back the two dirs

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58124 from myoungwon/wip-fix-66368

debian: add ceph-dedup-daemon to the ceph-test package

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #58358 from zdover23/wip-doc-2024-06-29-cephfs-fs-vols

doc/cephfs: take Anthony's suggestion

doc/cephfs: take Anthony's suggestion

Incorporate Anthony's suggestion in
https://github.com/ceph/ceph/pull/57567#discussion_r1607016050 into
doc/cephfs/fs-volumes.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58133 from TRYTOBE8TME/wip-rgw-null-ver-final

src/rgw: Deleting an object with null version fix

Merge pull request #55945 from lxbsz/wip-64679-new

mds/client: return -ENODATA when xattr doesn't exist for removexattr

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw/multisite: use bilog_flags in cls_rgw_bucket_unlink_instance as well.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #58300 from idryomov/wip-66300-test-fixup

test/librbd: avoid caching effects TestGroup.add_snapshot{,PP}

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58290 from guits/fix_nvme_loop

tests: make sure nvmetcli and nvme-cli are up to date

Merge pull request #58201 from idryomov/wip-66570

librbd: make diff-iterate in fast-diff mode aware of encryption

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58265 from idryomov/wip-rbd-pybind-utc-timestamps

pybind/rbd: change to return "aware" datetime objects

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58286 from rhcs-dashboard/clone-validation

mgr/dashboard: fix clone async validators with different groups

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

test/libcephfs: add ceph_test_libcephfs_vxattr

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

test/libcephfs: add removexattr test support

There have two ways to remove xattrs, removexattr() and setxattr()
with the XATTR_REPLACE flags set.

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

test/libcephfs: move FsCrypt xattr test to dedicated test file

Switch to use XATTR_CREATE.

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds/client: return -ENODATA when xattr doesn't exist for removexattr

The POSIX says we should return -ENODATA when the corresponding
attribute doesn't exist when removing it. While there is one
exception for the acl ones in the local filesystems, for exmaple
for xfs, which will treat it as success.

While in the MDS side there have two ways to remove the xattr:
sending a CEPH_MDS_OP_SETXATTR request by setting the 'flags' with
CEPH_XATTR_REMOVE and just issued a CEPH_MDS_OP_RMXATTR request
directly.

For the first one it will always return 0 when the corresponding
xattr doesn't exist, while for the later one it will return
-ENODATA instead, this should be fixed in MDS to make them to be
consistent.

Added a CEPH_XATTR_REMOVE2 new flags and will return -ENODATA errno
directly when the crresponding xattr doesn't exist. Just keeps the
old CEPH_XATTR_REMOVE flags to make it to be compatible with old
clients.

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

src/test/rgw: Adding a test for check

Addition of a test to rgw_multi to check the
working of deleting an object with null version.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw/multisite: add a flag to bilog_flags and use it to set/unset null version

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge PR #57619 into main

* refs/pull/57619/head:
qa/cephfs: use wait_for_daemon() instead of sleep()-ing
qa/cephfs: mark file system joinable for fs rename tests before unmounting clients

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57271 from tj90241/a-better-crc32c

common: Leverage a better CRC32C implementation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #57107 from ifed01/wip-ifed-async-compact

os/bluestore: enable async manual compactions

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #56334 from NitzanMordhai/wip-nitzan-wathcnotify3-timout-increase

test/librados: WatchNotify3 increase timeout

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57581 from adamemerson/wip-boost-1.85

build: Bump boost version to 1.85

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/crimson-rados: add seastore

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge PR #53755 into main

* refs/pull/53755/head:
PendingReleaseNotes: add note about CephFS set_vxattrs
doc/cephfs: Update docs to match remove functionality and respective vxattrs
qa: Add test coverage for vxattr behavior
qa: Add removexattr to support setfattr removal.
mds: Implement remove for ceph vxattrs

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #56476 into main

* refs/pull/56476/head:
qa: add README to every fs:upgrade sub-suite
qa: upgrade sub-suite mds_upgrade_sequence from n-1 and n-2 releases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57632 from cbodley/wip-http-mgr-locking

rgw/http: check 'done' under mutex

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

tests: make sure nvmetcli and nvme-cli are up to date

Otherwise jobs end up with the following failure:

```
2024-06-25T14:22:18.659 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to write to /dev/nvme-fabrics: Invalid argument
```

Also, the output of nvme list has changed so we have to update
qa/tasks/nvme_loop.py accordingly.

Fixes: https://tracker.ceph.com/issues/66707
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #58287 from pereman2/libradospp-example-doc

doc/rados: update how to install c++ header files

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #58299 from nbalacha/wip-nbalacha-doc-typos

doc/dev: Fix typos in encoding.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58036 from Suyashd999/core-false-positives

osd: suppressing two false clang-tidy detections

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #58279 from rhcs-dashboard/cephfs-nav-fix

mgr/dashboard: fix nav urls for cephfs and nfs

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

test/librbd: avoid caching effects TestGroup.add_snapshot{,PP}

After rollback started being tested in commit b3977c53c930
("test/librbd: make rollback in TestGroup.add_snapshot{,PP}
meaningful"), these tests can fail on comparing post-rollback
data to expected data if run with exclusive lock disabled.
This doesn't occur with exclusive lock enabled because the RBD
cache gets invalidated implicitly before releasing the lock.

While at it, pass LIBRADOS_OP_FLAG_FADVISE_FUA to avoid relying
on any cache settings that happen to be in effect.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: give groups unique names in TestGroup tests

Otherwise, a failing add_snapshot test causes add_snapshotPP test to
fail too, for example.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: drop unused variable in TestGroup.group_create

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/dev: Fix typos in encoding.rst

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #57709 from xxhdx1985126/wip-seastore-fix-parent_invalid-mappings

crimson/os/seastore: avoid getting wrong logical extents through "parent-invalid" lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58209 from connorfawcett/gcov-fix

nasm-wrapper: Ignore --coverage flag passed to nasm

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #52568 from smanjara/wip-rgw-mdlog-polling

rgw/multisite: metadata polling event based on unmodified mdlog_marker

Merge pull request #58229 from cbodley/wip-dout-fmt

dout: add macros for libfmt-style logging

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #58126 from NitzanMordhai/wip-nitzan-suites-rados-cephadm-stray-daemon

suites: adding ignore list for stray daemon

doc/rados: update how to install c++ header files

In this example librados2-devel only install C header files on fedora 40,
therefore I added libradospp-devel to the command to include C++ header files.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #57125 from xxhdx1985126/wip-65679

crimson/osd/pg: reset the snap mapper's backend when pg interval changes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: cephfs snapshot and clones e2e tests

Fixes: https://tracker.ceph.com/issues/65180
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 3d1a7c56ccb8eb6d988e034e99e199cd226b1704)

mgr/dashboard: fix clone async validators with different groups

Providing a way to dynamically update the async validator based on the
selector field so that when the selected value changes, the depended
field like the clone name gets validated again against the new value

Fixes: https://tracker.ceph.com/issues/66703
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix nav urls for cephfs and nfs

Fixes: https://tracker.ceph.com/issues/66697
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58270 from Svelar/adminsocket

test/admin_socket: increase slack

Merge pull request #58226 from zdover23/wip-doc-2024-06-24-rados-troubleshooting-osd-debugging-slow-requests

doc/rados: edit troubleshooting-osd.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

test/admin_socket: increase slack

fixes: https://tracker.ceph.com/issues/66638

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

test/crimson/seastore/test_transaction_manager: add a UT case for
detecting invalid lba mappings

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: fix lba mappings before getting
logical extents through them

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/btree: add interfaces to check whether the mappings'
parents have been modified

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction: minor bug fix

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction_manager: assert that the parents of the
mappings following which logical extents are to be retrieved are valid

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/cache: add an efficient method to check if extents are
viewable to transactions

Instead of searching the transaction's retired_set to determine whether
an extent has been retired, we add the transaction that's retiring an
extent to that extent's retired_transactions field and search that field
to do the check. Since the probability of multiple transactions retiring
the same extent is very low, this approach should be more cpu efficient.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/os/seastore/transaction: make Transaction::is_retired() an
accurate extent check

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

dout: add macros for libfmt-style logging

new dout macros use fmt::print() to write output directly to the
underlying ostream. this enables libfmt's printf-style formatting syntax
for ceph log output, without the string allocation overhead of:

    dout(10) << fmt::format("str={} count={}", str, count) << dendl;

which becomes:

    dout_fmt(10, "str={} count={}", str, count);

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #57986 from rhcs-dashboard/fix_duplicated_token_status_polling

mgr/dashboard: fix token status being polled continously

Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge PR #53503 into main

* refs/pull/53503/head:
qa: add tests for `mds last-seen` command
doc/cephfs: add documentation for `mds last-seen`
PendingReleaseNotes: add note on last-seen command
mon/MDSMonitor: add command to lookup when mds was last seen
mon/MDSMonitor: set birth time on FSMap during encode
pybind/mgr/dashboard: show context diff for openapi check

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #56876 from pponnuvel/no_crash_if_nfs_unavailable

mgr/nfs: Don't crash ceph-mgr if NFS clusters are unavailable.

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

pybind/rbd: change to return "aware" datetime objects

utcfromtimestamp() and utcnow() have been deprecated in Python 3.12.
Let's follow suit because it turns out that many datetime methods in
Python 3 interpret "naive" objects that we are currently returning as
local times.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #53849 from dvanders/dvanders_alma

install-deps.sh, do_cmake.sh: almalinux is another el flavour

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #58141 from ljflores/wip-tracker-65852

qa/suites/rados/thrash/workloads: remove cache tiering workload

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #58139 from bill-scales/fix_num_shards_repaired_stat

osd/ECBackend.cc: Fix double increment of num_shards_repaired stat

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57401 from kamoltat/wip-ksirivad-fix-65826

qa/tasks/mgr/test_progress.py: deal with pre-exisiting pool

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #56531 from rzarzynski/wip-bug-65183

mon, qa: suites override ec profiles with --yes_i_really_mean_it; monitors accept that

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #58192 from kshtsk/wip-suse-remove-zoned-namespace

os/bluestore: remove zoned namespace support

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #58243 from zdover23/wip-doc-2024-06-25-rados-ops-stretch-mode-credit

doc/rados: credit Prashant for a procedure

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

nasm-wrapper: Exclude --coverage flag from being passed to nasm

A previous fix for compiling with gcov code coverage enabled forgot to
stop the --coverage flag being passed to nasm during compile,
this breaks compilation as nasm doesn't recognise this flag.

Fixes: https://tracker.ceph.com/issues/66601
Signed-off-by: Connor Fawcett <connorfa@uk.ibm.com>

Merge pull request #58223 from Matan-B/wip-matanb-crimson-revert-56994

Revert "crimson/osd/osd_operation: fix dump_historic_slow_ops command…

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #58110 from rhcs-dashboard/bring-back-cluster-swticher

mgr/dashboard: fix (multi) cluster switcher disappearance after carbon UI shell

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>

Merge pull request #57930 from rhcs-dashboard/get-replication-info

mgr/dashboard: get and update bucket replication policies

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

doc/rados: credit Prashant for a procedure

Credit Prashant D for creating the stretch-mode workaround procedure for
retrieving the correct size of datacenters.

Follows: https://github.com/ceph/ceph/pull/58109

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #56052 into main

* refs/pull/56052/head:
qa/suites: ignore unresponsive client when the test passes
qa: enhance per-client labelled perf counters test

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #56429 into main

* refs/pull/56429/head:
mds: fix rank root doesn't insert root ino into its subtree map when starting
mds: flush mds log before finishing STATE_STARTING
mds/FSMap: go back to STARTING state when rank doesn't make it pass STARTING

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #57034 into main

* refs/pull/57034/head:
qa: cleanup snapshots before subvolume delete

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57085 into main

* refs/pull/57085/head:
mds: set the correct WRLOCK flag always in wrlock_force()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #57576 into main

* refs/pull/57576/head:
src/test: add testcase file for low level sync io code paths

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/dashboard: fix token status being polled continously

subscribe only once rather than continously subscribing

Fixes: https://tracker.ceph.com/issues/66665
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix (multi) cluster switcher disappearance after carbon UI

Fixes: https://tracker.ceph.com/issues/66538
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #57001 from rhcs-dashboard/cheroot-upgrade

mgr/dashboard: bump cheroot to > 10.0

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>

Merge pull request #57977 from athanatos/sjust/wip-66308-obc-locking

crimson: simplify obc loading by locking excl for load and demoting to needed lock

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58142 from adk3798/rgw-zone-create-error-handling

mgr/rgw: fix error handling in rgw zone create

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57339 from phlogistonjohn/jjm-smb-login-control

smb: add login control access parameters to share resource

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

doc/rados: edit troubleshooting-osd.rst

Make minor changes to the "Debugging Slow Requests" section of
doc/rados/troubleshooting/troubleshooting-osd.rst in preparation
for an expansion of this section in response to a reqeust from Joel
Davidow.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #57927 from rhcs-dashboard/expand-cluster-improvements

mgr/dashboard: Expand Cluster improvements

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56658 from afreen23/wip-nfs-route

mgr/dashboard: Fix NFS routing

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/osd/pg: reset the snap mapper's backend when pg interval changes

Fixes: https://tracker.ceph.com/issues/65679
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Revert "crimson/osd/osd_operation: fix dump_historic_slow_ops command works"

This reverts commit 834ab99efc6453f91183a47849f56617cf73c112.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/.../object_context_loader: add missing dpp argument to debug statement

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../tri_mutex: use hobject_t for tri_mutex::name debug value

Useful for distinguishing between tri_mutex instances on different clones.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../tri_mutex: no need to store waiter_name on waiter_t

It's going to be the same as tri_mutex::name for any waiter on
a tri_mutex.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../object_context: drop recovery_read_marker

This doesn't seem to serve a purpose with current crimson.  classic
uses ObjectState::recovery_read_marker to indicate that backfill
should be requeued upon wakeup, but that hasn't been necessary so
far in crimson.  We can reintroduce this if it becomes useful.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../object_context: drop unused maybe_get_excl

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../object_context: remove loading_mutex

Now that we acquire and demote an exclusive lock during loading without
blocking, a seperate loading_mutex shouldn't be necessary.

Signed-off-by: Samuel Just <sjust@redhat.com>