qa: add test for cephx path check on unlinked snapped dir tree

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add debugging for stray_prior_path

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #58407 from nbalacha/wip-nbalacha-rbd-qa-fixes

qa/workunits/rbd: minor rbd_mirror_helpers.sh fixes

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58403 from ivancich/wip-fix-bucket-sync-thread

rgw: fix bucket sync thread waiting inordinate amount of time

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58378 from YaZasnyal/fix/discard_buffer_size

rgw: DoS when QuotaExceeded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58119 from Suyashd999/fix-uam8

rgw_user: 'keys' is in undefined state after being moved.

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #58339 from clwluvw/s3website-no-host

rgw: use s3website REST when higher priority and no host header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58399 from cbodley/wip-rgw-deshard-encoding

cls/rgw: bump cls_rgw_reshard_entry decode version to match encode

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #57060 from awojno-bloomberg/wip-awojno-status-header

rgw: implement x-amz-replication-status for PENDING & COMPLETED

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/rbd: minor fixes

Corrected paths and added missing spaces.

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #57985 from thotz/add-storage-adminops-api

rgw/adminops: add option to provide storageclass adminops user apis

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #57858 from adamemerson/wip-66340

rgw/multisite: Fix use-after-move in retry logic in logbacking

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>
Reviewed-by: Suyash Dongre <suyashd999@gmail.com>

rgw: fix bucket sync thread waiting inordinate amount of time

A signed value was read in as an unsigned value, so -1 was interpreted
as a very large value. This made the thread wait period in the bucket
sync thread inordinately long, preventing bucket sync and dynamic
resharding (unless values appropriate for debugging were set).

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #58388 from sseshasa/wip-fix-perf-basic-failure

qa/tasks: Initialize 'monitoring_profiles' spec to an empty dict

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #58375 into main

* refs/pull/58375/head:
qa/crontab: convert fs main run to monthly at higher priority

Reviewed-by: Venky Shankar <vshankar@redhat.com>

cls/rgw: bump cls_rgw_reshard_entry decode version to match encode

9302fbb3f5416871c1978af5d45f3bf568c2c190 bumped the version in
ENCODE_START() but missed DECODE_START(). i don't think that would cause
any decode failures, unless we later raise ENCODE_START's compat_v above
2

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58081 from sajibreadd/wip-65861

notifications: report an error when persistent queue deletion failed

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #57537 from kchheda3/wip_fix_notification_caching

rgw/notification: Fix the caching issues of notification brokers, where the cache was not invalidated if topic attributes were changed

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

qa/tasks: Initialize 'monitoring_profiles' spec to an empty dict

Initialize 'monitoring_profiles' to an empty python dictionary instead of
'None' to prevent the cbt task from failing due to the TypeError exception
when attempting to iterate a 'NoneType'.

The bug was introduced as part of https://github.com/ceph/ceph/pull/51438
and commit e174c6e2cf6c90bd130320eb1f251b62d4b4d6c2.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
Fixes: https://tracker.ceph.com/issues/66799

Merge pull request #55564 from adk3798/cephadm-cert-store

mgr/cephadm: Add Cert/Key Store

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #58092 from zdover23/wip-doc-2024-06-18-start-os-recs

doc/start: remove mention of Centos 8 support

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dan Mick <dan.mick@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #58230 from rkachach/fix_secure_monitoring_check

mgr/cephadm: redeploy when some dependency daemon is add/removed

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58154 from tengjie5/tmp3

mgr: excute cmd 'ceph orch apply osd' returned without error info

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57975 from dvanders/dvanders_ipv4

cephadm: disable ms_bind_ipv4 if we will enable ms_bind_ipv6

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57848 from thegreenbear/cephadm-haproxy-bugged-config

cephadm/services/ingress: fixed keepalived config bug

Reviewed-by: Adam King <adking@rehat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #57338 from phlogistonjohn/jjm-mgr-module-cleanups

pybind/mgr: mgr_module.py pep8/flake8 cleanups & tox updates

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #57394 from phlogistonjohn/jjm-smb-module-testing

smb: improve smb module test coverage

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

doc/start: remove mention of Centos 8 support

Remove information so that it is clear that Centos 8 will no longer be
supported by Ceph.

Signed-off-by: Zac Dover <zac.dover@proton.me>

rgw: Changed discard buffer size

Signed-off-by: Artem Vasilev <art.a.vasilev@tbank.ru>

qa/crontab: convert fs main run to monthly at higher priority

The lab can't keep up with current nightly load; reduce fs runs on main branch
to once a month but raise priority so we have a reliable baseline.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #57238 from rhcs-dashboard/lifecycle-policies

mgr/dashboard: add RGW lifecycle management

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #58366 from zdover23/wip-doc-2024-07-01-cephfs-mount-prereqs

doc/cephfs: update cephfs-shell link

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #58260 from rhcs-dashboard/rgw-sync-policy

mgr/dashborad: RGW sync policy

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #56150 from xxhdx1985126/wip-seastore-teuthology

qa/suites/crimson-rados: add seastore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: add RGW lifecycle management

Fixes: https://tracker.ceph.com/issues/50327
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

mgr/dashborad: RGW sync policy

Fixes: https://tracker.ceph.com/issues/66576
Signed-off-by: Naman Munet <nmunet@redhat.com>

doc/cephfs: update cephfs-shell link

Repair the link to cephfs-shell.rst in doc/cephfs/cephfs-shell.rst that
was broken in https://github.com/ceph/ceph/pull/41165/ when
doc/cephfs/cephfs-shell.rst was moved to doc/man/8/cephfs-shell.rst.

This commit is made in response to a request by Lander Duncan that was
made on the [ceph-users] mailing list, and can be seen here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/F7V4CWLIYCAJ4JXI2JLNY6QPCFPR4SLA/

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58363 from zdover23/wip-doc-2024-07-01-start-hardware-recs

doc/start: s/overprovsioning/overprovisioning/

doc/start: s/overprovsioning/overprovisioning/

Restore a missing "i" in "overprovisioning" in
doc/start/hardware-recommendations.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58102 from yuvalif/wip-yuval-64690

rgw/amqp: fix valgrind errors on uninitialized memory

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/crimson-rados: disable seastore with default thrash

Currently we have 2 types of thrash tests: default and simple.
Seastore passes only the simple one.

The 2 types are seperated to diffrent dirs. `thrash` dir
will be tested with bluestore only.
`thrash_simple` will be tested with both backend.

Once Seastore passes default ones, we can merge back the two dirs

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58124 from myoungwon/wip-fix-66368

debian: add ceph-dedup-daemon to the ceph-test package

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #58358 from zdover23/wip-doc-2024-06-29-cephfs-fs-vols

doc/cephfs: take Anthony's suggestion

doc/cephfs: take Anthony's suggestion

Incorporate Anthony's suggestion in
https://github.com/ceph/ceph/pull/57567#discussion_r1607016050 into
doc/cephfs/fs-volumes.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58133 from TRYTOBE8TME/wip-rgw-null-ver-final

src/rgw: Deleting an object with null version fix

rgw: correct tag checking logic in bucket_exports_object

Signed-off-by: Alex Wojno <awojno@bloomberg.net>

Merge pull request #55945 from lxbsz/wip-64679-new

mds/client: return -ENODATA when xattr doesn't exist for removexattr

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw/multisite: use bilog_flags in cls_rgw_bucket_unlink_instance as well.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: PubObj uses RGWObjTags by value

stop wrapping RGWObjTags in a unique_ptr, and just use empty() or count() to determine whether any tags are present

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58300 from idryomov/wip-66300-test-fixup

test/librbd: avoid caching effects TestGroup.add_snapshot{,PP}

Reviewed-by: Ramana Raja <rraja@redhat.com>

rgw: use s3website REST when higher priority and no host header

If the Host header is missing, but the s3website API is enabled and
has a higher priority than the regular S3 API, then we should still
treat the request as a website request.

Fixes: https://tracker.ceph.com/issues/66748
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #58290 from guits/fix_nvme_loop

tests: make sure nvmetcli and nvme-cli are up to date

Merge pull request #58201 from idryomov/wip-66570

librbd: make diff-iterate in fast-diff mode aware of encryption

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58265 from idryomov/wip-rbd-pybind-utc-timestamps

pybind/rbd: change to return "aware" datetime objects

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58286 from rhcs-dashboard/clone-validation

mgr/dashboard: fix clone async validators with different groups

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

test/libcephfs: add ceph_test_libcephfs_vxattr

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

test/libcephfs: add removexattr test support

There have two ways to remove xattrs, removexattr() and setxattr()
with the XATTR_REPLACE flags set.

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

test/libcephfs: move FsCrypt xattr test to dedicated test file

Switch to use XATTR_CREATE.

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds/client: return -ENODATA when xattr doesn't exist for removexattr

The POSIX says we should return -ENODATA when the corresponding
attribute doesn't exist when removing it. While there is one
exception for the acl ones in the local filesystems, for exmaple
for xfs, which will treat it as success.

While in the MDS side there have two ways to remove the xattr:
sending a CEPH_MDS_OP_SETXATTR request by setting the 'flags' with
CEPH_XATTR_REMOVE and just issued a CEPH_MDS_OP_RMXATTR request
directly.

For the first one it will always return 0 when the corresponding
xattr doesn't exist, while for the later one it will return
-ENODATA instead, this should be fixed in MDS to make them to be
consistent.

Added a CEPH_XATTR_REMOVE2 new flags and will return -ENODATA errno
directly when the crresponding xattr doesn't exist. Just keeps the
old CEPH_XATTR_REMOVE flags to make it to be compatible with old
clients.

Fixes: https://tracker.ceph.com/issues/64679
Signed-off-by: Xiubo Li <xiubli@redhat.com>

src/test/rgw: Adding a test for check

Addition of a test to rgw_multi to check the
working of deleting an object with null version.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

rgw/multisite: add a flag to bilog_flags and use it to set/unset null version

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge PR #57619 into main

* refs/pull/57619/head:
qa/cephfs: use wait_for_daemon() instead of sleep()-ing
qa/cephfs: mark file system joinable for fs rename tests before unmounting clients

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #57271 from tj90241/a-better-crc32c

common: Leverage a better CRC32C implementation

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #57107 from ifed01/wip-ifed-async-compact

os/bluestore: enable async manual compactions

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #56334 from NitzanMordhai/wip-nitzan-wathcnotify3-timout-increase

test/librados: WatchNotify3 increase timeout

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #57581 from adamemerson/wip-boost-1.85

build: Bump boost version to 1.85

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/crimson-rados: add seastore

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge PR #53755 into main

* refs/pull/53755/head:
PendingReleaseNotes: add note about CephFS set_vxattrs
doc/cephfs: Update docs to match remove functionality and respective vxattrs
qa: Add test coverage for vxattr behavior
qa: Add removexattr to support setfattr removal.
mds: Implement remove for ceph vxattrs

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge PR #56476 into main

* refs/pull/56476/head:
qa: add README to every fs:upgrade sub-suite
qa: upgrade sub-suite mds_upgrade_sequence from n-1 and n-2 releases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #57632 from cbodley/wip-http-mgr-locking

rgw/http: check 'done' under mutex

Reviewed-by: Yuval Lifshitz <ylifshit@ibm.com>

tests: make sure nvmetcli and nvme-cli are up to date

Otherwise jobs end up with the following failure:

```
2024-06-25T14:22:18.659 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to write to /dev/nvme-fabrics: Invalid argument
```

Also, the output of nvme list has changed so we have to update
qa/tasks/nvme_loop.py accordingly.

Fixes: https://tracker.ceph.com/issues/66707
Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #58287 from pereman2/libradospp-example-doc

doc/rados: update how to install c++ header files

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #58299 from nbalacha/wip-nbalacha-doc-typos

doc/dev: Fix typos in encoding.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58036 from Suyashd999/core-false-positives

osd: suppressing two false clang-tidy detections

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

'keys' used after being moved.

Fixes: https://tracker.ceph.com/issues/66553
Signed-off-by: Suyash Dongre <suyashd999@gmail.com>

Merge pull request #58279 from rhcs-dashboard/cephfs-nav-fix

mgr/dashboard: fix nav urls for cephfs and nfs

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>

test/librbd: avoid caching effects TestGroup.add_snapshot{,PP}

After rollback started being tested in commit b3977c53c930
("test/librbd: make rollback in TestGroup.add_snapshot{,PP}
meaningful"), these tests can fail on comparing post-rollback
data to expected data if run with exclusive lock disabled.
This doesn't occur with exclusive lock enabled because the RBD
cache gets invalidated implicitly before releasing the lock.

While at it, pass LIBRADOS_OP_FLAG_FADVISE_FUA to avoid relying
on any cache settings that happen to be in effect.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: give groups unique names in TestGroup tests

Otherwise, a failing add_snapshot test causes add_snapshotPP test to
fail too, for example.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: drop unused variable in TestGroup.group_create

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/dev: Fix typos in encoding.rst

Signed-off-by: N Balachandran <nibalach@redhat.com>

Merge pull request #57709 from xxhdx1985126/wip-seastore-fix-parent_invalid-mappings

crimson/os/seastore: avoid getting wrong logical extents through "parent-invalid" lba mappings

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #58209 from connorfawcett/gcov-fix

nasm-wrapper: Ignore --coverage flag passed to nasm

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

mgr/cephadm: add migration for certs/keys to cert store

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move ingress cert to cert store

In an effort to try and organize our cert handling
and provide cert management features in the future

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: store iscsi cert and key in cert store

In an effort to organize our handling of certs and
offer cert management functionality in the future

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move rgw frontend ssl cert to cert store

In an effort to organize our certs and in the future
add cert management features.

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move node-exporter cert/key to cert store

We weren't storing this before, but we want to
add some cert management stuff later, so it's
good to start doing so

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move prometheus daemon cert to cert store

We weren't storing this before, but since we want to
add more options for cert management, it's good to start
doing so

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move alertmanager crt/key to cert store

We weren't actually even storing these before, but
given we want to be able to offer some more cert
management options in the future, it's good to start
doing so

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: add cert-store ls commands

To see what certs cephadm has stored in
its internal cert store

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move grafana cert/key to cert/key store

In an effort to try and better track what certs/keys
we manage with cephadm

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move agent endpoint root cert/key to cert store

Trying to move all certs/keys cephadm manages to
a central location

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: move service discovery cert/key to cert store

Trying to move all of the certs/keys cephadm manages
into one place.

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: add a Cert/Key tracking/storage class

The idea is to move storage/handling of certs from any misc.
spot in the mgr/cephadm codebase into a single class. This will
make it much easier to do things with the certs we have in
the future.

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #52568 from smanjara/wip-rgw-mdlog-polling

rgw/multisite: metadata polling event based on unmodified mdlog_marker

Merge pull request #58229 from cbodley/wip-dout-fmt

dout: add macros for libfmt-style logging

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #58126 from NitzanMordhai/wip-nitzan-suites-rados-cephadm-stray-daemon

suites: adding ignore list for stray daemon

doc/rados: update how to install c++ header files

In this example librados2-devel only install C header files on fedora 40,
therefore I added libradospp-devel to the command to include C++ header files.

Signed-off-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #57125 from xxhdx1985126/wip-65679

crimson/osd/pg: reset the snap mapper's backend when pg interval changes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: cephfs snapshot and clones e2e tests

Fixes: https://tracker.ceph.com/issues/65180
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 3d1a7c56ccb8eb6d988e034e99e199cd226b1704)