Merge pull request #59393 from anthonyeleven/caps-man-caps

doc/releases: Correct mimic.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #59253 from clwluvw/copy-source-attrs

rgw: load copy source bucket attrs in putobj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59318 from adk3798/cephadm-osd-original-weight-param

mgr/cephadm: add "original_weight" parameter to OSD class

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #59204 from tchaikov/wip-ceph-volume-deps

ceph-volume: add "packaging" to install_requires

Merge pull request #58990 from Matan-B/wip-matanb-fmt-draft

fmt: bump up version + related changes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge PR #55144 into main

* refs/pull/55144/head:
client: fix file cache cap leak which can stall async read call
test/client: test contiguous read for a non-contiguous write

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #56816 into main

* refs/pull/56816/head:
doc: mention the peer status failed when snapshot created on the remote filesystem.
qa: add test_cephfs_mirror_remote_snap_corrupt_fails_synced_snapshot
cephfs_mirror: update peer status for invalid metadata in remote snapshot

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com>

Merge PR #59166 into main

* refs/pull/59166/head:
mon/thrasher: set stopping

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases: Correct mimic.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #59348 from zdover23/wip-doc-2024-08-20-rados-ops-cache-tiering

doc/rados: document unfound object cache-tiering scenario

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #59362 from gbregman/main

mgr/cephadm: change SPDK RPC fields in nvmeof configuration

Merge pull request #59323 from yuvalif/wip-yuval-67514

test/rgw/notifications: don't check for full queue if topics expired

Reviewed-By: Casey Bodley <cbodley@ibm.com>

Merge pull request #54984 from NitzanMordhai/wip-nitzan-restful-un-boundary-keep-requests

mgr/rest: Trim  requests array and limit size

mgr/cephadm: change SPDK RPC fields in nvmeof configuration
Fixes https://tracker.ceph.com/issues/67629

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

python-common/ceph/deployment: change SPDK RPC fields in nvmeof configuration
Fixes https://tracker.ceph.com/issues/67629

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

doc/rados: document unfound object cache-tiering scenario

Explain how to deal with "unfound objects" when restarting OSDs in a
cache-tiered environment.

Fixes: https://tracker.ceph.com/issues/44286
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58460 from rkachach/fix_issue_oauth2_support

adding support for SSO based on auth2-proxy

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58860 from adk3798/cephadm-nvmeof-require-group

mgr/cephadm: require "group" parameter in nvmeof specs

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #59165 from NitzanMordhai/wip-nitzan-test-rados-tools-newline-trim

test: test_rados_tools compare output without trimming newline

doc/mgr/restful: update max_request config

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

PendingReleaseNotes: Adding note about rest module change and adding max_request option

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

mgr/rest: Trim request array and limit size

Presently, the requests array in the REST module has the potential to grow
indefinitely, leading to excessive memory consumption, particularly when
dealing with lengthy and intricate request results.

To address this issue, a limit will be imposed on the requests array within
the REST module.
This limitation will be governed by the `mgr/restful/x/max_requests` configuration
parameter specific to the REST module.
when submit_request called we will check request array if exceed max_request option
if it does we will check if the future trimmed request finished and log error
message in case we are trimming un-finished requests.

Fixes: https://tracker.ceph.com/issues/59580
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

Merge pull request #59153 from ajarr/wip-67436

rbd: fix CLI output of `rbd group snap info` command when a group snapshot with no member images

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Sunil Angadi <Sunil.Angadi@ibm.com>

Merge pull request #59292 from cyx1231st/wip-seastore-revert-decouple-ool-writes

Revert "crimson/os/seastore: wait ool writes in DeviceSubmission phase"

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #59241 from tobias-urdin/openstack-upperconstraints

qa: barbican: restrict python packages with upper-constraints

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/rgw/notifications: don't check for full queue if topics expired

there are other tests for queue length, so we can skip this check
if test takes too long.
also remove unnecessary delays from the test.

Fixes: https://tracker.ceph.com/issues/67514?tab=history
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

mgr/cephadm: add "original_weight" parameter to OSD class

Fixes: https://tracker.ceph.com/issues/67329
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #58961 from NitzanMordhai/wip-nitzan-dencoder-test-forward-incompat-fix

workunit/dencoder: dencoder test forward incompat fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58594 from jamiepryde/isa-xor-raid

erasure-code/isa: Use isa/raid's xor_gen() instead of the region_xor(…

Reviewed-by: Mark Nelson <mnelson@redhat.com>

qa: barbican: restrict python packages with upper-constraints

We install barbican by doing a pip install directly on the
cloned git repository but we don't honor the upper-constraints
from the OpenStack Requirements project that handles what
versions is supported.

This changes the pip install command that we issue when
installing barbican to honor the requirements for the
version (derived from the branch) that we use, in
this case it's the 2023.1 release upper-constraints [1].

This prevents us from pulling in untested Python packages.

This only updates Barbican because for the Keystone job
we dont directly issue pip but install using tox using the
`venv` environment which already by default sets the
constraints as you can see in [2].

[1] https://releases.openstack.org/constraints/upper/2023.1
[2] https://github.com/openstack/keystone/blob/stable/2023.1/tox.ini#L12

Fixes: https://tracker.ceph.com/issues/67444
Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

Merge pull request #59239 from yuvalif/wip-yuval-67513

Reviewed-By: Casey Bodley <cbodley@ibm.com>
test/rgw/notification: use real ip address instead of localhost

based on that comment:
https://tracker.ceph.com/issues/67206#note-6
the address used by the endpoint is taken as the real IP address of the
host where the test script is running and not localhost.

we also changed the rabbitmq-server conf to allow "guest"
user to connect over non localhost address

Fixes: https://tracker.ceph.com/issues/67206
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59200 from ifed01/wip-ifed-fix-store-test-col-ref

test/store_test: fix assertions due to unclosed collection refs.

Reviewd-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #59256 from zdover23/wip-doc-2024-08-17-cephfs-ceph-dokan-mount-point

doc/cephfs: s/mountpoint/mount point/

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #58995 from rhcs-dashboard/fix-66844-main

qa/mgr/dashboard: fix test race condition

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59212 from cyx1231st/wip-seastore-more-reports

crimson/os/seastore/cache: report lru usage/in/out with trans and extent type

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Revert "crimson/os/seastore: wait ool writes in DeviceSubmission phase"

This reverts commit c9e423facea79d42f0496264f267adee5d911b87.

The commit starts to submit OOL writes before submitting the journal
write, true, but it cannot guarantee that OOL writes finish before the
journal write.

Thus it is possible that during SeaStore restart, a journal record
appears valid but its dependent OOL records are partial written, which
leads to corruption.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59290 from anthonyeleven/mountpoint

doc: Harmonize 'mountpoint'

doc: Harmonize 'mountpoint'

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #59257 from zdover23/wip-doc-2024-08-17-cephfs-mount-point

doc/cephfs: s/mountpoint/mount point/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: s/mountpoint/mount point/

Change the string "mountpoint" to "mount point" in English-language
strings (as opposed to in commands, where the string "mountpoint"
sometimes appears and is correct).

cf. https://github.com/ceph/ceph/pull/58908#discussion_r1697715486 in
which page 345 of The IBM Style Guide is referenced to back up this
change.

This commit alters only English-language text and example commands in
which the string "{mount point}" is meant to be replaced. No commands
meant for cutting-and-pasting have been altered in this commit.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: s/mountpoint/mount point/

Change the string "mountpoint" to "mount point" in English-language
strings (as opposed to in commands, where the string "mountpoint"
sometimes appears and is correct).

cf. https://github.com/ceph/ceph/pull/58908#discussion_r1697715486
in which page 345 of The IBM Style Guide is referenced to back up this
change.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58355 from batrick/ceph-backport-fetchhead

script/ceph-backport: robustness adjustments for local git repo quirks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

test/rgw/notification: use real ip address instead of localhost

based on that comment:
https://tracker.ceph.com/issues/67206#note-6
the address used by the endpoint is taken as the real IP address of the
host where the test script is running and not localhost.

we also changed the rabbitmq-server conf to allow "guest"
user to connect over non localhost address

Fixes: https://tracker.ceph.com/issues/67206
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw: load copy source bucket attrs in putobj

PutObj is not loading the source bucket's attrs, which prevents
the associated policies from being loaded. As a result, the
permissions granted to the user to read from the source bucket are
not being properly evaluated.

Fixes: https://tracker.ceph.com/issues/67590
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #59167 from zdover23/wip-doc-2024-08-12-cephfs-file-layouts

doc/cephfs: improve "layout fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58896 into main

* refs/pull/58896/head:
client: flush the caps release in filesystem sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #59205 from xxhdx1985126/wip-seastore-find-pending-version

crimson/os/seastore/btree: fix minor corner case issue

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

client: flush the caps release in filesystem sync

We have hit a race between cap releases and cap revoke request
that will cause the check_caps() to miss sending a cap revoke ack
to MDS. And the client will depend on the cap release to release
that revoking caps, which could be delayed for some unknown reasons.

In Kclient we have figured out the RCA about race and we need
a way to explictly trigger this manually could help to get rid
of the caps revoke stuck issue.

Fixes: https://tracker.ceph.com/issues/67221
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #58415 from ljflores/wip-tracker-66809

qa/suites/upgrade: ignore PG_AVAILABILITY and MON_DOWN for quincy-x and reef-x upgrade suites

Merge pull request #59220 from ivoalmeida/carbon-datatable-cleanups

mgr/dashboard: carbon datatables impr and cleanups

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59218 from yuvalif/wip-yuval-67525

rgw/notifications: fixing radosgw-admin notification json

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59118 from xxhdx1985126/wip-crimson-backfill-cancellation

crimson/osd/backfill_state: support backfill cancellation

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/backfill_state: support backfilling cancellation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg_recovery: reset backfill_state when backfill finished

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57966 from xxhdx1985126/wip-crimson-concurrent-recover-missing

crimson/osd/osd_operations: make the "recover_missing" phase concurrent

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53151 from xxhdx1985126/wip-crimson-backfill-fixes

crimson/osd/backfill_state: fixes two corner cases in backfilling

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #58911 from yuvalif/wip-yuval-67229

test/cls_2pc_queue: prevent list+remove race between consumers

Reviewed-By: Casey Bodley <cbodley@ibm.com>

Merge pull request #59219 from yuvalif/wip-yuval-50610

doc/rgw/notification: persistent notification queue full behavior

Reviewed-By: Anthony D'Atri <anthony.datri@gmail.com>

crimson/os/seastore/cache: report lru usage/in/out with trans and extent type

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup periodical reporting

Consolidate time into a single place per SeaStore::Shard.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache/lru: renames

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: refine lru logics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: move counter_by_extent_t definition

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/seastore_types: unify checks to the extent types

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59225 from zdover23/wip-doc-2024-08-15-glossary-flapping-osd

doc/glossary: add "flapping OSD"

Merge pull request #59028 from cbodley/wip-67326

rgw/notify: visit() returns copy of owner string

Reviewed-by: Adam Emerson <aemerson@redhat.com>

doc/glossary: add "flapping OSD"

Add an entry for "Flapping OSD" to the glossary.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58448 from cbodley/wip-rgw-lc-async

cls/rgw: define lc ops in terms of ObjectOperation instead of IoCtx

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/dashboard: carbon datatables impr and cleanups

Fixes: https://tracker.ceph.com/issues/67544,
https://tracker.ceph.com/issues/67538,
https://tracker.ceph.com/issues/67542, https://tracker.ceph.com/issues/67545, https://tracker.ceph.com/issues/67546

Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

CMakeLists: set FMT_INSTALL OFF

See: https://github.com/fmtlib/fmt/pull/3264

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #58965 from linuxbox2/wip-lcgt-typo

rgwlc: fix typo in getlc (ObjectSizeGreaterThan)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rgw/notification: persistent notification queue full behavior

Fixes: https://tracker.ceph.com/issues/50610
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59169 from cbodley/wip-67464

rgw: revert account-related changes to get_iam_policy_from_attr()

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #57888 from liangmingyuanneo/wip-standalone-test-pg-repair

qa/standalone: bugfix for latecy repair after scrub

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #59210 from rhcs-dashboard/multi-cluster-overview-usage-template-fix

mgr/dashboard: fix multi-cluster usage bar error after carbon changes

mgr/cephadm: adding oauth2-proxy cephadm service

adding new oauth2-proxy service. The enable_auth flag enables SSO
authentication via the oauth2-proxy service. The user must ensure the
oauth2-proxy service is deployed before enabling this flag in the
mgmt-gateway service.

FQDN related changes: previously, we were obtaining the FQDN using a
call to the Python socket library run inside the container. While this
generally works, the FQDN returned inside a container can sometimes
differ from the one obtained outside the container. This discrepancy
could cause some issues. To ensure consistency, we now use the FQDN
from the inventory, which provides the correct value as recognized on the host.

Signed-off-by: Redouane Kachach <rkachach@ibm.com>

rgw/notifications: fixing radosgw-admin notification json

Fixes: https://tracker.ceph.com/issues/67525
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59168 from zdover23/wip-doc-2024-08-12-cephfs-cache-configuration

doc/cephfs: improve cache-configuration.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph.spec.in: spec files comments

```
RPM build errors:
    extra tokens at the end of %else directive in line 118:  %else # not fedora/rhel

    extra tokens at the end of %else directive in line 121:  %else # not x86_64
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd: use an hex-only format for snap_id when creating snapmapper entries

i.e. - without the optional text representation of NOSNAP & SNAPDIR

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
(cherry picked from commit 547b67fa684b50697b717b3fdcb88d71ae6956df)
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

test/common/test_hobject: fmt fixes

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson: fmt 11 fixes

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

seastar: cherry-pick fmt 11 fixes

Update seastar submodule to:
https://github.com/ceph/seastar/commits/wip-seastar-fmt-11-fixes/

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/scrub: missing 'const' in scrubber_common.h formatter

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd: fix scrub_backend formatters

Adding missing 'const', and placing in the fmt namespace.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd,common: add explicit formatter for shard_id_t

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common,include,msg: fmt 11 fixes

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

fmt: bump up version to 11.0.2

Move fmt submodule branch to
https://github.com/ceph/fmt/commits/wip-fmt-11.0.2/

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/os/seastore/cache: pass missing src to touch_extent()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup add_extent()

Move add_to_dirty() and touch_extent() out of add_extent(), this removes
duplicated calls to touch_extent() from the on_cache callback.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup remove_from_dirty()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: drop duplicated calls to touch_extent()

The extent is already PRESENT, which means it was already touched in
this transaction.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cached_extent: rename primary_ref_list

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: fix multi-cluster usage bar error after carbon changes

fixes: https://tracker.ceph.com/issues/67536

Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #59186 from rhcs-dashboard/replace-cluster-capacity-with-usage-bar

mgr/dashboard: replace individual cluster's capacity info with Usage bar in Multi-Cluster

Merge PR #59025 into main

* refs/pull/59025/head:
tools/rados: Fix extra NL in getxattr

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Gabriel Benhanokh <gbenhano@redhat.com>

Merge pull request #58485 from ivoalmeida/carbon-datatable

mgr/dashboard: replace ngx-datatable by carbon datatable

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore/btree: fix minor corner case issue

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

ceph-volume: add "packaging" to install_requires

in 0985e201, "packaging" was introduced as a runtime dependency of
ceph-volume, and `ceph.spec.in` was updated accordingly to note
this new dependency. but the debian packaging was not updated.

in 80edcd40, the missing dependency was added to debian/control as
one of ceph-volume's runtime dependency.

but dh_python3 is able to figure out the dependencies by reading
the egg's metadata of the ceph-volume python module. and as a
python project, ceph-volume is using its `setup.py` for
tracking its dependencies.

so in order to be more consistent, and keep all of its dependencies
in one place, let's move this dependency to setup.py . as the
packagings in both distros are able to figure the dependencies
from egg-info.

see also
- https://manpages.debian.org/testing/dh-python/dh_python3.1.en.html#dependencies
- https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_201x/#_automatically_generated_dependencies

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

qa/suites/upgrade: ignore MON_DOWN and PG_AVAILABILITY warnings in upgrade tests

Signed-off-by: Laura Flores <lflores@ibm.com>