crimson/.../replicated_recovery_backend: take excl lock while pushes commit

Fixes: https://tracker.ceph.com/issues/69412
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: route pushes earlier

Let ReplicatedRecoveryBackend::handle_recovery_op route pushes
between handle_push and handle_pull_response instead of
ReplicatedRecoveryBackend::handle_push.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicate_recovery_backend: remove unnecessary check

Already checked in handle_recovery_op.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../object_context_loader: add lock_excl_sync method

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../object_context_loader: add obc get_obc_manager variant

Avoids extra lookup.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: move do_transaction to _handle_pull_response

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: only call _committed_pushed_object if complete

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: convert handle_pull_response to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: convert _handle_pull_response to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: convert submit_push_data to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: convert prep_push_target to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../recovery_backend: convert to logging macros, some formatting changes

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../replicated_recovery_backend: convert to logging macros, some formatting changes

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: clarify RecoveryBackend::handle_backfill_op

Should make it a bit easier to find the two kinds of
message handlers.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #59050 from mohit84/issue_67270

mon: Create a local copy of scrub_state to avoid a crash

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #60984 from ljflores/wip-fix-telemetry-tests

qa/suites/upgrade/telemetry-upgrade: update telemetry tests

Merge pull request #61007 from kamoltat/wip-ksirivad-fix-68602

qa/tasks/thrashosds-health.yaml: Add osds down to ignorelist

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #58052 from jamiepryde/isal-default

core: Change the default plugin for Ceph erasure coded pools from Jerasure to ISA-L

Reviewed-by: Afreen Misbah <afreen23.git@gmail.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #56180 from yehudasa/wip-objecter-local-read

objecter: use configurable to set local and balanced reads

Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #61098 from anoopcs9/add-libcephfs-daemon-containerfile

Containerfile: Add `libcephfs-daemon` package

Merge pull request #61175 from ronen-fr/wip-rf-nvme-this

mon/nvme: fix unused lambda capture warnings
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: baum <alexander.indenbaum@gmail.com>

Merge pull request #61282 from oshrey16/wip-lua-oshrey

rgw/lua: add configurable runtime limit for the lua state

Merge pull request #61388 from AliMasarweh/wip-alimasa-bucket-logging-log-info

RGW\logging: added missing fields

Reviewed-by: yuvalif<ylifshit@redhat.com>

RGW\logging: adding transport layer fields

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #61249 from zdover23/wip-doc-2025-01-07-cephfs-snapshots

doc/cephfs: edit grammar in snapshots.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61267 from athanatos/sjust/wip-rep-pipeline-fix

crimson/.../replicated_request: fix pg lifetime in with_pg finally

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #61442 from zdover23/wip-doc-2025-01-18-cephfs-disaster-recovery-experts

doc/cephfs: edit disaster-recovery-experts (3 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61040 from rhcs-dashboard/smb-module-check

mgr/module: Smb module check

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

doc/cephfs: edit disaster-recovery-experts (3 of x)

Edit the fifth and sixth sections of
doc/cephfs/disaster-recovery-experts.rst in preparation for adding
deeper explanations of the contexts in which one should use the various
commands listed on that page.

The sections edited in this commit are

 - MDS Map Reset
 - Recovery From Mission Metadata Objects

Follows https://github.com/ceph/ceph/pull/61427

https://tracker.ceph.com/issues/69557

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61445 from zdover23/wip-doc-2025-01-19-cephfs-disaster-recovery-experts

doc/cephfs: disaster-recovery-experts cleanup

doc/cephfs: disaster-recovery-experts cleanup

Properly wrap a poorly-formatted paragraph that looks just awful in an
80-column viewport and change MDS to "MDS daemons" where the latter
makes the sentence a lot clearer.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61427 from zdover23/wip-doc-2025-01-17-cephfs-disaster-recovery-experts-2

doc/cephfs: edit disaster-recovery-experts (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

rgw/lua: add configurable runtime limit for the lua state

These changes improve the robustness and reliability of Lua script execution in the RADOS Gateway by enforcing configurable limit on execution time.

- Enhanced `lua_state_guard` to support configurable runtime limits for Lua scripts.
- Updated rgw.yaml.in to include new configuration options for Lua runtime limits.
- Added tests in `test_rgw_lua.cc` to verify Lua script execution with different runtime constraints.
- Updated Lua scripting documentation to reflect the new runtime limit configuration.

Signed-off-by: Oshry Avraham <oshryabiz@gmail.com>

doc/cephfs: edit disaster-recovery-experts (2 of x)

Edit the third and fourth sections of
doc/cephfs/disaster-recovery-experts.rst in preparation for adding
deeper explanations of the contexts in which one should use the various
commands listed on that page.

Follows https://github.com/ceph/ceph/pull/61426

https://tracker.ceph.com/issues/69557

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61426 from zdover23/wip-doc-2025-01-17-cephfs-disaster-recovery-experts

doc/cephfs: remove link to undeveloped tool

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

qa/suites/upgrade/telemetry-upgrade: ignore TELEMETRY_CHANGED warning

Even though we opt in right away after upgrading, which minimizes the chance
of this warning coming up, the cluster log still detects it sometimes and
fails the test. So, we can ignore it since it's expected for the warning
to come up for a few seconds in some cases.

Signed-off-by: Laura Flores <lflores@ibm.com>

qa/workunits: fix some details on telemetry workunits

1. For the telemetry upgrade tests, we want to
   make sure we enable the perf channel before we
   upgrade to test that out.

2. We also want to be sure to set up ident information.
   in the workunits used before the upgrade.

3. We can remove the "opt out" commands in the before-upgrade
   workunits since we want to stay opted in during the upgrade.
   We only opt out after all the checks are complete.

4. We mistakenly test "ceph telemetry show" twice when one
   of then should be "ceph telemetry show-all".

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #61440 from zdover23/wip-doc-2025-01-18-backport-61429-to-squid

doc/releases: edit warning for balancer issues

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/releases: edit warning for balancer issues

Rephrase the balancer-related warning issued to people upgrading to
Ceph 19.2.0.

Follows https://github.com/ceph/ceph/pull/61429

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

qa/suites/upgrade/telemetry-upgrade: replace quincy tests with squid tests

Since main is now pre-Tentacle, we should replace the quincy upgrade
path with squid.

I have kept the quincy workunits since they are still
referenced in the overall upgrade suite. I will remove these
in a separate PR since changing the overall upgrade tests reaches
beyond the scope of just telemetry.

Signed-off-by: Laura Flores <lflores@ibm.com>

qa/workunits: remove inaccurate reference to quincy from telemetry reef-->x test

This comment was mistakenly carried over in
https://github.com/ljflores/ceph/commit/6d25c053127f6aaaf942e674dca30ffd42658b9d.

Signed-off-by: Laura Flores <lflores@ibm.com>

qa/workunits: add new collections to telemetry upgrade workunits (reef)

This commit changes the workunit that tests the upgrade path from
reef to x.

The logic to verify collections is now much more robust. Previously,
the test relied on developers to make it aware of any new collections
it needed to verify. Now, I have added logic to check that the # of
known collections matches the # of reported collections. If it does not,
the test errors out, which will force developers to add any new
collections to the upgrade test along with adding it to the telemetry
code.

The test also verifies all collections instead of focusing only on new
collections that were added after the base collections.

Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #61429 from laimis9133/squid-doc-patch

doc/releases /squid: add warning for balancer issues

Merge pull request #61407 from clwluvw/replication-status

rgw: update attrs with replication status to reflect in response

Reviewed-by: Alex Wojno <awojno@bloomberg.net>

Merge pull request #61406 from clwluvw/checkpoint-dstbucket

rgw: set dest bucket id in bucket sync checkpoint

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #61271 from clwluvw/datasync-usermode-err

rgw: set retcode on verify_bucket_permission in RGWObjFetchCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61157 from aiivashchenko/fix_68195

rgw: add missing last_modified field to swift api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/releases/squid: add warning for balancer issues

This change is to update 19.2.0 Squid release documentation with a warning for users regarding potential placement group balancer issues.

Signed-off-by: Laimis Juzeliunas <laimis.juzeliunas@oxylabs.io>

Merge pull request #61387 from yuvalif/wip-yuval-69473

test/rgw/lua: install luarocks

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #59143 from Sean10/fix_rados_pybind_zero_param

pybind/rados: fix the incorrect order of offset,length in WriteOp.zero

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #61384 from sj14/sj/delete-policy-204

rgw: S3 Delete Bucket Policy should return 204 on success

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61370 from yehudasa/wip-69536

rgw: RGWRados::get_olh() needs to use the correct attr

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61285 from cbodley/wip-69462

rgw: /admin/user api dumps account-related user info

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #61244 from awojno-bloomberg/fix-bucket-sync-status

rgw: fix radosgw-admin use after free in bucket sync status

Reviewed-By: Adam Emerson <aemerson@ibm.com>
Reviewed-By: Oguzhan Ozmen <oozmen@bloomberg.net>

doc/cephfs: remove link to undeveloped tool

Remove a link to a tracker for a planned-but-never-developed tool that
would have facilitated the importation or exportation of a RADOS pool to
a file. This is a part of a project to improve the CephFS Disaster
Recovery documentation, which was requested by Eugen Block. See the
tracker issue below for more on this project.

Follows https://github.com/ceph/ceph/pull/61411

See https://github.com/ceph/ceph/pull/61411#discussion_r1918525109

https://tracker.ceph.com/issues/69557

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61411 from zdover23/wip-doc-2025-01-16-cephfs-disaster-recovery-experts

doc/cephfs: edit disaster-recovery-experts

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #59155 from badone/wip-tracker-67489-fixed-4-for-verify

qa/suites: Make fixed-4 an option for verify tests

Merge pull request #60556 from aclamk/wip-aclamk-bluefs-truncate-allocations-main

os/bluestore: Make truncate() drop unused allocations - addendum

Merge pull request #60363 from aclamk/wip-aclamk-fix-bluefs-bdev-expand

os/bluestore: Create additional bdev labels when expanding block device.

Merge pull request #61369 from laimis9133/patch-2

doc/radosgw/config-ref: fix lifecycle workload tuning description

Merge pull request #61412 from gbregman/main

mgr/cephadm/nvmeof: Add SPDK iobuf options field to NVMeOF configuration

Merge pull request #60265 from MaxKellermann/timeout_round_up

msg/async: round timeouts up (fix busy loops)

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #48539 from chrisphoffman/wip-57864

qa/tasks: Include stderr on tasks badness check

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/cephadm/nvmeof: Add SPDK iobuf options field to NVMeOF configuration
Fixes https://tracker.ceph.com/issues/69554

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

test/rgw/lua: install luarocks

since it is only a suggestion for debian.
this is a regression from: https://github.com/ceph/ceph/pull/60666

Fixes: https://tracker.ceph.com/issues/69473
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #61393 from abhishek-kane/mapping_email

Add Abhishek Kane into peoplemap, githubmap, organizationmap, mailmap

Reviewed-by: Adam King <adking@redhat.com>

doc/cephfs: edit disaster-recovery-experts

Edit the first two sections of doc/cephfs/disaster-recovery-experts.rst
in preparation for adding deeper explanations of the contexts in which
one should use the various commands listed on that page.

https://tracker.ceph.com/issues/69557

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61349 from VallariAg/wip-vallari-nvmeof-ceph-version

mon: do not show nvmeof in 'ceph versions' output

Merge pull request #60889 from anoopcs9/fix-invalid-access-mds

mds: Fix invalid access of mdr->dn[0].back()

Add Abhishek Kane into peoplemap, githubmap, organizationmap, mailmap

Signed-off-by: abhishek-kane <abhishek.kane@ibm.com>

mon: do not show nvmeof in 'ceph versions' output

NVMeoF gateway version is independent of ceph version
so 'ceph version' shows wrong nvmeof version in output
(i.e. instead of gateway version, it shows Ceph version).
Hence, remove nvmeof in 'ceph versions' output.

To check for gateway version, use 'gw info' command.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #61218 from dmick/wip-containerfile-ceph-version

container/Containerfile: replace CEPH_VERSION label for backward compat

rgw: update attrs with replication status to reflect in response

In handle_replication_status_header() when the COMPLETED status is
being set to object's attr, it needs to be reflected to the current
live attrs for the corresponding request so it can be reflected in
the first try than the second.

Fixes: https://tracker.ceph.com/issues/69549
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: set dest bucket id in bucket sync checkpoint

bucket_source_sync_checkpoint() requires bucket id to lookup for
full-sync object and the dest pipe doesn't have it. by replcaning
the bucket key in the pipe with the bucket info we will ensure the
bucket id is set.

Fixes: https://tracker.ceph.com/issues/69548
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #60847 from cbodley/wip-rgw-boto-headbucket-stats

examples/rgw: register boto extension for HeadBucket stats

Reviewed-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #61211 from yuvalif/wip-yuval-logging-tenant

rgw/logging: add more validations to conf, cross tenant support and cleanup

Reviewed-By: Adam Emerson <aemerson@ibm.com>, Ali Masarwe <ali.masarwa@ibm.com>

Merge pull request #61373 from anthonyeleven/fix-file-descriptions

AsyncMessenger.cc : improve error messages

Reviewed-by: Zac Dover <zac.dover@proton.me>

RGW\log: adding some missing members

Signed-off-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #61289 from ronen-fr/wip-rf-catcher-fix

common: fix md_config_cacher_t
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: S3 Delete Bucket Policy should return 204 on success

Currently, RGW returns a 200 on a successful DELETE on a bucket policy but
the S3 API expects a 204:

    https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketPolicy.html

The sample response in the example above is a 204.

This patch checks the op_ret in `RGWDeleteBucketPolicy::send_response()`
and on a success we turn it to a 204 (or STATUS_NO_CONTENT).

Fixes: https://tracker.ceph.com/issues/69539
Signed-off-by: Simon Jürgensmeyer <simon.juergensmeyer@hetzner-cloud.de>

Merge pull request #61072 from pecastro/doc_and_test_fixes

Doc and test fixes

AsyncMessenger.cc : improve error messages

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

qa/tasks: Include stderr on tasks badness check.

Make sure that first_in_ceph_log() doesn't return None (which is
treated as success/"no badness" by the caller) if the cluster log file
is missing.

Fixes: https://tracker.ceph.com/issues/57864
Co-authored-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Christopher Hoffman <choffman@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: RGWRados::get_olh() needs to use the correct attr

Fixes: https://tracker.ceph.com/issues/69536
Signed-off-by: Yehuda Sadeh <ysadehwe@ibm.com>

doc/radosgw/config-ref: fix lifecycle workload tuning description

This commit updates RGW Config Reference - Lifecycle Settings section. In particular it changes the section where two different tunings for two different workloads are suggested. It aims for a more understandable description as asked by the contributors.

Signed-off-by: Laimis Juzeliunas <laimis.juzeliunas@oxylabs.io>

Merge pull request #59739 from sabaini/fix/ceph-volume-ep

[ceph-volume] fix importlib.metadata compat

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60073 from mkogan1/wip-fix-rate-lim

rgw: fix user rate limit is not enforced w/ global rate limit set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61322 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_ostringstream_fix

common/io_exerciser: Ensure empty stringstream in ceph_test_rados_io_sequence RadosIO between ReST calls

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

qa/scrub: more delay when waiting for noscrub to take effect

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61352 from zdover23/wip-doc-2025-01-14-releases-quincy-eol

doc/releases: add actual_eol for quincy

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

common: config_cacher: use set::contains() instead of count()

moved out of the main commit to
facilitate backporting it to pre-C++20.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/scrub: change 'bin/ceph' to 'ceph'

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: fix md_config_cacher_t

In its get_tracked_conf_keys() member function, the
cacher (in the existing code) initializes a static
function-block variable ('keys'), and uses it for
registering the observer.

But the cacher is instantiated on the type of
the configuration value. Thus, multiple cacher
objects for which the configuration values are
of the same type - share the static 'keys'. Only
one of the observers is registered.

Note that the code could have been simplified
somewhat, if the signature of the
get_tracked_conf_keys() function
was changed to return 'const char* const *'.

Fixes: https://tracker.ceph.com/issues/69236
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61314 from aclamk/wip-aclamk-bluefs-truncate-fix

os/bluestore: Fix BlueFS::truncate()

Merge pull request #61261 from xxhdx1985126/wip-seastore-lba-backref-mapping

crimson/os/seastore: refactor LBAMapping

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60215 from MaxKellermann/osd_optimizations

OSD: Small optimizations

Merge pull request #61312 from adamemerson/wip-boost-1.86-fixes

{mgr, rbd, tests/os}: Fixes for Boost 1.86

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/.../replicated_request: work around gcc 11 bug in with_pg_interruptible

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #61284 from shraddhaag/wip-shraddhaag-fix-rados-monthrash

rados/monthrash: add PG_DEGRADED to ignorelist

doc/releases: add actual_eol for quincy

Add the actual EOL date for the Quincy release (it's 2025-01-13).

Signed-off-by: Zac Dover <zac.dover@proton.me>