rgw: update attrs with replication status to reflect in response

In handle_replication_status_header() when the COMPLETED status is
being set to object's attr, it needs to be reflected to the current
live attrs for the corresponding request so it can be reflected in
the first try than the second.

Fixes: https://tracker.ceph.com/issues/69549
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #60847 from cbodley/wip-rgw-boto-headbucket-stats

examples/rgw: register boto extension for HeadBucket stats

Reviewed-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #61211 from yuvalif/wip-yuval-logging-tenant

rgw/logging: add more validations to conf, cross tenant support and cleanup

Reviewed-By: Adam Emerson <aemerson@ibm.com>, Ali Masarwe <ali.masarwa@ibm.com>

Merge pull request #61373 from anthonyeleven/fix-file-descriptions

AsyncMessenger.cc : improve error messages

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61289 from ronen-fr/wip-rf-catcher-fix

common: fix md_config_cacher_t
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #61072 from pecastro/doc_and_test_fixes

Doc and test fixes

AsyncMessenger.cc : improve error messages

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #59739 from sabaini/fix/ceph-volume-ep

[ceph-volume] fix importlib.metadata compat

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #60073 from mkogan1/wip-fix-rate-lim

rgw: fix user rate limit is not enforced w/ global rate limit set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #61322 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_ostringstream_fix

common/io_exerciser: Ensure empty stringstream in ceph_test_rados_io_sequence RadosIO between ReST calls

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

qa/scrub: more delay when waiting for noscrub to take effect

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61352 from zdover23/wip-doc-2025-01-14-releases-quincy-eol

doc/releases: add actual_eol for quincy

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

common: config_cacher: use set::contains() instead of count()

moved out of the main commit to
facilitate backporting it to pre-C++20.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa/scrub: change 'bin/ceph' to 'ceph'

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: fix md_config_cacher_t

In its get_tracked_conf_keys() member function, the
cacher (in the existing code) initializes a static
function-block variable ('keys'), and uses it for
registering the observer.

But the cacher is instantiated on the type of
the configuration value. Thus, multiple cacher
objects for which the configuration values are
of the same type - share the static 'keys'. Only
one of the observers is registered.

Note that the code could have been simplified
somewhat, if the signature of the
get_tracked_conf_keys() function
was changed to return 'const char* const *'.

Fixes: https://tracker.ceph.com/issues/69236
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61314 from aclamk/wip-aclamk-bluefs-truncate-fix

os/bluestore: Fix BlueFS::truncate()

Merge pull request #61261 from xxhdx1985126/wip-seastore-lba-backref-mapping

crimson/os/seastore: refactor LBAMapping

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60215 from MaxKellermann/osd_optimizations

OSD: Small optimizations

Merge pull request #61312 from adamemerson/wip-boost-1.86-fixes

{mgr, rbd, tests/os}: Fixes for Boost 1.86

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61284 from shraddhaag/wip-shraddhaag-fix-rados-monthrash

rados/monthrash: add PG_DEGRADED to ignorelist

doc/releases: add actual_eol for quincy

Add the actual EOL date for the Quincy release (it's 2025-01-13).

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #61311 from zdover23/wip-doc-2025-01-10-releases-remove-quincy-from-active

doc/releases: remove Quincy from "Active"

Reviewed by: Laura Flores <lflores@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61208 from Yonatan-Zaken/restore_selinux_context_when_writing_remote_files_on_host

mgr/cephadm: Restore client file(s) default SELinux context type

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61099 from rkachach/fix_issue_69252

mgr/cephadm: fixing logic in cert-store save_cert method

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #61035 from ShwetaBhosale1/fix_issue_69192_getting_error_if_yaml_file_has_trailing_tab

mgr/cephadm: mgr orchestrator module raise exception if there is trailing tab in yaml file

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60888 from ShwetaBhosale1/fix_issue_69070_only_bind_ports_on_network_parameter_spec_for_alertmanager

mgr/cephadm: Add only_bind_ports_on_network spec parameter for Alertmanager

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60828 from rkachach/fix_prometheus_targets_discovery_mtls

mgr/cephadm: fixing config files for prometheus and alertmanager

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #60690 from ShwetaBhosale1/fix_issue_68695_ceph_orch_commands_does_not_work_for_deployment_name_osd

mgr/cephadm:  ceph orch <start/stop/restart> commands should work for service name osd  and command to updated service name for osd

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #58604 from adk3798/cephadm-rgw-no-multisite-traffic

mgr/cephadm: allow disabling rgw_run_sync_thread through spec

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

common/io_exerciser: Reorder lines in RadosIo send_command functions

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

librbd/migration/HttpClient: Use asio::ssl::stream

`beast::ssl_stream` is deprecated as of 1.86, and its loss of the move
constructor keeps it from compiling on that version.

Hopefully this passes tests on 1.85, too, or it will have to wait
until the boost bump.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #61272 from laimis9133/main

doc/radosgw/config-ref: fix lc worker thread tuning

Merge pull request #58924 from imtzw/tzw_ikey_lat

os/bluestore: record omapiter init latency

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #57617 from Matan-B/wip-matanb-fix-python-warnings

test/pybind/test_rados.py: fix warnings

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60278 from rzarzynski/wip-os-fastomapiter

os, osd: bring the lightweight OMAP iteration

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

common/io_exerciser: Clear the stringstream in RadosIO class

Ensures the stringstream is empty before sending a command in the RadosIO class

Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>

Merge pull request #61000 from MaxKellermann/no_boost

Use `std` instead of Boost

Merge pull request #61278 from VallariAg/wip-vallari-nvmeof-alerts-update

monitoring: Update nvmeof alert limits in config

os/bluestore: Fix BlueFS::truncate()

In `struct bluefs_fnode_t` there is a vector `extents` and
the vector `extents_index` that is a log2 seek cache.

Until modifications to truncate() we never removed extents from files.
Modified truncate() did not update extents_index.

For example 10 extents long files when truncated to 0 will have:
0 extents, 10 extents_index.
After writing some data to file:
1 extents, 11 extents_index.

Now, `bluefs_fnode_t::seek` will binary search extents_index,
lets say it located seek at item #3.
It will then jump up from #0 extent (that exists) to #3 extent which
does not exist at.
The worst part is that code is now broken, as #3 != extent.end().

There are 3 parts of the fix:
1) assert in `bluefs_fnode_t::seek` to protect against
   jumping outside extents
2) code in BlueFS::truncate to sync up `extents_index` with `extents`
3) dampening down assert in _replay to give a way out of cases
   where incorrect "offset 12345" (12345 is file size) instead of
   "offset 20000" (allocations occupied) was written to log.

Fixes: https://tracker.ceph.com/issues/69481
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #60762 from ljflores/wip-tracker-67181

qa/suites/rados/verify/validater: ignore OSD_DOWN

Merge pull request #60261 from MaxKellermann/PyModule_no_lock

mgr/PyModule: do not lock in get_name(), get_error_string()

rgw/logging: clean pending objects

* when source bucket is deleted
* when logging is disabled
* when logging conf changes
* when target bucket is deleted

also add "bucket logging info" admin command
returning logging conf for source bucket, and list
of source buckets for log bucket

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: retry attribuite set in case of race

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: log bucket must not have encryption

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: source and log bucket must be different

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: support source and destination buckets on different tenants

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

mgr: Work around bug in Boost MPI/Python

Thanks to the maintainers of the Arch User Repository Ceph PKGBUILD
for this fix.

Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #60958 from xxhdx1985126/wip-69120

crimson/osd/backfill_state: recheck whether needs to scan the primary at the end of Enqueuing::Enqueuing

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #60803 from xxhdx1985126/wip-69027

crimson/osd/replicate_backend: add standalone delete to the push queue if the object is deleted while it's being backfilled

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #60795 from taodd/fix-rbd-sparsify-on-ec-image-slow

osd: optimize extent comparison in PrimaryLogPG

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #61313 from VallariAg/wip-vallari-teuthology-docs

doc: improve tests-integration-testing-teuthology-workflow.rst

Merge pull request #61291 from cbodley/wip-69470

doc/rgw: document UserName requirements for account migration

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55259 from linuxbox2/wip-object-attributes

rgw: implement GetObjectAttributes

Merge pull request #60969 from idryomov/wip-69135

qa/suites/upgrade/reef-x: sync log-ignorelist with quincy-x

Reviewed-by: Laura Flores <lflores@ibm.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #60081 from anthonyeleven/anthonyeleven/tweak

src/exporter: improve usage message

rgw_attrs: fixes for 3 more iam/perms mappings

Help from Pritha and Casey appreciated

Plus, fix 3 explicit mapping tests in test_rgw_iam_policy.cc.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #61164 from chardan/wip-objectstore-ops-replayer-sign-warning

ops_replayer: fix signed/unsigned warning and add parameter range checking.

doc: improve tests-integration-testing-teuthology-workflow.rst

This commit adds:
1. workflow summary in the first section along with an image.
2. sub-section "Pushing to ceph-ci repository" to second section.
3. file doc/dev/developer_guide/testing_integration_tests/workflow.png

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

os/bluestore: bluefs unittest for truncate bug

Unittest showing 2 different flavours of problems:
1) bluefs log corruption
2) bluefs sigsegv

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

crimson/os/seastore: refactor LBAMapping

So that PhyscalNodeMapping doesn't contain LBAMapping specific
interfaces

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #61254 from kamoltat/wip-ksirivad-fix-stretch-mode-doc

doc/rados/operations/stretch-mode: Improve doc
Reviewed-by: zdover23
Reviewed-by: anthonyeleven

Merge pull request #61288 from adamemerson/wip-69303

rgw: Don't crash on exceptions from pool listing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tests: removed quincy from crontab

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc/releases: remove Quincy from "Active"

Remove "Quincy" from the list of "Active Releases" in
doc/releases/index.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rgw: document UserName requirements for account migration

clarify the following error when migrating a user into an account:

> could not modify user: unable to modify user, UserName contains invalid characters

Fixes: https://tracker.ceph.com/issues/69470
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Don't crash on exceptions from pool listing

Fixes: https://tracker.ceph.com/issues/69303
Signed-off-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #60330 from JonBailey1993/JonBailey1993/ceph_test_rados_io_sequence_inject_error

common/io_exerciser: Add support to ceph_test_rados_io_sequence injecting errors for testing how erasure coding deals with error scenarios

Reviewed-by: Ronen Friedman <rfriedma@ibm.com>

Merge pull request #52791 from clwluvw/location-constraint

rgw: check for location constraint on master zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59960 from smanjara/wip-fix-missing-http-data

rgw/multisite:  include request body when CreateBucket op is forwarded to master

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60881 from NitzanMordhai/wip-nitzan-ipv6-subnet-checks

common/pick_address: Add IPv6 support to is_addr_in_subnet

Merge pull request #60987 from drunkard/main

doc: fixes in doc building

Merge pull request #60883 from xxhdx1985126/wip-crimson-backfill-throttle

crimson/osd/pg_recovery: throttle backfills together with pglog based recoveries

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #61262 from VallariAg/fix-nvmeof-teuthology-basic-test

qa/workunits/nvmeof/basic_tests.sh: fix nvme list assert

Merge pull request #60623 from piyushagarwal1411/ceph-exporter

vstart: add ceph-exporter support

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rados/monthrash: add PG_DEGRADED to ignorelist

Issue: mon/test_mon_osdmap_prune.sh is failing due to
PG_DEGRADED state.

Cause: We are writing an objects to a newly created PG before
it has enough OSDs in acting set to create enough replicas.
This is expected behaviour, thus the solution is to add the
warning to ignorelist.

Fixes: https://tracker.ceph.com/issues/69288
Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

monitoring: Update nvmeof alert limits in config

Update these in config.libsonnet:
- NVMeoFMaxGatewaysPerGroup (4->8)
- NVMeoFMaxGatewaysPerCluster (4->32)
- NVMeoFMaxNamespaces (1024->2048)
- NVMeoFHighClientCount (32->128)

Also update prometheus_alerts.yml and test_alerts.yml
accordingly.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

Merge pull request #60891 from xxhdx1985126/wip-seastore-fadvise-backfill

crimson/os/seastore: add fadvise support to SeaStore and prevent recovery/backfill from polluting the cache of SeaStore

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc/rados/operations/stretch-mode: Improve doc

Added more content and rewrite some sections

Signed-off-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

doc/radosgw/config-ref: fix lc worker thread tuning

This commit updates RGW Config Reference - Lifecycle Settings section. In particular it addresses an incorrect suggestion to decrease parallel threads in the workers pool for a more aggressive/accelerated per-bucket lifecycle processing. A more aggressive lifecycle processing for a bucket containing higher number of objects is achieved by increasing, not decreasing parallel threads.
Current suggestion is miss-leading.

Fixes: https://tracker.ceph.com/issues/63659
Signed-off-by: Laimis Juzeliunas <laimis.juzeliunas@oxylabs.io>

Fix signed/unsigned warning and add parameter range checking.

Signed-off-by: Jesse F. Williamson <1643380+chardan@users.noreply.github.com>

Merge pull request #60848 from cbodley/wip-rgw-deprecate-iam-tenant

docs/rgw: deprecate tenant-based IAM in favor of accounts

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #61074 from chardan/wip-radowsgw-admin-jfw-restructure_file

rgw: migrate rgw_admin to new radosgw-admin directory

Merge pull request #60379 from cbodley/wip-librados-cancellation

librados/asio: forward asio cancellations to AioCompletion::cancel()

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

rgw/multisite: the create_bucket forward request omits the
the request body, thus missing some data if specified inside
CreateBucketConfiguration xml on the non-master zone.
also, now that we perform cksum validation against empty payloads,
such a request would fail with -ERR_AMZ_CONTENT_SHA256_MISMATCH due
to a zero content-length but a non-empty payload hash.
this fix ensures that request body is forwarded during create_bucket

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #61127 from VallariAg/wip-nvmeof-delete-healthcheck

mon/NVMeofGwMap: add healthcheck warning NVMEOF_GATEWAY_DELETING

Merge pull request #61136 from cbodley/wip-69301

rgw: don't use merge_and_store_attrs() when recreating a bucket

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #61255 from Matan-B/wip-matanb-crimson-seastore-deafult-2

common/options/crimson.yaml.in: fallback to Bluestore by default

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #61251 from guits/ceph-volume-69432

ceph-volume: fix loop devices support

vstart.sh: add support for launching the ceph-exporter daemon

Signed-off-by: Piyush Agarwal <piyushagarwal14.pa@gmail.com>

Merge pull request #61252 from guits/hints-create_id

ceph-volume: add python hints to util.prepare.create_id()

qa/workunits/nvmeof/basic_tests.sh: fix connect-all assert

There seems to be change in 'nvme list' json output
which caused failures in asserts after 'nvme connect-all'
command.

Signed-off-by: Vallari Agrawal <vallari.agrawal@ibm.com>

rgw:attrs: dump marker and max_parts only if sent

Suggested by Casey.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:attrs cap max_parts at 1000

Suggested by Casey.  This is a relatively small value, but let's
be conservative.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: GetObjAttrs enforces MaxSize and returns correct IsTruncated

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: Object::list_parts() starts after marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rados: get_part_obj_state() fixes accounted_size when uncompressed

the part head objects don't have a RGW_ATTR_MANIFEST attribute, so
get_obj_state_impl() isn't able to set the correct
RGWObjState::accounted_size unless RGW_ATTR_COMPRESSION provides one

get_part_obj_state() builds a fake manifest in memory to represent the
part and updates state.size accordingly, but it hadn't corrected the
value of state.accounted_size

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: GetObjAttrs calls rgw_s3_prepare_decrypt()

RGWGetObj::execute() relies on the virtual get_decrypt_filter() which is
overridden by RGWGetObj_ObjStore_S3

but when called from RGWGetObjAttrs_ObjStore_S3, there is no override.
add one that just calls rgw_s3_prepare_decrypt()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw_cksum: return version_id in all cases

The version_id member in RGWGetObjAttrs shadows the one in
RGWGetObj.

Just assign it before conditionally returning the version-id header
in RGWGetObjAttrs_ObjStore_S3::send_response().

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_cksum: permit fallback to checksum-type from create-multipart, in upload-part

There appear to be workloads that provide a checksum algorithm in
create-multipart-upload, but do not provide (what must be) the
corresponding algorithm when uploading the parts.  (complete-multipart-upload
has no checksum argument, the value is implicit.)

This behavior is inconsistent with at least some SDKs, but it is
possibly accepted behavior in AWS S3, and is not logically necessary,
since the originally supplied checksum type is already known.

Therefore, change the behavior of upload-part to fall back to a
checksum type that was sent with the corresponding create-multipart-upload
request, if one is present, rather than failing with InvalidRequest.

Fixes: https://tracker.ceph.com/issues/68513
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 6b487a4c6dbadf3f470c8b12ddd5f2521c6920c6)

rgw: defensive check for part cksum in try_sum_part_cksums

If a create-multipart-upload requested additional checksum but a
part upload omits it, fail gracefully.

Possibly we can fixup the checksum uploads and remove this scenario,
but coding defensively would still be correct.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>