rgw/s3: apply vhost logic to s3 only

the vhost-style transformations ran in RGWREST::preprocess() before we
even route the request, so applied to every REST API in radosgw

vhost-style requests are specific to the S3 API, so they should only
apply after being routed to RGWRESTMgr_S3

extract the vhost logic from RGWREST::proprocess() into
rgw_rest_transform_s3_vhost_style(), and call that only from
RGWRESTMgr_S3::get_resource_mgr_as_default()

url-decoding of request_uri into decoded_uri is now duplicated in
preprocess() to apply to all requests, then again after vhost-style
transforms the request_uri

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rest: use ceph::split() for rgw_enable_apis

avoid allocating a list of strings to parse the comma-separated
rgw_enable_apis configuration

the range returned by ceph::split() has no size() function, so change
the calculation to not require it - `size() - distance(begin(), pos)`
is the same thing as `distance(pos, end())`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/rest: add RGWRESTMgr_S3Website

move some s3website stuff out of RGWRESTMgr_S3::get_handler() into its
own nested RGWRESTMgr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/main: clean up use of rgw_enable_apis

replace `get_str_vec()` with `ceph::split()`

replace `map<string, bool>` with `set<string>`

replace `count(key) > 0` with `contains(key)`

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/iam: add s3:Get/PutAccountPublicAccessBlock actions

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: enforce BlockPublicAcls on CreateBucket

this check is unique to account-level PublicAccessBlock configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add helper for bucket + account PublicAccessBlock config

get_public_access_conf() takes an optional account, and checks
RGW_ATTR_PUBLIC_ACCESS on that in addition to the bucket. if both attrs
are found, return the union of their configurations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: req_state::public_access_block is not optional

a default-constructed PublicAccessBlockConfiguration (with all bool
members set to false) makes for a perfectly good empty state

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: PublicAccessBlockConfiguration as raw struct

we really don't need encapsulation for this

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/sal: make xattrs reachable through RGWAccountInfo

account info is exposed to requests through s->auth.identity->get_account(),
which returns optional<RGWAccountInfo> without access to xattrs

add an 'attrs' member variable to RGWAccountInfo and remove the separate
'attrs' argument from sal apis which read/write RGWAccountInfo::attrs directly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/s3: rename req_state::bucket_access_conf to public_access_block

the name reflects a PublicAccessBlock configuration that may either come
from the bucket (s3 PutPublicAccessBlock) or the account (s3control)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add s3control by default to rgw_enable_apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #65997 from YiteGu/fix-param-err-for-ratelimit-command

doc/rgw: use --bucket instead of --uid

Merge pull request #65772 from cbodley/wip-73348

rgw: fix 'bucket rm --bypass-gc' for copied objects

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #65775 from cbodley/wip-73361

rgw: RGWSI_Notify drains the finisher before deleting RGWWatchers

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #65996 from YiteGu/fix-desc-err-for-max-read-ops

doc/rgw: fix the description error of --max-read-ops

Merge pull request #64331 from Jayaprakash-ibm/wip-cputrace-addition

common: Add cputrace tool for scope-based CPU performance tracing

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

Merge pull request #65977 from rhcs-dashboard/fix-73563-main

mgr/dashboard: fix multisite wizard realm configuration mode

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Naman Munet <nmunet@redhat.com>

doc/rgw: use --bucket instead of --uid

Signed-off-by: YiteGu <guyite@bytedance.com>

doc/rgw: fix the description error of --max-read-ops

max-read-ops indicates the number of operations, not bytes.

signed-off-by: Yite Gu <guyite@bytedance.com>

Merge pull request #65408 from tchaikov/wip-suppress-asan-warnings

qa/{asan,lsan}.supp: suppress Python module and libstdc++ false positives

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #64768 from bluikko/doc-notifications-fixes-cephadm2

doc/radosgw: Small fixes and improvements in notifications.rst, part 2

Merge pull request #65899 from Matan-B/wip-matanb-crimson-user-docs

doc: Introduce Crimson User Guide

Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #64730 from benhanokh/dedup_throttle

rgw/dedup: add throttling mechanism

Merge pull request #65902 from yuvalif/wip-yuval-73434

rgw/logging: fix race condition when name update returns ECANCELED

Merge pull request #65585 from cbodley/wip-rgw-lc-bucket-already-run-today

rgw/lc: enforce consistent rgw_lc_debug_interval start times

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #60815 from rhcs-dashboard/fix-69029-main

mgr/cephadm: Use a persistent volume to store Loki DB

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #65982 from bluikko/doc-config-ref-formatting-radosgw

doc/radosgw: Small fixes in config-ref.rst

Merge pull request #64959 from edwinzrodriguez/ceph-wip-72442

rgw: Update buffer size for HMAC signature

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62787 from dang/wip-dang-getattrs

RGW - zipper - Remove target from get_obj_attrs()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common: use constexpr buffer size instead of fixed 256 for read_format in cputrace

Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>

Merge pull request #65926 from soumyakoduri/wip-skoduri-restore-tests

rgw/restore: increase read_thorugh restore period

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #65980 from idryomov/wip-debug-cls-fixup

cls/hello: debug_cls -> debug_objclass in the comment

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/radosgw: Small fixes in config-ref.rst

Use privileged prompt for CLI command.

Use title case consistently for section titles.

Fix a typo that caused RST syntax rendered as-is.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #65920 from nh2/docs-cephfs-inodes-on-replicated

doc/cephfs/createfs: Recommend default data pool on SSDs for non-EC

rgw: Update buffer size for HMAC signature and improve signature max size calculation

Move SIGNATURE_MAX_SIZE to public interface of AWSEngine::VersionAbstractor
Use SIGNATURE_MAX_SIZE to size buffer in get_v2_signature

Fixes: https://tracker.ceph.com/issues/72442
Signed-off-by: Edwin Rodriguez <edwin.rodriguez1@ibm.com>

cls/hello: debug_cls -> debug_objclass in the comment

debug_cls option doesn't exist -- it's spelled debug_objclass.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #65970 from cbodley/wip-cmake-arrow-thrift-boost-jammy

cmake: BuildArrow.cmake uses bundled thrift if system version < 0.17

Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #64762 from mkogan1/wip-ratelim-list-v001

rgw: rate limit for LIST and DELETE ops

Merge pull request #65168 from stackhpc/swap-tip-fix-wording

doc/start: fix wording in swap tip

Merge pull request #65869 from tchaikov/wip-crimson-aligned-dma-write

crimson/seastore: use DMA alignment for block size instead of stat

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: fix multisite wizard realm configuration mode

The option to select an existing realm for replication is not coming up in the wizard even though a realm created and eligible for replication.

Fixes: https://tracker.ceph.com/issues/73563
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #65973 from rhcs-dashboard/inline-edit-validator-fix

mgr/dashboard: fix generic form submit validator for inline edit

Reviewed-by: Naman Munet <nmunet@redhat.com>

Merge pull request #65707 from a16bitsysop/crimson-gcc15

crimson: admin: fix FTBFS with gcc 15

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

doc: Introduce Crimson User Guide

All of Crimson's documentation was included under `doc/dev`.
As we gradually lean towards a more user facing documentation such as
deployment, usage, Packaging and so on -- we should have a separated guide with
non dev related docs.

Developer docs would remain under dev/crimson/

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: fix generic form submit validator for inline edit

currently the validation error is being applied generically to the
parent formgroup which will set the whole form into an error state when
one of the inline editing is failing on a validation. So just changing
that to a single control.

Fixes: https://tracker.ceph.com/issues/73558
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #65954 from positiveEV/patch-2

doc/rados/operations: remove duplicate word

cmake: BuildArrow.cmake uses bundled thrift if system version < 0.17

the bump to arrow 17.0.0 broke the ubuntu jammy builds with:

In file included from /usr/include/thrift/transport/TTransport.h:25,
                 from /usr/include/thrift/protocol/TProtocol.h:28,
                 from /usr/include/thrift/TBase.h:24,
                 from /build/ceph-20.3.0-3599-g3d863d32/src/arrow/cpp/src/generated/parquet_types.h:14,
                 from /build/ceph-20.3.0-3599-g3d863d32/src/arrow/cpp/src/generated/parquet_constants.h:10,
                 from /build/ceph-20.3.0-3599-g3d863d32/src/arrow/cpp/src/generated/parquet_constants.cpp:7:
/usr/include/thrift/transport/TTransportException.h:23:10: fatal error: boost/numeric/conversion/cast.hpp: No such file or directory
   23 | #include <boost/numeric/conversion/cast.hpp>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

when comparing the gcc command line with arrow-15.0.0, the following argument
is no longer present:
> -isystem /build/ceph-20.3.0-3402-gb2db4947/obj-x86_64-linux-gnu/boost/include

arrow 17.0.0 seems to assume that thrift doesn't depend on boost anymore. a
comment in https://github.com/apache/arrow/issues/32266 claims that
> we don't need Boost with system Thrift 0.17.0 or later

but our jammy builds are stuck with libthrift-0.16.0. to reenable jammy builds,
instruct Arrow's cmake to use its bundled thrift dependency if our system thrift
version is < 0.17.0

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: Findthrift.cmake parses version string

when thrift headers are found, parse the version string into a
cmake variable and pass it to find_package_handle_standard_args

this will be needed by BuildArrow.cmake for some conditional logic

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #65774 from ljflores/wip-add-telemetry-pool-flag

mgr/telemetry: add 'ec_optimizations' flag to 'basic_pool_flags' collection

Merge pull request #65955 from gbregman/main

mgr/cephadm: Increase the default expiration time for NVMEoF subsyste…

Merge pull request #65891 from kshtsk/wip-workunit-drop-pipes

qa/tasks/workunit: fix no module named 'pipes'

Merge pull request #65967 from jmundack/doc-sort-csc-memer-list

doc: Update governance.rst to have member names sorted

doc: Update governance.rst to have member names sorted

Originally this list was sorted by first name, overtime new members were added at the bottom. Resorting them again by first name

Signed-off-by: Joseph Mundackal <jmundackal@bloomberg.net>

Merge pull request #64359 from mkogan1/wip-d3n-stripe-gt-chunk-py

rgw/qa/d3n: improve D3N test robustness

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #65367 from myoungwon/wip-nvmedevice-coroutine

crimson/os/seastore/rbm: switch rbm to coroutine

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #65935 from tchaikov/wip-debian-libxsimd-dev

debian/control: Add libxsimd-dev build dependency for vendored Arrow

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #65664 from cbodley/wip-73240

rgw/lc: LCOpAction_CurrentExpiration checks mtime for delete markers

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/cephadm: Increase the default expiration time for NVMEoF subsystem cache

Fixes: https://tracker.ceph.com/issues/73542
Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #65529 from mkogan1/wip-d4n-reserve

rgw/d4n: refactor unused conf rgw_d4n_l1_datacache_size

doc/rados/operations: remove duplicate word

Signed-off-by: positiveEV <50848762+positiveEV@users.noreply.github.com>

Merge pull request #65054 from BBoozmen/dequeue_op_latency_at_completion

OSD: add latency data to DEQUEUE_OP_END event

Merge pull request #64377 from ljflores/wip-tracker-70716

qa/tasks: Ignore expected `backfill_toofull` warning

Merge pull request #65616 from xxhdx1985126/wip-73159

crimson/osd/pg: create "submit_lock" unlocker before locking it

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #64756 from rzarzynski/wip-bug-67244

monc: synchronize tick() of MonClient with shutdown()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #65578 from Matan-B/wip-matanb-crimson-seastore-should_block

crimson/os/seastore/epm: Verify available segments on mkfs/mount

Reviewed-by: Samuel Just <sjust@redhat.com>

debian/control: Add libxsimd-dev build dependency for vendored Arrow

In commit e8460cbd, we introduced the "pkg.ceph.arrow" build profile to
support building with system Arrow packages. However, neither Debian nor
Ubuntu currently ships Arrow packages.

Since WITH_RADOSGW_SELECT_PARQUET is always enabled in debian/rules,
Arrow support is required for all builds. When the pkg.ceph.arrow profile
is not selected, the build uses vendored Arrow. With the recent change to
use AUTO mode for xsimd detection, Arrow will attempt to find system xsimd
>= 9.0.1. Adding libxsimd-dev as a build dependency ensures it's available
for Arrow to detect and use, reducing build time on supported distributions.

On distributions with insufficient xsimd versions (< 9.0.1), Arrow will
automatically fall back to its bundled version.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

cmake/BuildArrow: Use AUTO mode for xsimd dependency detection

Arrow requires xsimd >= 9.0.1 according to arrow/cpp/thirdparty/versions.txt.
Previously, we unconditionally set -Dxsimd_SOURCE=BUNDLED, forcing the use
of Arrow's vendored xsimd regardless of system package availability.

This commit changes to -Dxsimd_SOURCE=AUTO, which allows Arrow's
resolve_dependency mechanism to automatically:
1. Try to find system xsimd package
2. Check if version >= 9.0.1
3. Use system version if found and sufficient
4. Fall back to bundled version otherwise

This reduces build time and dependencies on systems with sufficient xsimd,
while maintaining compatibility with older distributions.

Distribution availability:
- Ubuntu Noble (24.04): libxsimd-dev 12.1.1 (✓ will use system)
- Ubuntu Jammy (22.04): libxsimd-dev 7.6.0 (✗ will use bundled)
- Debian Trixie (13): libxsimd-dev 13.2.0 (✓ will use system)
- CentOS Stream 9: xsimd-devel 7.4.9 (✗ will use bundled)

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #65236 from leonidc/delete_all_gws

nvmeofgw: fix issue of delete all gws from the pool/group

Merge pull request #64176 from tchaikov/wip-pybind-cb-types

pybind: fix callback function type compatibility for GCC 14/15

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Nitzan Mordhai <nmordech@ibm.com>

Merge pull request #65934 from tchaikov/cephadm-detect-pip

cephadm/build: Fix _has_python_pip() function check

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #65919 from phlogistonjohn/jjm-bwc-variants

build-with-container: build image variants

doc/cephfs/createfs: Recommend default data pool on SSDs for non-EC

Signed-off-by: Niklas Hambüchen <mail@nh2.me>

Merge pull request #65930 from cloudbehl/filesystem-grafana-fixes

monitoring: Fix Filesystem grafana dashboard units

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #65065 from ifed01/wip-ifed-fix-70390

os/bluestore: complete fix for extent spanning over shard boundaries

Merge pull request #65468 from Kushal-deb/kill-daemon

mgr/cephadm: configurable per-service stop timeout

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

cephadm/build: Fix _has_python_pip() function check

The _has_python_pip() function was incorrectly checking for the venv
module instead of pip, causing it to always return the wrong result.
This would prevent proper detection of whether pip is available during
the cephadm build process.

Fix by changing the module check from 'venv' to 'pip'.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #65848 from Hezko/nvmeof-cli-def-max-ns-num-on-subsys-create

mgr/dashboard: change the default max namespace from 4096 to None in subsystem add command

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #65791 from aainscow/fix_compiler_warnings

crimson/rgw: Fix clang compiler warnings.

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Edwin Rodriguez <erodriguez@lazyk-ranch.com>

Merge pull request #65829 from rhcs-dashboard/73379-fix-secure-monitoring-stack

mgr/dashboard : Fix secure-monitoring-stack creds issue

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #60361 from rhcs-dashboard/converting-badges-to-carbon-tag

mgr/dashboard: replace bootstrap badges with carbon tags

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

rgw/restore: increase read_thorugh restore period

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

script/build-with-container: add build image variants

Allow the user to control the content of the build image with a
high-level `--image-variant=` switch. Currently the supported values are
`default` (the same maximal image we have been generating) and
`packages` a slimmer image that avoids installing certain test-only
dependencies.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Dockerfile.build: make FOR_MAKE_CHECK a build argument

Set it only during install time.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

install-deps.sh: let FOR_MAKE_CHECK variable take precedence

Previously, the FOR_MAKE_CHECK variable could only enable installing
extra (test) dependencies when install-deps.sh was used and it was
ignored if `tty -s` exited true. This change allows FOR_MAKE_CHECK to
take precedence over the tty check and to specify one of true, 1, yes to
enable extra "for make check" deps or false, 0, no to explicitly disable
the extra deps.

Based-on-work-by: Dan Mick <dan.mick@redhat.com>
Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #65731 from koch-synyx/doc/dashboard-rsa-note

doc/mgr/dashboard: add note that only RSA keys are supported for TLS

Merge pull request #65531 from ShwetaBhosale1/fix_issue_72906_haproxy_changes_for_nfs_active_active_support

mgr/cephadm: Add stick table and haproxy peers in haproxy.cfg for NFS to support nfs active-active cluster

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #65851 from guits/cv-fix-db-pre-created-lvs

ceph-volume: lvm.Lvm.setup_metadata_devices refactor

Merge pull request #65898 from gbregman/main

mgr/cephadm: Limit the value of max-namespaces-per-subsystem.

Merge pull request #65909 from bluikko/doc-sphinx-warnings-plus-cephfs

doc/cephfs: Fix Sphinx warnings and other minor improvements

Merge pull request #65375 from ifed01/wip-ifed-bluefs-ratio

os/bluestore: report metadata/data ratio in BlueFS perf counter

Reviewed-by: Mark Nelson <mark.a.nelson@gmail.com>

Merge pull request #64682 from ifed01/wip-ifed-fix-zstd

compressor/zstd: compressor correct buffer end detection using get_remaining()

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

monitoring: Fix Filesystem grafana dashboard units

Fixes: https://tracker.ceph.com/issues/73521
Signed-off-by: Ankush Behl <cloudbehl@gmail.com>

Merge branch 'ceph:main' into main

Merge pull request #64009 from rzarzynski/wip-bug-70869

osd: prevent OSDMap::check_health() from asserting due to new OSDs found in subtree

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@ibm.com>

Merge pull request #65910 from anoopcs9/fix-mgr-smb-doc

doc/mgr: Fix a copy-paste error in smb.rst

doc/mgr: Fix a copy-paste error in smb.rst

Signed-off-by: Anoop C S <anoopcs@cryptolab.net>

doc/cephfs: Fix Sphinx warnings and other minor improvements

Fix 2 Sphinx warnings about problematic syntax in troubleshooting.rst:
add empty lines between paragraphs.

In fs-volumes.rst:

Fix 1 Sphinx warning about problematic syntax:
Fix indentation and change to ordered list instead of manually
hardcoding item numbers.

Use privileged bash prompt consistently for privileged commands.

Use the "[...] run a command of the following form:" for CLI examples
with placeholders.

Fix accidental emphasis on a list item by adding empty lines around
sub-list.

Use title case in section titles and trim overly long underline.

Capitalize UID, GID, Unicode, Ceph in text.

Use double backticks consistently for command line switches, data, etc.

Start with capital case and end in full stop if full sentence.

Linkify mention of quiesce dev doc and add a label for it in
doc/dev/mds_internals/quiesce.rst.

Other very minor changes like typo, remove space at end of line.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #65745 from guits/cv-drop-subprocess-call-get-devices

ceph-volume: use udev data instead of LVM subprocess in get_devices()

Merge pull request #65554 from matteopt/fix/udevdata-init-empty-file

ceph-volume: fix UdevData initialisation from empty /run/udev/data/* file