osd/scrub: making Scrub's fadvide flags a constant

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #62972 from laimis9133/laimis9133-compression-docs

doc/radosgw/compression: separate RGW and RADOS pool level compression

Merge pull request #62836 from athanatos/sjust/wip-crimson-repop-reply-ordering-69439

crimson: osd_operation cleanups and fix for MOSDRepOpReply ordering

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge branch 'main' into laimis9133-compression-docs

Signed-off-by: Laimis Juzeliūnas <58551069+laimis9133@users.noreply.github.com>

Merge pull request #59206 from ygtzf/bugfix-compress-use-isal

compressor: compressor_zlib_isal did not take effect in compression

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #62904 into main

* refs/pull/62904/head:
pybind/mgr/volumes: make casesensitive attr uniform in interface

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #63035 from Matan-B/wip-matanb-crimson-scan_for_backfill-fix

crimson/osd/recovery_backend: scan_for_backfill_primary correctly handle missing object

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #63033 from bluikko/doc-placement-formatting-radosgw

doc/radosgw: Promptify cmds and improve formatting in placement.rst

Merge pull request #63028 from bluikko/doc-d3n-formatting-radosgw

doc/radosgw: Improve formatting in d3n_datacache.rst

Merge pull request #63029 from bluikko/doc-admin-privprompts-radosgw

doc/radosgw: Use privileged prompt for CLI commands in admin.rst

Merge pull request #63032 from bluikko/doc-compression-typo-radosgw

doc/radosgw: Remove stray full stop mid-sentence in compression.rst

Merge pull request #62672 from adamemerson/wip-test-common-signed-comparison

test/common: Fix signed comparison

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #62670 from adamemerson/wip-not-before-queue-signed-comparison

common/not_before_queue: Fix signed comparison warning

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #61962 from clwluvw/replication-perms

rgw: add support replication actions in policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #62922 from saif-0987/refactor/testid-update-01

mgr/dashboard: Replace data-cy with data-testid for cypress IDs

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #63010 from ronen-fr/wip-rf-repair-62451

qa/standalone/scrub: fix expected outputs in repair tests

Reviewed-by: Adam Kupczyk <akupczyk@ibm.com>

crimson/osd/recovery_backend: scan_for_backfill_primary correctly handle
missing object

scan_for_backfill was seperated to scan_for_backfill_primary and
scan_for_backfill_replica.
The fix from:
https://github.com/ceph/ceph/pull/62837/commits/88432ebd7432c513ccd495e77425401beddb9953
was only copied to the replica version.

Fixes: https://tracker.ceph.com/issues/71124
Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62978 from afreen23/main

mgr/dashboard: Update translations

Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/radosgw: Promptify cmds and improve formatting in placement.rst

Use preformatted blocks with a privileged bash prompt instead of
hardcoding prompts in the beginning of each line for CLI commands.

Indent continuation lines of multi-line CLI example commands the same
way they are indented elsewhere.

Use inline code formatting consistently, add double-backticks for
inside text references to CLI commands, configuration data, etc.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/radosgw: Remove stray full stop mid-sentence in compression.rst

Remove a full stop that seems to be a typo in the middle of a sentence.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/radosgw: Use privileged prompt for CLI commands in admin.rst

Instead of not defining a prompt to use in CLI commands and falling back
to the default unprivileged prompt, use explicit privileged bash prompt
for CLI commands that require privileges.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/radosgw: Improve formatting in d3n_datacache.rst

Change to a full stop one comma that is followed by capital
case and looks like a separate sentence otherwise too.

Add missing inline code formatting consistently for file
names, config data, etc.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

qa/standalone/scrub: fix expected output in snaps repair tests

Specifically - TEST_corrupt_snapset_scrub_rep in osd-scrub-repair.sh.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson: add operation wrapper for MOSDRepOpReply

This should avoid reordering between cores.

Fixes: https://tracker.ceph.com/issues/69439
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: convert cross-core operations to use RemoteOperation

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: fix DynamicPerfStats usage in ClientRequest

ClientRequest::get_connection() return l_conn, which will be
null by the time PG::add_client_request_lat is called in
ClientRequest::do_process.  Modify get_connection() to
return a Connection& from whichever of l_conn or r_conn
isn't null.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../osd_operation.h: add RemoteOperation

Subsequent commits will switch various ops to inherit from
this thereby removing some boilerplate.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../osd.cc: convert active ops to start_pg_operation_active

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../pg_shard_manager: add start_pg_operation_active

Messages between OSDs for PGs that have already completed peering
require fewer checks than otherwise.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #62872 into main

* refs/pull/62872/head:
qa: add test for cloning with charmap
pybind/mgr/volumes: fix typo in casesensitive vxattr

Reviewed-by: Anoop C S <anoopcs@cryptolab.net>
Reviewed-by: Xavi Hernandez <xhernandez@gmail.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #62089 into main

* refs/pull/62089/head:
doc/dev/release-checklists: add vX.3.0 checklist item
doc/dev/release-checklist: check v20.0.0 tag

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #63011 into main

* refs/pull/63011/head:
.github: run verify-qa from base branch
 .github: run verify-qa when PR HEAD is updated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: utilize is_impersonating for forwarded sts requests

With the introduction of is_impersonating in SysReqApplier,
RoleApplier can now use the same mechanism to mark when a request
has been forwarded by a system user on behalf of another role (e.g.,
through STS) to mark it as a system request (s->system_request).

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

.github: run verify-qa from base branch

If the PR does not have the script, perhaps it does not run?

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

 .github: run verify-qa when PR HEAD is updated

Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

rgw: dont rate limit forwarded requests

rely on s->system_request to skip rate limiting on forwarded requests
as well as normal system user requests.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: use is_admin() for permission checks

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: override perms for admin on data sync

If pipe is in user mode and the user is admin, don't check for perms
and let it go.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: change is_admin_of() to is_admin()

As admin propery of a user is something global and nothing related
to any other owner, we don't need any comparision.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: make rgw_sync_pipe_params::user optional

In rgw_sync_pipe_params, the mode can be either system or user.
When in system mode, no user is involved, but the current
implementation holds an empty rgw_user, which can cause confusion
in pipe_rules::find_basic_info_without_tags().

With this change, rgw_user is now optional, ensuring that when no
user is involved, it is explicitly nullopt rather than an empty object.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

qa/rgw: add perm check test for copy obj between zonegroups

Make sure perms are evaluated properly for the source object.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

doc: add release note for new policy actions on replication

Fixes: https://tracker.ceph.com/issues/70093
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: remote copy obj pass rgwx-perm-check-uid for perm evaluation

When copying object from remote source (bucket from another zonegroup)
the perms of the source is not evaluated resulting in reading from
unauthorized buckets.
passing `rgwx-perm-check-uid` will let the source zone evaluates the
perm and close this bug.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: RGWRadosPutObj evals source bucket perm for backward compatibility

As of a3f40b4 we no longer evaluate perms locally for source bucket,
this could cause broken permission evaluation dusring upgrade as one
zone is not respecting the perm evaluation based on the `rgwx-perm-check-uid`
arg.

This can be dropped in T+2 release.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: make verify_bucket_permission functions const

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: give hint via header for perm evaluation in GetObj

Return `Rgwx-Perm-Checked` header as a hint for the destination zone
to know whether the perms where considered or not.
This is just a backward compatibility for upgrade and can be dropped
in T+2 release.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: rest client callback when all headers are passed

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: pass rgwx-perm-check-uid for multisite fetch object

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: GetObject(Version) not allowed to replicate sse-kms objects

To replicate objects encrypted via sse-kms objects,
s3:GetObjectVersionForReplication is required.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: take account GetObject(Version)Tagging when replicating

In case the uid has no permission to read tagging, the tags should
not be replicated.
Ref. https://docs.aws.amazon.com/AmazonS3/latest/userguide/setting-repl-config-perm-overview.html

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

qa/rgw: add test for source object perm check in multisite

Check whether the policies are honored on source object in source
zone when replicating.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: replication require lock perm if enabled

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: check source object replication by replication actions

Check for permissions of `s3:GetObjectVersionForReplication` in
addition to `s3:GetObject` and `s3:GetObjectVersion` when fetching
the object for multisite.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: export action_bit_string through header file

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: only allow system override if identity is not impersonating

Since multisite now delegates permission checks for source objects
to the source zone (a3f40b4), we need to avoid allowing system-level
overrides when the request is impersonating another identity.

SysReqApplier should only grant override permission if the request
is truly system-authenticated and not acting on behalf of another
user or role (i.e., no rgwx-perm-check-uid or rgwx-perm-check-role
in the request).

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: SysReqApplier overrides is_admin_of based on impersonation

SysReqApplier now returns true for is_admin_of() when the requester
was a system user and was not impersonating any user/role using
rgwx-perm-check-uid or rgwx-perm-check-role.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

qa/rgw: add test for new replication actions

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: support s3ReplicateTags perm on destination bucket for replication

Check for tag replication permission on destination bucket, so if
there was an explicit deny, donot include tags in the replicated
object.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: check for s3ReplicateObject perm on destination bucket for replication

Instead of s3:PutObject rely on s3:s3ReplicateObject permission to
check whether the user can replicate to the destination bucket.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: verify perm on delete replication

Check for s3:ReplicateDelete for replicating object deletes and
delete markers when pipe is set to user mode.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: move RGWUserPermHandler to header

So it can be used by others.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: weaning off RGWUserPermHandler from RGWDataSyncEnv

So it can be called by RGWAsyncRadosRequest classes not holding
sync_env.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: send bucket sync structs to bucket_sync.h

So it can be imported by headers like rgw_cr_rados.h that already
has dependency to rgw_data_sync.h.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

rgw: drop unused params passed to RGWStatRemoteObjCR by RGWObjFetchCR

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #56576 from pritha-srivastava/wip-rgw-assume-role-multisite

rgw/sts: correcting authentication in case s3 ops are directed to a primary from secondary after assumerole.

qa/standalone/scrub: fix expected output in replicated repair tests

Specifically - TEST_corrupt_scrub_replicated in osd-scrub-repair.sh.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #63004 from dasJ/fix/ceph-volume-split

ceph-volume: Fix splitting with too many parts

Merge pull request #63008 from bluikko/doc-compression-promptify-radosgw

doc/radosgw: Promptify CLI commands in compression.rst

Merge pull request #63007 from bluikko/doc-keystone-formatting-radosgw

doc/radosgw: Promptify commands and improve formatting in keystone.rst

Merge pull request #63006 from bluikko/doc-bucketpolicy-formatting-radosgw

doc/radosgw: Improve formatting in bucketpolicy.rst

doc/radosgw/compression: separate RGW and RADOS pool level compression

Add notes indicating a difference between compressions done by RGW and on RADOS pool level for better understanding.

Signed-off-by: Laimis Juzeliunas <laimis.juzeliunas@oxylabs.io>
Co-authored-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #63009 from bluikko/doc-bucketpolicy-addedin-radosgw

doc/radosgw: Use macro for release info in bucketpolicy.rst

Merge PR #61796 into main

* refs/pull/61796/head:
rgw/sts: Use client_id for assumerolewithwebidentityresponse
rgw/sts: adding validation of jwks_uri cert according
rgw/sts: fix to pick jwk which is of type
rgw/sts: adding code for JWT signature validation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/standalone/scrub: fix expected output in EC repair tests

Specifically - TEST_corrupt_scrub_erasure_* in osd-scrub-repair.sh.
This is required following recent changes to the EC code.

Fixes: https://tracker.ceph.com/issues/70851
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

doc/radosgw: Improve formatting in bucketpolicy.rst

Add a missing empty line before an unordered list that
caused it to be rendered as just normal text in a
single paragraph, instead of a list.

Use inline code for CLI command names and other
such identifiers.

Capitalize "s3" correctly as "S3" when it refers to
the S3 service/API.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #62657 from jamiepryde/ec-plugins-tidying-nits-and-bits

erasure-code: Handle review comments from #61804 to tidy up EC plugin changes

Merge pull request #53365 from AliMasarweh/wip-alimasa-keystone-auth-rgw

RGW: When using Keystone auth for RGW, include the Keystone user in ops log

Reviewed-by: Matthew Benjamin <mbenjamin@redhat.com>

doc/radosgw: Use macro for release info in bucketpolicy.rst

Don't spell out the release that added the feature, instead use RST
macro for it, like other documents do.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/radosgw: Promptify CLI commands in compression.rst

Use the bash prompt block for example CLI command.
Separate command output from the CLI command.

Use the correct privileged bash prompt on an already
promptified example command, instead of an
unprivileged bash prompt.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/radosgw: Promptify commands and improve formatting in keystone.rst

Use blocks with bash privileged command prompt for CLI
command examples. Separate example command output to a
preformatted block. Previously a hard-coded prompt in
some place inconsistently while no prompts in others.

Indent multi-line CLI command examples consistently.

Use Title Case consistency in section titles, instead
of some capitalized only first letter of title text.

Use double-backtick inline code for syntax strings,
HTTP header names etc as seems common.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

ceph-volume: Fix splitting with too many parts

The data can be anything and also contain a `=`, causing the line to
fail with `Too many values to unpack`. In my case, it failed with
`ID_FS_LABEL=pvc_name=rook-ceph-lvm-data-44f2gc`.

Regression was introduced here: https://github.com/ceph/ceph/pull/60006

Fixes: https://tracker.ceph.com/issues/71101
Signed-off-by: Janne Heß <janne@hess.ooo>

Merge pull request #62860 from joscollin/wip-D70953-cephfs-journal-tool

doc: update cephfs-journal-tool docs

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #62977 from tchaikov/mgr-std-variant

mgr: migrate from boost::variant to std::variant

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #62975 from tchaikov/libcephfs_proxy-silence-warning

libcephfs_proxy: remove arithmetic on void*

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Xavi Hernandez <xhernandez@gmail.com>

Merge pull request #62991 from tchaikov/common-aligned_storage

common: avoid using std::aligned_storage_t

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #62996 from ronen-fr/wip-rf-testtoofast

osd/scrub: always round up reported scrub duration

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr: migrate from boost::variant to std::variant

Replace `boost::variant` with `std::variant` as part of our effort to reduce
third-party dependencies in favor of C++ standard library alternatives.

Benefits include:
- Improved code readability and maintainability
- Reduced external dependency surface
- More consistent API usage with other components

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #62979 from anthonyeleven/formatting

doc/radosgw: Improve formatting in layout.rst

common: avoid using std::aligned_storage_t

std::aligned_storage_t was deprecated in C++23, to be prepared for
it, let's use alignas for the same behavior.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rgw/qa: added test case to assume a role after role creation
syncs, and then creating a bucket on both primary and secondary.
The test name is test_assume_role_after_sync.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: by-passing authentication using temp creds
in case the request is forwarded from secondary in
a multi-site setup. authenticating with the system
user creds of which are used to sign the request.
Permissions are still derived from the role.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

osd/scrub: always round up reported scrub duration

as expected by some tests, and clearer for the user.

Fixes: https://tracker.ceph.com/issues/68833
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #62970 from bluikko/doc-toc-sectionlevels2-radosgw

doc/radosgw: Fix section header level in config-ref.rst

Merge pull request #62986 from bluikko/doc-headerlines-radosgw

doc/radosgw: Fix length of section header underlines in oidc.rst

Merge pull request #62988 from bluikko/doc-oidc-examples-radosgw

doc/radosgw: Fix RST syntax rendered as text in oidc.rst

Merge pull request #62987 from bluikko/doc-sts-edit-radosgw

doc/radosgw: Improve formatting in STS.rst

rgw/sts: Use client_id for assumerolewithwebidentityresponse
if aud is not present in JWT.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding validation of jwks_uri cert according
to https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
for n&e which can be later used for all key types
(x5c, n&e).

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: fix to pick jwk which is of type
'sig' for signature validation of the token.

Fixes: https://tracker.ceph.com/issues/54562
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw/sts: adding code for JWT signature validation
using modulus and exponent for RSA group of algorithms.

A couple of issues and a fix have been suggested by
Pupu Toivonen (pupu.toivonen@csc.fi) in signature calculation
using modulus and exponent.

Fixes: https://tracker.ceph.com/issues/51018
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>