Merge pull request #54437 from Matan-B/wip-matanb-crimson-osdmap-trimming

crimson/osd: introduce osdmap trimming

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53712 from ronen-fr/wip-rf-move-1

osd: fix a missing 'noexcept' on a move ctor

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54482 from ronen-fr/wip-rf-repl-hp

osd/scrub: decouple being reserved from handling scrub requests

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54689 from PC-Admin/update-stretch-mode-docs

doc/rados: Update stretch mode docs.

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/rados: update "stretch mode"

Update stretch mode docs, min_size and max_size are no longer defined in
the CRUSH map and the example rule given will fail to compile.

Specify that the tiebreaker data centre cannot be defined in CRUSH as
this produces an error.

Signed-off-by: Michael Collins <perthserverplus@gmail.com>

osd: fix a missing 'noexcept' on a move ctor

as a non-default, non-noexcept move ctor is ignored by
stl containers.

See clang-tidy's performance-noexcept-move-constructor

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54752 from zdover23/wip-doc-2023-12-02-radosgw-multisite-formatting

doc/radosgw: fix formatting

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: fix formatting

Repair the formatting of a string that had a string inside backticks
that itself was inside double asterisks. The presence of the asterisks
around the entire string caused the backticks to appear in the rendered
documentation.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54526 from cbodley/wip-rgw-acl-cleanup

rgw/acl: clean up RGWAccessControlPolicy and friends

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54488 from cbodley/wip-versioned-variant

common: add versioned encodings for std::variant

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

tests/scrub: deactivate osd-scrub-dump stand-alone test

as the scrub reservation changes had made it obsolete.
Note - it is not an issue of fixing the test, but rather
that the tested functionality is no longer there.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common: add generate_test_instances() for std::variant

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: add versioned encodings for std::variant

adds two encoding strategies for `std::variant<>` under the namespaces
`ceph::versioned_variant` and `ceph::converted_variant`

these versioned encodings allow the variant to be extended with new
types, provided that they're always added to the end without changing
or removing existing types. because of this requirement, no default
encoding is provided for `std::variant`. callers must opt in to one
namespace or the other

the `converted_variant` encoding requires the variant's first type T
to use versioned encoding, and guarantees that the variant's encoding
is backward-compatible with T's

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54648 from rhcs-dashboard/fs-rename-fix

mgr/dashboard: fs rename only when fs is offline

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>

Merge pull request #54731 from zdover23/wip-doc-2023-11-30-rados-tshooting-log-and-debug-note

doc/rados: improve formatting of log-and-debug.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54730 from zdover23/wip-doc-2023-11-30-glossary-monitor-store

doc/glossary.rst: add "Monitor Store"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: improve formatting of log-and-debug.rst

Improve the arrangment of information in the section "Ceph Subsystems"
in doc/rados/troubleshooting/log-and-debug.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #54724 from ajarr/wip-63673

qa/workunits/rbd/cli_generic.sh: narrow race window

... when checking that rbd_support module command fails after blocklisting the module's client.

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: generalized code-block component

Fixes: https://tracker.ceph.com/issues/63608
Signed-off-by: Nizamudeen A <nia@redhat.com>

rgw/acl: remove virtual destructor of ACLGranteeType

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: ACLGrant uses variant for grantee types

use of `ACLGrant::get_id()` was awkward because most grantee types
returned nothing, but emails were returned as `struct rgw_user`. change
the internal representation into a variant, and expose getters for each
grantee type so callers can handle each type specifically. the encoded
format of `ACLGrant` remains unchanged

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: req_state stores ACLs by value instead of unique_ptr

we no longer rely on polymorphism for the s3/swift variants of
`RGWAccessControlPolicy`, so `req_state` can store `bucket_acl`,
`object_acl` and `user_acl` by value

most functions now take these acls by const- or mutable reference
instead of pointers since they won't be nullptr

some code paths won't initialize some of these bucket/object/user acl
variables, and we rely on `RGWAccessControlPolicy::verify_permissions()`
to return false for those because we won't match an empty owner or
array of grants

in only one case, `verify_user_permissions()` has to return true when
`user_acl` is uninitialized, because S3 doesn't have user acls so
uninitialized user acls should not deny access

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/s3: parse_policy() as free function

s3 acl parsing classes no longer inherit from the acl classes
themselves, and are all encapsulated in rgw_acl_s3.cc behind a single
rgw::s3::parse_policy() function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/s3: write_policy_xml() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/s3: create_policy_from_headers() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/s3: create_canned_acl() as free function

adds canned acls to base class RGWAccessControlPolicy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: use create_default() instead of empty canned acls

create_default() adds the same default acl, without the relation to s3
canned acls

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: remove RGWAccessControlPolicy subclasses

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: format_account_acl() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: create_account_policy() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: add_grants() as static free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: add_grants() uses user_to_grant() helper

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: format_container_acls() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: merge_policy() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: create_container_policy() as free function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: parse_grant() as static function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl/swift: parse acl header with ceph::split()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: add_grant() takes const ref

that also required fixing some const-correctness issues

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: remove CephContext members

remove the CephContext member variables and pass in
DoutPrefixProvider for logging where it's necessary

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused policy_bl_to_stream()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: remove unused virtual compare_group_name()

that was the only virtual function, so remove virtual destructors too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ACLOwner as raw struct

this was a class with two protected members, but with a getter/setter
for each. this 'encapsulation' only added unnecessary boilerplate

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: reorder rgw_user members for default operator<=>

the default operator<=> does a memberwise comparison the same that
rgw_user::compare() did, except that it compared `ns` before `id`

reorder the rgw_user members so that the default operator<=> can
replace compare() and the related comparison operators

replaces uses of rgw_user::compare() with operator== and !=

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/acl: remove empty base class ACLGrantee

an unnecessary base class of ACLGrantee_S3 which is just for s3
xml parsing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54304 from cyx1231st/wip-crimson-make-crosscore-send-ordered

crimson/osd: support to send messages concurrently with exclusive phases

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #54697 from yuvalif/wip-yuval-63314

rgw/kafka/amqp: fix race conditionn in async completion handlers

reviewed-by: cbodley

osd/scrub: handling unexpected scrub requests

if arriving while still handling a previous chunk request,
the handling of of the previous chunk request will be aborted.
No response is sent. The scrubber resets, then handles the new request.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: separate counters for primary vs. replica scrubs

The OSD limits the number of concurrent scrubs performed on its PGs.
This limit is now enforced separately for primary and replica scrubs.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: skip reserving replicas for high-priority scrubs

(The primary side of the required changes)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: decouple being reserved from handling scrub requests

For a replica, following this change:

* 'ReplicaActive' captures the state of the scrubber when
  acting as a replica, from peering to interval change;

* "being reserved" is just a flag maintained by ReplicaActive, and
  is no longer a prerequisite for handling scrub requests.

* each scrub request is now associated with its own 'token' value.

and the following minor simplification:

* the 'should we wait for pushes' decision is now part of the
  code executed on the transition from ReplicaIdle into ReplicaActiveOp.
  StartReplicaNoWait can now be discarded.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54305 from rhcs-dashboard/add-tags

mgr/dashboard: add tags to edit bucket

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54355 from nobuto-m/info-rbd-stats-pools

mgr/dashboard: info on why RBD graphs are empty

Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

doc/glossary.rst: add "Monitor Store"

Add the term "Monitor Store" to the glossary.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #53431 into main

* refs/pull/53431/head:
qa: add test cases to verify error reporting works as expected
mgr: fix some doc strings in object_format.py
mgr/tests: test returning error status works as expected
mgr: make object_format's Responder class capable of responding err status
mgr/nfs: report proper errno with err status

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54631 from rishabh-d-dave/mgr-vol-tests

qa/cephfs: improvements for name generators in test_volumes.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50359 from adamemerson/wip-remove-rgwsi_rados

rgw: Remove `RGWSI_RADOS`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54616 from rishabh-d-dave/mgr-vol-improvements

mgr/vol: make some improvements

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54308 from rishabh-d-dave/mgr-vol-flake-warnings

mgr/vol: fix flake8 warnings

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>

qa/workunits/rbd/cli_generic.sh: narrow race window

... when checking whether a rbd_support module command fails after
blocklisting the module's client.

In tests that check the recovery of the rbd_support module after its
client is blocklisted, the rbd_support module's client is
blocklisted using the `osd blocklist add` command. Next,
`osd blocklist ls` command is issued to confirm that the client is
blocklisted. A rbd_support module command is then issued and expected
to fail in order to verify that the blocklisting has affected the
rbd_support module's operations. Sometimes it was observed that before
this rbd_support module command reached the ceph-mgr, the rbd_support
module detected the blocklisting, recovered from it, and was able to
serve the command. To reduce the race window that occurs when trying to
verify that the rbd_support module's operation is affected by client
blocklisting, get rid of the `osd blocklist ls` command.

Fixes: https://tracker.ceph.com/issues/63673
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #54424 from phlogistonjohn/jjm-cephadm-daemon-to-container

cephadm: add a more generic daemon_to_container function

Reviewed-by: Adam King <adking@redhat.com>

rgw: Remove `RGWSI_RADOS`

And remove the last places that reference it.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_SysObj*`

Simply use the RADOS handle and `rgw_rados_ref` directly.

Also make `rgw::AccessListFilter` a std::function.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_User_RADOS`

Simply use the RADOS handle and `rgw_rados_ref` directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_Zone_Tools`

Simply use the RADOS handle directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_Zone`

Simply use the RADOS handle directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_Notify`

Simply use the RADOS handle directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_MDLog`

Simply use the RADOS handle and `rgw_rados_ref` directly.

Also move `async_processor` out from `RGWSI_RADOS` and into
`RGWServices_Def`. This is as good a place as it for any, for now, as
it's reachable by everyone who needs it and exists through the
lifetime of the process.

Eventually it's going to go away due to coroutinization, anyway.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_ConfigKey_RADOS`

Simply use the RADOS handle directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove `RGWSI_RADOS` from `RGWSI_Cls`

Simply use the RADOS handle and `rgw_rados_ref` directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Remove RGWSI_RADOS from RGWSI_BI_RADOS

Simply use the RADOS handle and `rgw_rados_ref` directly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Distentangle `rgw_rados_ref` from `RGWSI_RADOS`

`RGWSI_RADOS` has no reason to exist. `rgw_rados_ref`, on the other
hand, is fairly useful. Move `rgw_rados_ref` to
`store/rados/rgw_tools.h` along with a function to create one.

Then clean up the fallout.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #54619 from cbodley/wip-rgw-sal-get-stats-async

rgw/sal: clean up read_stats_async() callbacks

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #54582 from phlogistonjohn/jjm-cephadm-more-template-error

cephadm: add a custom template not found exception with diagnostic info

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/cephfs: improvements for name generators in test_volumes.py

Generate a name that is shorter and easier to remember.

Also, write a simpler, faster & better helper method for generating
unique names. This method will also have shorter and more concise name,
so this will be easier to type and easier to read.

Fixes: https://tracker.ceph.com/issues/63680
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #54618 from alimaredia/wip-remove-vstart-runner-radosgw-admin-task

qa: remove vstart runner from radosgw_admin task

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54578 from ronen-fr/wip-rf-dedup-clang

tools: modify ceph_dedup_tool to maintain Clang 15 compatibility

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/scrub: use OpCarryingEvent for scrub events carrying common data

Copied from a similar implementation in Crimson.
The payload includes the MOSDScrub* op, and the 'from' id.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Split from "osd/scrub: move responsibility for clearing the 'being reserved' state"

PeeringState: introduce optional on_replica_activate notification

The OSD will use it to activate the scrubber in replica mode.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #48483 from batrick/i48673

mds: revert standby-replay trimming changes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw/kafka/amqp: fix race conditionn in async completion handlers

Fixes: https://tracker.ceph.com/issues/63314
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

crimson/osd: update osd_beacon_report_interval on conf change

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: OSDBeacon to report lec

See: https://github.com/ceph/ceph/pull/14504

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: update TODO comments

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd/crimson/pg: support cluster_osdmap_trim_lower_bound()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: update superblock managment

introduce PerShardState::superblock to be updated on
each shard once the OSDSingletonState instance is updated.

Note: PerShardState::superblock is a dependency for
      PG::cluster_osdmap_trim_lower_bound().

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: introduce osdmap trimming

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_meta: add OSDMeta::remove_map

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/common/shared_lru: introduce cached_key_lower_bound()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54287 from Matan-B/wip-matanb-crimson-do_osd_ops_execute-v3

crimson/osd: execute `submit_error_log` as an ExclusivePhase

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54209 from VallariAg/wip-xml-scanner

qa: use Remote.run_unit_test and ValgrindScanner

Merge pull request #54685 from zdover23/wip-doc-2023-11-28-rados-tshooting-log-and-debug-small-repair

doc/rados: improve "Ceph Subsystems"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #54684 from zdover23/wip-doc-2023-11-28-rados-ops-pgs-typo

doc/rados: ops/pgs: s/power of 2/power of two

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: improve "Ceph Subsystems"

Improve the English in the subsection "Ceph Subsystems" in the section
"Subsystem, Log and Debug Settings" [sic] in
doc/rados/troubleshooting/log-and-debug.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/rados: ops/pgs: s/power of 2/power of two

Change "power of 2" to "power of two" in
doc/rados/operations/placement-groups.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #51229 from jsoref/spelling-rgw

rgw: Fix spelling

Merge pull request #53456 from petrutlucian94/dokan_case_insensitive

dokan: simple case insensitive emulation

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/vol: rectify variable name

Method "open_clone_subvolume_pair()" returns subvolumes, yet return
value is named as clone_volume. Rectify this mistake.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: remove vstart runner from radosgw_admin task

Fixes: https://tracker.ceph.com/issues/63583
Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #54621 from cbodley/wip-rgw-posix-warning

rgw/posix: fix unused variable warning

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw: fix spelling errors

* access
* acquire
* across
* algorithm
* alternatively
* ambiguous
* attributes
* authenticated
* automatically
* based
* between
* beware
* bootstrap
* bucket
* cache
* character
* check
* cleanup
* combinations
* committed
* comparison
* compatibility
* concurrent
* conditions
* conjunction
* connections
* considered
* contains
* correct
* creates
* crypto
* dependencies
* dequeued
* dequeuing
* derivative
* described
* determine
* deterministic
* detritus
* discrepancy
* display
* either way
* elements
* entirely
* entries
* error
* everything
* expected
* explicitly
* extend
* false
* fclose
* feature
* following
* foreign
* format
* forward
* functionality
* functions
* generated
* getting
* gracefully
* happened
* hashicorp
* healthcheck
* hierarchy
* however
* http
* identify
* idleness
* ignored
* incremental
* information
* initialization
* length
* locally
* macros
* malformed
* means
* mechanism
* multipart
* multiple
* must
* namespace
* necessary
* nonexistent
* notifications
* occurred
* official
* outputs
* outputted
* overridden
* ownership
* parallelism
* partition
* passthrough
* pause
* penguin
* perform
* permission
* populated
* position
* preferably
* prefixes
* presumably
* previous
* print
* quadruple
* reference
* regardless
* registry
* renewed
* request
* requires
* reshard
* resource
* result to
* satisfy
* secret
* separate
* separator
* setting
* shouldn't
* skipping
* some
* somehow
* something
* specific
* struct
* succeeded
* success
* successful
* successfully
* syntax
* testing
* truncated
* unambiguously
* unittest
* unmodified
* unnecessary
* valid_action
* verifying
* versioning
* visible
* whatever
* whenever

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>