doc/radosgw - edit admin.rst "set user rate limit"

Edit "Set User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #53999 into main

* refs/pull/53999/head:
PendingReleaseNotes: support for subvolumes and subvolume groups in snap_schedule
snap_schedule/tests: fix db upgrade issue
qa: add yaml for on demand subvol version testing
qa: add test cases for testing --subvol and --group arguments
mgr/volumes: conditionalize subvolume upgrade
mgr/volumes: ensure correct init of v1 subvol
mgr/snap_schedule: add subvol and subvol group arguments to cli
mds/snap_schedule: add subvolume group column management
mgr/volumes: add remote helper methods to fetch subvolume info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55101 from Matan-B/wip-matanb-crimson-snaptrim_event-cleanup

crimson/osd/osd_operations: snaptrim_event cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52815 from adk3798/upgrade-nfs-migration-check

qa/cephadm: mgr-nfs-upgrade, match any migration > 2

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #51963 from adk3798/async-timeout-fixup

mgr/cephadm: fixups for asyncio based timeout

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54721 from adk3798/pull-stable-compiled-cephadm

qa/cephadm: add support to pull compiled cephadm for stable branches

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55106 from zdover23/wip-doc-2024-01-09-radosgw-admin-rate-limit

doc/rgw: edit admin.rst - rate limit management

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: edit admin.rst - rate limit management

Edit the "Rate Limit Management" section in doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/osd/osd_operations: let pg reference destruct last

Prevent from SnapTrimEvent::handle destruct before SnapTrimEvent::pg.
The pg reference should be destructed last.

May releate to: https://tracker.ceph.com/issues/63647

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/background_recovery: cleanup unused logger

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: cleanup SubOpBlocker::emplace_back usage

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: using snap_trim_*_ret_t

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: move SubOpBlocker to seperate file

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54965 from rhcs-dashboard/osd-e2e-fix

mgr/dashboard: fix e2e failure related to landing page

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55012 from rhcs-dashboard/snapshot-creation

mgr/dashboard: subvolume snapshot creation form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #55100 from petrutlucian94/boost_ceph_mirror

win32_deps_build.sh: download Boost from Ceph mirror

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54812 from adk3798/osd-spec-needs-id

mgr/orchestrator: block OSD specs with no service id

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54948 from mgfritch/cephadm-build-rpm

cephadm: add runtime python minor version to RPM query macro

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54977 from pcuzner/fix-host-threads-output

orchestrator: Fix representation of CPU threads in host ls --detail command

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54978 from pcuzner/add-device-count-summary

orchestrator: Add summary line to orch device ls output

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54913 from jianwei1216/visualize_osd_superblock_oldest_map_newest_map_main

OSD: oldest_map and newest_map should be displayed directly

Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #54634 from NitzanMordhai/wip-nitzan-mgr-restful-mod-leaking-memory-results

mgr/BaseMgrModule: Optimize CPython Call in Finish Function

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54829 from awojno-bloomberg/wip-fix-role-delete-replication

rgw/iam: fix role deletion replication

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53156 from qiuxinyidian/rgw-dev

rgw: modify string match_wildcards with fnmatch

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: subvolume snapshot creation form

Fixes: https://tracker.ceph.com/issues/63934
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55009 from ronen-fr/wip-rf-actpr-requeue

osd/scrub: remove scrub_clear_state()

Reviewed-by: Samuel Just <sjust@redhat.com>

win32_deps_build.sh: download Boost from Ceph mirror

We've recently changed the Boost mirror since the previous one
became unavailable.

As pointed out in PR comments, it's even better to use the Ceph
hosted source in order to save Ceph lab bandwidth.

We're adding bzip2 as a dependency. xz-utils was missing, so we're
adding it to our list.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mgr/dashboard: fix e2e failure related to landing page

Fixes: https://tracker.ceph.com/issues/59142
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/osd/osd_operations/snaptrim_event: add op lifetime comment

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/osd/osd_operations/snaptrim_event: fix lifetime on finally()"

This reverts commit 84c5b6c578c1e64807eca5098d51193ee71e6106.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/osd/osd_operations/snaptrim_event: lifetime fixes"

This reverts commit abceb1652239629ed11187a5fc670a3b1a3a5bb1.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54870 from Svelar/crimson_g_conf

crimson: add set_val and rm_val to global conf

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52315 from ronen-fr/wip-rf-rgw-tasserts

test/rgw: annotating variables with maybe_unused

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55088 from cbodley/wip-make-dist-wget

Merge pull request #54587 from cbodley/wip-45736

rgw: add headers to guide cache update in 304 response

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54715 from rkachach/fix_issue_63678

mgr/rook: fixing inconsistencies on the dashboard 'services' view

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54676 from rkachach/fix_list_devices_issue

mgr/rook: fixing devices listing on rook

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

make-dist: don't use --continue option for wget

the boost jfrog mirror is broken and returns an HTML error page instead
of the archive. the file size of this page is 11534 bytes

when download_from() retries the download from download.ceph.com, the -c
option tells it to resume the download of the existing file. the
resulting boost_1_82_0.tar.bz2 ends up with the correct total file size
of 121325129 bytes, but the first 11534 bytes still correspond to the
HTML from jfrog. that causes the sha256sum mismatch

remove the -c option so that wget fetches the archive in its entirety

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/cephadm: mgr-nfs-upgrade, match any migration > 2

I believe this check was originally added because
the 2->3 migration migrated some nfs related bits. Since
then we've had to update the migration this checks
for every time we bump the max migration. This change
is intended to instead just have it check for a
miration > 2 so we don't have to keep updating it.

Signed-off-by: Adam King <adking@redhat.com>

qa/upgrade/reef-x: pull compiled cephadm to start upgrades from reef

The compiled zipapp cephadm that began in reef needs
to be pulled differently than the old single python script
cephadm from earlier releases. This commit updates the reef-x
upgrade suite to pull cephadm in this new way.

Signed-off-by: Adam King <adking@redhat.com>

qa/cephadm: support to pull stable branch compiled cephadm

This is to allow us to pull the latest build of
cephadm off of a stable branch (currently the only
valid option for that is reef, although this hopefully
will work with squid, T release, etc. in the future).

This should allow us to bootstrap cliusters based on
those stable branches for use in upgrade testing

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #55081 from petrutlucian94/boost_mirror

win32_deps_build.sh: change Boost URL

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #55075 from zdover23/wip-doc-2024-01-07-radosgw-admin-quota

doc/radosgw: edit admin.rst - quota

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit admin.rst - quota

Edit the "Quota Management" section (all of it) in
doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

win32_deps_build.sh: change Boost URL

The Boost download URL is broken again [1], we'll switch to a
working mirror.

[1] https://github.com/boostorg/boost/issues/842

Fixes: https://tracker.ceph.com/issues/63952
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

orchestrator: Add summary line to orch device ls

This patch just adds a summary line to the plain
text output of orch device ls when the --summary
switch is given. This helps to quickly understand your
device countswhen managing hosts with many devices.

Fixes: https://tracker.ceph.com/issues/63864
Signed-off-by: Paul Cuzner <pcuzner@ibm.com>

Merge pull request #54525 from myoungwon/wip-delta-overwrite-inplace-replay

crimson/os/seastore:  introduce inplace rewrite for RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54613 from Matan-B/wip-matanb-crimson-build-inc-maps

crimson/osd: Support incremental maps

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/scrub: remove scrub_clear_state()

Its functionality is now handled by the FSM when existing
'Session' or 'ActiveReplicaOp' states.
For a PG that is not active - no need to do anything.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54996 from ronen-fr/wip-rf-active-primary

osd/scrub: add a "clean primary" base state

Reviewed-by: Samuel Just <sjust@redhat.com>

PendingReleaseNotes: support for subvolumes and subvolume groups in snap_schedule

Signed-off-by: Milind Changire <mchangir@redhat.com>

snap_schedule/tests: fix db upgrade issue

Signed-off-by: Milind Changire <mchangir@redhat.com>

qa: add yaml for on demand subvol version testing

Signed-off-by: Milind Changire <mchangir@redhat.com>

qa: add test cases for testing --subvol and --group arguments

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/volumes: conditionalize subvolume upgrade

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/volumes: ensure correct init of v1 subvol

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/snap_schedule: add subvol and subvol group arguments to cli

Signed-off-by: Milind Changire <mchangir@redhat.com>

mds/snap_schedule: add subvolume group column management

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/volumes: add remote helper methods to fetch subvolume info

  * add remote method to fetch subvolume path
  * add remote method to list subvolumes
  * add remote method to fetch subvolume attributes

Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #55069 from ceph/wip-rgw-adminops-user-info-doc-update

doc: update rgw admin api req params for get user info

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: update rgw admin api req params for get user info

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #54993 from ljflores/wip-balancer-status-detail-documentation

Merge pull request #54890 from kamoltat/wip-ksirivad-fix-63389

src/mon/OSDMonitor.cc: more descriptive loggings for crc mismatch

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54708 from amathuria/wip-amathuria-bz-2134786-fix-norecover

mgr/pg_autoscaler: add check for norecover flag

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #52380 from kamoltat/wip-ksirivad-fix-tracker-59564

mon/ConnectionTracker.cc: disregard connection scores from mon_rank = -1

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #48175 from amathuria/wip-add-test-case-bz-2011756

DaemonServer.cc: fix config show command for RGW daemons

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: avoid "over clearing" queued_or_active flag

If two StartScrub messages are received in quick succession, the earlier
one might clear the queued_or_active flag as it fails for being from an old
interval.
When that happens - a 3'rd scrub request will actually be allowed to go
through, while the scrubber is still handling the second one.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: manage queue registration lifetime in the FSM

As the state of 'being registered in the OSDs scrub queue'
corresponds to the PrimaryActive FSM state.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55065 from zdover23/wip-doc-2024-01-05-radosgw-admin-add-remove-admin-caps

doc/radosrgw: edit admin.rst

doc/radosrgw: edit admin.rst

Edit "Add / Remove Admin Capabilities" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/cache: return false explicitly if delta is not replayed

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: check correct crc for inplace-rewritten extents  after replay is done

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore: fix overflow issue and make len > 0 when set_contents is called

set_contents causes the overflow at times because
alloc_extent is allowed to use uint32_t.

Specifically, in random_writes case, PADDING_SIZE is 256<<10,
whereas set_contents's len is uint16_t.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/epm: detect transaction confict during rbm write using OOL

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: introduce modified_region in DATA_BLOCK to keep track of modified region

It has a limitation to keep track of the modified region using the existing
deltas because we can not get the correct region in two cases: 1) a case where replay
is done and 2) duplicate_for_write. This commit introduces modified region
to solve the problem.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: write only modified region when rewriting the extent in an inplace manner

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore: add a random overwrite test to test_objec_date_handler

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cache: reset version to 1 if inplace write occurs before in prepare_record

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #54425 from linuxbox2/wip-lcnotify-cleanup

rgwlc:  additional lc notify cleanups

Merge pull request #54889 from phlogistonjohn/jjm-cephadm-build-with-wheels

cephadm: build with wheels, enabling pyyaml dependency

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54722 from phlogistonjohn/jjm-cephadm-sidecars

cephadm: add support for sidecar containers

Reviewed-by: Adam King <adking@redhat.com>

rgwlc:  even current object versions have a unique instance

Fixes: https://tracker.ceph.com/issues/63546
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc:  additional lc notify cleanups

Follow the same formula to build up obj_state and version_id
in all call sites.

Resolves: rhbz#2163667

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #50055 from cbodley/wip-rgw-polymorphic-executor

rgw/beast: switch back to polymorphic executor

Reviewed-by: Mark Kogan <mkogan@ibm.com>

Merge pull request #54903 from cbodley/wip-63829

radosgw-admin: 'zone set' won't overwrite existing default-placement

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #54505 from adamemerson/wip-lazyfifo-no-mutex-over-coro

rgw/multisite: Fix deadlock by not holding mutex over coroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49594 from Matan-B/wip-matanb-crimson-calc-subsets

crimson: Support Partial Object Recovery

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #55022 from yuvalif/wip-yuval-63901

rgw/kafka: make the connection idle timeout configurable

reviwed-by: cbodley

Merge pull request #54967 from yuvalif/wip-yuval-63859

rgw/notifications/lc: handle publish_commit() failures as warnings

reviewed-by: dang, thotz

crimson/osd: Keep track of modified_ranges

* `modifies_ranges` interval_set is added to osd_op_params_t
* keep track of modified_ranges while executing relevant ops
* Add `osd_op_params` parameter to `PGBackend::remove()`.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/replicated_recovery_backend: handle_pull_response() - recalc_subsets()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

mson/osd/replicated_recovery_backend: Introduce prep_push_to_repli…
…ca()

Intelligently push an object to a replica.  make use of existing
clones/heads and dup data ranges where possible.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/replicated_recovery_backend: seperate prep_push_to_replica and prep_push

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

rename prep_push to prep_push_to_replica

dismiss head_ctx

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/replicated_recovery_backend: prepare_pull to support calc_clone_subsets

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/object_metadata_helper: Introduce calc_*_subsets

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #55037 from ronen-fr/wip-rf-crush-dummy

test/crush: removed unused code

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #53319 from leonid-s-usov/vstart-localhost-for-real

vstart.sh: make sure that --localhost does only bind to 127.0.0.1 in all cases

Merge pull request #54883 from Svelar/memstore

os/memstore: deep copy on partial bufferptrs on setattrs

Merge pull request #54884 from Svelar/memory_is_zero_dev

inline_memory: optimized mem_is_zero for aarch64 by neon intrinsic