mgr/rook: always recreate kvm default network + fix groups refresh
Fixes: https://tracker.ceph.com/issues/64027
This change also includes:
- adding ~/.local/bin to path so behave binary can be found
- adding requirements.txt file for testing dependencies
- increasing timeout used to wait for tools deployment to 90s
- increasing timeout used to wait for kvm network to 20s

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #55120 from rhcs-dashboard/snapshot-delete

mgr/dashboard: delete cephfs snapshot

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: afreen23 <NOT@FOUND>

Merge pull request #54974 from fweimer-rh/c99

GCC 14 compatibility fixes

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #55039 from cyx1231st/wip-crimson-alienstore-submit-concurrently

crimson/os/alienstore: submit from the alien world concurrently

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: chunmei-liu <chunmei.liu@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

mgr/dashboard: delete cephfs snapshot

Fixes: https://tracker.ceph.com/issues/63990
Signed-off-by: Nizamudeen A <nia@redhat.com>

crimson/os/alienstore: submit from the alien world concurrently

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #53678 from cbodley/wip-qa-rgw-upgrade-reef

qa/rgw/upgrade: add reef as a starting release

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54882 from Svelar/isa_xor_dev

erasure-code/isa/xor_op: add neon-based region_xor implementation

Merge pull request #55142 from zdover23/wip-doc-2024-01-11-radosgw-user-rate-limit-ops

doc/radosgw - edit admin.rst "set user rate limit"

Merge pull request #55146 from zdover23/wip-doc-2024-01-11-radosgw-get-user-rate-limit

doc/radosgw: edit admin.rst "get user rate limit"

doc/radosgw: edit admin.rst "get user rate limit"

Edit "Get User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/radosgw - edit admin.rst "set user rate limit"

Edit "Set User Rate Limit" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge PR #53999 into main

* refs/pull/53999/head:
PendingReleaseNotes: support for subvolumes and subvolume groups in snap_schedule
snap_schedule/tests: fix db upgrade issue
qa: add yaml for on demand subvol version testing
qa: add test cases for testing --subvol and --group arguments
mgr/volumes: conditionalize subvolume upgrade
mgr/volumes: ensure correct init of v1 subvol
mgr/snap_schedule: add subvol and subvol group arguments to cli
mds/snap_schedule: add subvolume group column management
mgr/volumes: add remote helper methods to fetch subvolume info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #55101 from Matan-B/wip-matanb-crimson-snaptrim_event-cleanup

crimson/osd/osd_operations: snaptrim_event cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52815 from adk3798/upgrade-nfs-migration-check

qa/cephadm: mgr-nfs-upgrade, match any migration > 2

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #51963 from adk3798/async-timeout-fixup

mgr/cephadm: fixups for asyncio based timeout

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54721 from adk3798/pull-stable-compiled-cephadm

qa/cephadm: add support to pull compiled cephadm for stable branches

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #55106 from zdover23/wip-doc-2024-01-09-radosgw-admin-rate-limit

doc/rgw: edit admin.rst - rate limit management

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: edit admin.rst - rate limit management

Edit the "Rate Limit Management" section in doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/osd/osd_operations: let pg reference destruct last

Prevent from SnapTrimEvent::handle destruct before SnapTrimEvent::pg.
The pg reference should be destructed last.

May releate to: https://tracker.ceph.com/issues/63647

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/background_recovery: cleanup unused logger

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd: cleanup SubOpBlocker::emplace_back usage

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: using snap_trim_*_ret_t

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/osd_operations/snaptrim_event: move SubOpBlocker to seperate file

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54965 from rhcs-dashboard/osd-e2e-fix

mgr/dashboard: fix e2e failure related to landing page

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55012 from rhcs-dashboard/snapshot-creation

mgr/dashboard: subvolume snapshot creation form

Reviewed-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>

Merge pull request #55100 from petrutlucian94/boost_ceph_mirror

win32_deps_build.sh: download Boost from Ceph mirror

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54812 from adk3798/osd-spec-needs-id

mgr/orchestrator: block OSD specs with no service id

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54948 from mgfritch/cephadm-build-rpm

cephadm: add runtime python minor version to RPM query macro

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #54977 from pcuzner/fix-host-threads-output

orchestrator: Fix representation of CPU threads in host ls --detail command

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54978 from pcuzner/add-device-count-summary

orchestrator: Add summary line to orch device ls output

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #54913 from jianwei1216/visualize_osd_superblock_oldest_map_newest_map_main

OSD: oldest_map and newest_map should be displayed directly

Reviewed-by: Matan Breizman <Matan.Brz@gmail.com>

Merge pull request #54634 from NitzanMordhai/wip-nitzan-mgr-restful-mod-leaking-memory-results

mgr/BaseMgrModule: Optimize CPython Call in Finish Function

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #54829 from awojno-bloomberg/wip-fix-role-delete-replication

rgw/iam: fix role deletion replication

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53156 from qiuxinyidian/rgw-dev

rgw: modify string match_wildcards with fnmatch

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: subvolume snapshot creation form

Fixes: https://tracker.ceph.com/issues/63934
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #55009 from ronen-fr/wip-rf-actpr-requeue

osd/scrub: remove scrub_clear_state()

Reviewed-by: Samuel Just <sjust@redhat.com>

win32_deps_build.sh: download Boost from Ceph mirror

We've recently changed the Boost mirror since the previous one
became unavailable.

As pointed out in PR comments, it's even better to use the Ceph
hosted source in order to save Ceph lab bandwidth.

We're adding bzip2 as a dependency. xz-utils was missing, so we're
adding it to our list.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

mgr/dashboard: fix e2e failure related to landing page

Fixes: https://tracker.ceph.com/issues/59142
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@redhat.com>

crimson/osd/osd_operations/snaptrim_event: add op lifetime comment

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/osd/osd_operations/snaptrim_event: fix lifetime on finally()"

This reverts commit 84c5b6c578c1e64807eca5098d51193ee71e6106.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Revert "crimson/osd/osd_operations/snaptrim_event: lifetime fixes"

This reverts commit abceb1652239629ed11187a5fc670a3b1a3a5bb1.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #54870 from Svelar/crimson_g_conf

crimson: add set_val and rm_val to global conf

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #52315 from ronen-fr/wip-rf-rgw-tasserts

test/rgw: annotating variables with maybe_unused

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #55088 from cbodley/wip-make-dist-wget

Merge pull request #54587 from cbodley/wip-45736

rgw: add headers to guide cache update in 304 response

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #54715 from rkachach/fix_issue_63678

mgr/rook: fixing inconsistencies on the dashboard 'services' view

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54676 from rkachach/fix_list_devices_issue

mgr/rook: fixing devices listing on rook

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

make-dist: don't use --continue option for wget

the boost jfrog mirror is broken and returns an HTML error page instead
of the archive. the file size of this page is 11534 bytes

when download_from() retries the download from download.ceph.com, the -c
option tells it to resume the download of the existing file. the
resulting boost_1_82_0.tar.bz2 ends up with the correct total file size
of 121325129 bytes, but the first 11534 bytes still correspond to the
HTML from jfrog. that causes the sha256sum mismatch

remove the -c option so that wget fetches the archive in its entirety

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/cephadm: mgr-nfs-upgrade, match any migration > 2

I believe this check was originally added because
the 2->3 migration migrated some nfs related bits. Since
then we've had to update the migration this checks
for every time we bump the max migration. This change
is intended to instead just have it check for a
miration > 2 so we don't have to keep updating it.

Signed-off-by: Adam King <adking@redhat.com>

qa/upgrade/reef-x: pull compiled cephadm to start upgrades from reef

The compiled zipapp cephadm that began in reef needs
to be pulled differently than the old single python script
cephadm from earlier releases. This commit updates the reef-x
upgrade suite to pull cephadm in this new way.

Signed-off-by: Adam King <adking@redhat.com>

qa/cephadm: support to pull stable branch compiled cephadm

This is to allow us to pull the latest build of
cephadm off of a stable branch (currently the only
valid option for that is reef, although this hopefully
will work with squid, T release, etc. in the future).

This should allow us to bootstrap cliusters based on
those stable branches for use in upgrade testing

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #55081 from petrutlucian94/boost_mirror

win32_deps_build.sh: change Boost URL

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

pybind: Fix C type errors in Cython-generated Python bindings

Several Ceph APIs use bool * types, which correspond to
libcpp.bool * types in Cython.  The bint type has an incorrect
size 4 and cannot be used as a replacement.

This prevents a compilation failure with future compilers:

…-build/src/pybind/rbd/rbd.c: In function ‘__pyx_pf_3rbd_3RBD_104namespace_exists’:
…-build/src/pybind/rbd/rbd.c:42165:76: error: passing argument 3 of ‘rbd_namespace_exists’ from incompatible pointer type
42165 |         __pyx_v_ret = rbd_namespace_exists(__pyx_v__ioctx, __pyx_v__name, (&__pyx_v__exists));
      |                                                                           ~^~~~~~~~~~~~~~~~~
      |                                                                            |
      |                                                                            int *
In file included from …-build/src/pybind/rbd/rbd.c:1268:
…/src/include/rbd/librbd.h:1496:45: note: expected ‘_Bool *’ but argument is of type ‘int *’
 1496 |                                       bool *exists);
      |                                             ^

Signed-off-by: Florian Weimer <fweimer@redhat.com>

tracing: Fix C type errors in librados tracing

This fixes type errors like this:

In file included from /usr/include/lttng/tracepoint-event.h:69,
                 from …-build/include/tracing/librados.h:4143,
                 from …/src/tracing/librados.c:6
:
…-build/include/tracing/librados.h:
 In function ‘lttng_ust__event_probe__librados___rados_mon_command_exit’:
…-build/include/tracing/librados.h:477:9: error: initialization of ‘size_t’ {aka ‘long unsigned int’} from ‘size_t *’ {aka ‘long unsigned int *’} makes integer from pointer without a cast
  477 |         ceph_ctf_integerp(size_t, outslen, outslen)
      |         ^~~~~~~~~~~~~~~~~

GCC 14 will likely treat these type mismatches as an error
and fail the build.

Signed-off-by: Florian Weimer <fweimer@redhat.com>

Merge pull request #55075 from zdover23/wip-doc-2024-01-07-radosgw-admin-quota

doc/radosgw: edit admin.rst - quota

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/radosgw: edit admin.rst - quota

Edit the "Quota Management" section (all of it) in
doc/radosgw/admin.rst.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

win32_deps_build.sh: change Boost URL

The Boost download URL is broken again [1], we'll switch to a
working mirror.

[1] https://github.com/boostorg/boost/issues/842

Fixes: https://tracker.ceph.com/issues/63952
Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

orchestrator: Add summary line to orch device ls

This patch just adds a summary line to the plain
text output of orch device ls when the --summary
switch is given. This helps to quickly understand your
device countswhen managing hosts with many devices.

Fixes: https://tracker.ceph.com/issues/63864
Signed-off-by: Paul Cuzner <pcuzner@ibm.com>

Merge pull request #54525 from myoungwon/wip-delta-overwrite-inplace-replay

crimson/os/seastore:  introduce inplace rewrite for RBM

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #54613 from Matan-B/wip-matanb-crimson-build-inc-maps

crimson/osd: Support incremental maps

Reviewed-by: Samuel Just <sjust@redhat.com>

osd/scrub: remove scrub_clear_state()

Its functionality is now handled by the FSM when existing
'Session' or 'ActiveReplicaOp' states.
For a PG that is not active - no need to do anything.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #54996 from ronen-fr/wip-rf-active-primary

osd/scrub: add a "clean primary" base state

Reviewed-by: Samuel Just <sjust@redhat.com>

PendingReleaseNotes: support for subvolumes and subvolume groups in snap_schedule

Signed-off-by: Milind Changire <mchangir@redhat.com>

snap_schedule/tests: fix db upgrade issue

Signed-off-by: Milind Changire <mchangir@redhat.com>

qa: add yaml for on demand subvol version testing

Signed-off-by: Milind Changire <mchangir@redhat.com>

qa: add test cases for testing --subvol and --group arguments

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/volumes: conditionalize subvolume upgrade

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/volumes: ensure correct init of v1 subvol

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/snap_schedule: add subvol and subvol group arguments to cli

Signed-off-by: Milind Changire <mchangir@redhat.com>

mds/snap_schedule: add subvolume group column management

Signed-off-by: Milind Changire <mchangir@redhat.com>

mgr/volumes: add remote helper methods to fetch subvolume info

  * add remote method to fetch subvolume path
  * add remote method to list subvolumes
  * add remote method to fetch subvolume attributes

Signed-off-by: Milind Changire <mchangir@redhat.com>

Merge pull request #55069 from ceph/wip-rgw-adminops-user-info-doc-update

doc: update rgw admin api req params for get user info

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc: update rgw admin api req params for get user info

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #54993 from ljflores/wip-balancer-status-detail-documentation

Merge pull request #54890 from kamoltat/wip-ksirivad-fix-63389

src/mon/OSDMonitor.cc: more descriptive loggings for crc mismatch

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #54708 from amathuria/wip-amathuria-bz-2134786-fix-norecover

mgr/pg_autoscaler: add check for norecover flag

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>

Merge pull request #52380 from kamoltat/wip-ksirivad-fix-tracker-59564

mon/ConnectionTracker.cc: disregard connection scores from mon_rank = -1

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #48175 from amathuria/wip-add-test-case-bz-2011756

DaemonServer.cc: fix config show command for RGW daemons

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: avoid "over clearing" queued_or_active flag

If two StartScrub messages are received in quick succession, the earlier
one might clear the queued_or_active flag as it fails for being from an old
interval.
When that happens - a 3'rd scrub request will actually be allowed to go
through, while the scrubber is still handling the second one.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

osd/scrub: manage queue registration lifetime in the FSM

As the state of 'being registered in the OSDs scrub queue'
corresponds to the PrimaryActive FSM state.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #55065 from zdover23/wip-doc-2024-01-05-radosgw-admin-add-remove-admin-caps

doc/radosrgw: edit admin.rst

doc/radosrgw: edit admin.rst

Edit "Add / Remove Admin Capabilities" in doc/radosgw/admin.rst.

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore/cache: return false explicitly if delta is not replayed

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: check correct crc for inplace-rewritten extents  after replay is done

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore: fix overflow issue and make len > 0 when set_contents is called

set_contents causes the overflow at times because
alloc_extent is allowed to use uint32_t.

Specifically, in random_writes case, PADDING_SIZE is 256<<10,
whereas set_contents's len is uint16_t.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/epm: detect transaction confict during rbm write using OOL

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: introduce modified_region in DATA_BLOCK to keep track of modified region

It has a limitation to keep track of the modified region using the existing
deltas because we can not get the correct region in two cases: 1) a case where replay
is done and 2) duplicate_for_write. This commit introduces modified region
to solve the problem.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore: write only modified region when rewriting the extent in an inplace manner

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test/crimson/seastore: add a random overwrite test to test_objec_date_handler

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/os/seastore/cache: reset version to 1 if inplace write occurs before in prepare_record

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #54425 from linuxbox2/wip-lcnotify-cleanup

rgwlc:  additional lc notify cleanups

Merge pull request #54889 from phlogistonjohn/jjm-cephadm-build-with-wheels

cephadm: build with wheels, enabling pyyaml dependency

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #54722 from phlogistonjohn/jjm-cephadm-sidecars

cephadm: add support for sidecar containers

Reviewed-by: Adam King <adking@redhat.com>

rgwlc:  even current object versions have a unique instance

Fixes: https://tracker.ceph.com/issues/63546
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc:  additional lc notify cleanups

Follow the same formula to build up obj_state and version_id
in all call sites.

Resolves: rhbz#2163667

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #50055 from cbodley/wip-rgw-polymorphic-executor

rgw/beast: switch back to polymorphic executor

Reviewed-by: Mark Kogan <mkogan@ibm.com>

Merge pull request #54903 from cbodley/wip-63829

radosgw-admin: 'zone set' won't overwrite existing default-placement

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #54505 from adamemerson/wip-lazyfifo-no-mutex-over-coro

rgw/multisite: Fix deadlock by not holding mutex over coroutine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49594 from Matan-B/wip-matanb-crimson-calc-subsets

crimson: Support Partial Object Recovery

Reviewed-by: Samuel Just <sjust@redhat.com>