test/crimson/.../test_object_data_handler: add multiple clone/overwrite test case

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/.../test_object_data_handler.cc: add support for clones

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: add logging to remap_mappings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: add formatter for remap_entry_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../object_data_handler: fix LOG_PREFIX for do_clone

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: convert remap_mappings to use cursors

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: convert reserve_region to use cursor

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: simplify _update_mapping_ret, remove update_mapping_ret_bare_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert _update_mapping to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: remove update_refcount, simplify _update_mapping return value

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert remap_mappings to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert remap_mappings to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: convert clone_mapping to use cursors

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert clone_pin to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert clone_mapping to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert get_end_mapping to return LBACursorRef

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert get_end_mapping to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: remove LBAMapping update_mapping variant

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: remove get_mapping interfaces

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert get_pins to use get_cursors

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert get_extents_if_live and helpers to use cursors

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert get_extents_if_live to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: convert alloc_extents_ret to use LBACursor

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert alloc_data_extents to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert alloc_extents to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert rewrite_logical_extent to use get_cursor

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: switch alloc_extent_ret to LBACursorRef

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert reserve_region to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert reserve_region to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert alloc_extent to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert rewrite_logical_extent to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert get_pin to use get_cursor

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: remove remove_mappings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: rework _remove in terms of LBACursor interfaces

Removes the need for _remove_direct_mapping and _remove_indirect_mapping.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: avoid using LBAManager::ref_update_result_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: replace LBAManager::complete_indirect_lba_mapping with TransactionManager::complete_mapping

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: simplify get_extent_if_linked

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert _remove_indirect_mapping to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert _remove_direct_mapping to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../remove: convert to use cursor interfaces

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert remove to use cursor interfaces

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_types: add LBACursor::is_direct

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert _remove_indirect_mapping_only to use new interfaces

Also removes now unused LBAMapping::remove_indirect_mapping_only.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: remove unused get_ref_update_result

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: replace unused incref_extent with update_mapping_refcount

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: remove unused _incref_extent

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert remove_mapping to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: switch update_mapping to take a cursor, add adapter

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert update_mapping to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: introduce resolve_cursor_to_mapping

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager: add get_cursor interfaces

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert get_mappings to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert get_cursors to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../fixed_kv_btree: add get_btree helper

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_lba_manager: convert mkfs to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert remove to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert read_extent to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: convert get_pin to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../btree_types: allow get_refcount() on indirect cursor

It should always be 1, but there's no reason not to allow it.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #67018 from rhcs-dashboard/customize-side-panel

mgr/dashboard: side-panel enhancements

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #66761 from rhcs-dashboard/carbonize-delete-zone-modal

mgr/dashboard: carbonize-delete-zone-modal

Reviewed-by: Afreen Misbah <afreen@ibm.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge PR #66508 into main

* refs/pull/66508/head:
pybind: remove deprecated PyEval_InitThreads() calls

Reviewed-by: Patrick Donnelly <pdonnell@ibm.com>

Merge pull request #66370 from kshtsk/wip-main-drop-netstat

qa/workunits/rgw: drop netstat usage

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #67054 from djgalloway/revert-cephadm-image

Revert "qa/cephadm: add default container image name base"

Merge pull request #66846 from rhcs-dashboard/add-cephfs-mirror-endpoints

mgr/dashboard: add CephFS mirror endpoints

Reviewed-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

mgr/dashboard: side-panel enhancements

fixes: https://tracker.ceph.com/issues/74486

Signed-off-by: Naman Munet <naman.munet@ibm.com>

mgr/dashboard: add CephFS mirror endpoints

For: create peers, delete peers, create token, and check module status

Fixes: https://tracker.ceph.com/issues/74362
Signed-off-by: Pedro Gonzalez Gomez <pegonzal@ibm.com>

Merge pull request #66740 from xxhdx1985126/wip-seastore-rbm-no-backref

crimson/os/seastore: drop backrefs in cases of pure rbm seastores

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #67063 from idryomov/wip-74529

qa: don't assume that /dev/sda or /dev/vda is present in unmap.t

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #67048 from idryomov/wip-74513

qa: krbd_blkroset.t: eliminate a race in the open_count test

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #66622 from ivancich/wip-max-max-reshard-logs-listings

rgw: enforce max max for reshard logs listings

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mingyuan Liang <liangmingyuan@baidu.com>

Merge pull request #66673 from ivancich/wip-fix-radosgw-admin-obj-put

rgw: `radosgw-admin object put ...` broken w/ versioned buckets

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: enforce max max for reshard logs listings

Previously a list of reshard logs could be unlimited. We need to
enforce time limits on CLS ops.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #66623 from ivancich/wip-fix-bi-list-backwards

rgw: bucket index list can go backwards and may loop

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #67058 from sseshasa/wip-fix-iops-threshold-warning-74501

qa: Disable OSD benchmark from running for tests.

Reviewed-by: Laura Flores <lflores@ibm.com>

Merge pull request #67006 from bluikko/wip-doc-radosgw-ref-links-split1

doc/radosgw: change all intra-docs links to use ref (2 of 6)

qa: don't assume that /dev/sda or /dev/vda is present in unmap.t

Instead of hard-coding the block device name, use the block device that
is backing the filesystem that the test is running on.  We can be quite
sure it won't be an RBD device ;)

Fixes: https://tracker.ceph.com/issues/74529
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #66914 from afreen23/productive-card

mgr/dashboard: Add productive card component

Reviewed-by: Devika Babrekar <devika.babrekar@ibm.com>

qa: Disable OSD benchmark from running for tests.

Disable OSD bench from benchmarking the OSDs for teuthology tests. This is to
help prevent a cluster warning pertaining to the IOPS value not lying within
a typical threshold range from being raised.

The tests can rely on the built-in static values as defined by
osd_mclock_max_capacity_iops_[ssd|hdd] which should be good enough.

Fixes: https://tracker.ceph.com/issues/74501
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Revert "qa/cephadm: add default container image name base"

This reverts commit 1b155707a2b463240e5bf1c73f9ef0ade93cd7be.

This should have been a default in teuthology.yaml.  My reasoning for the revert is:
- This is hardcoding a value
- Folks wouldn't know this is configurable without looking at cephadm.py
- Instead of doing what the error message said to do, we removed the error message

Signed-off-by: David Galloway <david.galloway@ibm.com>

Merge pull request #67041 from Matan-B/wip-matanb-debug-container

container/build.sh: Use dedicated debug tags

qa: krbd_blkroset.t: eliminate a race in the open_count test

Even at QD=1, dd may take less than 10 seconds to work its way to the
end of a 10M image, producing "No space left on device" error instead
of the expected "Operation not permitted" error which is supposed to
arise from the device getting marked read-only while opened.

Fixes: https://tracker.ceph.com/issues/74513
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #66625 from kshtsk/wip-dnsmasq-preserve-nameservers

rgw/website: preserve nameservers for future use in dnsmasq

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #66648 from Matan-B/wip-matanb-crimson-flavor-overrides

qa/config,crontab: Adjust to Crimson flavor cleanup

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

qa/crontab/teuthology-cronjobs: Use debug flavor

Use the new Debug flavor introduced in https://github.com/ceph/ceph-build/pull/2497.
This should provide more nightly test coverage.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

qa/crontab/teuthology-cronjobs: Cleanup Crimson tentacle nightly

With https://github.com/ceph/ceph-build/pull/2497 merged we no loger
build Tentacle+Crimson regularly. As Crimson no longer backport changes
into Tentacle, there's no reason to keep testing it.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #67037 from tchaikov/wip-cmake-fix-cmake-ld

cmake: fix undefined PY_LDFLAGS in distutils_install_cython_module

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #66749 from ronen-fr/wip-rf-dumptargets

osd/scrub: improve scrub target dumps

Reviewed-by: Jon Bailey <jonathan.bailey1@ibm.com>

container/build.sh: Use dedicated debug tags

https://github.com/ceph/ceph-build/pull/2497 introduced a debug flavor.
This seems to cause conflicts with the image being pushed to quay as one
of the flavors might override the other.

Tag debug build containers explicitly.
Alternative solution would be to skip debug containers all together.
However. these might be useful for development purposes.

Note, prune-quay might also need to be updated once this is merged.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

container/build.sh: cleanup crimson flavors

We no longer have crimson-flavors see https://github.com/ceph/ceph-build/pull/2497

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

cmake: fix undefined PY_LDFLAGS in distutils_install_cython_module

The distutils_install_cython_module() function was using ${PY_LDFLAGS}
without defining it, causing the linker to fail with:

  /opt/rh/gcc-toolset-13/root/usr/libexec/gcc/x86_64-redhat-linux/13/ld:
  cannot find -lrados: No such file or directory

This bug was introduced in commit d22734f6cb0 which changed:
  set(ENV{LDFLAGS} "-L${CMAKE_LIBRARY_OUTPUT_DIRECTORY}")
to:
  set(ENV{LDFLAGS} "${PY_LDFLAGS}")

However, PY_LDFLAGS was only defined in distutils_add_cython_module(),
not in distutils_install_cython_module(). This meant that during the
install phase, LDFLAGS was set to an empty string, and the linker
couldn't find librados.so and other Ceph libraries in the build
directory.

The bug was exposed by commit 719b74984605b490f23004eb41583a22c934c5fb
which changed rados.pxd to use C preprocessor conditionals (#ifdef
BUILD_DOC) instead of Cython's compile-time IF statements. This meant
the build now required proper linking during the install phase.

Fix by defining PY_LDFLAGS in distutils_install_cython_module():

  set(PY_LDFLAGS "${CMAKE_SHARED_LINKER_FLAGS} -L${CMAKE_LIBRARY_OUTPUT_DIRECTORY}")

This keeps CMAKE_SHARED_LINKER_FLAGS as a space-separated string and
appends the library directory flag, avoiding issues with semicolon
conversion.

Fixes: d22734f6cb0
Signed-off-by: Kefu Chai <k.chai@proxmox.com>

doc/radosgw: change all intra-docs links to use ref (2 of 6)

Part 2 of 6 to make backporting easier. Depends on part 1.

Use the the ref role for all remaining links in doc/radosgw/ with the
exception of config-ref.rst which will depend on changes to rgw.yaml.in.

The external link definitions syntax being removed is intended for
linking to external websites and not for intra-docs links. Validity of
ref links will be checked during the docs build process.

Add labels for links targets if necessary.
Remove unused external link definitions in the modified files.

Use confval instead of literal text for 2 configuration keys in
vault.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #66944 from bluikko/wip-doc-radosgw-ref-links1

doc/radosgw: change all intra-docs links to use ref (1 of 6)

doc/radosgw: change all intra-docs links to use ref (1 of 6)

Part 1 of 6 to make backporting easier. Many of the following parts
depend on this.

Use the the ref role for all remaining links in doc/radosgw/ with the
exception of config-ref.rst which will depend on changes to rgw.yaml.in.

The external link definitions syntax being removed is intended for
linking to external websites and not for intra-docs links. Validity of
ref links will be checked during the docs build process.

Add labels for links targets if necessary.
Remove unused external link definitions in the modified files.

Use confval instead of literal text for 2 configuration keys in
vault.rst.
Use Ceph Object Gateway consistently in multisite.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #67020 from Matan-B/wip-matanb-crimson-asan-fixes-v2

Revert "common/options: fix heap-use-after-free by using never-destro…

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Kefu Chai <k.chai@proxmox.com>

Merge pull request #66998 from ronen-fr/wip-rf-nextdeepscrub

osd/scrub: removing the unused next_deepscrub_interval()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #66847 from adk3798/cephadm-teuth-assume-default-image-base

qa/cephadm: add default container image name base

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #66938 from ivancich/wip-orphan-list-empty

rgw: rgw-orphan-list can continue with empty intermediate file(s)

Merge pull request #66975 from imran-imtiaz/dashboard

mgr/dashboard: improve RBD group API endpoint documentation

Revert "common/options: fix heap-use-after-free by using never-destroyed static"

```
Jan 20 09:27:16 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]: AddressSanitizer:DEADLYSIGNAL
Jan 20 09:27:16 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]: =================================================================
Jan 20 09:27:16 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]: ==3==ERROR: AddressSanitizer: stack-overflow on address 0x7b512f6c8dd8 (pc 0x0000046e7a72 bp 0x7b512de7c900 sp 0x7b512f6c8dd8 T0)
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #0 0x0000046e7a72 in get_global_options() (/usr/bin/ceph-osd-crimson+0x46e7a72) (BuildId: 2a86043f51c9be9cb19801e276fb3ee36239556a)
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #1 0x0000046e540e in build_options() (/usr/bin/ceph-osd-crimson+0x46e540e) (BuildId: 2a86043f51c9be9cb19801e276fb3ee36239556a)
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #2 0x0000033b7949 in get_ceph_options() (/usr/bin/ceph-osd-crimson+0x33b7949) (BuildId: 2a86043f51c9be9cb19801e276fb3ee36239556a)
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #3 0x000003440540 in md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool) (/usr/bin/ceph-osd-crimson+0x3440540) (BuildId: 2a860>
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #4 0x0000046856a8 in crimson::common::ConfigProxy::ConfigProxy(EntityName const&, std::basic_string_view<char, std::char_traits<char> >) (/usr>
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #5 0x000000eb6cb5 in seastar::shared_ptr_count_for<crimson::common::ConfigProxy>::shared_ptr_count_for<EntityName&, std::__cxx11::basic_string>
..
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #40 0x000000ed6434 in seastar::future<int> seastar::futurize<int>::apply<crimson::osd::_get_early_config(int, char const**)::{lambda()#1}::ope>
Jan 20 09:27:17 ceph-node-0 ceph-e818662e-f5e1-11f0-b263-525400908ba7-osd-1[12300]:     #41 0x000000ed672b in seastar::async<crimson::osd::_get_early_config(int, char const**)::{lambda()#1}::operator()() const::{lambda()#1}>(seast>
```

This reverts commit 1ab0a8cb726cb730954294423acec887b92fa5b0.

Fixes: https://tracker.ceph.com/issues/74481
Signed-off-by: Matan Breizman <mbreizma@redhat.com>