test/crimson/seastore/test_object_data_handler: remap pins through the
transaction that created the pins

Fixes: https://tracker.ceph.com/issues/65610
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 02acc0a71891ccadff68c60eaaf65bfa2c8bab4c)

crimson/os/seastore/btree: assert that the transaction getting logical
extents from pins are the one that created the pins

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit eae1b3172ee81a01403dfe8722ef6fcfee621032)

Merge pull request #57513 from Matan-B/wip-57129-squid

squid: crimson/os/seastore/transaction_manager: fix write pipeline phase leak

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57509 from Matan-B/wip-56383-squid

squid: crimson/os/seastore/btree: clean up `FixedKVLeafNode::get_logical_child()`

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57508 from Matan-B/wip-56875-squid

squid: crimson/osd/osd_meta: load incremental osdmap from "inc_osdmap.XXX"

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57506 from Matan-B/wip-56353-squid

squid: crimson/os/seastore: avoid new allocation when overwriting data in RBM for performance

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57504 from Matan-B/wip-56611-squid

squid: crimson/osd/replicated_recovery_backend: prepare_pull use pg_info

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57503 from Matan-B/wip-56912-squid

squid: crimson/common/operation: fix and move exit() after entering the next phase

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57501 from Matan-B/wip-56775-squid

squid: crimson/osd: implement basic reactor-utilization stats report to log

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57500 from Matan-B/wip-56627-squid

squid: crimson/os/seastore: alloc mapping with refcount when rewriting logical extents

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57499 from Matan-B/wip-55847-squid

squid: crimson: convert some of client_request to use coroutines

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57413 from neha-ojha/wip-squid-rc

squid: src/ceph_release: dev -> rc

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #57532 from rhcs-dashboard/wip-66092-squid

squid: mgr/k8sevents: update V1Events to CoreV1Events

Reviewed-by: Laura Flores <lflores@ibm.com>

doc/dev/release-checklists.rst: squid milestone added

Signed-off-by: Neha Ojha <nojha@redhat.com>

.github/milestone.yml: add squid

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit bca45c7df2c26b105401b299161f553ffe4e32c1)

Merge pull request #57547 from zdover23/wip-doc-2024-05-19-backport-57542-to-squid

doc/cephfs: Squid and later - subvolume quiesce

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #57540 from zdover23/wip-doc-2024-05-18-backport-57534-to-squid

squid: doc/cephfs: edit fs-volumes.rst (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: Squid and later - subvolume quiesce

Add a note to the "Subvolume quiesce" section that says that the
information in the section applies only to the Squid and later releases
of Ceph. This is included here so that I don't overwrite the Reef and
Quincy documentation with irrelevant information, and so that I don't
overwrite the Squid information with blank space where the "Subvolume
quiesce" section should be.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit be63ca6d037a954caacb3b9737332e886a1f762f)

doc/cephfs: edit fs-volumes.rst (2 of x)

Edit doc/cephfs/fs-volumes to the section "Cloning Snapshots" (but not
including the section "Cloning Snapshots".

Follows https://github.com/ceph/ceph/pull/57415

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0a3981a011d5e768ab8a5782884283b6650af28a)

Merge pull request #57520 from cbodley/wip-66069-squid

squid: cmake: disable WITH_QATLIB/ZIP on non-x86

Reviewed-by: Ken Dreyer <kdreyer@ibm.com>

mgr/k8sevents: update V1Events to CoreV1Events

centos9 only provides kubernetes 26.1.0 as base dep and hence the
k8sevents code needs to be updated accordingly. the api changes happened
in kuberenetes while 19.0.0 was released

Fixes: https://tracker.ceph.com/issues/65627
Fixes: https://tracker.ceph.com/issues/64981
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 6af964719217d720e6c2fd1ba2a607f6255d2604)

cmake: disable WITH_QATLIB/ZIP on non-x86

This feature is only relevant to x86 hosts.

Signed-off-by: Ken Dreyer <kdreyer@ibm.com>
Fixes: https://tracker.ceph.com/issues/66016
Signed-off-by: Ken Dreyer <kdreyer@ibm.com>
(cherry picked from commit 487cd2fddbab784269af9f48206a130e63f1eca3)

crimson/os/seastore/transaction_manager: fix write pipeline phase leak

At present, if a transaction gets interrupted right after it enters
WritePipeline::ReserveProjectedUsage and before any later continuations
get executed, WritePipeline::ReserveProjectedUsage will be locked
forever.

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 6a6f340c09a8dfe4565e298db11a30345ef7f82f)

crimson/os/seastore/btree: clean up `FixedKVLeafNode::get_logical_child()`

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 6e4f52a8ff93484d156f1d9095608a55e9c46aed)

crimson/osd/osd_meta: load incremental osdmap from "inc_osdmap.XXX"

Fixes: https://tracker.ceph.com/issues/65474
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 69271aef7002d0a76b43e8d1fc0667355119823e)

crimson/seastore: add a TODO comment regarding is_data_stable()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 5df6ffc79b415def0b7984096413e9562e83399a)

common/options: correct explanation regarding delta_based_overwrite

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 387c0c89992b5a28a7c5bba9f720a951aee1673b)

test/crimson/seastore: rename set/unset_overwrite_threshold() to enable/disable_delta_based_overwrite()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 24c0c198a5dd57070f210dc69dd83481df73fa15)

crimson/os/seastore: avoid new allocation when overwriting data in RBM for performance

In 4K random write test, after seastore is filled up by 4MB extents,
current implementation performs deep copy in duplicate_for_write(), resulting in
significant performance degradation by 80%.
Therefore, this commit changes the deep copy behavior for bufferptr during the overwrite
situation to shallow copy, leaving the original data untouched.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 76b87855b4a6fb60a1fae59792aeebff3e8762d3)

crimson/osd/replicated_recovery_backend: prepare_pull to use pg_info

Don't use peer's info on prepare_pull.

Fixes: https://tracker.ceph.com/issues/65200
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 7586abfad239d433049fb714d6fb8f2530a1b9c6)

crimson/common/operation: fix and move exit() after entering the next phase

If exit/unlock the barrier before entering the next phase, it is
possible for the next request to exit the barrier at the same time, and
enters the next phase first, causing reorder issues.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 078768ff961abad12cf3e7f19190a6f986fc5fce)

crimson/osd: implement basic reactor-utilization stats report to log

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 99909d45a282fb9dc89c6e8c98f4b866e67b09cb)

crimson/os/seastore: alloc mapping with refcount when rewriting logical extents

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
(cherry picked from commit 9269e68eacc825ccbe023e325c405bc0636d6b58)

crimson/os/seastore/btree_lba_manager: update_refcount returns the refcount of intermediate mapping

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
(cherry picked from commit 513bb4f3f21b739ea3f751680716b0fbcda9c3e1)

crimson/os/seastore/btree_lba_manager: cleanup methods that return std::pair

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
(cherry picked from commit 36f96ca000ab8839b7649c32f6d53ea88f9d9700)

crimson/os/seastore: introduce extent_ref_count_t

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
(cherry picked from commit bbee55a105ffb31d01c9a024325b8eb06d8f56e7)

crimson/os/seastore: remove unused return value of RecordScanner::scan_valid_records

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
(cherry picked from commit 1a914f64070299ab5712ce389096052709b72622)

crimson/common/interruptible_future: add discard_result

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
(cherry picked from commit 870441bb0c02992923ac6e0b6373b6d3770b470b)

crimson/.../client_request: work around gcc bz98401

See included comment for details.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1388173764408d047f2fbd5cf7b33dcf42971a03)

crimson/.../client_request: work around gcc bz101244 and bz102217

See included comment for details.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 90dcbfa0fa9d8a46f8075a7e6f2329652f0a312f)

crimson/.../client_request: convert ClientRequest::do_process to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 30237472bea485cb57f5c14031fb545aed97da15)

crimson/.../client_request: convert process_op to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit a4905fb3e2f2400656f0b436275e9fa745f14cc1)

crimson/.../client_request: make reply_op_error return interruptible_future<>

co_await can't implicitely convert it.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit ecf5795b6666f122fa3bf2771e83b066e611ade9)

crimson/.../client_request: convert ClientRequest::process_pg_op to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 0de3613072659cfdcb41ea7b1df5865425774b3b)

crimson/.../client_request: convert with_pg_process_interruptible coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 904a9937b6560f9a1f936f7f79badc56d8949dd1)

crimson/.../client_request: factor out with_pg_interruptible

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit b521cc3d822c39084f72c4f22f621928a184d731)

crimson/.../client_request: don't pass Ref<PG> by reference

If we only need a reference to the PG, pass a PG&.  Passing Ref<PG>&
makes it easy to inadvertently std::move() the passed value from
a caller.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 4164b08887c2b2c7a9df0cd80a681bb61be28ee6)

crimson/.../client_request: rename with_pg_int to with_pg_process

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 13042dde1134983f074f5bd4cf4c912d47d0193c)

crimson/.../client_request.cc: move message decode check to with_pg

We only need to do this once, no need to recheck on requeue.

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 10a9a11f5d535044f081964dc6b79813aa9bb624)

Merge pull request #57316 from Matan-B/wip-56027-squid

squid: doc/dev/crimson: add Code Walkthroughs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57484 from Matan-B/wip-56534-squid

squid: crimson: introduce INTERNAL_PG_LOCAL_NS, use for snapmapper

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57483 from Matan-B/wip-56378-squid

squid: crimson/osd/osd_operations/snaptrim_event: update PG's stats

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57482 from Matan-B/wip-56246-squid

squid: crimson/os/seastore/btree: check for reserved ptrs when determining children stability

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57481 from Matan-B/wip-55947-squid

squid: crimson/os/seastore: handle OP_COLL_MOVE_RENAME

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57480 from Matan-B/wip-55708-squid

squid: crimson/osd/main: Enable multicore cluster messenger

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57270 from Matan-B/wip-55632-squid

squid: debian/control: add new dependencies protobuf for crismon

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57287 from Matan-B/wip-55751-squid

squid: test/crimson/test_socket: complete write-side errors in test_unexpected_down()

Reviewed-by: Samuel Just <sjust@redhat.com>

common/hobject: make hobject_t formatter methods const

this is an alternative to 6021129f0c5c1c133436e471741c73f6e315c5dd
Since https://github.com/ceph/ceph/pull/55583 is not backported to S.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

crimson/.../scrub_events: use PGBackend::list_objects in ScrubScan::run

Fixes: https://tracker.ceph.com/issues/64975
Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit e69883ad2a05afab9d9ab02b098ad2418a4eec8a)

crimson/.../scrub_events: convert ScrubScan::run to coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 0d042f9d18fb191c3c8d6dbbf7c5bae54b8ed0fb)

crimson/.../scrub_events: use PGBackend::list_objects in ScrubFindRange::run

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 9f39124f33e878e321b330e58e97efc0a1cf1242)

crimson/.../scrub_events: convert ScrubFindRange::run to a coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 5b0416ae100443f43361e1308e2d28e2763ba725)

crimson: move snapmapper object into INTERNAL_PG_LOCAL_NS

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 1b7e3c4fb773a6e0a16145114013a07aa85edcc5)

crimson/osd/pg_backend: introduce INTERNAL_PG_LOCAL_NS, skip in PGBackend::list_objects

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit f32b37a599592d35ea948e66e7cfdfef7113b8ac)

crimson/osd/pg_backend: convert PGBackend::list_objects to a coroutine

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 4f056a4c1e9f01e4b04c96e85ebcfa29e6469adc)

crimson/osd/pg_backend: generalize PGBackend::list_objects, add overloads

Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 22a628534307d35e3fa90fb36a98f81cbf7cb7f6)

crimson/osd/osd_operations/snaptrim_event: update PG's stats

Fixes: https://tracker.ceph.com/issues/63307
Signed-off-by: junxiang Mu <1948535941@qq.com>
(cherry picked from commit 4a180f559a3fff1615664898603fc832cfd8d72c)

crimson/os/seastore/btree: always check the stability of extents within
the current transaction's view

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 7dcd74ac098201d2ea6f9d1c723f9944ade6bf33)

crimson/os/seastore/cached_extent: add comments to elaborate why
MUTATION_PENDING and under-io extents are "stable"

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit b1c59ca077597e14a053d47d4961b12ba7a2a1b0)

crimson/os/seastore/btree: check for reserved ptrs when determining
children stability

Fixes: https://tracker.ceph.com/issues/64957
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 7aa972def9f53fd9ed2f708aa87ce83362765cbf)

test/crimson/seastore: add the test case for renaming objects

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 2520927590ac48783cb35feb570285ec066964a4)

crimson/os/seastore: handle OP_COLL_MOVE_RENAME

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 70245c5b207eca6349920d1877ea237ec687f93d)

crimson/osd/osd: handle_scrub_message on any core

After enabling multicore cluster messenger, there is no
guarantee which core will handle the request.
See: OSDSingletonState::send_to_osd()

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit cbad3e4517e91476b19a8766f98240c81df40a36)

crimson/osd/main: Enable multicore cluster messenger

Taken from: f78e99c

Co-authored-by: Yingxin Cheng <yingxin.cheng@intel.com>
Co-authored-by: Chunmei Liu <chunmei.liu@intel.com>
Co-authored-by: Xinyu Huang <xinyu.huang@intel.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit ab239ad6cc5eb1cb4ca71d5b409bc4001124342f)

Merge pull request #57347 from Matan-B/wip-56375-squid

squid: crimson/.../scrub_events: fix progress lifetime in deep_scan_object

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57328 from Matan-B/wip-55840-squid

squid: crimson/os/seastore/omap_manager: expand omap tree nodes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57327 from Matan-B/wip-55887-squid

squid: crimson/os/seastore: omap_get_values with range [start, "infinite") should return all results from "start"

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57325 from Matan-B/wip-55889-squid

squid: crimson/os/seastore/async_cleaner: add comments to clarify what kind of extents would be considered dead by the cleaner

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57321 from Matan-B/wip-55770-squid

squid: CMakeLists: disable Seastar_IO_URING to fix seastar unittest timeout

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57323 from Matan-B/wip-55801-squid

squid: crimson/os/alienstore: use homebrew counting_semaphore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57320 from Matan-B/wip-56028-squid

squid: crimson/os/seastore: cleanups around lba parent-child pointer

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57317 from Matan-B/wip-55888-squid

squid: crimson/os/seastore: cleanup and use LBAMapping::is_stable()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57290 from Matan-B/wip-55806-squid

squid: crimson/os/seastore: copy attrs and omaps when cloning objects

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57324 from Matan-B/wip-52267-squid

squid: crimson/osd: put snapmapper's key-value pairs into dedicated objs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57281 from Matan-B/wip-55684-squid

squid: crimson: fixes for stack-use-after-return on recent clang and gcc versions

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57350 from Matan-B/wip-56217-squid

squid: test/crimson/sesastore/test_block: add constructor to construct

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57349 from Matan-B/wip-56462-squid

squid: crimson/os/seastore: fix use after free error

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57348 from Matan-B/wip-56332-squid

squid: crimson/os/pg_map: allow multiple shards to create new pg mappings at the same time

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57346 from Matan-B/wip-56374-squid

squid: crimson/osd/osd.h: declare osdmap after OSDSingletonState

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57345 from Matan-B/wip-56250-squid

squid: crimson/os/seastore: cache metadata during trimming to prevent from disk read

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57344 from Matan-B/wip-55846-squid

squid: crimson: add coroutine support for errorated and interruptible futures

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57326 from Matan-B/wip-55855-squid

squid: crimson/os/seastore: adjust SeaStore::_omap_set_kvs() params

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57322 from Matan-B/wip-55981-squid

squid: crimson/os/seastore/collection_manager: allow empty-delta-buffer collection nodes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57315 from Matan-B/wip-55977-squid

squid: crimson/os/seastore: allow to remap the dirty extent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57314 from Matan-B/wip-55536-squid

squid: osd/SnapMapper: Crimson - fix OSDriver::get_next

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>

Merge pull request #57312 from Matan-B/wip-55910-squid

squid: osd/SnapManager: Crimson - fix std::string constructed from 0

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57311 from Matan-B/wip-55946-squid

squid: crimson/os/seastore: make less verbose log when testing performance on RBM

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57289 from Matan-B/wip-55772-squid

squid: crimson/os/seastore: two attrs related bug fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57288 from Matan-B/wip-55705-squid

squid: crimson/os/seastore: fix wrong-positioned asserts

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #57286 from Matan-B/wip-55407-squid

squid: crimson/osd/shard_services: make sure that only up/acting members can create pgs

Reviewed-by: Samuel Just <sjust@redhat.com>