crimson/tools/store_nbd: read logical extents via
TransactionManager::read_pin()

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 33b56a04d51443f5f035bfc267da16d4a5dfe28c)

crimson/os/seastore/cache: add comment about backref_extent_entry_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 62974a65897edce2afdb16ba99baea0193a0d4a6)

test/crimson/seastore: complement lba test with logical extents

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 3c4f8c761333dcc1a4e24e73f37808720c8f684f)

test/crimson/seastore: check intra-fixedkv-btree parent->child trackers during unittests

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 302bc3c2d95cd36012008e334444f689ddb6694f)

crimson/os/seastore/btree: drop btree_pin_set_t

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 4a3dfc0f630d6e635bd82801e0107be78d3d2c6d)

crimson/os/seastore/transaction_manager: follow leaf<->logical extent pointers to read extent

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 89c2d0b3af7feb4124b7a5ac7f54026c205663b5)

crimson/os/seastore/lba_manager: link lba leaf nodes with logical extents by pointers

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit cce850d75609c7c34bce0920e4e12ba9b9513229)

crimson/os/seastore/btree: "templatize" btree leaf node to distinguish leaf nodes with(out) children

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 55e1924e3818e7fb574893372ba7dfad4fa54014)

crimson/os/seastore/btree: link fixed-kv-btree and root_block with pointers

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 4d9b60e75047fa0d7e5acce601bc15f49d253349)

crimson/os/seastore: more debug logs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 25b001db29bbd58c627f3ab64428f5aaac14a399)

crimson/os/seastore/backref_manager: retrieve live backref extents throught the backref tree

After involving intra-fixed-kv-btree parent-child pointers, we need to keep the
invariant that it's only when extents are not in transactions' read_set that
we can directly query cache with inspecting the transaction

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 45440fadd20fa21deaddbe6db4e0c4e84015c9bf)

crimson/os/seastore/btree: avoid searching transactions' read_set when retrieving btree nodes

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit c29051c4c747c5b8415b8600affc24c0025507fb)

crimson/os/seastore/btree: search fixed-kv-btree by parent<->child pointers

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 7c3305f0149808f800dc6c852d4f1f755a490ec4)

crimson/os/seastore/cache: invalidate out-dated extent when initiating Cache

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 686d1206532200733d69b9430227bdb6bf2d9eb3)

crimson/os/seastore/cached_extent: improve the representation of "has_been_invalidated"

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 1b4c591ef51920211de978f490b4125177571e08)

crimson/os/seastore/btree: don't go to leaf nodes when updating internal mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit a86c7bd651bc7a89a49e57f9a87e8baeb4291c41)

crimson/os/seastore/btree: introduce parent<->child pointers for fixed-kv-btree nodes

maintain correct parent<->child pointers when modifying the btree

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 71051f997fca8ff0fdcd55586e3150bffa2a916a)

crimson/os/seastore/cached_extent: add on_invalidated interface

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit e5aabe6c306061b2ce3aff5ca30a9049c1a7f9bf)

crimson/os/seastore/cached_extent: add on_replace_prior interface

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit c56af30a9c199b4a5d2cba9d2208b8c1d8a6abfb)

crimson/os/seastore/cached_extent: duplicate_for_write accepts the transaction ref as its parameter

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 06ae16141470a9052224fbe76c842fa13bbf4520)

crimson/os/seastore/cached_extent: record transaction id in pending CachedExtents

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 8240be8f324b244061b17bb2c1676b192a12dca2)

Merge pull request #51673 from Matan-B/wip-50546-reef-2

reef: crimson/osd: Support librados::OPERATION_ORDERSNAP

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #51649 from adk3798/reef-require-image-flag-inspect

reef: cephadm: require --image is passed to inspect-image

Reviewed-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #51650 from adk3798/reef-cephadm-byte-compile

reef: cephadm: call compile_dir to byte compile zipapp py files

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51614 from adk3798/wip-59631-reef

reef: mgr/cephadm: support for nfs backed by VIP

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51613 from adk3798/wip-59550-reef

reef: mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51342 from k0ste/wip-59634-reef

reef: pybind/mgr: improve error when inbuf is given to commands that don't use it

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #51693 from zdover23/wip-doc-2023-05-23-backport-51682-to-reef

reef: doc/glossary: update bluestore entry

Merge pull request #51672 from k0ste/wip-61342-reef

reef: mgr/prometheus: fix pool_objects_repaired and daemon_health_metrics format

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #51636 from Matan-B/wip-50584-reef

reef: qa/suites/crimson-rados/thrash: Enable supported tests and ops

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51625 from Matan-B/wip-47974-reef

reef: crimson/os/seastore: evict cold data to slower devices

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #51588 from Svelar/reef-vstart-filestore-cleanup

reef: vstart: remove filestore options and config snippets

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #51644 from Matan-B/wip-51167-reef

reef: crimson/qa: make crimson run multicore in teuthology test

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51641 from Matan-B/wip-50457-reef

reef: qa/suites/crimson-rados/rbd: Add new rbd image format api tests

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51675 from Matan-B/wip-47637-reef

reef: crimson/osd: bring support for client blocklisting

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51674 from Matan-B/wip-51198-reef

reef: crimson/osd/snaptrim_event: Handle missing clone obc case

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/suites/crimson-rados/thrash: Enable supported thrashers

Balanced/Localized reads are now supported.
snap_remove and rollback are supported as well.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit b888cfa3da956aad527d05c3dbe507efb368363f)

qa/suites/crimson-rados/thrash: Add snap_remove/create weights

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 8839f829d60e0f0a3e6feca37b8eaa167bccdfd0)

qa/suites/crimson-rados: Remove rados_api_test.yaml.disabled

This test is already included in:
crimson-rados/basic/tasks/rados_api_tests.yaml

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit bb23ddc5dc31ced752db0310d9857ba268005182)

Merge pull request #51635 from Matan-B/wip-49980-reef

reef: crimson/os/seastore/journal: generalize recordsubmitter

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51629 from Matan-B/wip-50491-reef

reef: test/librados/snapshots*: Enable Crimson supported tests

Reviewed-by: Laura Flores <lflores@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

doc/glossary: update bluestore entry

Update the BlueStore entry in the glossary, explaining that as of Reef
BlueStore and only BlueStore (and not FileStore) is the storage backend
for Ceph.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit bcee264276128f622c35e3aab81fdecb2b8afc10)

test/crimson/seasatore: add evction test

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit c53446df687dd82860ba012f89ae783be8a4f51c)

crimson/os/seastore/EPM: cleanup alloc_new_extent

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 5bc650a0605ef9559ac32fe70930d446652c2d1d)

crimson/os/seasotre/segment_manager_group: ensure the types of devices are same

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 61156355385aaa06cf9d15870a8830256874714d)

crimson/os/seastore: add EPHEMERAL_COLD device type

rename SEGMENTED_EPHEMERAL to EPHEMERAL_MAIN

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit b40304092b6ce8b29e97437642928cc84afa1be5)

crimson/os/seastore/EPM/BackgroundProcess: introduce more eviction policy

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 45c53a26c662b2c23efdecaaa186b6ac8811c2ea)

crimson/os/seastore/EPM/BackgroundProcess: reserve space on cold cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 4aea0e266de43281a5d8c24fe91034bdaae45d98)

crimson/os/seastore: get segment info from cold segment provider

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 7b0ccecbced037bbceca707d62d81fe2accc5a39)

crimson/os/seastore: support mark_space_{free, used} on cold_cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 783c5e7662006c6efca7ade795e6e731b0b18332)

src/vstart.sh: support specify the type of secondary device

doc/crimson: add options and example about SeaStore

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit b40f6421f0d6f7966e13941ba13e70b5639bce93)

crimson/os/seastore: change the category of COLL_BLOCK

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 9283b72e476f306a8926aa2f7ae191ca8705fcd4)

crimson/os/seastore: specify the device type explicitly when constructing BlockSegmentManager

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 1e515e8561519ded43b1046942057d705441ee22)

crimson/os/seastore: make segment_manager work with HDD

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 89939e7ad7c1dffaa1c842be601fa372d074a86a)

crimson/os/seasotre: add CLEANER_COLD transaction type

rename CLEANER to CLEANER_MAIN

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 0d20f1595a0c839e780540b61ff68f52be29a99b)

crimson/os/seastore: init cold segment cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 5876ef35b651ab5ac94f891382553d25f641f19e)

crimson/os/seastore/SegmentCleaner: distinguish the cold cleaner from the normal

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 483e36826802745e68855f0f0f2a89d199903011)

crimson/os/seastore/EPM/BackgroundProcess: add cold_cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 343f579748d9856a7f8fde49025db91cdce1367d)

crimson/os/seastore: move ool segment seq allocator to EPM

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 944c1253bf52fb6f7d773b4bfeb5f7cc8f0ffebc)

crimson/os/seastore/EPM: refactor reserve process

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 86e2f4b24bff98b775a76289aa3fd382333d42ad)

crimson/os/seastore/EPM/BackgroundProcess: rename cleaner to main_cleaner

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 3adb17bf04252d9c1afd9050ed1e876bd6c654b9)

Merge pull request #51623 from Matan-B/wip-50317-reef

reef: crimson/os/seastore/epm: also do prepare_write for rbm

Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: modify FreeBSD objectstore to memstore.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>
(cherry picked from commit 27e66f506f10dca7018c10e3e1804f0b3c217b8e)

vstart.sh:drop filestore support

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>
(cherry picked from commit 41f2af4f5e70accd3da1a5761f73bda8786b7f4f)

Merge pull request #51677 from zdover23/wip-doc-2023-05-23-backport-51646-to-reef

reef: doc/dev/crimson: Improve crimson.rst

Merge pull request #51678 from zdover23/wip-doc-2023-05-23-backport-51651-to-reef

reef: doc: note deprecation of Cache Tiering in Reef

mgr/cephadm: asyncio based universal timeout for ssh/cephadm commands

Since we already have make use of asyncio for our ssh commands,
we can use asyncio's timeout on waiting for concurrent futures to complete
as a way to have universal timeouts on our cephadm commands.
This change also creates a contextmanager that will catch any asyncio.TimeoutError.
Using the contextmanager along with calls to the wait_async function
will catch any timeout exception raised and convert it into an appropriate
OrchetratorError including information about what and where for the timeout
if it was provided (host where run, what command). This allows us to guarantee a
background ssh command eventually returns and inform users of any
timeouts by raising a health warning or logging the error instead
of sitting idle indefinitely

Fixes: https://tracker.ceph.com/issues/54024
Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit adf13c82ebb75a132709aa0b0bbc1d4fef7643a7)

mgr/cephadm: pass --timeout to cephadm binary commands

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 00d4203ec60ed42840b1b318af49e4e697bd04f2)

Merge pull request #51612 from adk3798/wip-59548-reef

reef: mgr/cephadm: allow setting mon crush locations through mon service spec

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #51611 from adk3798/wip-59546-reef

reef: cephadm: reschedule haproxy from an offline host

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #51208 from guits/wip-59255-reef

reef: ceph-volume: fix drive-group issue that expects the batch_args to be a string

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51205 from guits/wip-59516-reef

reef: ceph-volume: fix batch refactor issue

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51194 from guits/wip-59523-reef

reef: ceph-volume: quick fix in zap.py

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #51191 from guits/wip-59519-reef

reef: orchestrator: add `--no-destroy` arg to `ceph orch osd rm`

Reviewed-by: Adam King <adking@redhat.com>

doc: note deprecation of Cache Tiering in Reef

Add a note to the documentation that cache tiering is deprecated in
Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 7f9eaf7a8ef95d611df81463424cfd06472db356)

doc/dev/crimson: Improve crimson.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 524901d3cfbf73cc86284a2623ada493836c13ec)

Merge pull request #51593 from cbodley/wip-61219-reef

reef: rgw: use string_view to parse Accept header

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #51592 from cbodley/wip-61298-reef

reef: qa/rgw: add POOL_APP_NOT_ENABLED to log-ignorelist

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #51569 from amathuria/wip-61232-reef

reef: qa/tasks: Changing default mClock profile to high_recovery_ops

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #51409 from kamoltat/wip-ksirivad-reef-backport-50857

reef:mon/Monitor.cc: exit function if !osdmon()->is_writeable()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #50977 from yaarith/wip-59390-reef

reef: mgr/telemetry: add leaderboard description and documentation

Merge pull request #51357 from rhcs-dashboard/wip-59633-reef

reef: mgr/cephadm: update monitoring stack versions

Reviewed-by: Adam King <adking@redhat.com>

crimson/osd: drop the OSDMap caching from PG

`PG` directly aggregates both `OSDMap` and `PeeringState` which
maintains its own `OSDMap` instance. This duplication leads to
issues as the `PG::osdmap` never gets updated.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit d8655c08ab51e2c5b1437c7a46498d5450a1bac0)

crimson/osd: implement watcher blocklisting

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 24098bf9bbf66c64f48cb56187143102ffe13230)

crimson/osd: check for blocklisted clients

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 03b6c21b5366ea5d837de3059e70146209dcb335)

crimson/osd/pg: PG::on_active_actmap() clear PG_STATE_SNAPTRIM_ERROR

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit ad129f39e75ed9a128c5bc2bc52d575a01f1b0ea)

crimson/osd/pg: PG::on_active_actmap() fix SnapTrimEvent stop condition

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 03ff8f206ab681eb1518e7e73d737bce40ab7a15)

crimson/osd/osd_operations/snaptrim_event: Handle missing clone obc

The case where resolve_oid is not able to return the clone oid (in order
to load the clone obc) should result in an SNAPTRIM_ERROR and stop the loop
iteration. Without taking this case into account an infinite loop of snap
trim events will be created.
The ENOENT error is delegated upwards to be handled by PG::on_active_actmap().

Fixes: https://tracker.ceph.com/issues/59231
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 5f1f3625908331e164274621f2e0b092efda0452)

crimson/osd/osd_operations/snaptrim_event: Add log lines and formatting

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 51c70dc95dd6a0ce2d9044a0907e7fa46fc02619)

test/librados/snapshots_cxx: Enable OrderSnap for Crimson

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit ee4cebedfe61e5181c38d6be153b68cf56b74c61)

crimson/osd/osd_operations/client_request: Support librados::OPERATION_ORDERSNAP

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 749e0ec558b1fc0415fcc14afee88180d6f3e106)

crimson/osd/client_request: Move writer SnapContext initialization

Move the initialization from PG::do_osd_ops() to ClientRequest::do_process()
in order to allow for ORDERSNAP error checking before do_osd_ops. (See next commit)

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit b5ef10628179d164736cf406c5d5ed9abeb6e945)

mgr/prometheus: fix pool_objects_repaired and daemon_health_metrics format

mgr/prometheus: fix pool_objects_repaired and daemon_health_metrics format

- fix "error reading metrics for http://****:***/metrics: reading text format failed: text format parsing error in line 2010: second HELP line for metric name "ceph_pool_objects_repaired" error

- rename label name "poolid" to "pool_id" like all other metrics
- change type for the "daemon_health_metrics" to gauge

Fixes: https://tracker.ceph.com/issues/59505
Signed-off-by: banuchka <tyrchenok@gmail.com>
(cherry picked from commit 95d5303e11d6cba5ec4e71716af8931bcc2b9767)

Merge pull request #51645 from Matan-B/wip-51448-reef

reef: crimson/osd/scheduler/mclock_scheduler: Fix OSD unable to start

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51643 from Matan-B/wip-50411-reef

reef: crimson/osd: start operations asynchrously

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51642 from Matan-B/wip-51333-reef

reef: crimson/osd/objclass: Fix compilation warning

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51640 from Matan-B/wip-50227-reef

reef: crimson/osd/heartbeat: share osdmap to peers when necessary

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51639 from Matan-B/wip-51202-reef

reef: crimson/osd/pg: make clone object's version consistent with pglog

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51638 from Matan-B/wip-51180-reef

reef: crimson/osd/snaptrim_event: get SnapSetContext from head obc

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #51637 from Matan-B/wip-50530-reef

reef: crimson/osd: Handle rollback to head obejct

Reviewed-by: Samuel Just <sjust@redhat.com>