Merge pull request #53962 from Matan-B/wip-53051-reef

reef: crimson/os/seastore/omap_manager: fix the entry leak issue in BtreeOMapManager::omap_list()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53961 from Matan-B/wip-52128-reef

reef: crimson/os/seastore/journal/cbj: generalize scan_valid_records()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53960 from Matan-B/wip-53023-reef

reef: crimson/os/seastore: fix daggling reference of oid in SeaStore::Shard::stat()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53959 from Matan-B/wip-52757-reef

reef: crimson/os/seastore: retire_extent_addr clean up

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53958 from Matan-B/wip-52655-reef

reef: crimson/ertr: assert on invocability of func provided to safe_then()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53955 from Matan-B/wip-52446-reef

reef: crimson/os/seastore/transaction_manager: data loss issues

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53951 from Matan-B/wip-51179-reef

reef: crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53949 from Matan-B/wip-51916-reef

reef: crimson/net: support connections in multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53947 from Matan-B/wip-52217-reef

reef: crimson/os/seastore/omap_manager: correct editor settings

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53946 from Matan-B/wip-52090-reef

reef: crimson/os/seastore/btree: should add left's size when merging levels…

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53937 from Matan-B/wip-50598-reef

reef: crimson/osd: add embedded suppression ruleset for LSan

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53953 from Matan-B/wip-52369-reef

reef: crimson/os/seastore: add metrics about total invalidated transactions

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53952 from Matan-B/wip-51770-reef

reef: crimson/os/seastore/rbm: make rbm support multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53950 from Matan-B/wip-51852-reef

reef: crimson/osd/watch: Add logs around Watch/Notify

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53948 from Matan-B/wip-52032-reef

reef: crimson/osd: don't ignore start_pg_operation returned future

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53945 from Matan-B/wip-51980-reef

reef: crimson/os/seastore: fix in check_node

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53944 from Matan-B/wip-51961-reef

reef: Crimson/osd: Disable concurrent MOSDMap handling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53943 from Matan-B/wip-51945-reef

reef: crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53941 from Matan-B/wip-51948-reef

reef: crimson/os/seastore/cached_extent: add prepare_commit interface

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53940 from Matan-B/wip-51380-reef

reef: crimson/osd/heartbeat: Improve maybe_share_osdmap behavior

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53939 from Matan-B/wip-51750-reef

reef: crimson/os/seastore/backref_manager: scan backref entries by journal seq

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53938 from Matan-B/wip-51475-reef

reef: crimson/os/seastore: fix ceph_assert in segment_manager.h

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53935 from Matan-B/wip-51355-reef

reef: crimson/os/seastore: enable SMR HDD

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53797 from smanjara/ragweed-test-reef

reef: rgw/upgrade: point upgrade suites to ragweed ceph-reef branch

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53773 from cbodley/wip-63057-reef

reef: rgw: improve buffer list utilization in the chunkupload scenario

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53770 from cbodley/wip-63054-reef

reef: rgw: fix SignatureDoesNotMatch when extra headers start with 'x-amz'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53767 from cbodley/wip-63051-reef

reef: rgw: s3website doesn't prefetch for web_dir() check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53765 from cbodley/wip-63048-reef

reef: rgw: fix rgw rate limiting RGWRateLimitInfo class decode_json max_rea…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53760 from cbodley/wip-63060-reef

reef: rgw: fix unwatch crash at radosgw startup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #53756 from cbodley/wip-63042-reef

reef: [CVE-2023-43040] rgw: Fix bucket validation against POST policies

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #53681 from mkogan1/wip-t62141-reef

reef: valgrind: UninitCondition under __run_exit_handlers suppression

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54026 from zdover23/wip-doc-2023-10-16-backport-54022-to-reef

reef: doc/rados: Edit troubleshooting-osd (3 of x)

doc/rados: Edit troubleshooting-osd (3 of x)

Edit doc/rados/troubleshooting/troubleshooting-osd.rst (3 of x).

Follows https://github.com/ceph/ceph/pull/53986.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 01b5aa5d27a97a02cef03b9eb0ee9dbcd55684f1)

Merge pull request #54008 from zdover23/wip-doc-2023-10-14-backport-54006-to-reef

doc/rados: remove FileStore material (in Reef)

doc/rados: remove FileStore material (in Reef)

Remove FileStore-related material because FileStore is not supported in
Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit eab0be5ca513ca2dfaeb31911c175e88bcf4f9e4)

Merge pull request #54007 from zdover23/wip-doc-2023-10-14-backport-54005-to-reef-2

reef: doc/dev: remove cache-pool

doc/dev: remove cache-pool

Remove cache-pool, which is deprecated in Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 073b0919ed9e413a04a7e12b7a0c23b5e058cd52)

Merge pull request #54000 from zdover23/wip-doc-2023-10-13-backport-53986-to-reef

reef: doc/rados: Edit troubleshooting-osd (2 of x)

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

doc/rados: Edit troubleshooting-osd (2 of x)

Edit doc/rados/troubleshooting/troubleshooting.rst (2 of x).

Follows https://github.com/ceph/ceph/pull/53936.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 37e7099267996a3075b4902a10a19d94fc738c08)

Merge pull request #53458 from rhcs-dashboard/wip-62602-reef

reef: mgr/dashboard: remove unnecessary failing hosts e2e

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #53982 from zdover23/wip-doc-2023-10-12-backport-53936-to-reef

reef: doc/rados: edit troubleshooting-osd (1 of x)

doc/rados: edit troubleshooting-osd (1 of x)

Edit doc/rados/troubleshooting/troubleshooting-osd.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0fc663d793fe34e42e19b251fcc79b5af52581c6)

crimson/os/seastore/omap_manager: fix the entry leak issue in
BtreeOMapManager::omap_list()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit cf105932a7fadf67ec494587b078bbfb4caf4d26)

crimson/os/seastore/journal/cbj: remove return_record and read_record

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 39ea5add3a81c32263533ec8f01590efeceb2c86)

crimson/os/seastore/journal/cbj: check if batched records are written at once before rolling

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit ef1a6a90e0131cdf3bb96ace835f845ae23ba0ce)

crimson/os/seastore: genralize read_validate_record_metadata and read_validate_data

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 15ae64c3311378f6055d312c2b5f64b46912d899)

crimson/os/seastore/journal/cbj: check if the first record in CBJ is written after rolling or not

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit a10f2855da3bc1c25f664158af1cc79d70bdef57)

crimson/os/seastore/journal/cbj: introduce magic value to identify that written records are correct

The cbj would replay invalid records if the same write pattern (mkfs -> write -> mkfs -> write) repeats.
On the second mkfs(), current cbj wrongly recognizes the old records are valid beceause they have
correct seq. number at appropriate offset even though the records are expired.

To solve this, this commit introduces random value to identify the journal  during mkfs()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 942ea6487f59e52d88050fb8deb8a2e6c0ca83c3)

crimson/os/seastore: introduce generalized scan_valid_records for RBM

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit a580e516965ff8ae57dbaab202854f9ebe38ba5f)

crimson/os/seastore: introduce RecordScanner to generalize scan_valid_records()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 22c747826b398cc0a0d68aab9bba1fd096a2de9d)

crimson/os/seastore: fix daggling reference of oid in
SeaStore::Shard::stat()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 0ee9bb21be89bdcfb3e6cfb224c256d6eccd9f38)

crimson/os/seastore: retire_extent_addr clean up

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit b99a1956bdcb879a3e5440e617593bae43fafd62)

crimson/ertr: assert on invocability of func provided to safe_then()

The goal is to fail with an easier-to-understand error on failed
`static_assert()` than cryptic template substitution errors when
the function provided to e.g. `safe_then()` doesn't match the type
carried within an errorated future.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 83687f2cc86897d60a460575a7ab2acbd2f9170c)

crimson/os/seastore/object_data_handler: add ObjectDataHandler(extent_len_t)

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit c1fdb2a8a41c6f0f204c980c4f3af7cfac28723d)

crimson/os/seastore/cache: correct log prefix for get_absent_extent()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 5a563d63e754d5163424486dbd0d665165615794)

crimson/os/seastore: add metrics about total invalidated transactions

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 6265452c50c2bbd0283a66d03d6a8538c2285293)

src/test/crimson/seastore/nvmedevice: update a test case based on rbm multi shard change

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 7a15d484c552a177585954cdb4e2e226a43c1035)

crimson/os/seastore/rbm: make rbm support multiple shards

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 1d6beef975144210ae29626e61aeb284aff30d53)

test/crimson/seastore: test if lazy read work correctly

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 9d973efe32c089351ec756232d3699b4c9fe1f64)

test/crimson/seastore: test overwrite pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit efd3cdc71056baabba291d7a359dda6dbd9121b1)

test/crimson/seastore: add test for TransactionManager::remap_pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit b5931d2795cfe44b96c6ee9347514b002197bb5b)

crimson/os/seastore: check if length exist in mark_space_used and mark_space_free

to avoid misuse of buffer length and data length when there is a no actual data extent

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit dc4a23cef3787099dcc17f99b05f4ec1a63eca9a)

crimson/os/seastore: deepcopy the buffer of exist clean extent when duplicate_for_write

Beacuse exist clean extent shares buffer with original clean extent.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 026066a7e5c4cc1bc04ed91003c553146a0b60ef)

crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 5d00ec0d00ddefe5b82ce82d2acfa134d964a883)

crimson/os/seastore/transaction_manager: implement remap_pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 55d33b69b004adc6fe6740872122311ca9372c0c)

crimson/os/seastore/transaction_manager: implement alloc_remapped_extent

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit cf6e46e57afc44b46d7855dcb5849724f6ddfcbb)

crimson/os/seastore: support no data extent in read path

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit f19de7c77f7d6b783802f6c18e2e9fe106c90243)

crimson/os/seastore: let CachedExtent::ptr optional

separate buffer length and data length, since no buffer extent is allowed

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 7cb44d350b68002123c3c2f97a888cc42260f522)

crimson/osd/watch: don't print bufferlists

The bufferlists clutter the logs and don't infrom of anything useful:
```
DEBUG 2023-05-26 23:36:29,780 [shard 2] osd - send_completion sending notify replies: {notify_reply_t{watcher_gid=4220, watcher_cookie=139786421930752, bl=buffer::list(len=0,

)}, notify_reply_t{watcher_gid=4220, watcher_cookie=139786422032096, bl=buffer::list(len=0,

)}}
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 3548489f622c50ccaa1d8c351e420873743ed980)

crimson/osd/watch: Add logs around Watch/Notify

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit b1c05f9acecfd2dffb57d63f42fec9da156ff366)

crimson/osd/heartbeat: relax the order of replacement reset and accept

With the new implementation in messenger, the order of replacement reset
and accept events cannot be determined because they are from different
connections.

Modify the heatbeat logic to tolerate the both cases.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit a8709464232fc5cc28032716f67c754971420e6b)

crimson/net: cleanup, rename is_fixed_cpu

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 2fed9397d5f1a93e01d02b5914931420c4f28c37)

crimson/net: dispatch the initial acceptance from the socket core

It's meaningless to dispatch the initial acceptance always from the
msgr core.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 1e8a39f359d8481bb3038b461063c87384bf3ee8)

crimson/net: keep the order of cross-core events in msgr v2

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit bdd89b68eaf7a64b11aeb8c40ad5060698e03541)

crimson/net: expose is_fixed_cpu configuration at Messenger level

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 6a01988894c11d54d3f5c6eb0d8078da0a8836cc)

crimson/net: misc cleanups with logs around cross-core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 77268679631adaad2408eac62d3d599e2acf6059)

crimson/net: convert all interactions between protocol and io-handler to be cross-core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 0b60209df64804d2a2f4121216cbaf058b38515b)

crimson/net: prevent racing in protocol to switch core and to call io-handler interfaces

Otherwise, calling io-handler interfaces may result in wrong core/order.

This needs to take special care to handle preemptive cases such as
closing and replacing.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 5b8003386d718b62d10cced6659ed2461042e1f5)

crimson/net: implement logic to move a io-handler to a new sid

Note that it is inevitable that the user can mark down the connection
while the protocol is trying to move the connection to another core.

In that case, the implementation should tolerate the racing, which
finally needs to cleanup resources correctly and dispatch reasonable
events.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit b10f0bd3ccc919bfbe4d063bebabe651faf60812)

crimson/net: expose the connection working shard to users

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit b2a456d6b86e0105397f66fb1576acb37d650c56)

crimson/net: notify the new connection shard upon accept/connect

Users may need to know the new connection shard prior to message
dispatching. Otherwise, there will be no chance for user to do any
related preparations.

This is still a placeholder before multi-core messegner is enabled.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 18d2c30e675a7445888570658fdca9aeb6299b2d)

crimson/net: make io-handler in/out dispatching aware of being switched

To prevent the previous shard-states racing on the shared data
structures after switching to a new shard-states.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 77fef01fa8c25c480ec64d9c566a690e119bf8f8)

crimson/net: allow io-handler to dispatch in/out independently based on ctx

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit ca30d02ccb40c70c8a8608af5ef5f339d343cb79)

crimson/net: make send/keepalive keep trying until the shard is correct

It will be possible that during cross-core send/keepalive, the
connection shard is changed on-the-fly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit bea8fb50fdb362a3738fefed43fe7f196e2547cd)

crimson/net: factor out IOHandler::shard_states_t which will be switchable atomically

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 37f1456027abf38516ddf11f9c3d8f210ca0d91f)

crimson/net: misc cleanups to io handler implementations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 8d0f5943aaa91c220011c67fc43a8aa51c1df52b)

crimson/net: misc cleanups to protocol v2 implementations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 6658ba17ef1bcc89b8ff5e74bfc7c7c0b8ef8fac)

crimson/net: notify and update io_states from io_handler to protocol asynchronously

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit d20fb8304085299d8ec434145d77d1d43af5aae3)

crimson/net: introduce a simpler reset_peer_state() with cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 23c13306776229c237395ccad3b6947a328c2519)

crimson/net: clear FrameAssemblerV2 at replace

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 51636d9954cbbc8c34ed5f0a275ccc08d9899cf1)

crimson/net: adjust the IO path in FrameAssemblerV2 with the foreign socket

FrameAssemblerV2 and Socket may in different cores during handshake.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit d1b4323606a7bdf3ab0db6eca444c1c346c11354)

crimson/net: allow FrameAssemblerV2 to check and shutdown a foreign socket

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 50bc62a9da8f18fae1cff86ce8c91f6125a96225)

crimson/net: check that FrameAssemblerV2 is working in the expected core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit c47dc1d0e46add1d9f284d41112e9b772b0b36b9)

crimson/net: use SocketFRef

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 23c7a85084d9934e252c0c94a6bb39bd3534a861)

crimson/net: check that Socket only works in its core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit d5a9f0eda34f08e9eadbdd5d5e2e26147d27f759)

crimson/net: wrap message with foreign-ptr in the send path

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit db07ae9d5cf8e6ac8031eec0689c74a9d5f55aff)

crimson/net: change the static IS_FIXED_CPU to runtime

Otherwise the messenger implementation needs to be templated.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 58491826d54bbbfeeab1543337c8ff327218db74)

crimson/net: decouple tests from SocketConnection class

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 7797cabe2ab38eea30eca51614398badef8bada1)

crimson/net: hide seastar packet and temporary_buffer inside Socket

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 4dab5cf87ae3561fd67d2538b33eb98b6b7d43de)

crimson/net: introduce IOHandler::sid and the related assertions

Note that Connection and Messenger may be destructed in any order, so
keep a copy of msgr_sid in SocketConnection.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 3b47c9a632db4ab04d0a185c598e18b2c1219ee5)