Merge pull request #53762 from cbodley/wip-63045-reef

reef: rgw/keystone: EC2Engine uses reject() for ERR_SIGNATURE_NO_MATCH

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #53896 from cbodley/wip-62744-reef

reef: qa/rgw: add new POOL_APP_NOT_ENABLED failures to log-ignorelist

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #54037 from vshankar/wip-61828-reef

reef: qa: assign file system affinity for replaced MDS

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54035 from vshankar/wip-63174-reef

reef: mds: adjust pre_segments_size for MDLog when trimming segments for st…

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53918 from kotreshhr/wip-63165-reef

reef: pybind/mgr/volumes: log mutex locks to help debug deadlocks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53906 from vshankar/wip-62797-reef

reef: qa: move nfs (mgr/nfs) related tests to fs suite

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Adam King adking@redhat.com

Merge pull request #53827 from rishabh-d-dave/wip-63094-reef

reef: qa/cephfs: fix build failure for mdtest project

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53825 from rishabh-d-dave/wip-63096-reef

reef: qa/cephfs: fix ior project build failure

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53666 from lxbsz/wip-62521

reef: client: move the Inode to new auth mds session when changing auth cap

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53661 from lxbsz/wip-62524

reef: ceph: allow xlock state to be LOCK_PREXLOCK when putting it

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53624 from lxbsz/wip-62515

reef: qa/cephfs: switch to python3 for centos stream 9

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53572 from kotreshhr/wip-62405-reef

reef:    mgr/volumes: Fix pending_subvolume_deletions in volume info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53558 from batrick/wip-62733-reef

reef: mds: add event for batching getattr/lookup

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53553 from batrick/wip-62899-reef

reef: qa: lengthen shutdown timeout for thrashed MDS

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #53548 from batrick/wip-62901-reef

reef: mds: log message when exiting due to asok command

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Christopher Hoffman <choffman@redhat.com>

Merge pull request #53497 from lxbsz/wip-62860

reef: mds: fix deadlock between unlinking and linkmerge

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #54118 from aaSharma14/wip-63272-reef

reef: mgr/dashboard: add port and zone endpoints to import realm token form in rgw multisite

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #54137 from zdover23/wip-doc-2023-10-22-backport-54136-to-reef

reef: doc/rgw: pubsub capabilities reference was removed from docs

Merge pull request #54105 from Matan-B/wip-46062-reef

reef: crimson: Enable tcmalloc when using seastar

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53966 from Matan-B/wip-53305-reef

reef: crimson/os/seastore/onode_manager: populate value recorders of onodes to be erased

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/onode_manager: populate value recorders of onodes to
be erased

Otherwise, the following modification sequence with the same transaction
might lead to onode extents' crc inconsistency during journal replay:

1. modify the last mapping in an onode extent;
2. erase the last mapping in that onode extent.

During journal replay, if the first modification is not recorded in the
delta, the onode extent's content would be inconsistent with that before
the system reboot

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 6c236bb63a9a49c7f9e2dab03a5fe0bb016560f0)

Merge pull request #54098 from Matan-B/wip-53455-reef

reef: crimson/os/seastore/cache: don't add EXIST_CLEAN extents to lru

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54097 from Matan-B/wip-53501-reef

reef: crimson/os/seastore: create page aligned bufferptr in copy ctor of CachedExtent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54096 from Matan-B/wip-53433-reef

reef: crimson/tools/perf_staged_fltree: fix compile error

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54095 from Matan-B/wip-52896-reef

reef: crimson/net: consolidate messenger implementations and enable multi-shard UTs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54093 from Matan-B/wip-53046-reef

reef: crimson/tools/perf_crimson_msgr: randomize client nonce

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54092 from Matan-B/wip-51141-reef

reef: crimson/os/seastore: OP_CLONE in seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54091 from Matan-B/wip-52306-reef

reef: crimson/osd: fixes and cleanups around multi-core OSD

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rgw: pubsub capabilities reference was removed from docs

most pubsub docs were removed in
0fecbe2476afbb06de7ea731b7c92066ae40cfc4
however, the text removed described functionality that is only
supported bia pubsub

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>
(cherry picked from commit c479f2e6caa1f8f736eedb6f0036dbaa4649a204)

Merge pull request #54130 from zdover23/wip-doc-2023-10-21-backport-54107-to-reef

reef: .github: Clarify checklist details

.github: Clarify checklist details

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 1ca356a916910a4016d5d3460d68ea3732f3bad9)

Merge pull request #54114 from zdover23/wip-doc-2023-10-20-backport-54084-to-reef

reef: doc/rados: edit troubleshooting-pg (2 of x)

Merge pull request #53924 from adk3798/wip-63169-reef

reef: mgr/cephadm: fix upgrades with nvmeof

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53851 from batrick/wip-63082-reef

reef: mds/FSMap: allow upgrades if no up mds

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #53484 from batrick/wip-62852-reef

reef: qa: ignore expected cluster warning from damage tests

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #53101 from batrick/wip-62553-reef

reef: libcephsqlite: fill 0s in unread portion of buffer

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #52460 from batrick/wip-62022-reef

reef: pybind/mgr: reopen database handle on blocklist

Reviewed-by: Xiubo Li <xiubli@redhat.com>

mgr/dashboard: add port and zone endpoints to import realm token form in
rgw multisite

Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 84ec19442b2db4f4e389810efbc01674d9824408)

doc/rados: edit troubleshooting-pg (2 of x)

Edit doc/rados/troubleshooting/troubleshooting-pg.rst (1 of x).

Follows https://github.com/ceph/ceph/pull/54052.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit d0cdfc1d72c4edefb060f8ba9ab130189617cab1)

Merge pull request #54111 from zdover23/wip-doc-2023-10-20-backport-54099-to-reef

reef: doc: Documentation about main Ceph metrics

doc: Documentation about main Ceph metrics

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Co-authored-by: Zac Dover <zac.dover@proton.me>
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@ibm.com>
(cherry picked from commit 9bb63bdc8969f2ecdfeedfc8396890ad59f0d796)

Merge pull request #53922 from adk3798/reef-mgr-cephadm-image-base

reef: mgr/cephadm: fix default image base in reef

Reviewed-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #53919 from ajarr/wip-63156-reef

reef: librbd: kick ExclusiveLock state machine stalled waiting for lock from reacquire_lock()

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #53864 from adk3798/wip-61963-reef

reef: Adding rollback mechanism to handle bootstrap failures

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #53861 from ljflores/wip-63125-reef

reef: osd: fix: slow scheduling when item_cost is large

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #53534 from kamoltat/wip-ksirivad-reef-backport-54136

reef:pybind/mgr/pg_autoscaler: Use bytes_used for actual_raw_used

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #53093 from sseshasa/wip-62546-reef

reef: osd/scheduler/mClockScheduler: Use same profile and client ids for all clients to ensure allocated QoS limit consumption.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: Enable tcmalloc when using seastar

classic-osds have always caused significant memory fragmentation
when using the libc memory allocator due to the way that Ceph
tends to utilize memory.  In recent testing, crimson-osd was found
to use 25-27GB of RAM with the stock 3GB bluestore cache settings
(osd_memory_target is only used when tcmalloc is available).  Upon
further testing, it was found that the classic OSD is even worse,
using between 32-33GB of RAM after a 5 minute 4K sequential
write test when using libc malloc.

The good news is that it appears that crimson-osd is able to use
tcmalloc for alienstore without significant modification. Better
still, it drastically reduces memory usage.  In the same test that
resulted in 25GB RSS memory usage for crimson-osd with libc malloc,
a tcmalloc linked version took around 9GB (with an 8GB
osd_memory_target).  Since we do not yet (afaik) expose classic OSD
debugging in crimson it is tough to tell why we are still a little
over, but it's clear that for alienstore we are going to need to
use tcmalloc as we do in classic.

Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit d884a45aea8d5e0a5fa1689f2fd3167d68292893)

Revert "[CHERRY-PICKED FOR TESTING ONLY] crimson: Enable tcmalloc when using seastar"

This reverts commit 380bc6d2e4ec62e724612dbac57ffdbd82511f9d.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 96c6c2681faa616314582ad3e9ccef71ee6d5f69)

Merge pull request #54057 from Matan-B/wip-51875-reef

reef: crimson/osd/lsan_suppressions.cc: Add MallocExtension::Initialize()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #53957 from Matan-B/wip-52529-reef

reef: crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54021 from Matan-B/wip-48052-reef

reef: qa/suites/crimson-rados: bring backfill testing

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it
with ReplicateRequest

This commit is a follow-up for e14142894922b403344d7f08315b67a291b9a27a
and f4d336a2dbf864fe798ca7f500d44576d3f626d5

Fixes: https://tracker.ceph.com/issues/61943
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 0b7c47db78cc22c7cf25c50b430674e17b01a601)

crimson/os/seastore/cache: replace is_clean by is_stable_clean wherever
possible

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit f60f318450fd8118b96e3dbae29cccc68ed1ae19)

crimson/os/seastore/cache: don't add EXIST_CLEAN extents to lru

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 88ca65776bd7248152b55360d4621107ce0caeb9)

crimson/os/seastore: create page aligned bufferptr in copy ctor of CachedExtent

Signed-off-by: Zhang Song <zhangsong02@qianxin.com>
(cherry picked from commit 07577e3bc14423acec391a85ee79ddc6b1f269a1)

crimson/tools/perf_staged_fltree: fix compile error

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 251288410c3ad8c763e386bec92eedb327801a4d)

test/crimson: increase smp settings for test_socket and test_messenger

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit e44ce1a892b721ae4a5a8d9e3525ff2c95cb1448)

test/crimson/test_messenger: add delay to suite markdown

markdown() needs delay to be propagated to the primary core.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit ec8722f2128a92f5ef76f1c0cd14e33ec8a5c295)

test/crimson/test_messenger: enable multi-core messenger in FailoverSuite

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 290dd2d68df8289352438ca9167ed1d18ddcdd9a)

test/crimson/test_messenger: support cross-core protocol-level tests

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit d65ba2d018b14d87252f50baa890350a6e7b38e3)

crimson/net: support cross-core blocking for interceptor

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit bcf20d6771f5adf608fcf44b47073c133c6b0b35)

crimson/net: should not check shutdown from a different core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 74e9098833e578d7514b8ab1f7a4588a052885a4)

crimson/net: rework interceptor to support cross-core interceptions

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 6be5e630ed79e09e0b77c481856f56a6b27ea562)

test/crimson/test_messenger: relax the ordering checks to tracked_conn

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 4515c34977f8c54d459b0140305e24c9bb29a455)

test/crimson/test_messenger: move all connection state checks to the protocol level

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 933745cd815f7cc81438a89b3ba9ac5fc2ef61b7)

crimson/net: don't miss keepalive during message sweeping

The flags need to be updated atomically to prevent missing them during
message sweeping.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 9bf4eeaa923351dc8f53b7cc3a02c2e8483c054e)

test/crimson/test_messenger: drop test_concurrent_dispatch()

The only purpose of test_concurrent_dispatch() is to verify that 2
messages are all received. Revise test_echo() for the same purpose, and
drop test_concurrent_dispatch().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit a4f6c82a9a0897aa1cc5ff02c10b660fcbd6eba0)

test/crimson/test_messenger: implement multi-shard test_echo::test_state::Client

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 9b5182779d76043c428e664f201acca5986aab7c)

test/crimson: add missing calls to ConfigProxy::start()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 9d817d33d564a4240ba576c378046edee54a599c)

crimson/net: adjust dispatcher interface about cross-core notifications

Due to that we aren't able to determine cross-core ordering:
* Move ms_handle_connect/accept() to be called in the new shard, so it
  will notify before ms_dispatch() in the same core;
* Introduce another ms_handle_shard_change() when the current core is
  changed;

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 6bab7e698db8b7f3bec240952ed33a2bb9918a20)

test/crimson/test_messenger: implement multi-shard test_echo::test_state::Server

Also introduces ShardedGates.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit ec6acfc352b1bd57b7e0d87114bbf51f2c0190b9)

crimson/tools/perf_crimson_msgr: randomize client nonce

In case we want to start multiple clients simultaneously.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 91c65e22387aa4b7c572712f4a3028ed6e8d40f9)

crimson/os/seastore: debug outputs

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>
(cherry picked from commit 6a583aad13e5fc26bbfc7374b6c292f0dc77b662)

crimson/os/seastore/object_data_handler: take indirect offset into
account when reading object data

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 9cdcd06caadef6cdca64a974c6525ba300c1e289)

crimson/os/seastore/transaction_manager: don't remove intermediate
mapping when remapping pins

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 4b14549e6ebd6aa63e9006d3c487e139731817fc)

crimson/os/seastore/transaction_manager: remap indirect mappings

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit e2c0ebec76aec12e8af6558fff22657acf64bd5d)

crimson/os/seastore/transaction_manager: remove extents for physical lba mappings if necessary
when removing indirect lba mappings

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>
(cherry picked from commit d6bfd9455daa4ea5f0c86d0a931b808c5c60b3f5)

crimson/os/seastore/lba_manager: decrease refcount of the physical lba
mappings when removing indirect lba mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 31204c2e1cb05ac1476abbcc5d5198f38e326095)

crimson/os/seastore/onode_manager: erase onodes from the OnodeTree at the moment of erase_onode

Signed-off-by: xuxuehan <xuxuehan@qianxin.com>
(cherry picked from commit e5ee75ee85a7c6f5de6823986eeb30fe44349c47)

crimson/os/seastore/object_data_handler: don't reserve lba space when truncating objects to zero size

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 826bb50fb3c4d279fa06a648940aebf78303f82d)

test/crimson/seastore: add OP_CLONE unittest case

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 9993b99c1c86bbfd1248db6f621cde7dc966b5d5)

crimson/os/seastore/lba_manager: take lba entry indirection into account when getting lba mappings

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 48023c7f67600cbbccc49f6499f0df3526e1b882)

crimson/os/seastore: implement OP_CLONE

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 9f303cde7b9d7aca14a2024948079b4280168a2d)

crimson/os/seastore/lba_manager: lba map value may contain laddr

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit f11504cd5d3fbba01cf9dbf11345e2c15001f947)

crimson/osd: cleanup ShardDispatcher

ShardDispatcher logically equals to seastar::sharded<PGShardManager>,
drop unnecessary members.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 4ecc58d0ff372420a780c41bcaa975a25719c207)

crimson/osd: avoid unnecessary foreign copy in ms_dispatch()

Foreign-copy also implies that requests may be out-of-order, which is
incorrect in the data path.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 436706c4612bada876682df967cd40187805e823)

crimson/osd: make sure OSD is constructing in the primary core

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit d6c5f05933e279607576dc2d6968c67248528813)

Merge pull request #54020 from Matan-B/wip-52784-reef

reef: qa/suites/crimson-rados: add centos9 to supported distros

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54019 from Matan-B/wip-52598-reef

reef: qa/suites/crimson-rados: Use centos8 for testing

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54063 from Matan-B/wip-52592-reef

reef: crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54062 from Matan-B/wip-52420-reef

reef: crimson/os/seastore/zbd: zbdsegmentmanager write path fixes.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54061 from Matan-B/wip-52390-reef

reef: crimson/os/object_data_handler: splitting right side doesn't mean splitting only one extent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54060 from Matan-B/wip-52223-reef

reef: crimson/osd/osd_operations: consistent naming to pipeline users

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54059 from Matan-B/wip-52091-reef

reef: crimson/tools/perf_crimson_msgr: integrate multi-core msgr with various improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54058 from Matan-B/wip-51147-reef

reef: Crimson/osd: support multicore osd

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #54069 from rhcs-dashboard/wip-63231-reef

reef: mgr/dashboard: disable hosts field while editing the filesystem

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>

Merge pull request #53921 from ljflores/wip-63168-reef

reef: qa/suites/upgrade: fix env indentation in stress-split upgrade tests

Merge pull request #54079 from zdover23/wip-doc-2023-10-19-backport-54051-to-reef

reef: doc: Update mClock QOS documentation to discard osd_mclock_cost_per_*

doc: Update mClock QOS documentation to discard osd_mclock_cost_per_*

The cost parameters (osd_mclock_cost_per_*) have been removed.
The cost of an operation is now determined using the random IOPS
and maximum sequential bandwidth capability of the OSD's underlying device.

Fixes: https://tracker.ceph.com/issues/58529
Signed-off-by: tanchangzhi <544463199@qq.com>
(cherry picked from commit a7502cb5125dda8d052dc3991ed5d7013606aca2)