]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
21 months agoMerge pull request #53957 from Matan-B/wip-52529-reef
Matan Breizman [Thu, 19 Oct 2023 12:28:26 +0000 (15:28 +0300)]
Merge pull request #53957 from Matan-B/wip-52529-reef

reef: crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54021 from Matan-B/wip-48052-reef
Matan Breizman [Thu, 19 Oct 2023 08:34:38 +0000 (11:34 +0300)]
Merge pull request #54021 from Matan-B/wip-48052-reef

reef: qa/suites/crimson-rados: bring backfill testing

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agocrimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it 53957/head
Xuehan Xu [Wed, 19 Jul 2023 07:52:22 +0000 (15:52 +0800)]
crimson/osd/osd_operations: add pipeline to LogMissingRequest to sync it
with ReplicateRequest

This commit is a follow-up for e14142894922b403344d7f08315b67a291b9a27a
and f4d336a2dbf864fe798ca7f500d44576d3f626d5

Fixes: https://tracker.ceph.com/issues/61943
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 0b7c47db78cc22c7cf25c50b430674e17b01a601)

21 months agoMerge pull request #54020 from Matan-B/wip-52784-reef
Matan Breizman [Thu, 19 Oct 2023 07:11:10 +0000 (10:11 +0300)]
Merge pull request #54020 from Matan-B/wip-52784-reef

reef: qa/suites/crimson-rados: add centos9 to supported distros

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54019 from Matan-B/wip-52598-reef
Matan Breizman [Thu, 19 Oct 2023 07:10:39 +0000 (10:10 +0300)]
Merge pull request #54019 from Matan-B/wip-52598-reef

reef: qa/suites/crimson-rados: Use centos8 for testing

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54063 from Matan-B/wip-52592-reef
Matan Breizman [Thu, 19 Oct 2023 07:09:26 +0000 (10:09 +0300)]
Merge pull request #54063 from Matan-B/wip-52592-reef

reef: crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54062 from Matan-B/wip-52420-reef
Matan Breizman [Thu, 19 Oct 2023 07:07:46 +0000 (10:07 +0300)]
Merge pull request #54062 from Matan-B/wip-52420-reef

reef: crimson/os/seastore/zbd: zbdsegmentmanager write path fixes.

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54061 from Matan-B/wip-52390-reef
Matan Breizman [Thu, 19 Oct 2023 07:07:02 +0000 (10:07 +0300)]
Merge pull request #54061 from Matan-B/wip-52390-reef

reef: crimson/os/object_data_handler: splitting right side doesn't mean splitting only one extent

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54060 from Matan-B/wip-52223-reef
Matan Breizman [Thu, 19 Oct 2023 07:06:34 +0000 (10:06 +0300)]
Merge pull request #54060 from Matan-B/wip-52223-reef

reef: crimson/osd/osd_operations: consistent naming to pipeline users

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54059 from Matan-B/wip-52091-reef
Matan Breizman [Thu, 19 Oct 2023 07:05:40 +0000 (10:05 +0300)]
Merge pull request #54059 from Matan-B/wip-52091-reef

reef: crimson/tools/perf_crimson_msgr: integrate multi-core msgr with various improvements

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54058 from Matan-B/wip-51147-reef
Matan Breizman [Thu, 19 Oct 2023 07:03:09 +0000 (10:03 +0300)]
Merge pull request #54058 from Matan-B/wip-51147-reef

reef: Crimson/osd: support multicore osd

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #54069 from rhcs-dashboard/wip-63231-reef
Nizamudeen A [Thu, 19 Oct 2023 04:45:24 +0000 (10:15 +0530)]
Merge pull request #54069 from rhcs-dashboard/wip-63231-reef

reef: mgr/dashboard: disable hosts field while editing the filesystem

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ankush Behl <cloudbehl@gmail.com>
Reviewed-by: ivoalmeida <NOT@FOUND>
21 months agoMerge pull request #53921 from ljflores/wip-63168-reef
Zack Cerza [Wed, 18 Oct 2023 20:57:35 +0000 (14:57 -0600)]
Merge pull request #53921 from ljflores/wip-63168-reef

reef: qa/suites/upgrade: fix env indentation in stress-split upgrade tests

21 months agoMerge pull request #54079 from zdover23/wip-doc-2023-10-19-backport-54051-to-reef
Anthony D'Atri [Wed, 18 Oct 2023 20:04:55 +0000 (16:04 -0400)]
Merge pull request #54079 from zdover23/wip-doc-2023-10-19-backport-54051-to-reef

reef: doc: Update mClock QOS documentation to discard osd_mclock_cost_per_*

21 months agodoc: Update mClock QOS documentation to discard osd_mclock_cost_per_* 54079/head
tanchangzhi [Tue, 17 Oct 2023 08:48:51 +0000 (16:48 +0800)]
doc: Update mClock QOS documentation to discard osd_mclock_cost_per_*

The cost parameters (osd_mclock_cost_per_*) have been removed.
The cost of an operation is now determined using the random IOPS
and maximum sequential bandwidth capability of the OSD's underlying device.

Fixes: https://tracker.ceph.com/issues/58529
Signed-off-by: tanchangzhi <544463199@qq.com>
(cherry picked from commit a7502cb5125dda8d052dc3991ed5d7013606aca2)

21 months agoMerge pull request #53654 from mcv21/wip-62970-reef
Yuri Weinstein [Wed, 18 Oct 2023 18:50:17 +0000 (11:50 -0700)]
Merge pull request #53654 from mcv21/wip-62970-reef

reef: Debian: update to dh compat 12, fix more serious packaging errors, correct copyright syntax

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
21 months agoMerge pull request #53823 from cfsnyder/wip-62944-reef
Casey Bodley [Wed, 18 Oct 2023 17:52:36 +0000 (18:52 +0100)]
Merge pull request #53823 from cfsnyder/wip-62944-reef

reef: rgw: add radosgw-admin bucket check olh/unlinked commands

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #54071 from zdover23/wip-doc-2023-10-18-backport-54028-to-reef
Anthony D'Atri [Wed, 18 Oct 2023 16:49:42 +0000 (12:49 -0400)]
Merge pull request #54071 from zdover23/wip-doc-2023-10-18-backport-54028-to-reef

reef: doc/start: Modernize and clarify hardware-recommendations.rst

21 months agoMerge pull request #54073 from zdover23/wip-doc-2023-10-18-backport-54052-to-reef
Anthony D'Atri [Wed, 18 Oct 2023 16:49:00 +0000 (12:49 -0400)]
Merge pull request #54073 from zdover23/wip-doc-2023-10-18-backport-54052-to-reef

reef: doc/rados: edit troubleshooting-pg.rst (1 of x)

21 months agoMerge pull request #53229 from ifed01/wip-ifed-snapdiff-ref
Yuri Weinstein [Wed, 18 Oct 2023 16:22:31 +0000 (09:22 -0700)]
Merge pull request #53229 from ifed01/wip-ifed-snapdiff-ref

reef: cephfs: implement snapdiff

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
21 months agoMerge pull request #52850 from lxbsz/wip-61165
Yuri Weinstein [Wed, 18 Oct 2023 16:21:22 +0000 (09:21 -0700)]
Merge pull request #52850 from lxbsz/wip-61165

reef: client: issue a cap release immediately if no cap exists

Reviewed-by: Venky Shankar <vshankar@redhat.com>
21 months agoMerge pull request #52847 from lxbsz/wip-62192
Yuri Weinstein [Wed, 18 Oct 2023 16:20:35 +0000 (09:20 -0700)]
Merge pull request #52847 from lxbsz/wip-62192

reef: mds: do not send split_realms for CEPH_SNAP_OP_UPDATE msg

Reviewed-by: Venky Shankar <vshankar@redhat.com>
21 months agorgw: fix output formatting of bucket index check admin api 53823/head
Cory Snyder [Mon, 25 Sep 2023 10:06:41 +0000 (10:06 +0000)]
rgw: fix output formatting of bucket index check admin api

The bucket index check admin API was previously returning invalid
JSON.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 32fb6a1a68398a99324b2e64ebe3bcf3a9ccf02a)

Conflicts:
src/rgw/driver/rados/rgw_bucket.cc

Cherry-pick notes:
- Conflicts due to check_bad_index_multipart taking optional_yield param on main

21 months agorgw: fix radosgw-admin bucket check stat calculation bug
Cory Snyder [Fri, 22 Sep 2023 21:08:25 +0000 (21:08 +0000)]
rgw: fix radosgw-admin bucket check stat calculation bug

Fixes a regression with radosgw-admin bucket check stat
calculation and bucket reshard stat calculation when
there are objects that have transitioned from unversioned
to versioned. The bug was introduced in
152aadb71b61c53a4832a1c8cf82fce3d64b68d1.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 4728daa5557bfb79a608dd903b8630e2b15fcb2c)

21 months agorgw: add test case to reproduce bucket check stats bug for versioned bucket
Cory Snyder [Fri, 22 Sep 2023 21:00:46 +0000 (21:00 +0000)]
rgw: add test case to reproduce bucket check stats bug for versioned bucket

Reproduces a regression where radosgw-admin bucket check incorrectly counts
objects that started as unversioned and later transitioned to versioned.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 340522f9aed50d65137568c1f9dcf4b1e7945a79)

21 months agorgw: radosgw-admin bucket check should only print index entries with --check-objects...
Cory Snyder [Fri, 22 Sep 2023 08:35:16 +0000 (08:35 +0000)]
rgw: radosgw-admin bucket check should only print index entries with --check-objects flag

Printing all index entries can be very time consuming for large
buckets and the inability to switch this behavior off makes it
cumbersome to use the command for fixing bucket stats. This was
also preventing the command from outputting recalculated bucket
stats when the --fix flag wasn't specified.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 6b057fe55413c0eaf9959f006584cba6cc4c192a)

21 months agorgw: prevent another leftover bucket index olh entry scenario
Cory Snyder [Thu, 21 Sep 2023 19:27:51 +0000 (19:27 +0000)]
rgw: prevent another leftover bucket index olh entry scenario

If a call to bucket_index_link_olh or bucket_index_unlink_instance
fails, its associated pending xattr may have prevented the olh object
from being removed by another thread. We should do a best effort
cleanup attempt for this case by calling update_olh before returning
an error to the caller.

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 570adec5bb8142f5baf1f05f0040e8afdb11ec05)

21 months agorgw: fix rgw versioned bucket stat accounting during reshard and check index
Cory Snyder [Thu, 7 Sep 2023 17:23:14 +0000 (17:23 +0000)]
rgw: fix rgw versioned bucket stat accounting during reshard and check index

Fixes: https://tracker.ceph.com/issues/62760
Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 152aadb71b61c53a4832a1c8cf82fce3d64b68d1)

21 months agoqa/workunits/rgw: add tests that reproduce bucket stats inconsistency bugs
Cory Snyder [Thu, 7 Sep 2023 14:43:23 +0000 (14:43 +0000)]
qa/workunits/rgw: add tests that reproduce bucket stats inconsistency bugs

Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit b79dcf640ac2cc3dacf1b87bbe351db823c445d0)

21 months agorgw: add radosgw-admin bucket check olh/unlinked commands
Cory Snyder [Thu, 25 May 2023 16:59:04 +0000 (16:59 +0000)]
rgw: add radosgw-admin bucket check olh/unlinked commands

Adds commands to radosgw-admin for checking for and fixing leftover
entries in the bucket index (and associated RADOS objects).

Fixes: https://tracker.ceph.com/issues/62075
Signed-off-by: Cory Snyder <csnyder@1111systems.com>
(cherry picked from commit 9b2042a0750109f575b02c9c4b127dd941994662)

Conflicts:
src/rgw/driver/rados/rgw_rados.cc

Cherry-pick notes:
- Conflicts due to multiple methods takeing optional_yield param on main

21 months agoMerge pull request #53329 from vshankar/tr-62583
Yuri Weinstein [Wed, 18 Oct 2023 14:55:26 +0000 (07:55 -0700)]
Merge pull request #53329 from vshankar/tr-62583

reef: mds: blocklist clients with "bloated" session metadata

Reviewed-by: Xiubo Li <xiubli@redhat.com>
21 months agoMerge pull request #53271 from mchangir/wip-59408-reef
Yuri Weinstein [Wed, 18 Oct 2023 14:54:48 +0000 (07:54 -0700)]
Merge pull request #53271 from mchangir/wip-59408-reef

reef: cephfs_mirror: correctly set top level dir permissions

Reviewed-by: Xiubo Li <xiubli@redhat.com>
21 months agoMerge pull request #53241 from batrick/wip-62661-reef
Yuri Weinstein [Wed, 18 Oct 2023 14:54:19 +0000 (07:54 -0700)]
Merge pull request #53241 from batrick/wip-62661-reef

reef: mds: drop locks and retry when lock set changes

Reviewed-by: Xiubo Li <xiubli@redhat.com>
21 months agoMerge pull request #53187 from batrick/wip-62609-reef
Yuri Weinstein [Wed, 18 Oct 2023 14:52:36 +0000 (07:52 -0700)]
Merge pull request #53187 from batrick/wip-62609-reef

reef: mgr: register OSDs in ms_handle_accept

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
21 months agoMerge pull request #53183 from batrick/wip-62518-reef
Yuri Weinstein [Wed, 18 Oct 2023 14:51:41 +0000 (07:51 -0700)]
Merge pull request #53183 from batrick/wip-62518-reef

reef: mds: acquire inode snaplock in open

Reviewed-by: Xiubo Li <xiubli@redhat.com>
21 months agoMerge pull request #53138 from mihalicyn/wip-62570-reef
Yuri Weinstein [Wed, 18 Oct 2023 14:51:19 +0000 (07:51 -0700)]
Merge pull request #53138 from mihalicyn/wip-62570-reef

reef: ceph_fs.h: add separate owner_{u,g}id fields

Reviewed-by: Xiubo Li <xiubli@redhat.com>
21 months agoMerge pull request #52975 from batrick/wip-62422-reef
Yuri Weinstein [Wed, 18 Oct 2023 14:50:01 +0000 (07:50 -0700)]
Merge pull request #52975 from batrick/wip-62422-reef

reef: mds: dump locks when printing mutation ops

Reviewed-by: Xiubo Li <xiubli@redhat.com>
21 months agodoc/rados: edit troubleshooting-pg.rst (1 of x) 54073/head
Zac Dover [Tue, 17 Oct 2023 09:13:01 +0000 (19:13 +1000)]
doc/rados: edit troubleshooting-pg.rst (1 of x)

Edit doc/rados/troubleshooting/troubleshooting-pg.rst (1 of x).

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit fbde0a441ee753c06afe3723584d6c9686ceef21)

21 months agodoc/start: Modernize and clarify hardware-recommendations.rst 54071/head
Anthony D'Atri [Mon, 16 Oct 2023 01:27:44 +0000 (21:27 -0400)]
doc/start: Modernize and clarify hardware-recommendations.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
(cherry picked from commit 83bd3a8dfb83c46a2ba84d9f309a6dbe72c64ea9)

21 months agoMerge pull request #53954 from Matan-B/wip-52266-reef
Matan Breizman [Wed, 18 Oct 2023 07:04:47 +0000 (10:04 +0300)]
Merge pull request #53954 from Matan-B/wip-52266-reef

reef: crimson/mgr: Fix config show command

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agomgr/dashboard: disable hosts field while editing the filesystem 54069/head
Nizamudeen A [Fri, 13 Oct 2023 07:47:05 +0000 (13:17 +0530)]
mgr/dashboard: disable hosts field while editing the filesystem

Even though the Placement field was disabled, the Host field was still
showing up in UI while Editing. That option is not possible in fs form.

Fixes: https://tracker.ceph.com/issues/63193
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit b18a1441bebdb5c997ed3970d0a53dd39cebd277)

21 months agocrimson/net: set TCP_NODELAY according to ms_tcp_nodelay 54063/head
Xuehan Xu [Sun, 23 Jul 2023 09:33:16 +0000 (17:33 +0800)]
crimson/net: set TCP_NODELAY according to ms_tcp_nodelay

There are cases in which we need low latency of an individual op, like
updating RBD image's object map which would block other ops for a
object not created yet. We need to make sure the rbd-object-map-update
like ops have as low latency as possible

Fixes: https://tracker.ceph.com/issues/62098
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit e21cfa770570f7e7410b830b11edee17e80dead2)

21 months agocrimson/os/seastore/zbd: Split iovs in case of bigger buffer sizes. 54062/head
Aravind Ramesh [Tue, 11 Jul 2023 04:09:36 +0000 (06:09 +0200)]
crimson/os/seastore/zbd: Split iovs in case of bigger buffer sizes.

In Seastore's ZBDSegmentManager, during long running write workloads
if write buffer size exceeds max buf size(IOV_MAX, 16MB),
it was resulting in an assert:
ceph/src/include/buffer.h:1189 : In function 'void ceph::buffer::v15_2_0::list::prepare_iov(VectorT*) const [with VectorT = std::vector<iovec>]', ceph_assert(%s)

This case is handled in BlockSegmentManager by splitting the
IO vectors to appropriate sizes and assert does not happen.

Updated the ZBDSegmentManager to similarly split IO vectors to
appropriate size.

Signed-off-by: Aravind Ramesh <Aravind.Ramesh@wdc.com>
(cherry picked from commit c15c0de4886a839ffebfd22d8204e7d1ca8a65b2)

21 months agocrimson/os/seastore/zbd: increase padding byte buffer size.
Aravind Ramesh [Wed, 5 Jul 2023 16:58:24 +0000 (18:58 +0200)]
crimson/os/seastore/zbd: increase padding byte buffer size.

Before closing a segment, seastore writes a tail information
at the end of the segment, since ZBD zones are serially write
required, if a zone is not fully written to the end, it writes
padding bytes till the tail section.
Increase the size of write chunks to reduce the number of writes.

Signed-off-by: Aravind Ramesh <aravind.ramesh@wdc.com>
(cherry picked from commit 181e6394014dcdca3d1d0cea5f35e92545cb84b9)

21 months agocrimson/os/object_data_handler: splitting right side doesn't mean 54061/head
Xuehan Xu [Tue, 11 Jul 2023 08:22:59 +0000 (16:22 +0800)]
crimson/os/object_data_handler: splitting right side doesn't mean
splitting only one extent

Fixes: https://tracker.ceph.com/issues/61952
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>
(cherry picked from commit 735357bcf5c1b765da661e4934d53d78661795a6)

21 months agocrimson/osd/osd_operations/pg_advance_map: peering_request_pg_pipeline getter 54060/head
Matan Breizman [Tue, 27 Jun 2023 07:05:28 +0000 (07:05 +0000)]
crimson/osd/osd_operations/pg_advance_map: peering_request_pg_pipeline getter

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 9f8159d9e017f5e36db0eb440284bc742a1fd5a5)

21 months agocrimson/osd/osd_operations: consistent naming to pipeline users
Matan Breizman [Wed, 28 Jun 2023 09:07:26 +0000 (09:07 +0000)]
crimson/osd/osd_operations: consistent naming to pipeline users

Rename pp/bp to client_pp or peering_pp.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 9f3dc8db7195b6a42499b46c7dbea872d30788b9)

21 months agocrimson/tools/perf_crimson_msgr: client don't skip core 0 if specified explicitly 54059/head
Yingxin Cheng [Mon, 3 Jul 2023 08:19:51 +0000 (16:19 +0800)]
crimson/tools/perf_crimson_msgr: client don't skip core 0 if specified explicitly

Usually, we skip core 0 to avoid unwanted performance impacts, but that
may not be always the case. Add the configuration not to skip core 0.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit a2df1d0942b4f756a78de4608a464948415c92a1)

21 months agocrimson/tools/perf_crimson_msgr: don't create client/server if not necessary
Yingxin Cheng [Mon, 26 Jun 2023 08:49:36 +0000 (16:49 +0800)]
crimson/tools/perf_crimson_msgr: don't create client/server if not necessary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit a07a02d0bc5e65bf9bd38aba3536c394450340aa)

21 months agocrimson/tools/perf_crimson_msgr: report server-side conns/iops/throughput
Yingxin Cheng [Fri, 16 Jun 2023 07:34:04 +0000 (15:34 +0800)]
crimson/tools/perf_crimson_msgr: report server-side conns/iops/throughput

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 5c14be46c8d9f2920c6e2e22dee82070e9a7f092)

21 months agocrimson/tools/perf_crimson_msgr: use lowres clock in the hot path
Yingxin Cheng [Thu, 15 Jun 2023 02:49:46 +0000 (10:49 +0800)]
crimson/tools/perf_crimson_msgr: use lowres clock in the hot path

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 237e92d3dd076e65b95917e348b55126da6f0bcc)

21 months agocrimson/tools/perf_crimson_msgr: integrate multi-core messenger as server
Yingxin Cheng [Wed, 31 May 2023 07:38:15 +0000 (15:38 +0800)]
crimson/tools/perf_crimson_msgr: integrate multi-core messenger as server

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit b04daeadf5671911f83ccbbea7bf791ced5f7a51)

21 months agocrimson/tools/perf_crimson_msgr: configure clients and client connections
Yingxin Cheng [Wed, 31 May 2023 07:31:59 +0000 (15:31 +0800)]
crimson/tools/perf_crimson_msgr: configure clients and client connections

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 9aab9143aeee94363e1bbaab311bc93ffdf45856)

21 months agocrimson/tools/perf_crimson_msgr: change depth to depth-per-job
Yingxin Cheng [Wed, 31 May 2023 07:07:49 +0000 (15:07 +0800)]
crimson/tools/perf_crimson_msgr: change depth to depth-per-job

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 938b82ab4f4abdbdc57f01086edc1d1288ac15fd)

21 months agocrimson/tools/perf_crimson_msgr: add server multi core option
Yingxin Cheng [Thu, 25 May 2023 06:18:58 +0000 (14:18 +0800)]
crimson/tools/perf_crimson_msgr: add server multi core option

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 0d170bd9df853180908de79258f4ad0eeb0bbc6e)

21 months agocrimson/tools/perf_crimson_msgr: decrease sampling rate
Yingxin Cheng [Thu, 25 May 2023 06:17:38 +0000 (14:17 +0800)]
crimson/tools/perf_crimson_msgr: decrease sampling rate

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 1d4e9678feeb6e02eb5dd8bf7bf715dd1f770884)

21 months agocrimson/tools/perf_crimson_msgr: support stop signal in server mode
Yingxin Cheng [Thu, 25 May 2023 06:16:16 +0000 (14:16 +0800)]
crimson/tools/perf_crimson_msgr: support stop signal in server mode

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 6d6043fa70c60e0e0d37c12fe3b05439246155f3)

21 months agocrimson/tools/perf_crimson_msgr: collect server reactor utilizations without client
Yingxin Cheng [Thu, 25 May 2023 06:12:41 +0000 (14:12 +0800)]
crimson/tools/perf_crimson_msgr: collect server reactor utilizations without client

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 430bcaa5e52e68feb0794fad668cb471eec60ccf)

21 months agocrimson/tools/perf_crimson_msgr: collect server reactor utilizations with client
Yingxin Cheng [Thu, 25 May 2023 06:09:26 +0000 (14:09 +0800)]
crimson/tools/perf_crimson_msgr: collect server reactor utilizations with client

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 291cc302a92d215b3a20d3a4efd8e4df0e65b6a5)

21 months agocrimson/tools/perf_crimson_msgr: report client reactor utilizations
Yingxin Cheng [Thu, 25 May 2023 05:58:46 +0000 (13:58 +0800)]
crimson/tools/perf_crimson_msgr: report client reactor utilizations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 50a80470afb0d63c623ae0f52ebfa803dcd37451)

21 months agocrimson/tools/perf_crimson_msgr: report aggregated client summary
Yingxin Cheng [Thu, 25 May 2023 05:48:12 +0000 (13:48 +0800)]
crimson/tools/perf_crimson_msgr: report aggregated client summary

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit c3f08a757fca12d89e5825b014589f13a8e85a2d)

21 months agocrimson/tools/perf_crimson_msgr: rearrange client cores
Yingxin Cheng [Thu, 25 May 2023 05:42:26 +0000 (13:42 +0800)]
crimson/tools/perf_crimson_msgr: rearrange client cores

Move client cores to the largest smp shards, and reserve core-0 to
reduce performance impacts.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 18bc828f3ec9be843e48eae2630ebe9be3c924a0)

21 months agocrimson/tools/perf_crimson_msgr: misc cleanups
Yingxin Cheng [Thu, 25 May 2023 04:11:47 +0000 (12:11 +0800)]
crimson/tools/perf_crimson_msgr: misc cleanups

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 5371e9bcef10059c1887d7ef3dace25bf0c32552)

21 months agocrimson/tools/perf_async_msgr: print configurations
Yingxin Cheng [Thu, 25 May 2023 03:53:43 +0000 (11:53 +0800)]
crimson/tools/perf_async_msgr: print configurations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 45586840ea56c32e6d4b7c55305b6500fae72415)

21 months agocrimson/tools/perf_async_msgr: support configure ms_async_op_threads
Yingxin Cheng [Thu, 25 May 2023 03:52:05 +0000 (11:52 +0800)]
crimson/tools/perf_async_msgr: support configure ms_async_op_threads

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 8580dcbf4ed33468778bcad1b71308a6e8d26c88)

21 months agocrimson/osd: modify the interface to integrate with milticore-msg 54058/head
chunmei [Fri, 23 Jun 2023 04:35:38 +0000 (04:35 +0000)]
crimson/osd: modify the interface to integrate with milticore-msg

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 00a5eaea4661e6fb23d174b2f367222541b24f21)

21 months agocrimson/osd: remove seastar::now() to gurantee sequence in pipline
chunmei [Fri, 23 Jun 2023 04:25:53 +0000 (04:25 +0000)]
crimson/osd: remove seastar::now() to gurantee sequence in pipline

in debug build, if wait_barrier is seastar::now, will cause reactor
schedule the following continuation away, but can't schedule them back
in the same order as by which they are scheduled away, so the continuations
are out of order here.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit c7628de4d230cd08d536d7e0c35bf6822cc7146e)

21 months agocrimson/osd: make OSDState and OSDMap_Gate sharded
chunmei [Fri, 23 Jun 2023 03:35:09 +0000 (03:35 +0000)]
crimson/osd: make OSDState and OSDMap_Gate sharded

put them into PerShardState to avoid cross core calling for normal cases.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 324191130ab77a06777f3a71b4c6bf995a4e5fe7)

21 months agocrimson/osd: change OSD::ms_dispatch to take an ConnectionFRef
chunmei [Sat, 8 Apr 2023 08:58:15 +0000 (08:58 +0000)]
crimson/osd: change OSD::ms_dispatch to take an ConnectionFRef

Shortly, we will need to be able to pass a ConnectionRef from another core.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit d657afc7a76d14882152d6f38da5adeaa39e55e1)

21 months agocrimson/osd: relax requirement that PGShardManager methods be invoked on PRIMARY_CORE
chunmei [Fri, 31 Mar 2023 04:11:27 +0000 (04:11 +0000)]
crimson/osd: relax requirement that PGShardManager methods be invoked on PRIMARY_CORE

Proxy uses of osd_singleton_state to PRIMARY_CORE as needed.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 6bb4f00a9d006f99b21104bf46b78dff269a116f)

21 months agocrimson/osd: enable PGShardMapping access from all cores
chunmei [Fri, 31 Mar 2023 02:49:33 +0000 (02:49 +0000)]
crimson/osd: enable PGShardMapping access from all cores

Previously, all accesses (incuding loookups) had to occur on core 0.  Now that
we want to be able to dispatch from all cores, we need PGShardManager to be
accessible from all cores.  To that end, we now proxy updates to core 0, but maintain
local copies of the map so that cores can perform local lookups.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit a3052969bfcfd9329b2baf7883765642d09ff038)

21 months agocrimson/osd: make osd sharded
chunmei [Fri, 23 Jun 2023 03:18:15 +0000 (03:18 +0000)]
crimson/osd: make osd sharded

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 5f82fbbc72f840fa8642015a120485648aaf507c)

21 months agoMerge pull request #53956 from Matan-B/wip-52221-reef
Matan Breizman [Tue, 17 Oct 2023 16:00:33 +0000 (19:00 +0300)]
Merge pull request #53956 from Matan-B/wip-52221-reef

reef: crimson/os/seastore/collection_manager: fill CollectionNode::decoded on clean reads

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53942 from Matan-B/wip-51034-reef
Matan Breizman [Tue, 17 Oct 2023 15:59:03 +0000 (18:59 +0300)]
Merge pull request #53942 from Matan-B/wip-51034-reef

reef: crimson/osd: fix ENOENT on accessing RadosGW user's index of buckets

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53168 from leonid-s-usov/bp/cap-throttle-event/reef
Leonid S. Usov [Tue, 17 Oct 2023 13:44:00 +0000 (16:44 +0300)]
Merge pull request #53168 from leonid-s-usov/bp/cap-throttle-event/reef

reef: mds/Server: mark a cap acquisition throttle event in the request

21 months agoMerge pull request #52899 from leonid-s-usov/backport/bulk-data-pool/reef
Leonid S. Usov [Tue, 17 Oct 2023 13:43:42 +0000 (16:43 +0300)]
Merge pull request #52899 from leonid-s-usov/backport/bulk-data-pool/reef

reef: Consider setting "bulk" autoscale pool flag when automatically creating a data pool for CephFS

21 months agoMerge pull request #53969 from Matan-B/wip-53432-reef
Matan Breizman [Tue, 17 Oct 2023 13:10:42 +0000 (16:10 +0300)]
Merge pull request #53969 from Matan-B/wip-53432-reef

reef: crimson/vstart: default seastore_device_size will be out of space f…

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53968 from Matan-B/wip-53230-reef
Matan Breizman [Tue, 17 Oct 2023 13:10:06 +0000 (16:10 +0300)]
Merge pull request #53968 from Matan-B/wip-53230-reef

reef: crimson/os/seastore/cbj: fix a potential overflow bug on segment_seq

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53967 from Matan-B/wip-53232-reef
Matan Breizman [Tue, 17 Oct 2023 13:09:49 +0000 (16:09 +0300)]
Merge pull request #53967 from Matan-B/wip-53232-reef

reef: test/crimson/seastore/rbm: add sub-tests regarding RBM to the existing tests

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53965 from Matan-B/wip-52204-reef
Matan Breizman [Tue, 17 Oct 2023 13:09:16 +0000 (16:09 +0300)]
Merge pull request #53965 from Matan-B/wip-52204-reef

reef: crimson/osd/object_context: consider clones found as long as they're in SnapSet::clones

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53964 from Matan-B/wip-53082-reef
Matan Breizman [Tue, 17 Oct 2023 13:08:52 +0000 (16:08 +0300)]
Merge pull request #53964 from Matan-B/wip-53082-reef

reef: crimson/admin/admin_socket: remove path file if it exists

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53963 from Matan-B/wip-53031-reef
Matan Breizman [Tue, 17 Oct 2023 13:08:34 +0000 (16:08 +0300)]
Merge pull request #53963 from Matan-B/wip-53031-reef

reef: crimson/osd/pg: check if backfill_state exists when judging objects'

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53962 from Matan-B/wip-53051-reef
Matan Breizman [Tue, 17 Oct 2023 13:08:10 +0000 (16:08 +0300)]
Merge pull request #53962 from Matan-B/wip-53051-reef

reef: crimson/os/seastore/omap_manager: fix the entry leak issue in BtreeOMapManager::omap_list()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53961 from Matan-B/wip-52128-reef
Matan Breizman [Tue, 17 Oct 2023 13:07:43 +0000 (16:07 +0300)]
Merge pull request #53961 from Matan-B/wip-52128-reef

reef: crimson/os/seastore/journal/cbj: generalize scan_valid_records()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53960 from Matan-B/wip-53023-reef
Matan Breizman [Tue, 17 Oct 2023 13:07:14 +0000 (16:07 +0300)]
Merge pull request #53960 from Matan-B/wip-53023-reef

reef: crimson/os/seastore: fix daggling reference of oid in SeaStore::Shard::stat()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53959 from Matan-B/wip-52757-reef
Matan Breizman [Tue, 17 Oct 2023 13:06:51 +0000 (16:06 +0300)]
Merge pull request #53959 from Matan-B/wip-52757-reef

reef: crimson/os/seastore: retire_extent_addr clean up

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53958 from Matan-B/wip-52655-reef
Matan Breizman [Tue, 17 Oct 2023 13:05:48 +0000 (16:05 +0300)]
Merge pull request #53958 from Matan-B/wip-52655-reef

reef: crimson/ertr: assert on invocability of func provided to safe_then()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53955 from Matan-B/wip-52446-reef
Matan Breizman [Tue, 17 Oct 2023 13:05:01 +0000 (16:05 +0300)]
Merge pull request #53955 from Matan-B/wip-52446-reef

reef: crimson/os/seastore/transaction_manager: data loss issues

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53951 from Matan-B/wip-51179-reef
Matan Breizman [Tue, 17 Oct 2023 13:04:31 +0000 (16:04 +0300)]
Merge pull request #53951 from Matan-B/wip-51179-reef

reef: crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53949 from Matan-B/wip-51916-reef
Matan Breizman [Tue, 17 Oct 2023 13:04:05 +0000 (16:04 +0300)]
Merge pull request #53949 from Matan-B/wip-51916-reef

reef: crimson/net: support connections in multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53947 from Matan-B/wip-52217-reef
Matan Breizman [Tue, 17 Oct 2023 12:46:25 +0000 (15:46 +0300)]
Merge pull request #53947 from Matan-B/wip-52217-reef

reef: crimson/os/seastore/omap_manager: correct editor settings

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53946 from Matan-B/wip-52090-reef
Matan Breizman [Tue, 17 Oct 2023 12:46:08 +0000 (15:46 +0300)]
Merge pull request #53946 from Matan-B/wip-52090-reef

reef: crimson/os/seastore/btree: should add left's size when merging levels…

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53937 from Matan-B/wip-50598-reef
Matan Breizman [Tue, 17 Oct 2023 12:45:48 +0000 (15:45 +0300)]
Merge pull request #53937 from Matan-B/wip-50598-reef

reef: crimson/osd: add embedded suppression ruleset for LSan

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53953 from Matan-B/wip-52369-reef
Matan Breizman [Tue, 17 Oct 2023 11:38:25 +0000 (14:38 +0300)]
Merge pull request #53953 from Matan-B/wip-52369-reef

reef: crimson/os/seastore: add metrics about total invalidated transactions

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53952 from Matan-B/wip-51770-reef
Matan Breizman [Tue, 17 Oct 2023 11:37:59 +0000 (14:37 +0300)]
Merge pull request #53952 from Matan-B/wip-51770-reef

reef: crimson/os/seastore/rbm: make rbm support multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53950 from Matan-B/wip-51852-reef
Matan Breizman [Tue, 17 Oct 2023 11:36:49 +0000 (14:36 +0300)]
Merge pull request #53950 from Matan-B/wip-51852-reef

reef: crimson/osd/watch: Add logs around Watch/Notify

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53948 from Matan-B/wip-52032-reef
Matan Breizman [Tue, 17 Oct 2023 11:34:49 +0000 (14:34 +0300)]
Merge pull request #53948 from Matan-B/wip-52032-reef

reef: crimson/osd: don't ignore start_pg_operation returned future

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53945 from Matan-B/wip-51980-reef
Matan Breizman [Tue, 17 Oct 2023 11:29:58 +0000 (14:29 +0300)]
Merge pull request #53945 from Matan-B/wip-51980-reef

reef: crimson/os/seastore: fix in check_node

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53944 from Matan-B/wip-51961-reef
Matan Breizman [Tue, 17 Oct 2023 11:28:09 +0000 (14:28 +0300)]
Merge pull request #53944 from Matan-B/wip-51961-reef

reef: Crimson/osd: Disable concurrent MOSDMap handling

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53943 from Matan-B/wip-51945-reef
Matan Breizman [Tue, 17 Oct 2023 11:27:31 +0000 (14:27 +0300)]
Merge pull request #53943 from Matan-B/wip-51945-reef

reef: crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Reviewed-by: Samuel Just <sjust@redhat.com>