]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
21 months agocrimson/osd: modify the interface to integrate with milticore-msg 54058/head
chunmei [Fri, 23 Jun 2023 04:35:38 +0000 (04:35 +0000)]
crimson/osd: modify the interface to integrate with milticore-msg

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 00a5eaea4661e6fb23d174b2f367222541b24f21)

21 months agocrimson/osd: remove seastar::now() to gurantee sequence in pipline
chunmei [Fri, 23 Jun 2023 04:25:53 +0000 (04:25 +0000)]
crimson/osd: remove seastar::now() to gurantee sequence in pipline

in debug build, if wait_barrier is seastar::now, will cause reactor
schedule the following continuation away, but can't schedule them back
in the same order as by which they are scheduled away, so the continuations
are out of order here.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit c7628de4d230cd08d536d7e0c35bf6822cc7146e)

21 months agocrimson/osd: make OSDState and OSDMap_Gate sharded
chunmei [Fri, 23 Jun 2023 03:35:09 +0000 (03:35 +0000)]
crimson/osd: make OSDState and OSDMap_Gate sharded

put them into PerShardState to avoid cross core calling for normal cases.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 324191130ab77a06777f3a71b4c6bf995a4e5fe7)

21 months agocrimson/osd: change OSD::ms_dispatch to take an ConnectionFRef
chunmei [Sat, 8 Apr 2023 08:58:15 +0000 (08:58 +0000)]
crimson/osd: change OSD::ms_dispatch to take an ConnectionFRef

Shortly, we will need to be able to pass a ConnectionRef from another core.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit d657afc7a76d14882152d6f38da5adeaa39e55e1)

21 months agocrimson/osd: relax requirement that PGShardManager methods be invoked on PRIMARY_CORE
chunmei [Fri, 31 Mar 2023 04:11:27 +0000 (04:11 +0000)]
crimson/osd: relax requirement that PGShardManager methods be invoked on PRIMARY_CORE

Proxy uses of osd_singleton_state to PRIMARY_CORE as needed.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 6bb4f00a9d006f99b21104bf46b78dff269a116f)

21 months agocrimson/osd: enable PGShardMapping access from all cores
chunmei [Fri, 31 Mar 2023 02:49:33 +0000 (02:49 +0000)]
crimson/osd: enable PGShardMapping access from all cores

Previously, all accesses (incuding loookups) had to occur on core 0.  Now that
we want to be able to dispatch from all cores, we need PGShardManager to be
accessible from all cores.  To that end, we now proxy updates to core 0, but maintain
local copies of the map so that cores can perform local lookups.

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit a3052969bfcfd9329b2baf7883765642d09ff038)

21 months agocrimson/osd: make osd sharded
chunmei [Fri, 23 Jun 2023 03:18:15 +0000 (03:18 +0000)]
crimson/osd: make osd sharded

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit 5f82fbbc72f840fa8642015a120485648aaf507c)

21 months agoMerge pull request #53956 from Matan-B/wip-52221-reef
Matan Breizman [Tue, 17 Oct 2023 16:00:33 +0000 (19:00 +0300)]
Merge pull request #53956 from Matan-B/wip-52221-reef

reef: crimson/os/seastore/collection_manager: fill CollectionNode::decoded on clean reads

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53942 from Matan-B/wip-51034-reef
Matan Breizman [Tue, 17 Oct 2023 15:59:03 +0000 (18:59 +0300)]
Merge pull request #53942 from Matan-B/wip-51034-reef

reef: crimson/osd: fix ENOENT on accessing RadosGW user's index of buckets

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53168 from leonid-s-usov/bp/cap-throttle-event/reef
Leonid S. Usov [Tue, 17 Oct 2023 13:44:00 +0000 (16:44 +0300)]
Merge pull request #53168 from leonid-s-usov/bp/cap-throttle-event/reef

reef: mds/Server: mark a cap acquisition throttle event in the request

21 months agoMerge pull request #52899 from leonid-s-usov/backport/bulk-data-pool/reef
Leonid S. Usov [Tue, 17 Oct 2023 13:43:42 +0000 (16:43 +0300)]
Merge pull request #52899 from leonid-s-usov/backport/bulk-data-pool/reef

reef: Consider setting "bulk" autoscale pool flag when automatically creating a data pool for CephFS

21 months agoMerge pull request #53969 from Matan-B/wip-53432-reef
Matan Breizman [Tue, 17 Oct 2023 13:10:42 +0000 (16:10 +0300)]
Merge pull request #53969 from Matan-B/wip-53432-reef

reef: crimson/vstart: default seastore_device_size will be out of space f…

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53968 from Matan-B/wip-53230-reef
Matan Breizman [Tue, 17 Oct 2023 13:10:06 +0000 (16:10 +0300)]
Merge pull request #53968 from Matan-B/wip-53230-reef

reef: crimson/os/seastore/cbj: fix a potential overflow bug on segment_seq

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53967 from Matan-B/wip-53232-reef
Matan Breizman [Tue, 17 Oct 2023 13:09:49 +0000 (16:09 +0300)]
Merge pull request #53967 from Matan-B/wip-53232-reef

reef: test/crimson/seastore/rbm: add sub-tests regarding RBM to the existing tests

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53965 from Matan-B/wip-52204-reef
Matan Breizman [Tue, 17 Oct 2023 13:09:16 +0000 (16:09 +0300)]
Merge pull request #53965 from Matan-B/wip-52204-reef

reef: crimson/osd/object_context: consider clones found as long as they're in SnapSet::clones

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53964 from Matan-B/wip-53082-reef
Matan Breizman [Tue, 17 Oct 2023 13:08:52 +0000 (16:08 +0300)]
Merge pull request #53964 from Matan-B/wip-53082-reef

reef: crimson/admin/admin_socket: remove path file if it exists

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53963 from Matan-B/wip-53031-reef
Matan Breizman [Tue, 17 Oct 2023 13:08:34 +0000 (16:08 +0300)]
Merge pull request #53963 from Matan-B/wip-53031-reef

reef: crimson/osd/pg: check if backfill_state exists when judging objects'

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53962 from Matan-B/wip-53051-reef
Matan Breizman [Tue, 17 Oct 2023 13:08:10 +0000 (16:08 +0300)]
Merge pull request #53962 from Matan-B/wip-53051-reef

reef: crimson/os/seastore/omap_manager: fix the entry leak issue in BtreeOMapManager::omap_list()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53961 from Matan-B/wip-52128-reef
Matan Breizman [Tue, 17 Oct 2023 13:07:43 +0000 (16:07 +0300)]
Merge pull request #53961 from Matan-B/wip-52128-reef

reef: crimson/os/seastore/journal/cbj: generalize scan_valid_records()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53960 from Matan-B/wip-53023-reef
Matan Breizman [Tue, 17 Oct 2023 13:07:14 +0000 (16:07 +0300)]
Merge pull request #53960 from Matan-B/wip-53023-reef

reef: crimson/os/seastore: fix daggling reference of oid in SeaStore::Shard::stat()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53959 from Matan-B/wip-52757-reef
Matan Breizman [Tue, 17 Oct 2023 13:06:51 +0000 (16:06 +0300)]
Merge pull request #53959 from Matan-B/wip-52757-reef

reef: crimson/os/seastore: retire_extent_addr clean up

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53958 from Matan-B/wip-52655-reef
Matan Breizman [Tue, 17 Oct 2023 13:05:48 +0000 (16:05 +0300)]
Merge pull request #53958 from Matan-B/wip-52655-reef

reef: crimson/ertr: assert on invocability of func provided to safe_then()

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53955 from Matan-B/wip-52446-reef
Matan Breizman [Tue, 17 Oct 2023 13:05:01 +0000 (16:05 +0300)]
Merge pull request #53955 from Matan-B/wip-52446-reef

reef: crimson/os/seastore/transaction_manager: data loss issues

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53951 from Matan-B/wip-51179-reef
Matan Breizman [Tue, 17 Oct 2023 13:04:31 +0000 (16:04 +0300)]
Merge pull request #53951 from Matan-B/wip-51179-reef

reef: crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53949 from Matan-B/wip-51916-reef
Matan Breizman [Tue, 17 Oct 2023 13:04:05 +0000 (16:04 +0300)]
Merge pull request #53949 from Matan-B/wip-51916-reef

reef: crimson/net: support connections in multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53947 from Matan-B/wip-52217-reef
Matan Breizman [Tue, 17 Oct 2023 12:46:25 +0000 (15:46 +0300)]
Merge pull request #53947 from Matan-B/wip-52217-reef

reef: crimson/os/seastore/omap_manager: correct editor settings

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53946 from Matan-B/wip-52090-reef
Matan Breizman [Tue, 17 Oct 2023 12:46:08 +0000 (15:46 +0300)]
Merge pull request #53946 from Matan-B/wip-52090-reef

reef: crimson/os/seastore/btree: should add left's size when merging levels…

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53937 from Matan-B/wip-50598-reef
Matan Breizman [Tue, 17 Oct 2023 12:45:48 +0000 (15:45 +0300)]
Merge pull request #53937 from Matan-B/wip-50598-reef

reef: crimson/osd: add embedded suppression ruleset for LSan

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53953 from Matan-B/wip-52369-reef
Matan Breizman [Tue, 17 Oct 2023 11:38:25 +0000 (14:38 +0300)]
Merge pull request #53953 from Matan-B/wip-52369-reef

reef: crimson/os/seastore: add metrics about total invalidated transactions

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53952 from Matan-B/wip-51770-reef
Matan Breizman [Tue, 17 Oct 2023 11:37:59 +0000 (14:37 +0300)]
Merge pull request #53952 from Matan-B/wip-51770-reef

reef: crimson/os/seastore/rbm: make rbm support multiple shards

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53950 from Matan-B/wip-51852-reef
Matan Breizman [Tue, 17 Oct 2023 11:36:49 +0000 (14:36 +0300)]
Merge pull request #53950 from Matan-B/wip-51852-reef

reef: crimson/osd/watch: Add logs around Watch/Notify

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53948 from Matan-B/wip-52032-reef
Matan Breizman [Tue, 17 Oct 2023 11:34:49 +0000 (14:34 +0300)]
Merge pull request #53948 from Matan-B/wip-52032-reef

reef: crimson/osd: don't ignore start_pg_operation returned future

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53945 from Matan-B/wip-51980-reef
Matan Breizman [Tue, 17 Oct 2023 11:29:58 +0000 (14:29 +0300)]
Merge pull request #53945 from Matan-B/wip-51980-reef

reef: crimson/os/seastore: fix in check_node

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53944 from Matan-B/wip-51961-reef
Matan Breizman [Tue, 17 Oct 2023 11:28:09 +0000 (14:28 +0300)]
Merge pull request #53944 from Matan-B/wip-51961-reef

reef: Crimson/osd: Disable concurrent MOSDMap handling

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53943 from Matan-B/wip-51945-reef
Matan Breizman [Tue, 17 Oct 2023 11:27:31 +0000 (14:27 +0300)]
Merge pull request #53943 from Matan-B/wip-51945-reef

reef: crimson/osd: fix Notify life-time mismanagement in Watch::notify_ack

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53941 from Matan-B/wip-51948-reef
Matan Breizman [Tue, 17 Oct 2023 11:21:04 +0000 (14:21 +0300)]
Merge pull request #53941 from Matan-B/wip-51948-reef

reef: crimson/os/seastore/cached_extent: add prepare_commit interface

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53940 from Matan-B/wip-51380-reef
Matan Breizman [Tue, 17 Oct 2023 11:20:38 +0000 (14:20 +0300)]
Merge pull request #53940 from Matan-B/wip-51380-reef

reef: crimson/osd/heartbeat: Improve maybe_share_osdmap behavior

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53939 from Matan-B/wip-51750-reef
Matan Breizman [Tue, 17 Oct 2023 11:19:53 +0000 (14:19 +0300)]
Merge pull request #53939 from Matan-B/wip-51750-reef

reef: crimson/os/seastore/backref_manager: scan backref entries by journal seq

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53938 from Matan-B/wip-51475-reef
Matan Breizman [Tue, 17 Oct 2023 11:19:10 +0000 (14:19 +0300)]
Merge pull request #53938 from Matan-B/wip-51475-reef

reef: crimson/os/seastore: fix ceph_assert in segment_manager.h

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53935 from Matan-B/wip-51355-reef
Matan Breizman [Tue, 17 Oct 2023 11:15:39 +0000 (14:15 +0300)]
Merge pull request #53935 from Matan-B/wip-51355-reef

reef: crimson/os/seastore: enable SMR HDD

Reviewed-by: Samuel Just <sjust@redhat.com>
21 months agoMerge pull request #53797 from smanjara/ragweed-test-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:37:47 +0000 (12:37 -0700)]
Merge pull request #53797 from smanjara/ragweed-test-reef

reef: rgw/upgrade: point upgrade suites to ragweed ceph-reef branch

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #53773 from cbodley/wip-63057-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:37:03 +0000 (12:37 -0700)]
Merge pull request #53773 from cbodley/wip-63057-reef

reef: rgw: improve buffer list utilization in the chunkupload scenario

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #53770 from cbodley/wip-63054-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:35:12 +0000 (12:35 -0700)]
Merge pull request #53770 from cbodley/wip-63054-reef

reef: rgw: fix SignatureDoesNotMatch when extra headers start with 'x-amz'

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #53767 from cbodley/wip-63051-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:34:05 +0000 (12:34 -0700)]
Merge pull request #53767 from cbodley/wip-63051-reef

reef: rgw: s3website doesn't prefetch for web_dir() check

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #53765 from cbodley/wip-63048-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:33:06 +0000 (12:33 -0700)]
Merge pull request #53765 from cbodley/wip-63048-reef

reef: rgw: fix rgw rate limiting RGWRateLimitInfo class decode_json max_rea…

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #53760 from cbodley/wip-63060-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:32:15 +0000 (12:32 -0700)]
Merge pull request #53760 from cbodley/wip-63060-reef

reef: rgw: fix unwatch crash at radosgw startup

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #53756 from cbodley/wip-63042-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:31:29 +0000 (12:31 -0700)]
Merge pull request #53756 from cbodley/wip-63042-reef

reef: [CVE-2023-43040] rgw: Fix bucket validation against POST policies

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
21 months agoMerge pull request #53681 from mkogan1/wip-t62141-reef
Yuri Weinstein [Mon, 16 Oct 2023 19:30:28 +0000 (12:30 -0700)]
Merge pull request #53681 from mkogan1/wip-t62141-reef

reef: valgrind: UninitCondition under __run_exit_handlers suppression

Reviewed-by: Casey Bodley <cbodley@redhat.com>
21 months agoMerge pull request #54026 from zdover23/wip-doc-2023-10-16-backport-54022-to-reef
Anthony D'Atri [Sun, 15 Oct 2023 23:54:09 +0000 (19:54 -0400)]
Merge pull request #54026 from zdover23/wip-doc-2023-10-16-backport-54022-to-reef

reef: doc/rados: Edit troubleshooting-osd (3 of x)

21 months agodoc/rados: Edit troubleshooting-osd (3 of x) 54026/head
Zac Dover [Sun, 15 Oct 2023 12:43:39 +0000 (22:43 +1000)]
doc/rados: Edit troubleshooting-osd (3 of x)

Edit doc/rados/troubleshooting/troubleshooting-osd.rst (3 of x).

Follows https://github.com/ceph/ceph/pull/53986.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 01b5aa5d27a97a02cef03b9eb0ee9dbcd55684f1)

21 months agoMerge pull request #54008 from zdover23/wip-doc-2023-10-14-backport-54006-to-reef
Anthony D'Atri [Fri, 13 Oct 2023 20:31:28 +0000 (16:31 -0400)]
Merge pull request #54008 from zdover23/wip-doc-2023-10-14-backport-54006-to-reef

doc/rados: remove FileStore material (in Reef)

21 months agodoc/rados: remove FileStore material (in Reef) 54008/head
Zac Dover [Fri, 13 Oct 2023 18:40:26 +0000 (04:40 +1000)]
doc/rados: remove FileStore material (in Reef)

Remove FileStore-related material because FileStore is not supported in
Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit eab0be5ca513ca2dfaeb31911c175e88bcf4f9e4)

21 months agoMerge pull request #54007 from zdover23/wip-doc-2023-10-14-backport-54005-to-reef-2
Anthony D'Atri [Fri, 13 Oct 2023 20:15:24 +0000 (16:15 -0400)]
Merge pull request #54007 from zdover23/wip-doc-2023-10-14-backport-54005-to-reef-2

reef: doc/dev: remove cache-pool

21 months agodoc/dev: remove cache-pool 54007/head
Zac Dover [Fri, 13 Oct 2023 18:17:44 +0000 (04:17 +1000)]
doc/dev: remove cache-pool

Remove cache-pool, which is deprecated in Reef.

Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 073b0919ed9e413a04a7e12b7a0c23b5e058cd52)

21 months agoMerge pull request #54000 from zdover23/wip-doc-2023-10-13-backport-53986-to-reef
zdover23 [Fri, 13 Oct 2023 14:01:03 +0000 (00:01 +1000)]
Merge pull request #54000 from zdover23/wip-doc-2023-10-13-backport-53986-to-reef

reef: doc/rados: Edit troubleshooting-osd (2 of x)

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>
21 months agodoc/rados: Edit troubleshooting-osd (2 of x) 54000/head
Zac Dover [Thu, 12 Oct 2023 08:33:58 +0000 (18:33 +1000)]
doc/rados: Edit troubleshooting-osd (2 of x)

Edit doc/rados/troubleshooting/troubleshooting.rst (2 of x).

Follows https://github.com/ceph/ceph/pull/53936.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 37e7099267996a3075b4902a10a19d94fc738c08)

21 months agoMerge pull request #53458 from rhcs-dashboard/wip-62602-reef
Nizamudeen A [Fri, 13 Oct 2023 05:35:08 +0000 (11:05 +0530)]
Merge pull request #53458 from rhcs-dashboard/wip-62602-reef

reef: mgr/dashboard: remove unnecessary failing hosts e2e

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
21 months agoMerge pull request #53982 from zdover23/wip-doc-2023-10-12-backport-53936-to-reef
Anthony D'Atri [Thu, 12 Oct 2023 09:02:48 +0000 (05:02 -0400)]
Merge pull request #53982 from zdover23/wip-doc-2023-10-12-backport-53936-to-reef

reef: doc/rados: edit troubleshooting-osd (1 of x)

21 months agodoc/rados: edit troubleshooting-osd (1 of x) 53982/head
Zac Dover [Wed, 11 Oct 2023 10:38:55 +0000 (20:38 +1000)]
doc/rados: edit troubleshooting-osd (1 of x)

Edit doc/rados/troubleshooting/troubleshooting-osd.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 0fc663d793fe34e42e19b251fcc79b5af52581c6)

21 months agocrimson/vstart: add --seastore-device-size option in vstart.sh command line 53969/head
chunmei [Thu, 14 Sep 2023 06:21:05 +0000 (06:21 +0000)]
crimson/vstart: add --seastore-device-size option in vstart.sh command line

default seastore_device_size will be out of space for smp >28

Signed-off-by: chunmei <chunmei.liu@intel.com>
(cherry picked from commit cf69c4f4946884266d6a42c92eef20e74771c3fa)

21 months agocrimson/os/seastore/cbj: fix a potential overflow bug 53968/head
Myoungwon Oh [Thu, 31 Aug 2023 02:43:24 +0000 (11:43 +0900)]
crimson/os/seastore/cbj: fix a potential overflow bug

CircularBoundedJournal is similar to circular queue, so segment_seq
increases after rolling. However, current implementation always
increases segment_seq_t when rolling occurs, resulting in
the overflow if segment_seq_t hits MAX_SEG_SEQ.

To mitigate this, this commit changes the type of the segment_seq_t
to uint64_t.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 36982b66574ff277e04a02d5b9a28e80d0790778)

21 months agocrimson/os/seastore/object_data_handler: consider a RBM case when checking if write... 53967/head
Myoungwon Oh [Fri, 25 Aug 2023 08:55:36 +0000 (17:55 +0900)]
crimson/os/seastore/object_data_handler: consider a RBM case when checking if write can be merged

RBM's paddr always indicates physical address, which means it doesn't have the dealayed.
So, this commit adds a condition that checks if given paddr is used for ongoing write.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 73de8937f65706c9c3dd2e36f6504196d0796b01)

21 months agotest/crimson/seastore/rbm: add sub-tests regarding RBM to the existing tests
Myoungwon Oh [Thu, 31 Aug 2023 05:13:07 +0000 (14:13 +0900)]
test/crimson/seastore/rbm: add sub-tests regarding RBM to the existing tests

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 670a19f27761c937ad8d07abf9bb7c94ed4a3b00)

21 months agocrimson/osd/osd_operations/snaptrim_event: don't process snaptrim events 53965/head
Xuehan Xu [Sun, 6 Aug 2023 07:45:24 +0000 (15:45 +0800)]
crimson/osd/osd_operations/snaptrim_event: don't process snaptrim events
of the same PG concurrently

Concurrently triming snaps of the same PG may lead to the following
problem:

1. fiber 1 get objects to trim for PG 1.x and snapid 1.d, which involves
   object OBJ;
2. fiber 2 get objects to trim for PG 1.x and snapid 1.c, which also
   involes object OBJ;
3. fiber 1 removes snap 1.c since it has been removed in the osdmap
4. fiber 2 try to get obc for object OBJ and find that OBJ doesn't have
   snap for id 1.c

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 267babe435a54130eda5770a537420fb516e2a8f)

21 months agocrimson/osd/object_context: consider clones found as long as they're in
Xuehan Xu [Tue, 18 Jul 2023 06:14:50 +0000 (14:14 +0800)]
crimson/osd/object_context: consider clones found as long as they're in
SnapSet::clones

Fixes: https://tracker.ceph.com/issues/61825
Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 957fd43e4b09a7607d4db52fe9674be905f76de8)

21 months agocrimson/osd/object_context_loader: rename with_head_and_clone_obc to
Xuehan Xu [Tue, 18 Jul 2023 06:12:59 +0000 (14:12 +0800)]
crimson/osd/object_context_loader: rename with_head_and_clone_obc to
with_clone_obc_direct

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit ec646eb22bfd30f6cf19c86fe24c233001272047)

21 months agocrimson/admin/admin_socket: remove path file if it exists 53964/head
Matan Breizman [Wed, 23 Aug 2023 08:10:32 +0000 (08:10 +0000)]
crimson/admin/admin_socket: remove path file if it exists

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 374de2a7622bf3fc75eceb24e39cf36a27ed8d1b)

21 months agocrimson/osd/pg: check if backfill_state exists when judging objects' 53963/head
Xuehan Xu [Thu, 17 Aug 2023 03:09:05 +0000 (11:09 +0800)]
crimson/osd/pg: check if backfill_state exists when judging objects'
degradation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 04764838b5395a4e4d0c872f2d2a433fc4aec074)

21 months agocrimson/os/seastore/omap_manager: fix the entry leak issue in 53962/head
Xuehan Xu [Fri, 18 Aug 2023 11:56:50 +0000 (19:56 +0800)]
crimson/os/seastore/omap_manager: fix the entry leak issue in
BtreeOMapManager::omap_list()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit cf105932a7fadf67ec494587b078bbfb4caf4d26)

21 months agocrimson/os/seastore/journal/cbj: remove return_record and read_record 53961/head
myoungwon oh [Wed, 16 Aug 2023 11:58:49 +0000 (11:58 +0000)]
crimson/os/seastore/journal/cbj: remove return_record and read_record

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 39ea5add3a81c32263533ec8f01590efeceb2c86)

21 months agocrimson/os/seastore/journal/cbj: check if batched records are written at once before...
myoungwon oh [Mon, 14 Aug 2023 08:10:04 +0000 (08:10 +0000)]
crimson/os/seastore/journal/cbj: check if batched records are written at once before rolling

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit ef1a6a90e0131cdf3bb96ace835f845ae23ba0ce)

21 months agocrimson/os/seastore: genralize read_validate_record_metadata and read_validate_data
myoungwon oh [Fri, 11 Aug 2023 14:47:21 +0000 (14:47 +0000)]
crimson/os/seastore: genralize read_validate_record_metadata and read_validate_data

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 15ae64c3311378f6055d312c2b5f64b46912d899)

21 months agocrimson/os/seastore/journal/cbj: check if the first record in CBJ is written after...
myoungwon oh [Fri, 11 Aug 2023 04:52:23 +0000 (04:52 +0000)]
crimson/os/seastore/journal/cbj: check if the first record in CBJ is written after rolling or not

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit a10f2855da3bc1c25f664158af1cc79d70bdef57)

21 months agocrimson/os/seastore/journal/cbj: introduce magic value to identify that written recor...
myoungwon oh [Sat, 22 Jul 2023 14:52:14 +0000 (14:52 +0000)]
crimson/os/seastore/journal/cbj: introduce magic value to identify that written records are correct

The cbj would replay invalid records if the same write pattern (mkfs -> write -> mkfs -> write) repeats.
On the second mkfs(), current cbj wrongly recognizes the old records are valid beceause they have
correct seq. number at appropriate offset even though the records are expired.

To solve this, this commit introduces random value to identify the journal  during mkfs()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 942ea6487f59e52d88050fb8deb8a2e6c0ca83c3)

21 months agocrimson/os/seastore: introduce generalized scan_valid_records for RBM
myoungwon oh [Sat, 22 Jul 2023 06:03:04 +0000 (06:03 +0000)]
crimson/os/seastore: introduce generalized scan_valid_records for RBM

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit a580e516965ff8ae57dbaab202854f9ebe38ba5f)

21 months agocrimson/os/seastore: introduce RecordScanner to generalize scan_valid_records()
myoungwon oh [Thu, 20 Jul 2023 02:50:48 +0000 (02:50 +0000)]
crimson/os/seastore: introduce RecordScanner to generalize scan_valid_records()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 22c747826b398cc0a0d68aab9bba1fd096a2de9d)

21 months agocrimson/os/seastore: fix daggling reference of oid in 53960/head
Xuehan Xu [Wed, 16 Aug 2023 11:37:59 +0000 (19:37 +0800)]
crimson/os/seastore: fix daggling reference of oid in
SeaStore::Shard::stat()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 0ee9bb21be89bdcfb3e6cfb224c256d6eccd9f38)

21 months agocrimson/os/seastore: retire_extent_addr clean up 53959/head
Xinyu Huang [Wed, 2 Aug 2023 02:09:52 +0000 (10:09 +0800)]
crimson/os/seastore: retire_extent_addr clean up

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit b99a1956bdcb879a3e5440e617593bae43fafd62)

21 months agocrimson/ertr: assert on invocability of func provided to safe_then() 53958/head
Radosław Zarzyński [Wed, 26 Jul 2023 10:42:45 +0000 (12:42 +0200)]
crimson/ertr: assert on invocability of func provided to safe_then()

The goal is to fail with an easier-to-understand error on failed
`static_assert()` than cryptic template substitution errors when
the function provided to e.g. `safe_then()` doesn't match the type
carried within an errorated future.

Signed-off-by: Radosław Zarzyński <rzarzyns@redhat.com>
(cherry picked from commit 83687f2cc86897d60a460575a7ab2acbd2f9170c)

21 months agocrimson/os/seastore/collection_manager: fill CollectionNode::decoded on 53956/head
Xuehan Xu [Wed, 28 Jun 2023 07:18:27 +0000 (15:18 +0800)]
crimson/os/seastore/collection_manager: fill CollectionNode::decoded on
clean reads

Otherwise, deltas of collection nodes can't be replayed correctly,
because the data of nodes is not loaded to CollectionNode::decoded

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 5eb473b99c1ea75fd6c3807d3f54eaec1018d383)

21 months agocrimson/os/seastore/object_data_handler: add ObjectDataHandler(extent_len_t) 53955/head
Xuehan Xu [Fri, 14 Jul 2023 03:59:06 +0000 (11:59 +0800)]
crimson/os/seastore/object_data_handler: add ObjectDataHandler(extent_len_t)

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit c1fdb2a8a41c6f0f204c980c4f3af7cfac28723d)

21 months agocrimson/os/seastore/cache: correct log prefix for get_absent_extent()
Xuehan Xu [Fri, 14 Jul 2023 03:56:12 +0000 (11:56 +0800)]
crimson/os/seastore/cache: correct log prefix for get_absent_extent()

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>
(cherry picked from commit 5a563d63e754d5163424486dbd0d665165615794)

21 months agocrimson/os/seastore: add metrics about total invalidated transactions 53953/head
Zhang Song [Sun, 9 Jul 2023 04:22:58 +0000 (12:22 +0800)]
crimson/os/seastore: add metrics about total invalidated transactions

Signed-off-by: Zhang Song <zhangsong325@gmail.com>
(cherry picked from commit 6265452c50c2bbd0283a66d03d6a8538c2285293)

21 months agosrc/test/crimson/seastore/nvmedevice: update a test case based on rbm multi shard... 53952/head
myoungwon oh [Sat, 17 Jun 2023 08:27:17 +0000 (08:27 +0000)]
src/test/crimson/seastore/nvmedevice: update a test case based on rbm multi shard change

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 7a15d484c552a177585954cdb4e2e226a43c1035)

21 months agocrimson/os/seastore/rbm: make rbm support multiple shards
myoungwon oh [Fri, 26 May 2023 08:29:50 +0000 (08:29 +0000)]
crimson/os/seastore/rbm: make rbm support multiple shards

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
(cherry picked from commit 1d6beef975144210ae29626e61aeb284aff30d53)

21 months agotest/crimson/seastore: test if lazy read work correctly 53951/head
Xinyu Huang [Fri, 26 May 2023 11:05:08 +0000 (11:05 +0000)]
test/crimson/seastore: test if lazy read work correctly

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 9d973efe32c089351ec756232d3699b4c9fe1f64)

21 months agotest/crimson/seastore: test overwrite pin
Xinyu Huang [Fri, 21 Apr 2023 14:22:16 +0000 (14:22 +0000)]
test/crimson/seastore: test overwrite pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit efd3cdc71056baabba291d7a359dda6dbd9121b1)

21 months agotest/crimson/seastore: add test for TransactionManager::remap_pin
Xinyu Huang [Fri, 21 Apr 2023 14:19:34 +0000 (14:19 +0000)]
test/crimson/seastore: add test for TransactionManager::remap_pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit b5931d2795cfe44b96c6ee9347514b002197bb5b)

21 months agocrimson/os/seastore: check if length exist in mark_space_used and mark_space_free
Xinyu Huang [Fri, 21 Apr 2023 14:42:00 +0000 (14:42 +0000)]
crimson/os/seastore: check if length exist in mark_space_used and mark_space_free

to avoid misuse of buffer length and data length when there is a no actual data extent

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit dc4a23cef3787099dcc17f99b05f4ec1a63eca9a)

21 months agocrimson/os/seastore: deepcopy the buffer of exist clean extent when duplicate_for_write
Xinyu Huang [Wed, 19 Apr 2023 01:46:31 +0000 (01:46 +0000)]
crimson/os/seastore: deepcopy the buffer of exist clean extent when duplicate_for_write

Beacuse exist clean extent shares buffer with original clean extent.

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 026066a7e5c4cc1bc04ed91003c553146a0b60ef)

21 months agocrimson/os/seastore: realize lazy read in split overwrite with overwrite refactor
Xinyu Huang [Fri, 7 Apr 2023 00:54:36 +0000 (00:54 +0000)]
crimson/os/seastore: realize lazy read in split overwrite with overwrite refactor

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 5d00ec0d00ddefe5b82ce82d2acfa134d964a883)

21 months agocrimson/os/seastore/transaction_manager: implement remap_pin
Xinyu Huang [Fri, 21 Apr 2023 15:52:24 +0000 (15:52 +0000)]
crimson/os/seastore/transaction_manager: implement remap_pin

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 55d33b69b004adc6fe6740872122311ca9372c0c)

21 months agocrimson/os/seastore/transaction_manager: implement alloc_remapped_extent
Xinyu Huang [Fri, 21 Apr 2023 15:10:49 +0000 (15:10 +0000)]
crimson/os/seastore/transaction_manager: implement alloc_remapped_extent

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit cf6e46e57afc44b46d7855dcb5849724f6ddfcbb)

21 months agocrimson/os/seastore: support no data extent in read path
Xinyu Huang [Sat, 10 Jun 2023 02:16:29 +0000 (10:16 +0800)]
crimson/os/seastore: support no data extent in read path

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit f19de7c77f7d6b783802f6c18e2e9fe106c90243)

21 months agocrimson/os/seastore: let CachedExtent::ptr optional
Xinyu Huang [Tue, 21 Feb 2023 06:18:29 +0000 (06:18 +0000)]
crimson/os/seastore: let CachedExtent::ptr optional

separate buffer length and data length, since no buffer extent is allowed

Signed-off-by: Xinyu Huang <xinyu.huang@intel.com>
(cherry picked from commit 7cb44d350b68002123c3c2f97a888cc42260f522)

21 months agocrimson/osd/watch: don't print bufferlists 53950/head
Matan Breizman [Wed, 31 May 2023 11:47:45 +0000 (11:47 +0000)]
crimson/osd/watch: don't print bufferlists

The bufferlists clutter the logs and don't infrom of anything useful:
```
DEBUG 2023-05-26 23:36:29,780 [shard 2] osd - send_completion sending notify replies: {notify_reply_t{watcher_gid=4220, watcher_cookie=139786421930752, bl=buffer::list(len=0,

)}, notify_reply_t{watcher_gid=4220, watcher_cookie=139786422032096, bl=buffer::list(len=0,

)}}
```

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit 3548489f622c50ccaa1d8c351e420873743ed980)

21 months agocrimson/osd/watch: Add logs around Watch/Notify
Matan Breizman [Wed, 31 May 2023 08:45:53 +0000 (08:45 +0000)]
crimson/osd/watch: Add logs around Watch/Notify

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
(cherry picked from commit b1c05f9acecfd2dffb57d63f42fec9da156ff366)

21 months agocrimson/osd/heartbeat: relax the order of replacement reset and accept 53949/head
Yingxin Cheng [Thu, 29 Jun 2023 05:25:28 +0000 (13:25 +0800)]
crimson/osd/heartbeat: relax the order of replacement reset and accept

With the new implementation in messenger, the order of replacement reset
and accept events cannot be determined because they are from different
connections.

Modify the heatbeat logic to tolerate the both cases.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit a8709464232fc5cc28032716f67c754971420e6b)

21 months agocrimson/net: cleanup, rename is_fixed_cpu
Yingxin Cheng [Tue, 27 Jun 2023 02:31:11 +0000 (10:31 +0800)]
crimson/net: cleanup, rename is_fixed_cpu

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 2fed9397d5f1a93e01d02b5914931420c4f28c37)

21 months agocrimson/net: dispatch the initial acceptance from the socket core
Yingxin Cheng [Fri, 16 Jun 2023 07:35:22 +0000 (15:35 +0800)]
crimson/net: dispatch the initial acceptance from the socket core

It's meaningless to dispatch the initial acceptance always from the
msgr core.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
(cherry picked from commit 1e8a39f359d8481bb3038b461063c87384bf3ee8)