]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
3 years agocrimson/os/alienstore: fix nullptr deref in OnCommit::finish(). 43262/head
Radoslaw Zarzynski [Wed, 22 Sep 2021 16:42:19 +0000 (16:42 +0000)]
crimson/os/alienstore: fix nullptr deref in OnCommit::finish().

`seastar::engine()` is available only for Seastar's threads;
it shouldn't be called outside of a reactor thread.
Unfortunately, this assumption is violated in `AlienStore`
where `OnCommit::finish()`, executed from a finisher thread
of `BlueStore`, calls `alien()` on `seastar::engine()`.
The net effect are crashes like the following one:

```
INFO  2021-09-22 14:26:33,214 [shard 0] osd - operator() writing superblock cluster_fsid 1d8f7908-2ebf-4a91-ae70-f445668c126b osd_fsid 4da9fe9a-1da5-4ea9-aa79-a1178165ede5         [381/1839]
Segmentation fault.
Backtrace:
 0# print_backtrace(std::basic_string_view<char, std::char_traits<char> >) at /home/rzarzynski/ceph1/build/../src/crimson/common/fatal_signal.cc:80
 1# FatalSignal::signaled(int, siginfo_t const&) at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/ostream:570
 2# FatalSignal::install_oneshot_signal_handler<11>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /home/rzarzynski/ceph1/build/../src/crimson/common/fatal_signal.cc:
62
 3# 0x00007F16BBA13B30 in /lib64/libpthread.so.0
 4# (anonymous namespace)::OnCommit::finish(int) at /home/rzarzynski/ceph1/build/../src/crimson/os/alienstore/alien_store.cc:53
 5# Context::complete(int) at /home/rzarzynski/ceph1/build/../src/include/Context.h:100
 6# Finisher::finisher_thread_entry() at /home/rzarzynski/ceph1/build/../src/common/Finisher.cc:65
 7# 0x00007F16BBA0915A in /lib64/libpthread.so.0
 8# clone in /lib64/libc.so.6
Dump of siginfo:
  ...
  si_addr: 0x10
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/common: add safe_then_unpack() to errorated futures. 43247/head
Radoslaw Zarzynski [Tue, 21 Sep 2021 21:28:45 +0000 (21:28 +0000)]
crimson/common: add safe_then_unpack() to errorated futures.

It was a prerequisite for another commit I finally thrown
away. However, this little bit can be still be useful even
for the sake of compliance with the interruptible variant.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/common: assert_moveable() doesn't depend on 3rd party's always_false<>.
Radoslaw Zarzynski [Tue, 21 Sep 2021 21:28:02 +0000 (21:28 +0000)]
crimson/common: assert_moveable() doesn't depend on 3rd party's always_false<>.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/net: fix dangling addrvec in bind(), the repeat_until_value() part. 43243/head
Radoslaw Zarzynski [Mon, 20 Sep 2021 13:59:30 +0000 (13:59 +0000)]
crimson/net: fix dangling addrvec in bind(), the repeat_until_value() part.

This is a follow-up to commit 480273082718230d64d2c068db1b22f44997bdc8.

`seastar::do_with()` doesn't extend the life-time of the callable's

```cpp
auto
do_with_impl(T1&& rv1, T2&& rv2, More&&... more) {
    // ...
    auto&& just_values = cherry_pick_tuple(idx(), std::move(all));
    auto&& just_func = std::move(std::get<nr>(std::move(all)));
    // ...
    auto task = std::apply(
        [](auto&&... x) {
            return std::make_unique<internal::do_with_state<value_tuple, ret_type>>(std::forward<decltype(x)>(x)...);
        },
        std::move(just_values));
    auto fut = std::apply(just_func, task->data());
    // ...
}
```

closure. This means the `addrs` needs to be captured by value also
by the lambda passed to `repeat_until_value()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agoMerge pull request #43154 from ceph/wip-yuriw-pacific-16.2.6-RN
David Galloway [Fri, 17 Sep 2021 13:38:54 +0000 (09:38 -0400)]
Merge pull request #43154 from ceph/wip-yuriw-pacific-16.2.6-RN

doc: 16.2.6 Release Notes

3 years agoMerge pull request #43197 from anthonyeleven/anthonyeleven/stale-read
zdover23 [Fri, 17 Sep 2021 05:52:34 +0000 (15:52 +1000)]
Merge pull request #43197 from anthonyeleven/anthonyeleven/stale-read

doc: clarify stale_read and placement-groups wording

Reviewed-by: Zac Dover <zac.dover@gmail.com>
3 years agoMerge PR #43122 into master
Patrick Donnelly [Fri, 17 Sep 2021 01:47:40 +0000 (21:47 -0400)]
Merge PR #43122 into master

* refs/pull/43122/head:
qa: add test for standby-replay marking rank damaged
MDSMonitor: handle damaged from standby-replay
mds: add config to mark rank damaged in standby-replay
include: unset std::hex after printing CompatSet
mds: refactor iterator lookup
mds: harden rank lookup

Reviewed-by: Venky Shankar <vshankar@redhat.com>
3 years agodoc: clarify stale_read and placement-groups wording 43197/head
Anthony D'Atri [Thu, 16 Sep 2021 23:40:33 +0000 (16:40 -0700)]
doc: clarify stale_read and placement-groups wording

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
3 years agoMerge pull request #43011 from rzarzynski/wip-crimson-opsequencer-ooo-lesser-footprint
Kefu Chai [Thu, 16 Sep 2021 23:23:14 +0000 (07:23 +0800)]
Merge pull request #43011 from rzarzynski/wip-crimson-opsequencer-ooo-lesser-footprint

crimson/osd: bring OpRegistry-based OpSequencer to handle out-of-order ClientRequests

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
3 years agoMerge pull request #43192 from rzarzynski/wip-crimson-net-dangling-bindvec
Samuel Just [Thu, 16 Sep 2021 19:22:18 +0000 (12:22 -0700)]
Merge pull request #43192 from rzarzynski/wip-crimson-net-dangling-bindvec

crimson/net: fix dangling addrvec in SocketMessenger::bind().

Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agodoc/releases/pacific.rst: Add note about Bullseye 43154/head
David Galloway [Thu, 16 Sep 2021 17:57:16 +0000 (13:57 -0400)]
doc/releases/pacific.rst: Add note about Bullseye

3 years agocrimson/net: fix dangling addrvec in SocketMessenger::bind(). 43192/head
Radoslaw Zarzynski [Thu, 16 Sep 2021 17:04:28 +0000 (17:04 +0000)]
crimson/net: fix dangling addrvec in SocketMessenger::bind().

`SocketMessenger::bind()` takes the address vector by `const&`.
while the callers in `OSD` are passing a temporary:

  ```cpp
  cluster_msgr->bind(pick_addresses(CEPH_PICK_ADDRESS_CLUSTER))
    // ...
  public_msgr->bind(pick_addresses(CEPH_PICK_ADDRESS_PUBLIC))
    // ...
  ```

  ```cpp
  entity_addrvec_t pick_addresses(int what) {
    // ...
    return addrs;
  }
  ```

The net result is a life-time mismatch and crashes like the following
one:

  ```
  WARN  2021-09-10 20:01:57,791 [shard 0] ms - [osd.0(client) v2:172.17.0.1:6800/3172806564@59988 >> mgr.? v2:172.17.0.7:6800/3846062660] waiting 3.2 seconds ...
  /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/stl_iterator.h:820:17: runtime error: reference binding to misaligned address 0x000041b58ab3 for type 'const struct entity_addr_t', which requires 4 byte alignment
  0x000041b58ab3: note: pointer points here
  <memory cannot be printed>
  /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/stl_vector.h:1132:16: runtime error: reference binding to misaligned address 0x000041b58ab3 for type 'const struct value_type', which requires 4 byte alignment
  0x000041b58ab3: note: pointer points here
  <memory cannot be printed>
  /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-7486-g27cb19ed/rpm/el8/BUILD/ceph-17.0.0-7486-g27cb19ed/src/msg/msg_types.h:561:22: runtime error: reference binding to misaligned address 0x000041b58ab3 for type 'const struct entity_addr_t', which requires 4 byte alignment
  0x000041b58ab3: note: pointer points here
  <memory cannot be printed>
  Segmentation fault on shard 0.
  ```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agodoc/releases/pacific.rst: add core release notes origin/wip-yuriw-pacific-16.2.6-RN
Neha Ojha [Thu, 16 Sep 2021 16:49:38 +0000 (16:49 +0000)]
doc/releases/pacific.rst: add core release notes

Signed-off-by: Neha Ojha <nojha@redhat.com>
3 years agocrimson/osd: rework OpSequencer for out-of-order op execution. 43011/head
Radoslaw Zarzynski [Wed, 1 Sep 2021 12:57:54 +0000 (12:57 +0000)]
crimson/osd: rework OpSequencer for out-of-order op execution.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agoMerge pull request #43180 from athanatos/sjust/wip-52630
Samuel Just [Thu, 16 Sep 2021 03:59:33 +0000 (20:59 -0700)]
Merge pull request #43180 from athanatos/sjust/wip-52630

crimson/net/SocketMessenger: include sleep.hh

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agoMerge pull request #43178 from athanatos/sjust/wip-52623
Samuel Just [Thu, 16 Sep 2021 03:59:09 +0000 (20:59 -0700)]
Merge pull request #43178 from athanatos/sjust/wip-52623

crimson/os/seastore/cache: Cache::get_root check for invalid

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agoMerge PR #43044 into master
Sage Weil [Thu, 16 Sep 2021 02:50:56 +0000 (22:50 -0400)]
Merge PR #43044 into master

* refs/pull/43044/head:
mgr/rook: add unit tests to Rook module
mgr/rook: translate placement spec to node selector and vice versa

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
3 years agoMerge PR #43172 into master
Sage Weil [Thu, 16 Sep 2021 02:48:36 +0000 (22:48 -0400)]
Merge PR #43172 into master

* refs/pull/43172/head:
qa/tasks/kubeadm: modify (do not clobber) daemon.json

Reviewed-by: Joseph Sawaya <jsawaya@redhat.com>
3 years agocrimson/net/SocketMessenger: include sleep.hh 43180/head
Samuel Just [Thu, 16 Sep 2021 02:31:52 +0000 (02:31 +0000)]
crimson/net/SocketMessenger: include sleep.hh

4a00a145 added a call to seastar::sleep.

Fixes: https://tracker.ceph.com/issues/52630
Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agoMerge PR #43022 into master
Patrick Donnelly [Thu, 16 Sep 2021 02:37:26 +0000 (22:37 -0400)]
Merge PR #43022 into master

* refs/pull/43022/head:
doc/cephfs: add mds_dir_max_entries option into mds-config-ref.rst
mds: add mds_dir_max_entries config option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge PR #42943 into master
Patrick Donnelly [Thu, 16 Sep 2021 02:36:18 +0000 (22:36 -0400)]
Merge PR #42943 into master

* refs/pull/42943/head:
mon: do not quickly mark mds laggy when MON_DOWN
mon: refactor quorum age calculation

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
3 years agoMerge PR #42846 into master
Patrick Donnelly [Thu, 16 Sep 2021 02:35:37 +0000 (22:35 -0400)]
Merge PR #42846 into master

* refs/pull/42846/head:
mds: Add new flag to MClientSession

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge PR #42737 into master
Patrick Donnelly [Thu, 16 Sep 2021 02:34:58 +0000 (22:34 -0400)]
Merge PR #42737 into master

* refs/pull/42737/head:
mds: ensure that we send the btime in cap messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
3 years agoMerge PR #42719 into master
Patrick Donnelly [Thu, 16 Sep 2021 02:34:23 +0000 (22:34 -0400)]
Merge PR #42719 into master

* refs/pull/42719/head:
mgr/volumes: Fix permission during subvol creation with mode

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agocrimson/os/seastore/cache: Cache::get_root check for invalid 43178/head
Samuel Just [Wed, 15 Sep 2021 21:23:29 +0000 (21:23 +0000)]
crimson/os/seastore/cache: Cache::get_root check for invalid

As with Cache::get_extent, we need to check that the ref
is still valid after wait_io since it's not in the
read set yet.

Fixes: https://tracker.ceph.com/issues/52623
Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agoMerge PR #42584 into master
Patrick Donnelly [Thu, 16 Sep 2021 02:33:18 +0000 (22:33 -0400)]
Merge PR #42584 into master

* refs/pull/42584/head:
doc: fix `daemon status` interface (exclude file system name)
test: adjust mirroring tests for `daemon status` change
mgr/mirroring: `daemon status` command does not require file system name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agoMerge PR #42541 into master
Patrick Donnelly [Thu, 16 Sep 2021 02:32:31 +0000 (22:32 -0400)]
Merge PR #42541 into master

* refs/pull/42541/head:
client: switch to use ceph_assert instead of assert
mds: switch to use ceph_assert() instead of assert()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agocrimson/os/seastore/cache: rename invalidate(Transaction&, CachedExtent&)
Samuel Just [Thu, 16 Sep 2021 01:37:31 +0000 (18:37 -0700)]
crimson/os/seastore/cache: rename invalidate(Transaction&, CachedExtent&)

This particular overload is really only declaring the Transaction
conflicted -- it assumes the extent is already invalid.  Rename to
mark_transaction_conflicted instead to avoid confusion with
invalidate(CachedExtent&).

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agoqa/tasks/kubeadm: modify (do not clobber) daemon.json 43172/head
Sage Weil [Wed, 15 Sep 2021 15:53:32 +0000 (10:53 -0500)]
qa/tasks/kubeadm: modify (do not clobber) daemon.json

Otherwise we blow away the mirror config.

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agodoc: updated the change log with three additional PRs merged on top
Yuri Weinstein [Wed, 15 Sep 2021 15:42:33 +0000 (15:42 +0000)]
doc: updated the change log with three additional PRs merged on top

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
3 years agodoc: dashboard motd change
Ernesto Puerta [Tue, 14 Sep 2021 17:18:00 +0000 (19:18 +0200)]
doc: dashboard motd change

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
3 years agodoc: note simpler MDS upgrade procedure for v16.2.6
Patrick Donnelly [Tue, 14 Sep 2021 15:12:10 +0000 (11:12 -0400)]
doc: note simpler MDS upgrade procedure for v16.2.6

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
3 years agodoc: 16.2.6 Release Notes
Yuri Weinstein [Mon, 13 Sep 2021 21:11:15 +0000 (21:11 +0000)]
doc: 16.2.6 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
3 years agoMerge pull request #43146 from jianwei1216/fix-partial-recovery-become-whole-object...
Kefu Chai [Wed, 15 Sep 2021 15:17:29 +0000 (23:17 +0800)]
Merge pull request #43146 from jianwei1216/fix-partial-recovery-become-whole-object-recovery-for-master

osd: fix partial recovery become whole object recovery after restart osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
3 years agoMerge pull request #42465 from tchaikov/wip-ceph-volume
Kefu Chai [Wed, 15 Sep 2021 13:52:42 +0000 (21:52 +0800)]
Merge pull request #42465 from tchaikov/wip-ceph-volume

debian, rpm: split ceph-volume into a separated package

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
3 years agoMerge pull request #43019 from rhcs-dashboard/fix-52028-master
Ernesto Puerta [Wed, 15 Sep 2021 13:40:58 +0000 (15:40 +0200)]
Merge pull request #43019 from rhcs-dashboard/fix-52028-master

mgr/dashboard: Incorrect MTU mismatch warning

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: p-se <NOT@FOUND>
3 years agoMerge pull request #43115 from mgfritch/cephadm-rm-repo-no-podman
Sebastian Wagner [Wed, 15 Sep 2021 06:38:48 +0000 (08:38 +0200)]
Merge pull request #43115 from mgfritch/cephadm-rm-repo-no-podman

cephadm: skip podman check during `rm-repo`

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
3 years agoMerge pull request #43129 from cyx1231st/wip-seastore-optimize-onode
Samuel Just [Wed, 15 Sep 2021 04:44:22 +0000 (21:44 -0700)]
Merge pull request #43129 from cyx1231st/wip-seastore-optimize-onode

crimson/os/seastore: minimize the static size of onode_layout_t

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43106 from rzarzynski/wip-crimson-msgr-bind
Samuel Just [Wed, 15 Sep 2021 04:41:46 +0000 (21:41 -0700)]
Merge pull request #43106 from rzarzynski/wip-crimson-msgr-bind

crimson: add support for ms_bind_retry_{delay,count}

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agoMerge pull request #43157 from liu-chunmei/seastore-binary-search
Samuel Just [Wed, 15 Sep 2021 04:41:15 +0000 (21:41 -0700)]
Merge pull request #43157 from liu-chunmei/seastore-binary-search

crimson/seastore:: add binary search for lba and omap iterator search

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore: minimize the static size of onode_layout_t 43129/head
Yingxin Cheng [Fri, 10 Sep 2021 08:37:05 +0000 (16:37 +0800)]
crimson/os/seastore: minimize the static size of onode_layout_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson: exclude oid when encode/decode object_info_t to/from object store
Yingxin Cheng [Fri, 10 Sep 2021 08:34:10 +0000 (16:34 +0800)]
crimson: exclude oid when encode/decode object_info_t to/from object store

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/seastore:: add binary search for lba and omap iterator search 43157/head
chunmei-liu [Mon, 13 Sep 2021 23:02:57 +0000 (16:02 -0700)]
crimson/seastore:: add binary search for lba and omap iterator search

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
3 years agoMerge pull request #43158 from athanatos/sjust/wip-seastore-omap
Samuel Just [Tue, 14 Sep 2021 20:36:49 +0000 (13:36 -0700)]
Merge pull request #43158 from athanatos/sjust/wip-seastore-omap

crimson/os/seastore: SeaStore::get_omap_iterator should seek_to_first

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #43161 from rzarzynski/wip-crimson-fix-no-bs
Samuel Just [Tue, 14 Sep 2021 20:36:24 +0000 (13:36 -0700)]
Merge pull request #43161 from rzarzynski/wip-crimson-fix-no-bs

crimson/os: allow to build crimson when WITH_BLUESTORE=OFF

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agorpm, debian: drop xmlstarlet from build deps 42465/head
Kefu Chai [Wed, 4 Aug 2021 11:20:15 +0000 (19:20 +0800)]
rpm, debian: drop xmlstarlet from build deps

xmlstarlet is not used anymore while performing "make check", so drop it
from the build dependency list.

it is still used by the teuthology test though, so keep it as a
dependency of ceph-test.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agoceph.spec: remove parted from ceph-base deps
Kefu Chai [Wed, 4 Aug 2021 11:00:01 +0000 (19:00 +0800)]
ceph.spec: remove parted from ceph-base deps

parted is only used by ceph-volume

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agoceph.spec, debian: move deps from base to ceph-volume
Kefu Chai [Wed, 28 Jul 2021 07:32:58 +0000 (15:32 +0800)]
ceph.spec, debian: move deps from base to ceph-volume

this change makes util-linux, xfsprogs and e2fsprogs runtime deps of ceph-volume

ceph-volume uses blkid and lsblk, which are in turn packaged by
util-linux.

util-linux were added as build dependency to fulfill the needs of
ceph-disk. and we tested ceph-disk as part of "make check", since
ceph-disk was dropped, there is no need to have util-linux as
build dependency anymore.

the same applies to e2fsprogs and xfsprogs

see also a99177834120e7a2c4592054f6a8b8736e0ffb92

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agodebian/control: make cryptsetup-bin a ceph-volume runtime dep
Kefu Chai [Wed, 28 Jul 2021 07:16:19 +0000 (15:16 +0800)]
debian/control: make cryptsetup-bin a ceph-volume runtime dep

was originally added by 6a45b8ed041514816d09b2808d2d390eadbd0dd9 and
cfcaceac44d6b7b7c55e81d0bfb05f4893f3b1d0 as a dependency of ceph-disk.
although, ceph-disk was replaced by ceph-volume, cryptsetup-bin is still
used.

but it should be a runtime dependency of ceph-volume. and it is not a
build dependency of ceph, so it's removed from the build dependency list
as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agoceph.spec.in: make cryptsetup a ceph-volume runtime dep.
Kefu Chai [Wed, 28 Jul 2021 07:14:42 +0000 (15:14 +0800)]
ceph.spec.in: make cryptsetup a ceph-volume runtime dep.

cryptsetup is solely used by ceph-volume, so it should be a runtime
dependency of ceph-volume, not ceph-base.

and remove it fomr build-dependency list, as we don't use this tool when
building ceph.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agoceph.spec.in: split ceph-volume into a separated package
Kefu Chai [Fri, 23 Jul 2021 11:19:43 +0000 (19:19 +0800)]
ceph.spec.in: split ceph-volume into a separated package

ceph-volume is a tool implemented in pure python, so it would be better
to make it a architecture independent package for better
maintainability.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agodebian: split ceph-volume into a separated package
Kefu Chai [Fri, 23 Jul 2021 10:55:18 +0000 (18:55 +0800)]
debian: split ceph-volume into a separated package

ceph-volume is a tool implemented in pure python, so it would be better
to make it a architecture independent package for better
maintainability.

in this change

* ceph-volume is extracted out into a separated package
* ceph-volume depends on ceph-osd, as it deploys it and relies on
  an already-installed ceph-osd in the system.
* ceph-osd recommends ceph-volume. as ceph-osd can be used as a
  standalone package. but ceph-volume enhances it. also, to ensure
  the existing users to get ceph-volume installed along with
  ceph-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agorpm,deb: make parted runtime dependency of ceph-osd
Kefu Chai [Fri, 23 Jul 2021 10:36:37 +0000 (18:36 +0800)]
rpm,deb: make parted runtime dependency of ceph-osd

we don't run parted when building ceph, instead, it is a runtime
dependency of ceph-volume, which is packaged in ceph-osd.

so in this change

* parted is removed from build dependency list
* parted is added as a runtime depedency of ceph-osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>
3 years agocrimson/os: add the preamble to futurized_store.cc. 43161/head
Radoslaw Zarzynski [Tue, 14 Sep 2021 15:05:32 +0000 (15:05 +0000)]
crimson/os: add the preamble to futurized_store.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/os: allow to build crimson when WITH_BLUESTORE=OFF.
Radoslaw Zarzynski [Tue, 14 Sep 2021 13:48:58 +0000 (13:48 +0000)]
crimson/os: allow to build crimson when WITH_BLUESTORE=OFF.

This lets to ensure nobody is accidentally linking with the alienized
version of `common`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson: switch try_bind() to bind(). 43106/head
Radoslaw Zarzynski [Thu, 9 Sep 2021 06:16:09 +0000 (06:16 +0000)]
crimson: switch try_bind() to bind().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/net: bind() respects now ms_bind_retry_{delay,count}.
Radoslaw Zarzynski [Thu, 9 Sep 2021 06:11:59 +0000 (06:11 +0000)]
crimson/net: bind() respects now ms_bind_retry_{delay,count}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/net: make try_bind() aware about address_not_available.
Radoslaw Zarzynski [Thu, 9 Sep 2021 06:09:54 +0000 (06:09 +0000)]
crimson/net: make try_bind() aware about address_not_available.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/net: decouple listen_ertr from bind_ertr.
Radoslaw Zarzynski [Wed, 8 Sep 2021 19:30:17 +0000 (19:30 +0000)]
crimson/net: decouple listen_ertr from bind_ertr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocrimson/net: decouple listen_ertr from FixedCPUServerSocket.
Radoslaw Zarzynski [Wed, 8 Sep 2021 19:27:44 +0000 (19:27 +0000)]
crimson/net: decouple listen_ertr from FixedCPUServerSocket.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agoMerge pull request #43109 from guits/guits-cv-24044
Guillaume Abrioux [Tue, 14 Sep 2021 08:00:14 +0000 (10:00 +0200)]
Merge pull request #43109 from guits/guits-cv-24044

ceph-volume: util/prepare fix osd_id_available()

3 years agocrimson/os/seastore: SeaStore::get_omap_iterator should seek_to_first 43158/head
Samuel Just [Tue, 14 Sep 2021 04:26:33 +0000 (21:26 -0700)]
crimson/os/seastore: SeaStore::get_omap_iterator should seek_to_first

Add debugging as well.

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agovstart.sh: default to msgr2 for crimson
Samuel Just [Tue, 14 Sep 2021 04:27:24 +0000 (21:27 -0700)]
vstart.sh: default to msgr2 for crimson

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agoMerge pull request #43123 from liu-chunmei/lba_metrics_iter
Samuel Just [Tue, 14 Sep 2021 03:33:31 +0000 (20:33 -0700)]
Merge pull request #43123 from liu-chunmei/lba_metrics_iter

crimson/seastore: add lba tree metrics for lba tree profiling

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
3 years agocrimson/seastore: add lba tree metrics for lba tree profiling 43123/head
chunmei-liu [Thu, 9 Sep 2021 05:25:45 +0000 (22:25 -0700)]
crimson/seastore: add lba tree metrics for lba tree profiling

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
3 years agoMerge pull request #43152 from athanatos/sjust/wip-simplify-delayed
Samuel Just [Tue, 14 Sep 2021 01:31:47 +0000 (18:31 -0700)]
Merge pull request #43152 from athanatos/sjust/wip-simplify-delayed

crimson/sesatore: simplify epm related transaction structures, fix some bugs

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
3 years agocrimson/os/seastore/transaction: add trace debugging for write_set 43152/head
Samuel Just [Sat, 11 Sep 2021 04:28:24 +0000 (21:28 -0700)]
crimson/os/seastore/transaction: add trace debugging for write_set

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore/cached_extent: fix unused variable warning
Samuel Just [Sat, 11 Sep 2021 05:44:26 +0000 (05:44 +0000)]
crimson/os/seastore/cached_extent: fix unused variable warning

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore/transaction_manager: fix delayed alloc addr in rewrite_logical_extent
Samuel Just [Sat, 11 Sep 2021 04:11:24 +0000 (21:11 -0700)]
crimson/os/seastore/transaction_manager: fix delayed alloc addr in rewrite_logical_extent

Setting nlextent's to the addr of the original extent violates invariants
in the intrusive_set we're using for ExtentIndex and generally breaks the
assumption in cache that paddrs are unique until reuse.  This means we'll
do an extra update_mapping for now -- we can refactor SegmentCleaner to
avoid it later.

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore/scanner: fix budget_used ref in scan_valid_records
Samuel Just [Sat, 11 Sep 2021 04:08:57 +0000 (21:08 -0700)]
crimson/os/seastore/scanner: fix budget_used ref in scan_valid_records

Fixes: https://tracker.ceph.com/issues/52582
Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore/cache: more get_extent debugging
Samuel Just [Fri, 10 Sep 2021 22:41:55 +0000 (22:41 +0000)]
crimson/os/seastore/cache: more get_extent debugging

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore/extent_placement_manager: clarify else branch delayed_alloc_or_oo...
Samuel Just [Fri, 10 Sep 2021 22:12:46 +0000 (15:12 -0700)]
crimson/os/seastore/extent_placement_manager: clarify else branch delayed_alloc_or_ool_write

Also shorten long line.

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore: replace fresh_block_list with inline/ool_block_list
Samuel Just [Fri, 10 Sep 2021 01:11:05 +0000 (18:11 -0700)]
crimson/os/seastore: replace fresh_block_list with inline/ool_block_list

With this change, Transaction::write_set is the union of mutually exclusive
lists
- delayed_alloc_list, inline_block_list, and mutated_block_list prior to
  calling ExtentPlacementManager::delayed_alloc_or_ool_write

and

- inline_block_list, ool_block_list, and mutated_block_list after
  calling ExtentPlacementManager::delayed_alloc_or_ool_write

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agoMerge PR #43136 into master
Sage Weil [Mon, 13 Sep 2021 22:28:43 +0000 (18:28 -0400)]
Merge PR #43136 into master

* refs/pull/43136/head:
qa/tasks/kubeadm: change calico encap to IPIPCrossSubnet
qa/suites/orch/rook/smoke: add host networking to matrix
qa/tasks/rook: fix shadowing of config arg in rook_cluster()

Reviewed-by: Joseph Sawaya <jsawaya@redhat.com>
3 years agoMerge pull request #42417 from rzarzynski/wip-bl-optimize-emptybptrs
Ilya Dryomov [Mon, 13 Sep 2021 21:00:17 +0000 (23:00 +0200)]
Merge pull request #42417 from rzarzynski/wip-bl-optimize-emptybptrs

common/bl, tests: optimize carriage handling in bufferlist::c_str()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
3 years agoqa/tasks/kubeadm: change calico encap to IPIPCrossSubnet 43136/head
Sage Weil [Mon, 13 Sep 2021 18:43:20 +0000 (13:43 -0500)]
qa/tasks/kubeadm: change calico encap to IPIPCrossSubnet

Signed-off-by: Sage Weil <sage@newdream.net>
3 years agocrimson/os/seastore/cached_extent: assert !parent_index in ExtentIndex::insert
Samuel Just [Fri, 10 Sep 2021 01:07:52 +0000 (18:07 -0700)]
crimson/os/seastore/cached_extent: assert !parent_index in ExtentIndex::insert

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/so/seastore: clarify ExtentIndex clear()
Samuel Just [Fri, 10 Sep 2021 01:07:31 +0000 (18:07 -0700)]
crimson/so/seastore: clarify ExtentIndex clear()

ExtentIndex::clear() needs to handle clearing parent_index.

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agocrimson/os/seastore/transaction: remove delayed_set
Samuel Just [Fri, 3 Sep 2021 20:22:46 +0000 (20:22 +0000)]
crimson/os/seastore/transaction: remove delayed_set

delayed_set isn't necessary.  Delayed extents have distinct paddr segments
already, so it's simpler to have a single index for all pending extents.

Signed-off-by: Samuel Just <sjust@redhat.com>
3 years agotests/bl: verify the carriage handling in bufferlist::c_str(). 42417/head
Radoslaw Zarzynski [Tue, 20 Jul 2021 10:32:36 +0000 (10:32 +0000)]
tests/bl: verify the carriage handling in bufferlist::c_str().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocommon/bl: abstract from _carriage checking in c_str().
Radoslaw Zarzynski [Thu, 5 Aug 2021 00:04:57 +0000 (00:04 +0000)]
common/bl: abstract from _carriage checking in c_str().

This is intended to enchance readability and sligthly
generalize the previous, `_carriage`-aware optimization.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agocommon/bl: c_str() considers _carriage now.
Radoslaw Zarzynski [Mon, 19 Jul 2021 14:06:20 +0000 (14:06 +0000)]
common/bl: c_str() considers _carriage now.

It could be that `_carriage` points on empty bptr which is
also stored in the `_buffers` container -- this happens
when we'are trying to preserve the appendable space to not
waste memory and avoid new allocation.
Before the change this approach was imposing unncessary
rebuilding on `c_str()`. Now we take into consideration
the special case when a bufferlist has the extra, empty
bptr at its end.

Fixes: https://tracker.ceph.com/issues/51725
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 years agoceph-volume: util/prepare fix osd_id_available() 43109/head
Guillaume Abrioux [Thu, 9 Sep 2021 08:23:43 +0000 (10:23 +0200)]
ceph-volume: util/prepare fix osd_id_available()

The current check only allows to request an OSD id that exists but
marked as 'destroyed'.
With this small fix, we can now use `--osd-id` with an id that doesn't
exist at all.

Fixes: https://tracker.ceph.com/issues/50880
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
3 years agomgr/rook: add unit tests to Rook module 43044/head
Joseph Sawaya [Fri, 3 Sep 2021 14:28:57 +0000 (10:28 -0400)]
mgr/rook: add unit tests to Rook module

This commit creates a unit test folder for rook, configurates tox
and creates a unit test to test the translation of placement specs
to node selectors and vice versa.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
3 years agomgr/rook: translate placement spec to node selector and vice versa
Joseph Sawaya [Fri, 3 Sep 2021 14:24:24 +0000 (10:24 -0400)]
mgr/rook: translate placement spec to node selector and vice versa

This commit creates methods for translating PlacementSpecs to NodeSelectors
and vice versa, this is a general method that would be used by orch commands that
have a placement option. It's able to translate specs that specify a label, hosts
and the host_pattern '*'.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>
3 years agoMerge pull request #43141 from sebastian-philipp/doc-orch-rm
Sebastian Wagner [Mon, 13 Sep 2021 14:38:01 +0000 (16:38 +0200)]
Merge pull request #43141 from sebastian-philipp/doc-orch-rm

doc/cephadm: Removing a service

Reviewed-by: Michael Fritch <mfritch@suse.com>
3 years agoMerge PR #42998 into master
Sage Weil [Mon, 13 Sep 2021 14:21:32 +0000 (10:21 -0400)]
Merge PR #42998 into master

* refs/pull/42998/head:
qa/tasks/rook: add testing for host label add/rm
mgr/rook: host add/rm label in rook orchestrator

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
3 years agoMerge pull request #43147 from krunerge/fix_typo
Kefu Chai [Mon, 13 Sep 2021 13:45:11 +0000 (21:45 +0800)]
Merge pull request #43147 from krunerge/fix_typo

doc/dev/crimson/poseidonstore: fix typo

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
3 years agoMerge pull request #43063 from guits/guits-trailing-spaces
Dimitri Savineau [Mon, 13 Sep 2021 13:13:07 +0000 (09:13 -0400)]
Merge pull request #43063 from guits/guits-trailing-spaces

ceph-volume: remove trailing spaces in code

3 years agoMerge pull request #42951 from ronen-fr/wip-ronenf-pg-dtor
Ronen Friedman [Mon, 13 Sep 2021 11:23:52 +0000 (14:23 +0300)]
Merge pull request #42951 from ronen-fr/wip-ronenf-pg-dtor

osd/scrub: destruct the scrubber shortly before the PG is destructed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>-
Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agodoc/dev/crimson/poseidonstore: fix typo 43147/head
krunerge [Mon, 13 Sep 2021 11:12:36 +0000 (19:12 +0800)]
doc/dev/crimson/poseidonstore: fix typo

Signed-off-by: krunerge <krunerge@tencent.com>
3 years agoosd: fix partial recovery become whole object recovery after restart osd 43146/head
Jianwei Zhang [Mon, 13 Sep 2021 10:13:18 +0000 (18:13 +0800)]
osd: fix partial recovery become whole object recovery after restart osd

support SERVER_OCTOPUS feature for pg_missing_item::encode()

Fixes: https://tracker.ceph.com/issues/52583
Signed-off-by: Jianwei Zhang <jianwei1216@qq.com>
3 years agoMerge pull request #42555 from hualongfeng/retire_error_change
Ilya Dryomov [Mon, 13 Sep 2021 10:04:34 +0000 (12:04 +0200)]
Merge pull request #42555 from hualongfeng/retire_error_change

librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
3 years agoMerge pull request #43038 from majianpeng/librbd-pwl-exclusive-lock
Ilya Dryomov [Mon, 13 Sep 2021 09:44:09 +0000 (11:44 +0200)]
Merge pull request #43038 from majianpeng/librbd-pwl-exclusive-lock

librbd: require exclusive lock for reads if pwl cache is enabled

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
3 years agolibrbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry 42555/head
Feng Hualong [Wed, 28 Jul 2021 03:42:44 +0000 (11:42 +0800)]
librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

When retiring, m_blocks_to_log_entries doesn't remove
corresponding write_entry (should be `*it` not `entry`)
that will be retired. It leads to read error. And
there should also consider discard entries.

Fixes: https://tracker.ceph.com/issues/52579
Signed-off-by: Feng Hualong <hualong.feng@intel.com>
3 years agoMerge pull request #43143 from sebastian-philipp/further-reading
Sebastian Wagner [Mon, 13 Sep 2021 07:39:16 +0000 (09:39 +0200)]
Merge pull request #43143 from sebastian-philipp/further-reading

doc/cephadm: Add lots of links to other chapters

Reviewed-by: Zac Dover <zac.dover@gmail.com>
3 years agoMerge pull request #41869 from ronen-fr/wip-ronenf-fmt-scrub
Ronen Friedman [Sun, 12 Sep 2021 13:32:44 +0000 (16:32 +0300)]
Merge pull request #41869 from ronen-fr/wip-ronenf-fmt-scrub

common, osd: add fmtlib formatting of some OSD types

Reviewed-by: Kefu Chai <kchai@redhat.com>
3 years agoMerge pull request #43056 from Matan-B/wip-matanb-scrub-passkey
Ronen Friedman [Sun, 12 Sep 2021 12:47:52 +0000 (15:47 +0300)]
Merge pull request #43056 from Matan-B/wip-matanb-scrub-passkey

osd: Replacing friend declarations for passkey interface

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
3 years agoMerge pull request #42684 from ronen-fr/wip-ronenf-scrub-token
Ronen Friedman [Sun, 12 Sep 2021 12:40:22 +0000 (15:40 +0300)]
Merge pull request #42684 from ronen-fr/wip-ronenf-scrub-token

osd/scrub: tag replica scrub messages to identify stale events

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
3 years agodoc/cephadm: Add lots of links to other chapters 43143/head
Sebastian Wagner [Sat, 11 Sep 2021 18:02:44 +0000 (20:02 +0200)]
doc/cephadm: Add lots of links to other chapters

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>