crimson/os/seastore: better epm output

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #41803 from xxhdx1985126/wip-extent-placement-manager-2

crimson/os/seastore: add extent placement manager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43105 from liu-chunmei/clean-alien-in-store-nbd

crimson/tools: clean alien in store-nbd

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools: clean alien in store-nbd

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #42604 from sseshasa/wip-skip-osd-benchmark

osd: Add config option to skip running the osd benchmark during init and update documentation.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43052 from dsavineau/ceph_volume_lvm_deactive_remove_all_arg

ceph-volume: remove --all ref from deactivate help

Merge pull request #43045 from sebastian-philipp/qa-tox-import-yaml

qa: tox.ini: verify yaml syntax

Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #42859 from mgfritch/cephadm-check-host-no-engine

mgr/cephadm: show unhandled exceptions during host add

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #42989 from sebastian-philipp/post-remove-osd-key-missing

mgr/cephadm: Add OSDService.post_remove()

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #43010 from mgfritch/cephadm-log-thread-ident

cephadm: add thread ident to log messages

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43048 from dsavineau/ceph_volume_lvm_migrate_no_systemd

ceph-volume: support no_systemd with lvm migrate

Merge pull request #43051 from dsavineau/ceph_volume_lvm_migrate_docs

doc/ceph-volume: add lvm migrate/new-db/new-wal

Merge pull request #43050 from dsavineau/fix_raw_list_with_lsblk

ceph-volume: fix raw list with logical partition

Merge pull request #43006 from CongMinYin/fix-assert-in-handle_flushed_sync_point

librbd/cache/pwl: don't clear next_sync_point_entry prematurely

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43074 from wangshuaich/rbd-children-crash-fix

librbd: fix use-after-free on ictx in list_descendants()

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

crimson/os/seastore: better debug outputs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: allow getting empty delta from omap extents

there are chances that a transaction gets invalidated after some of
its extents have their delta prepared.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add ExtentIndex::clear_and_dispose

When a transaction is interrupted and needs to repeat, its reset_preserve_handle() method
is called to clear various extent set and list. The problem is the clear operation call
ExtentIndex::clear() instead of ExtentIndex::erase(), which would leave CachedExtent::parent_index
still pointing to the write_set/delayed_set while the CachedExtent is no longer linked to
those sets. This would make CachedExtent::~CachedExtent() to try to erase itself from
CachedExtent::parent_index even it's not linked, which would cause failures in the successive
operations

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/store_nbd: involve the extent placement manager and the scanner

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: make fresh extents go though the extent placement manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: deal with transaction conflicts when doing mkfs

Cleaner transactions can conflict with mkfs transactions, retry mkfs transactions
when that happens

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: set journal_tail_target during replay

This is a bug fix, otherwise if crimson-osd boot up multiple times without
filling up more than one segment, segments may be used up and can't be
reclaimed as they would have the same journal tail

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: set root block to clean at the initialization phase

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: exclude ool extents from the judgement is_in_journal

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: use extent placement manager to rewrite extents

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: refactor extent rewrite procedure

Move logical extents' rewrite from LBManager to TransactionManager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson: add tracing mode for crimson-osd

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add Scanner to scan extents

As there will be two kinds of segments to be scanned, those created by the journal
and those created by the extent placement manager. We need a common module to scan
extents of both of these two kinds of segments

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add extent placement manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: a new condition variable

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: extract record related code into seastore_types.{h,cc}

Extent Placement Manager will be using record too.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: move JournalSegmentProvider to SegmentProvider

This provider will serve not only for the journal, but also the extent placement
manager from now on.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43084 from athanatos/sjust/wip-lba-pin

crimson/seastore: lba pin debugging and fix

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43083 from athanatos/sjust/wip-52532

crimson/os/seastore/.../lba_btree: fix handle_split internal nodes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43066 from rzarzynski/wip-crimson-fatal-reraise

 crimson/common: explicitly reraise handled signal in FatalSignal

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: add logging for invalid pin

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: pin may not match unless paddr does

The same laddr might be reallocated with a different size.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: improve add_pin debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree: fix handle_split internal nodes

Internal node pointers aren't actually allowed to point to end() -- that's
specific to the leaf pointer.

Fixes: https://tracker.ceph.com/issues/52532
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/fs_driver: fix FuturizedStore::create() call

Introduced: 8ec97762c4809c25d3b38ac15b942dfd710e3d29
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree: clarify handle_split, find_insertion comments

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: report correct error for ictx->state->close()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: fix use-after-free on ictx in list_descendants()

Ictx is deleted when "ictx->state->open()" and "ictx->state->close()"
fail, and then "lderr(ictx->cct)" crashes.

Fixes: https://tracker.ceph.com/issues/52522
Signed-off-by: Wang ShuaiChao <wangshuaich@chinatelecom.cn>

Merge pull request #43070 from krunerge/fix-children-list-log-text

cls/rbd: fix log text for children list

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43067 from tchaikov/wip-crimson-alien

crimson/osd: use reactor::alien to send message to alien

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42933 from p-se/mgr-prom-disable-cache

mgr/prometheus: offer ability to disable cache

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

crimson/osd: use reactor::alien to send message to alien

simpler this way.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

seastar: pick up change to add reactor::alien()

so we don't need to pass the reference to alien all the way down to
AlienStore.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

ceph-volume: test the lsblk command in list.py

Let's test we use the expected args when we build the `lsblk` command
to list the devices present on the node.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: fix raw list with logical partition

This is a regression introduced by 9212420, when the host is using a
logical partition then lsblk reports that partition as a child from the
physical device.
That logical partition is prefixed by the `└─` character.

This leads the `raw list` subcommand to show the lsblk error on the stderr.

```
$ ceph-volume raw list
{}
 stderr: lsblk: `-/dev/sda1: not a block device
```

The lsblk command output looks like:

```
$ lsblk --paths --output=NAME --noheadings
/dev/sda
└─/dev/sda1
/dev/sdb
/dev/sdc
/dev/sdd
```

Using the `--list` option with lsblk solves the issue.

```
$ lsblk --list --paths --output=NAME --noheadings
/dev/sda
/dev/sda1
/dev/sdb
/dev/sdc
/dev/sdd
```

Fixes: https://tracker.ceph.com/issues/52504
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge PR #43017 into master

* refs/pull/43017/head:
doc: fix typos

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43009 from fengjiankui121/pvs_repair

ceph-volume: pvs --noheadings replace pvs --no-heading

qa: tox.ini: verify yaml syntax

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cls/rbd: fix log text for children list

Signed-off-by: krunerge <krunerge@tencent.com>

Merge PR #42529 into master

* refs/pull/42529/head:
qa: verify rank 0 does not fail during journal repair tests
qa: avoid stopping/restarting mds in journal repair tests

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43065 from tchaikov/wip-doc-build

admin/doc-requirements: use funcparserlib from github

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42919 from sebastian-philipp/cephadm-async-close-conn

mgr/cephadm: Also make ssh._reset_con async

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Melissa Li <li.melissa.kun@gmail.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #42992 from ifed01/wip-ifed-fix-migrate

os/bluestore: fix bluefs migrate command

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

crimson/common: explicitly reraise handled signal in FatalSignal.

Over the current approach where we just reset the handler to
default and allow CPU to re-execute the segfaulting instruction,
the explicit `::reraise()` is:
1. immune to a race condition if muliple threads run into
   troubles the same time;
2. easier to understand and similar to the classic OSD.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: s/sig/signum/ in FatalSignal for better readability.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42965 from ronen-fr/wip-ronenf-split-child

osd: adding missing data to register_and_wake_split_child() logs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42953 from ronen-fr/wip-ronenf-handled-objs

osd/scrub: stats_of_handled_objects(): reduce irrelevant logs

Reviewed-by: Amnon Hanuhov <ahanukov@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #43061 from sebastian-philipp/cephadm-no-asyncssh

pybind/mgr: tox.ini add nooptional testenv

Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

admin/doc-requirements: use funcparserlib from github

funcparserlib is pulled in as a dependency by blockdiag. the latest version of
funcparserlib available on pypi is v0.3.6 which is not compatible with
Python3.8.

in this change, funcparserlib is installed from github instead to
address the build failure like:

  File "/home/docs/checkouts/readthedocs.org/user_builds/ceph/envs/41855/lib/python3.8/site-packages/sphinxcontrib/seqdiag.py", line 26, in <module>
    import seqdiag.utils.rst.nodes
  File "/home/docs/checkouts/readthedocs.org/user_builds/ceph/envs/41855/lib/python3.8/site-packages/seqdiag/utils/rst/nodes.py", line 16, in <module>
    from blockdiag.utils.rst import nodes
  File "/home/docs/checkouts/readthedocs.org/user_builds/ceph/envs/41855/lib/python3.8/site-packages/blockdiag/utils/rst/nodes.py", line 21, in <module>
    import blockdiag.builder
  File "/home/docs/checkouts/readthedocs.org/user_builds/ceph/envs/41855/lib/python3.8/site-packages/blockdiag/builder.py", line 16, in <module>
    from blockdiag import parser
  File "/home/docs/checkouts/readthedocs.org/user_builds/ceph/envs/41855/lib/python3.8/site-packages/blockdiag/parser.py", line 43, in <module>
    from funcparserlib.parser import (a, finished, forward_decl, many, maybe, skip,
  File "/home/docs/checkouts/readthedocs.org/user_builds/ceph/envs/41855/lib/python3.8/site-packages/funcparserlib/parser.py", line 123
    except NoParseError, e:
                       ^
SyntaxError: invalid syntax

once https://github.com/vlasovskikh/funcparserlib/issues/65 is
addressed, we should drop this change.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr/prometheus: offer ability to disable cache

Fixes: https://tracker.ceph.com/issues/52414
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

librbd/cache/pwl: don't clear next_sync_point_entry prematurely

In SyncPointLogOperation::clear_earlier_sync_point(),
sync_point->log_entry->next_sync_point_entry was prematurely set to
nullptr in clear_earlier_sync_point(). It is in write op stage, but
next_sync_point_entry is used in writeback stage in
handle_flushed_sync_point().

handle_flushed_sync_point() may pass a nullptr
cause assert in m_work_queue.The solution is to move the statement
that set next_sync_point_entry to nullptr after it is used.

Fixes: https://tracker.ceph.com/issues/52465
Signed-off-by: Yin Congmin <congmin.yin@intel.com>

Merge pull request #43034 from rubenk/fix-unmapped-bytes-help-text

common/PriorityCache: fix help text for unmapped_bytes metric

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

pybind/mgr: tox.ini remove duplicated `tox` env

as this is the same as `py3`. Avoid executing it twice

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

pybind/mgr: tox.ini add nooptional testenv

To verify cephadm properly works as expected without asyncssh installed

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

pybind/mgr: tox.ini: norecursedirs instead explicitly list modules

Simplify things by not listing all modules individually.

Also: Fixed two new doctests
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43047 from idryomov/wip-rbd-mirror-peer-profile-doc

doc: adjust client.rbd-mirror-peer mon cap and add details about site names

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42840 from TRYTOBE8TME/wip-rgw-kafka-timeout

Wip rgw kafka timeout

Merge pull request #42875 from pleiadesian/feature-lua

examples: add elasticsearch and prometheus examples

doc: fix typos

Signed-off-by: wangxinyu <wangxinyu@inspur.com>

Merge pull request #42870 from ktdreyer/cmake-thread-libs-init

cmake: link Threads::Threads instead of CMAKE_THREAD_LIBS_INIT

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42960 from wjwithagen/wjw-fix-cephfs-lock

tools/cephfs_mirror: fix lock declaratie/locking

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42962 from tchaikov/wip-cmake-ppc

arch,cmake: compile ppc.c on all powerpc machines

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

ceph-volume: remove --all ref from deactivate help

Until the `--all` feature is implemented.

This was partially removed in c13901f but not from the help command.

Fixes: https://tracker.ceph.com/issues/50109
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #43043 from tchaikov/crimson/os/cleanup

crimson/os: use structured binding in loop

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

doc/ceph-volume: add lvm migrate/new-db/new-wal

The inital PR was implementing those new commands only adds the man page
and not the online documentation.

Fixes: https://tracker.ceph.com/issues/51814
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

ceph-volume: support no_systemd with lvm migrate

The `ceph-volume lvm migrate/new-db/new-wal` commands don't support
running on non systemd systems or within containers.
Like other ceph-volume commands (lvm activate/batch/zap or raw activate)
we also need to be able to use the --no-systemd flag.

Fixes: https://tracker.ceph.com/issues/51854
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

doc: adjust client.rbd-mirror-peer mon cap

On top of "profile rbd" permissions, "profile rbd-mirror-peer" also
allows getting rbd/mirror and setting rbd/mirror/peer/* config keys.
This is what "rbd mirror pool peer bootstrap create" does.

Fixes: https://tracker.ceph.com/issues/50970
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: add some details about rbd-mirror site names

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43035 from zdover23/wip-doc-2021-09-03-rados-bootstrap-options

doc: rados/ceph-conf.rst bootstrap options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #43012 into master

* refs/pull/43012/head:
script: ceph-debug-docker.sh deals with the centos-repos -> centos-linux-repos transition.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43040 from sebastian-philipp/qa-deadsymlinks

qa/tox.ini: check for dead symlinks

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #42836 from ivancich/wip-fix-bi-test-error

rgw: fix bucket index list test error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/os: use structured binding in loop

also avoid using `map[key] = val` for setting an item in map, as, if
he key does not exist in map, `map[key]` would have to create a value
using its default ctor, and then call the `operator=(bufferlist&&)` to
set it.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge PR #43032 into master

* refs/pull/43032/head:
backport-create-issue: set backport priority

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #43039 from sebastian-philipp/rm-kubic

qa/distros: Remove stale kubic distros

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43021 from sebastian-philipp/rhel_8.3_container_tools_rhel8.yaml

qa/distros/podman: Add rhel_8.3_container_tools_rhel8.yaml

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tox.ini: check for dead symlinks

IMO the amount of symlinks we have to manually maintain
is tedious and error prone. Any ideas on improving thing?

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/distros: Remove stale kubic distros

Cause they're broken

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

qa/distros/podman: Add rhel_8.rhel_8.4_container_tools_3.0.yaml

mainly for the cephfs suite

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43030 from dsavineau/cephadm_fix_prometheus_spelling

cephadm: fix prometheus namespace spelling

Reviewed-by: Sage Weil <sage@newdream.net>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

ceph-volume: pvs --noheadings replace pvs --no-heading

Fixes: https://tracker.ceph.com/issues/52482
Signed-off-by: FengJiankui <fengjiankui@inspur.com>

Merge PR #42873 into master

* refs/pull/42873/head:
qa/tasks/rook: add OSD creation to Rook QA

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #43018 from cyx1231st/wip-seastore-onode-tree-alloc-hint

crimson/onode-staged-tree: improve laddr hints during allocation

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: rados/ceph-conf.rst bootstrap options

This is the editorial syntax and elegance PR for the "Bootstrap Options"
section in the "Configuring Ceph" chapter of the RADOS Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

common/PriorityCache: fix help text for unmapped_bytes metric

Signed-off-by: Ruben Kerkhof <ruben@42on.com>