ceph-volume: cleanup code

Simplify the logic and fix a typo.

Signed-off-by: Satoru Takeuchi <satoru.takeuchi@gmail.com>

Merge pull request #35652 from rzarzynski/wip-crimson-thread-cleanup

crimson: clean-up the crimson/thread directory

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35632 from dzafman/wip-46064

tools: Add statfs operation to ceph-objecstore-tool

Reviewed-by: Neha Ojha <nojha@redhat.com>

tools: Add statfs operation to ceph-objecstore-tool

Fixes: https://tracker.ceph.com/issues/46064
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #35329 from trociny/wip-quiesce_complete

librbd: quiesce/unquiesce API improvements

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #35649 into master

* refs/pull/35649/head:
vstart: allow specifying a different ganesha binary
vstart: handle multiple ganesha's more sanely

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

vstart: allow specifying a different ganesha binary

Signed-off-by: Jeff Layton <jlayton@redhat.com>

vstart: handle multiple ganesha's more sanely

Currently, if you have more than one ganesha daemon, it clobbers the
log, config and pidfiles from the previous daemons. Give them each
their own files.

While we're in here, move the log and pidfile to the out/ directory
with all of the other logfiles/pidfiles.

Fixes: https://tracker.ceph.com/issues/46079
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35650 from trociny/wip-46076

test/rbd_mirror: fix race on test shut down

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35631 from tchaikov/wip-cmake-cleanup

cmake: use add_compile_options() and add_definitions()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35630 from tchaikov/wip-systemd-subvars

cmake: move include(GNUInstallDirs) up

Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: move Throttle to src/crimson/common.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: make ThreadPool specific to AlienStore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/rbd_mirror: fix race on test shut down

in TestMockImageReplayerSnapshotReplayer.UnlinkRemoteSnapshot

Fixes: https://tracker.ceph.com/issues/46076
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35637 from bk201/wip-46067

stop.sh: do not block script when there is no running cluster

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #35603 from dillaman/wip-46034

mgr/prometheus: automatically discover RBD pools for stats gathering

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35574 from tspmelo/wip-tslint

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35634 from mgfritch/cephadm-manpage-shell-mount

doc/man: add missing cephadm shell `--mount` option

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cmake: add definitions using add_definitions()

also, add definitions as a list instead of a single string for better
readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use add_compile_options()

instead of appending compile flags to CMAKE_C_FLAGS, use
add_compile_options(), as COMPILE_OPTIONS is a list, it'd simpler to
append options to it and to access it in a structured way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use string(APPEND...) for appending cflags

* use string(APPEND...) for less repeatings
* `-rdynamic` is a linker option, so it should be added to
  `CMAKE_EXE_LINKER_FLAGS` for adding symbols to dynamic
  symbol tables so they can be consumed by plugins.

Signed-off-by: Kefu Chai <kchai@redhat.com>

stop.sh: do not block script when there is no running cluster

A query for current fsid is called inside `do_killcephadm`. This blocks
the script when there is no running cluster. The fix avoids entering the
function if cephadm command fails or returns no daemons.

The change also hides the following output for non-cephadm environments:

```
Unable to locate any of ['podman', 'docker']
```

Fixes: https://tracker.ceph.com/issues/46067
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35469 from xxhdx1985126/wip-crimson-block-io-for-recovery

crimson: block io for recovery ---- recovery part 2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35626 into master

* refs/pull/35626/head:
mds: fix whitespace

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/man: add missing cephadm shell `--mount` option

also clean-up `--mount` help usage

Signed-off-by: Michael Fritch <mfritch@suse.com>

crimson/osd: share map with clients when osdmap epoch doesn't match

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: do not assert in start_backfill_ops

osd_recovery_max_single_start is not necessarily used up in start_recovery_ops, and
start_backfill_ops could be invoked even there's no need to backfill, so do not assert
here, just return.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: fix client op version issue

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/monc: take care of connection lost when sending messages

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: send "mark me down" messages to monitor when shutting down

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: restart ongoing client_request after peering

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: block io for object recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

Merge pull request #35507 from aclamk/wip-bluestore-avl-description

BlueStore, added descriptions for AVL allocator parameters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: move include(GNUInstallDirs) up

so systemd/CMakeLists.txt can have access to the variables defined by
it. quote from https://cmake.org/cmake/help/latest/command/include.html.

> Variable reads and writes access the scope of the caller (dynamic
> scoping).

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35623 from liewegas/wip-bloomberg

doc/foundation: add bloomberg

mds: fix whitespace

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/foundation: add bloomberg

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: Fix tslint warnings

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve typescript lint

Fixes: https://tracker.ceph.com/issues/46000
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35514 from dzafman/wip-41564

mon: Warn when too many reads are repaired on an OSD

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34933 from tspmelo/wip-remove-entry

mgr/dashboard: Remove entryComponents

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #35505 from tspmelo/wip-ng-dropdown

mgr/dashboard: Use ng-bootstrap for Dropdown

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #32111 into master

* refs/pull/32111/head:
test_cephfs_shell: run cephfs-shell with conf file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge PR #35539 into master

* refs/pull/35539/head:
vstart_runner: set omit_sudo to True by default

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #35608 into master

* refs/pull/35608/head:
cephfs_features: add metric collection feature support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35578 from mgfritch/cephadm-infer-fsid-ukn

cephadm: skip any invalid fsid(s) during infer

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35567 from ceph/cephadm_annotate_scheduler

mgr/cephadm: annotate scheduling code with comments

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35492 from liewegas/wip-bocloud

doc/foundation: drop BoCloud

Merge pull request #35581 from bk201/ganesha-doc-pool-existence

mgr/cephadm: document pool requirement for NFS-Ganesha service

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Alexandra Settle <asettle@suse.com>

Merge pull request #35225 from ishanrai05/bootstrap-variables

mgr/dashboard: use bootstrap variables for colors

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/prometheus: automatically discover RBD pools for stats gathering

To simplify configuration, the existing 'rbd_stats_pools' config option
now accepts '*' as a wildcard for the pool and/or namespace portion of
the <pool>[/<namespace>] spec.

Fixes: https://tracker.ceph.com/issues/46034
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35559 from xiexingguo/wip-split-child-stat

osd/PeeringState: fix info.stats.up[acting] of splitting child

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: yanjun <yan.jun8@zte.com.cn>

test_cephfs_shell: run cephfs-shell with conf file

And set colors to False and debug to True in this conf file.

stdout captured by Python code in this testsuite can contain the colour
codes along with the expected string output. Setting colors to False in
cephfs-shell.conf should fix this.

Setting debug to False should help with analyzing test failures since
it enables printing messages that helps debugging (stacktrace for
cephfs-shell code, for example).

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35395 from kshtsk/wip-registry-conf

qa/tasks/cephadm: setup site based container registry

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35277 from aclamk/wip-limit-wal-sharding

kv/RocksDBStore: Set size for WALs in sharding mode.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #35547 from adk3798/cephadm-45097

cephadm: UX: Change error message when 'orch host add <host>' fails

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #35552 from jan--f/systemd-osd-libexec

systemd/ceph-osd: ceph-osd-prestart.sh now lives in /usr/libexec

Reviewed-by: Neal Gompa <ngompa13@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35558 from xiexingguo/wip-45991

osd/PG: fix history.same_interval_since of merge target again

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35497 from ifed01/wip-ifed-remove-wal-preext

os/bluestore: remove preextended WAL support.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35549 from mgfritch/cephadm-iscsi-strip-version

cephadm: strip whitespace from iscsi version

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #34643 from aclamk/wip-fix-bluefs-log-dump-v2

os/bluestore/BlueFS: Make log_dump possible without opening bluefs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35146 from wjwithagen/wjw-fix-shufflerandom

rgw: replace deleted std function: randomshuffle

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35609 from tchaikov/wip-mgr-volumes-nfs-fstring

mgr/volumes/nfs: use f-string for creating Exception

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #35466 from ishanrai05/quotes

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35614 from tchaikov/wip-install-deps-enable-required-crb-repo

install-deps.sh: do not enable unavailable repos

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #35295 from rhcs-dashboard/fix-snapshot-dropdown

mgr/dashboard: cropped actions menu in nested details

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35493 from mgfritch/orch-service-spec-validate

python-common: fix ServiceSpec validation

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

install-deps.sh: do not enable unavailable repos

should be specific when enabling codeready-builder repos, there is
chance that some repos are just not available, while the required one
is. for instance, "codeready-builder-for-rhel-8-x86_64-eus-debug-rpms"
might not be available. and in that case, `install-deps.sh` just fails.

so in this change, only the required one is enabled. see also
https://fedoraproject.org/wiki/EPEL

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/volumes/nfs: use f-string for creating Exception

for better understanding the reason of the failure

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephfs_features: add metric collection feature support

In client side we need to known whether mds support the metric
collection or not, or when mds server receieves the metric message
from client side and if it couldn't recognise, the mgr will close
the socket connection directly.

Fixes: https://tracker.ceph.com/issues/46041
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge PR #32288 into master

* refs/pull/32288/head:
test_cephfs_shell: remove test_every_shell_cmd_at_invocation
test_cephfs_shell: catch specific exception instead of catching all
cephfs-shell: set return value before exiting
test_cephfs_shell: re-arrange TestCephFSShell's methods
cephfs-shell: re-arrange code under __main__
test_cephfs_shell: add new test for shell cmd at invocation
cephfs-shell: launch shell only if there are no args following

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

osd/PeeringState: fix history.same_interval_since of merge target again

The symptom looks much like we see in
https://tracker.ceph.com/issues/37654.

The root cause is that both merge source and target could be
fabricated PGs (aka placeholders), hence merge target's
same_interval_since could remain 0 after merge.

Fix by adjusting history.same_interval_since to last_epoch_clean
reported by these PGs were found to be ready for merge.
This peer is going to be ignored/purged by primary anyway later
when peering is done.

Fixes: https://tracker.ceph.com/issues/45991
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #35307 from aclamk/wip-rocksdb-sharding-dump

kv/RocksDBStore: Fixes dump of batch commited to rocksdb

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #35562 from tchaikov/wip-drop-copr-repo

install-deps.sh: drop ktdreyer/ceph-el8 copr repo

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #35602 from tchaikov/wip-vstart-ganesha-comment

vstart.sh: update comment to use ganesha from CentOS Storage SIG

Reviewed-by: Varsha Rao <varao@redhat.com>

test: flush_pg_stats() ignore OSDs that don't respond to getting sequence

This eliminates bogus errors in the logs and returned from flush_pg_stats()

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Warn when too many reads are repaired on an OSD

Include test case
Configurable by setting mon_osd_warn_num_repaired (default 10)
Ignore new health warning with random eio injection test

Fixes: https://tracker.ceph.com/issues/41564
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #35510 from mgfritch/cephadm-bootstrap-timeout

cephadm: increase is_available timeout 30s -> 60s

Merge pull request #35585 from tchaikov/wip-test-mgr-dashboard-smoke-random-port

test/mgr/mgr-dashboard-smoke: listen on random ports

Reviewed-by: Laura Paduano <lpaduano@suse.com>

test/librbd: make TestLibRBD.QuiesceWatchError pass

when a duplicate quiesce notification is received

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35366 from tspmelo/wip-npm-20-05

mgr/dashboard: Update npm packages

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35557 from dillaman/wip-librbd-asio

librbd: initial boost asio integration

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35412 from sebastian-philipp/doc-cephadm-concepts-fqdn

doc/cephadm: Add chapter about FQDNs

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: document pool requirement for NFS-Ganesha service

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

vstart.sh: update comment to use ganesha from CentOS Storage SIG

see https://download.nfs-ganesha.org/3/LATEST/RHEL/

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/mgr/mgr-dashboard-smoke: listen on random ports

in hope to reduce the chance to listen an already used port while
running multiple "make check" instances in parallel.

See-also: http://tracker.ceph.com/issues/36737
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: use bootstrap variables for colors

Fixes: https://tracker.ceph.com/issues/38891
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>

mgr/dashboard: Use a unified quoting style

Use single quotes across all UI elements such as notifications

Fixes: https://tracker.ceph.com/issues/36070
Signed-off-by: Ishan Rai <ishanrai05@gmail.com>

Merge pull request #35524 from sebastian-philipp/cephadm-rm-container-storage

cephadm: call `podman rm --storage`

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35317 from dsavineau/doc_radosgw_key_typo

doc/radosgw/admin: fix key subuser typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35541 into master

* refs/pull/35541/head:
doc/dev: mention -l, -e and -n among useful options
doc/dev: add info on how to retest QA changes without rebuilding

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephadm: skip any invalid fsid(s) during infer

Fixes: https://tracker.ceph.com/issues/45999
Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: sort the list of inferred fsids

$ cephadm shell
ERROR: Cannot infer an fsid, one must be specified: ['1d5df33f-eb94-4a4f-b192-1d5e770ed0e7', 'unknown']
$ cephadm shell
ERROR: Cannot infer an fsid, one must be specified: ['unknown', '1d5df33f-eb94-4a4f-b192-1d5e770ed0e7']

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: UX: Change error message when 'orch host add <host>' fails

Instead of printing out a traceback if adding the host fails
during bootstrapping process, should now print error message
telling user host failed to be added

Fixes: https://tracker.ceph.com/issues/45097
Signed-off-by: Adam King <adking@redhat.com>

doc/cephadm: Add chapter about FQDNs

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Update npm packages

Fixes: https://tracker.ceph.com/issues/45949
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35561 from tchaikov/wip-crimson-disable-bluestore-with-allocator

crimson/os: disable alienized bluestore if using seastar allocator

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/cephadm: annotate scheduling code with comments

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35472 from varshar16/wip-vstart-stop-cephadm

stop.sh: Kill cephadm deployed daemons

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: call `podman rm --storage`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>