mgr/cephadm:Document the cephadm config-check feature

This patch updates the docs to describe the config-check feature,
describing how these checks can be enabled and managed.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:fix to resolve mypy issue

Build of ceph metadata needed addition type hints.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:add unit test for the lookup_check helper

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Drop active healthcheck during a disable request

The healthcheck could already be active when the admin attempts
to disable it. This patch removes the related healthcheck if it's set
during a config-check disable request.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Added helper function to return a specific healthcheck

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:unit test added for nics better than most

checks that we're not raising a healthcheck for a host if it's
nic speed it better than the rest!

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:skip an alert if the linkspeed is better than most

The logic was issuing a healthcheck if the linkspeed was different
to the majority. But if the difference is good (i.e. better!) we should
not be raising a healthcheck

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:fix mypy warning

Since switching how the roles for a host are determining the
type hint was missed..this patch addresses that

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Remove check from ceph metadata gathering

A check for _ceph_get_server was included for unit testing, but
the tests have been updated to make this obsolete.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Add unit test for hosts without public network NIC

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Minor updates to address review comments

Changes to reflect review comments
- picked up on subscribed = unknown state
- using get_daemon_types() call
- use log.exception more
- changed logic and errors from the public_network check

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Added CLI interface for the configuration checker

Patch to add CLI commands to show and manage the state of the
configuration checker feature

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Multiple updates related to the addition of the CLI

Some changes needed to support the introduction of the CLI commands
used to manage the cephadm checks. For example, the main Cephadm
check class now interacts with the keystore directly to determine
status, and provides support for commands like ls to list the
check definitions. In addition the main class now handles existing
configuration checks and ensure that the stored state in the keystore
matches the checks defined by the module

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Moved 'ownership' of the checker to cephadm

Initial implementation used the Serve class as the owner of the
configuration checker. This patch moves the checker up to the
cephadm module itself, to make the CLI command logic cleaner

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Unit tests updated to account for upgrades

Upgrades may change the config checks, so the tests now validate
that new checks and old bogus checks are handled correctly

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Updates to CephadmConfigChecks class

Multiple updates to ensure
- mgr health checks are raised correctly
- checks are independent and may be disabled

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:Adds unit tests for the CephadmConfigChecks class

Add unit tests to test suite to verify functionality. The unit tests use
a sample host definition and scale that to simulate a cluster to run
the tests against

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:add module option to enable configuration checks

Adds config_checks_enabled (bool) option

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm:added ceph version consistency check

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: added config checker to main serve loop

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: adding check logic

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: resolve rebase conflicts

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #39601 from badone/wip-ceph_test_rados_api_watch_notify-enoent-failure

tests: ceph_test_rados_api_watch_notify: Allow for reconnect

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39680 into master

* refs/pull/39680/head:
mds: allow `fs authorize` command to work

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39710 into master

* refs/pull/39710/head:
qa: run fs:verify on all distros

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39725 into master

* refs/pull/39725/head:
qa: delete all fs during tearDown

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39732 from mychoxin/resolve_compile_error

test/librbd: resolve compile error on centos

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39731 from mychoxin/use_unified_zero_func

rbd: use portable zero-ing memory function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39521 from yuvalif/wip-yuval-fix-49322

rgw/notification: add exception handling for persistent notification thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/librbd: resolve compile error on centos

Signed-off-by: YuanXin <yuanxin@didiglobal.com>

Merge pull request #39743 from tchaikov/wip-crimson-cleanup

crimson/osd: capturing this in a better way

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39652 from athanatos/sjust/wip-crimson-omap

crimson: btree omap_manager rework with other misc fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39631 into master

* refs/pull/39631/head:
osd/OSDMap: don't warn on NEW osd ids
mon/OSDMonitor: behave if inc map sets weight on not-yet-existing OSD
mon/OSDMonitor: do not mark newly created OSDs OUT

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39696 into master

* refs/pull/39696/head:
qa/suites/rados/monthrash: crank up client side debugging w/ mon thrashing

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: don't warn on NEW osd ids

If we allocate a new OSD, don't raise a health alert about it.

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/osd: keep ClientRequest alive during `seastar::repeat()`

before this change, `this` is wrapped by a smart pointer which is
in turn captured by multiple continuations. the continuation with the
longest life cycle is the one passed to `seastar::repeat()`. where
the underlying `repeater` captures the continuation as its member variable.
and `repeater` is not destroyed before `seastar::repeat()` returns.

so, we only need to capture `opref` in the continuation passed to
`seastar::repeat()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use finally for capturing this

* no need to capture this, it is already enclosed by `ref`
* use finally for capturing `ref`, and for printing out the "complete"
  logging message. we could use a different branch for printing out
  the failure message if any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reduce the indent

instead of using a lambda, use a future variable for reducing the indent
level.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: use portable zero-ing memory function

Signed-off-by: YuanXin <yuanxin@didiglobal.com>

Merge PR #39455 into master

* refs/pull/39455/head:
doc/man/8/ceph: document --max option
src/test/osd/safe-to-destroy: adjust test
ceph: print command output to stdout even on error
mgr/DaemonServer: include details in 'osd ok-to-stop' output
mgr: add --max <n> to 'osd ok-to-stop' command
mgr: relax osd ok-to-stop condition on degraded pgs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39629 into master

* refs/pull/39629/head:
crush/CrushWrapper: rebuild shadow tree on 'osd crush reweight-subtree'
crush/CrushWrapper: update shadow trees on update_item()

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39674 into master

* refs/pull/39674/head:
mon/OSDMonitor: fix safety/idempotency of {set,rm}-device-class

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #39728 into master

* refs/pull/39728/head:
mon/PGMap: include the per-pool usage breakdown in pg dump

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39705 from smithfarm/wip-fix-s390x-build-regression

rpm: re-disable SUSE lttng build on z390x

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39733 from mgfritch/cephadm-test-multiple-digest

cephadm: add multi-digest test

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: add multi-digest test

quick unit test for inspect of a container with multiple digests

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39648 from liewegas/cephadm-spec-config

mgr/cephadm: add config section to ServiceSpec

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/man/8/ceph: document --max option

Signed-off-by: Sage Weil <sage@newdream.net>

src/test/osd/safe-to-destroy: adjust test

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: print command output to stdout even on error

Currently in the case where the mon returns a command error code, we print
the error stream and Error ... message but not the command output.  Usually
there isn't any, so we haven't noticed until now, but there is not reason
why shouldn't return both an error code and some output.

Restructure the code so that the error message goes *after* the JSON output,
where it will be a bit more obvious to the user (if the stdout scrolled
the terminal, for instance).  (This is not a change in behavior since
previously we weren't seeing the stdout at all.)

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39686 into master

* refs/pull/39686/head:
qa/ceph_fuse: don't use createfs anymore while mounting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39664 into master

* refs/pull/39664/head:
pybind/cephfs: DT_REG and DT_LNK values are wrong

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #39574 into master

* refs/pull/39574/head:
client: wake up the front pos waiter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39294 into master

* refs/pull/39294/head:
client: add testcase for ceph_ll_lookup_vino
client: add ceph_ll_lookup_vino
client: make _lookup_ino take a vinodeno_t
client: stop doing unnecessary work in ll_lookup_inode

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #39538 into master

* refs/pull/39538/head:
mgr/telemetry: fix 'channel_indent' typo
mgr/telemetry: check if 'ident' channel is active

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39429 into master

* refs/pull/39429/head:
qa/tasks/mgr/dashboard: identify cluster in telemetry test

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa: delete all fs during tearDown

During the ceph task Unwind, the MDS are stopped. If any file system
still exists, we will see failover messages in the cluster log.

Fixes: https://tracker.ceph.com/issues/49510
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39709 into master

* refs/pull/39709/head:
qa: use tcmalloc with valgrind in fs:valgrind

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39714 from zdover23/wip-doc-cephadm-typo-togeter-2021-Feb-26

doc/cephadm: s/togeter/together/

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: validate config options at apply time

Make sure config options are valid/exist.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: disallow managed options in ServiceSpec config section

If a service is managing a config option, prevent the user from specifying
it in the spec.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: add config section to ServiceSpec

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39644 from liewegas/cephadm-service-id

cephadm: include service_name in unit.meta file

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: s/togeter/together

Exactly as it says in the title; this is a typo
I noticed while reading over the cephadm guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39713 from tchaikov/wip-mgr-rbd-support-type

mgr/rbd_support: fix type annotation of mirror_info

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39551 from sebastian-philipp/doc-cephadm-reduce-orch-cli

doc/cephadm: Restructure orchestrator docs

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: remove Orchestrator CLI from cephadm toc

not much useful information there anymore.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move host labels to host mgmt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group MDS sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Add iscsi

Fixes https://tracker.ceph.com/issues/49488

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group NFS sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: rename monitoring chapter title

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group MON sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: make custom containers its own chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group RGW mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move scheduler topic to service mgmt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move unmanaged=true to service-mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group general service mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group OSD mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Move FQDN chapter to host mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Move SSH config from operations to host-mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group host mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39692 from zdover23/wip-doc-dev-t8y-debugging-analyzing-rewrite-2021-Feb-25

doc/dev: t8y - rewrite "a. & d. a t8y job"

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: t8y - rewrite "a. & d. a t8y job"

This PR rewrites the "Analyzing and Debugging a
Teuthology Job" section of the "Debugging Tips"
page of the Teuthology User Guide in the Developer
Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39614 from sebastian-philipp/cephadm-again-test-ubuntu_2004_podman

qa/cephadm: Test again podman on ubuntu 20.04

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rbd_support: fix type annotation of mirror_info

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38604 from cybozu/doc-fix-the-meaning-of-min_size

doc: fix the meaning of default_min_size of pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39684 from xxhdx1985126/wip-alienstore-parallelism-improvement

crimson/os/alienstore: improve alienstore's write parallelism

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39602 from tchaikov/wip-mgr-flux-ann

mgr: add type annotations to BaseMgrModule and influx

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39371 from tchaikov/wip-mgr-rbd-support-ann

mgr/rbd_support: add type annotation and cleanups

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

crimson/os/alienstore: improve alienstore's write parallelism

replace the grand per store tp_mutex with a finer grained per-collection
lock for better concurrency

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/futurized_store: fix return comment on omap_get_values

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/omap_manager: rework to be null safe and adjust interfaces

Adjusts a few things in omap_manager/btree:
- fix to be null safe, simplify/rename iterator interface
- update interface methods to match futurized_store.h
- misc whitespace fixes

Signed-off-by: Samuel Just <sjust@redhat.com>

mon/PGMap: include the per-pool usage breakdown in pg dump

We started accounting for this ages ago, but we don't expose it anywhere.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39666 from zdover23/wip-doc-dev-t8y-rerunning-tests-rewrite-2021-Feb-24

doc/dev: rewrite t8y "re-running tests"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39694 from zdover23/wip-doc-dev-t8y-debugging-triaging-rewrite-2021-Feb-25

doc/dev: rewrite t8y "triaging" section

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39695 from zdover23/wip-doc-dev-t8y-debugging-adding-sections-2021-Feb-26

doc/dev: add t8y debug section names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39697 from zdover23/wip-doc-dev-t8y-debug-reporting-the-issue-rewrite-2021-Feb-26

doc/dev: rewrite t8y "reporting the issue"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39707 from petrutlucian94/overlapped

rbd: fix mingw 5.0.3 compatibility issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: run fs:verify on all distros

It's believed this is no longer a problem now that we use tcmalloc.

Fixes: https://tracker.ceph.com/issues/49391
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>