Merge PR #39696 into master

* refs/pull/39696/head:
qa/suites/rados/monthrash: crank up client side debugging w/ mon thrashing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39455 into master

* refs/pull/39455/head:
doc/man/8/ceph: document --max option
src/test/osd/safe-to-destroy: adjust test
ceph: print command output to stdout even on error
mgr/DaemonServer: include details in 'osd ok-to-stop' output
mgr: add --max <n> to 'osd ok-to-stop' command
mgr: relax osd ok-to-stop condition on degraded pgs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39629 into master

* refs/pull/39629/head:
crush/CrushWrapper: rebuild shadow tree on 'osd crush reweight-subtree'
crush/CrushWrapper: update shadow trees on update_item()

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39674 into master

* refs/pull/39674/head:
mon/OSDMonitor: fix safety/idempotency of {set,rm}-device-class

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #39728 into master

* refs/pull/39728/head:
mon/PGMap: include the per-pool usage breakdown in pg dump

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39705 from smithfarm/wip-fix-s390x-build-regression

rpm: re-disable SUSE lttng build on z390x

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39733 from mgfritch/cephadm-test-multiple-digest

cephadm: add multi-digest test

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: add multi-digest test

quick unit test for inspect of a container with multiple digests

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39648 from liewegas/cephadm-spec-config

mgr/cephadm: add config section to ServiceSpec

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/man/8/ceph: document --max option

Signed-off-by: Sage Weil <sage@newdream.net>

src/test/osd/safe-to-destroy: adjust test

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: print command output to stdout even on error

Currently in the case where the mon returns a command error code, we print
the error stream and Error ... message but not the command output.  Usually
there isn't any, so we haven't noticed until now, but there is not reason
why shouldn't return both an error code and some output.

Restructure the code so that the error message goes *after* the JSON output,
where it will be a bit more obvious to the user (if the stdout scrolled
the terminal, for instance).  (This is not a change in behavior since
previously we weren't seeing the stdout at all.)

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39686 into master

* refs/pull/39686/head:
qa/ceph_fuse: don't use createfs anymore while mounting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39664 into master

* refs/pull/39664/head:
pybind/cephfs: DT_REG and DT_LNK values are wrong

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #39574 into master

* refs/pull/39574/head:
client: wake up the front pos waiter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39294 into master

* refs/pull/39294/head:
client: add testcase for ceph_ll_lookup_vino
client: add ceph_ll_lookup_vino
client: make _lookup_ino take a vinodeno_t
client: stop doing unnecessary work in ll_lookup_inode

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #39538 into master

* refs/pull/39538/head:
mgr/telemetry: fix 'channel_indent' typo
mgr/telemetry: check if 'ident' channel is active

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39429 into master

* refs/pull/39429/head:
qa/tasks/mgr/dashboard: identify cluster in telemetry test

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #39709 into master

* refs/pull/39709/head:
qa: use tcmalloc with valgrind in fs:valgrind

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39714 from zdover23/wip-doc-cephadm-typo-togeter-2021-Feb-26

doc/cephadm: s/togeter/together/

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: validate config options at apply time

Make sure config options are valid/exist.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: disallow managed options in ServiceSpec config section

If a service is managing a config option, prevent the user from specifying
it in the spec.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: add config section to ServiceSpec

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39644 from liewegas/cephadm-service-id

cephadm: include service_name in unit.meta file

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: s/togeter/together

Exactly as it says in the title; this is a typo
I noticed while reading over the cephadm guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39713 from tchaikov/wip-mgr-rbd-support-type

mgr/rbd_support: fix type annotation of mirror_info

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39551 from sebastian-philipp/doc-cephadm-reduce-orch-cli

doc/cephadm: Restructure orchestrator docs

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: remove Orchestrator CLI from cephadm toc

not much useful information there anymore.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move host labels to host mgmt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group MDS sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Add iscsi

Fixes https://tracker.ceph.com/issues/49488

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group NFS sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: rename monitoring chapter title

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group MON sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: make custom containers its own chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group RGW mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move scheduler topic to service mgmt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move unmanaged=true to service-mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group general service mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group OSD mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Move FQDN chapter to host mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Move SSH config from operations to host-mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group host mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39692 from zdover23/wip-doc-dev-t8y-debugging-analyzing-rewrite-2021-Feb-25

doc/dev: t8y - rewrite "a. & d. a t8y job"

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: t8y - rewrite "a. & d. a t8y job"

This PR rewrites the "Analyzing and Debugging a
Teuthology Job" section of the "Debugging Tips"
page of the Teuthology User Guide in the Developer
Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39614 from sebastian-philipp/cephadm-again-test-ubuntu_2004_podman

qa/cephadm: Test again podman on ubuntu 20.04

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rbd_support: fix type annotation of mirror_info

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38604 from cybozu/doc-fix-the-meaning-of-min_size

doc: fix the meaning of default_min_size of pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39684 from xxhdx1985126/wip-alienstore-parallelism-improvement

crimson/os/alienstore: improve alienstore's write parallelism

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39602 from tchaikov/wip-mgr-flux-ann

mgr: add type annotations to BaseMgrModule and influx

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39371 from tchaikov/wip-mgr-rbd-support-ann

mgr/rbd_support: add type annotation and cleanups

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

crimson/os/alienstore: improve alienstore's write parallelism

replace the grand per store tp_mutex with a finer grained per-collection
lock for better concurrency

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

mon/PGMap: include the per-pool usage breakdown in pg dump

We started accounting for this ages ago, but we don't expose it anywhere.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39666 from zdover23/wip-doc-dev-t8y-rerunning-tests-rewrite-2021-Feb-24

doc/dev: rewrite t8y "re-running tests"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39694 from zdover23/wip-doc-dev-t8y-debugging-triaging-rewrite-2021-Feb-25

doc/dev: rewrite t8y "triaging" section

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39695 from zdover23/wip-doc-dev-t8y-debugging-adding-sections-2021-Feb-26

doc/dev: add t8y debug section names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39697 from zdover23/wip-doc-dev-t8y-debug-reporting-the-issue-rewrite-2021-Feb-26

doc/dev: rewrite t8y "reporting the issue"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39707 from petrutlucian94/overlapped

rbd: fix mingw 5.0.3 compatibility issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: use tcmalloc with valgrind in fs:valgrind

Follow-up: dc64ccf063f1ddf843fae4d2621ec25187f56135
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rbd: fix mingw 5.0.3 compatibility issue

GetOverlappedResultEx isn't avaialable when using mingw 5.0.3
(default on Ubuntu Bionic).

Since we don't really need the extra parameters, we'll switch
to GetOverlappedResult.

We're doing this mostly for testing purposes as the mingw 5.0.3
runtime is known to be broken.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

cephadm: provide meta during bootstrap

This is slightly gross, but we need ctx.meta_json for the bootstrap case,
which deploys a mon and mgr.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: put service_name in unit.meta and use it when available

Inferring service_name from the daemon name is error-prone.

Fixes: https://tracker.ceph.com/issues/46219
Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: accept arbitrary dict via --meta-json

e.g., --meta-json '{"foo": "bar"}'

Signed-off-by: Sage Weil <sage@newdream.net>

rpm: re-disable SUSE lttng build on z390x

This partially reverts 2b1e646f7aade3135a98c505111ac7ebef5e93a6 which
mistakenly changed a line inside an "%if 0%{?suse_version}" conditional.

Fixes: 2b1e646f7aade3135a98c505111ac7ebef5e93a6
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #39649 from Daniel-Pivonka/fixorphaninitialdaemons

cephadm: fix bug in orphan-initial-daemons logic

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #39613 from sebastian-philipp/DaemonDescriptionStatus

mgr/orch: Add DaemonDescriptionStatus

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #39672 from ceph/wip-159-notes

doc/releases/octopus.rst: 15.2.9 Release Notes

doc/dev: rewrite t8y "reporting the issue"

This commit rewrites the "Reporting the Issue"
section, reflecting my better understanding of the
various tools and parties in play. I also added a
TL;DR-style summary (or argument) paragraph at the
top of the section.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa/suites/rados/monthrash: crank up client side debugging w/ mon thrashing

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev: add t8y debug section names

This commit adds two section headings to the
Debugging section of the Teuthology Guide:

  1. Viewing Test Results
  2. Identifying Failed Jobs

The text as it stood before seemed to call out for
this explicit sectioning of material, so I've answered
that call.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: rewrite t8y "triaging" section

This commit simplifes and clarifies the "Triaging
the Cause of Failure" section in the Teuthology
Guide in the Developer Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev: rewrite t8y "re-running tests"

This PR rewrites the "re-running tests" section
so that its elegance and readability are improved.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa/ceph_fuse: don't use createfs anymore while mounting

createfs was removed codebase since it didn't created FS regardless of
its value.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39598 from tchaikov/wip-crimson-op-sequencer

crimson/osd: refactor OperationRepeatSequencer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39659 from tchaikov/wip-pybind-rados-cleanup

pybind/rados: remove rados_nobjects_list_next() from .pxi

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/{prometheus,telemetry}: appease mypy

update to adapt the type annotation of MgrModule.list_servers()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/ceph_module: add type annotation to BaseMgrModule

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: use empty data points to signal shutting down

to appease mypy, which prefer a queue with elements of the same type.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: avoid casting variable

self.config is a dict with elements of different types, to appease mypy,
we would need to cast the indexed item to the expected type. another way
is to use f-string, so the variable are always printed using `__str__`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: use get_module_option() for typed option

get_module_option() cast the option to the type claimed in its defition,
also, the returned value are always validated before it is originally
set. so there is no need to repeat this in mgr module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: use "N/A" for unknown hostname

in theory, there is chance that get_metadata() returns None, so let use
"N/A" in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: add typing annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: specify range of threads

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: specify the type of options

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/mgr_module correct typing of self_test

self_test() is allowed to return str

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: define commands uing CLICommand

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/influx: define options using Option

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38937 from myoungwon/fix-snap-refcount

src/test: fix to avoid fail notification when testing manifest refcount

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #39610 from tchaikov/wip-mgr-set-option

mgr: raise if set_module_option() fails

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39320 from ifed01/wip-ifed-fix-huge-bluefs-write

os/bluestore: fix huge(>4GB) writes from RocksDB to BlueFS.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #39600 from tchaikov/wip-mgr-zabbix-desc

mgr/zabbix: add desc to commands

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39676 from liewegas/fix-kv

mon/KVMonitor: fix 'osd new' cross-service commit

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38914 into master

* refs/pull/38914/head:
qa: bump osd heartbeat grace for ffsb workload

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #39498 into master

* refs/pull/39498/head:
test: use std::atomic<bool> instead of volatile for cb_done var

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39183 into master

* refs/pull/39183/head:
mon: simplify fs 'services' view of 'ceph status'
mds: add volumes + status to data: section of 'ceph status'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/rados: remove rados_nobjects_list_next() from .pxi

this is a follow-up of da5d4c813ffdd391da54dcf5022763616bab4b21

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39668 from zdover23/wip-doc-dev-t8y-rewrite-naming-ceph-ci-branch-2021-Feb-25

doc/dev: t8y-naming-ceph-ci-branch rewrite

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39140 from sseshasa/wip-dmclock-profiles-improvements

osd: Refinements to mclock built-in profiles implementation.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #38921 from lixiaoy1/pwl_teuthology

qa: add tests for persistent writeback cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/KVMonitor: fix 'osd new' cross-service commit

When we converted ConfigKeyService to KVMonitor, we didn't correctly
change this to propose_pending(), which mean that the kv change wasn't
captured in the paxos transaction.

Fixes: bb7ebc41532aeb23cff2241ab07b3f01c2f57ddd
Fixes: https://tracker.ceph.com/issues/49460
Signed-off-by: Sage Weil <sage@newdream.net>