ceph-volume: avoid calling zap_lv with a LV-less VG

Fixes: https://tracker.ceph.com/issues/44125
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33233 from tchaikov/wip-crimson-misc

crimson: fix build with GCC-10

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33221 from liewegas/fix-44065

ceph.spec: Recommend (but do not require) podman

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: correct type of callback func

`md_config_obs_t` is defined as
```
using md_config_obs_t = ceph::md_config_obs_impl<ConfigProxy>;
```
in `common/config_obs.h`. it takes advantage of a fact that
somebody exposes the correct version of `ConfigProxy` to the global
namespace. this is intended to fulfill the needs of other components
which expects `md_config_obs_t`. otherwise we need to specify
`ceph::md_config_obs_impl<ceph::ConfigProxy>` or
`ceph::md_config_obs_impl<crimson::common::ConfigProxy>` depending on
if we are programming crimson or not.

but in this case, we are actually defining
`crimson::common::ConfigProxy`, so it'd be better to define
`md_config_obs_t` explicitly instead relying on "somebody" which exposes
`ConfigProxy`. and `ConfigObserver` is defined using the current
`ConfigProxy`, so it's more correct and more readable than using
`md_config_obs_t` defined in `common/config_obs.h`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable concepts in boost::asio

GCC-10 and Clang choke when compiling a concept constrained with
its template parameter, which is in turn another concept. as a
workaround of the bug of boost::asio, we should disable concepts
support in it. but it's nice to enable it when the compiler is
able to use concepts to do some compile time checkings even the
concepts are not compliant to C++20.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33174 from jan--f/c-v-functional-centos8

ceph-volume: use centos8 for functional testing

Merge pull request #33224 from howard0su/clean

src: polish the wording

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33203 from howard0su/wip_fix_dpdk

msg/async/dpdk: Fix build when DPDK enabled

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec: Recommend (but do not require) podman

1- Strictly speaking, docker is okay too.
2- We don't want podman (or docker) installed inside the container image,
but we do need the cephadm package.

Fixes: https://tracker.ceph.com/issues/44065
Signed-off-by: Sage Weil <sage@redhat.com>

src: polish the wording

Signed-off-by: Jun Su <howard0su@gmail.com>

Merge PR #33211 into master

* refs/pull/33211/head:
qa/tasks/mgr/dashboard/test_orchestrator: support addr attribute in inventory

Reviewed-by: Sebastian Wagner <swagner@suse.com>

debian/control: make cephadm recommend (but not require) docker

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33191 into master

* refs/pull/33191/head:
cephadm: shell: allow -e

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: shell: allow -e

Set environment variables for shell commands.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33207 into master

* refs/pull/33207/head:
doc/index: remove quick start from front page for now
doc/bootstrap: warn that cephadm is not for production yet

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/index: remove quick start from front page for now

1- wait until this works on a production version,
2- maybe a shorter procedure, or just the initial steps and then
link off to bootstrap.rst?

This effectively reverts 79040c2 (PR #32531)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/bootstrap: warn that cephadm is not for production yet

We can remove this when octopus is released, and/or adjust the language.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33214 from yuriw/wip-yuriw-crontab-master

qa/tests: reduced runs for nautilus, added runs for octopus

ceph-volume: port lvm functional tests to centos

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: port batch functional tests to centos8

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #33139 into master

* refs/pull/33139/head:
mgr/cephadm: catch exception in _check_hosts()
mgr/cephadm: do not crash module on exception in serve thread

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33175 into master

* refs/pull/33175/head:
mgr/cephadm: Add some more type annotations
mgr/orchestrator: unify StatelessServiceSpec and StatefulServiceSpec

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33189 into master

* refs/pull/33189/head:
mgr/cephadm: fix redeploy (etc) of crash containers
cephadm: do not use special unit, naming for crash agent
cephadm: 'crash' is a ceph daemon

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/tests: reduced runs for nautilus, added runs for octopus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #33176 from jmolmo/bz_43972

mgr/rook: Fix error creating OSD's

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks/mgr/dashboard/test_orchestrator: support addr attribute in inventory

The missing attribute `addr` causes error when loading dummy data.

Fixes: https://tracker.ceph.com/issues/44054
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #33027 into master

* refs/pull/33027/head:
os/bluestore/BlueFS: reclaim unused too-granular extents first
os/bluestore/BlueFS: track bluefs owned extents that are too granular
os/bluestore/BlueFS: verify that fnodes respect the bluefs alloc_size
os/bluestore/BlueFS: align bluefs' owned extents to alloc_size

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #33188 into master

* refs/pull/33188/head:
qa/workunits/cephtool/test.sh: delete test_erasure pool

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33187 from dillaman/wip-44066

qa/workunits/rbd: override CEPH_ARGS when initializing the site name

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33002 from dillaman/wip-43613

rbd-mirror: initial snapshot-based mirroring bootstrap logic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33054 from cbodley/wip-rgw-pubsub-syncstop

rgw: use new Stopped state for special handling of 'bucket sync disable'

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

msg/async/dpdk: Fix build when DPDK enabled

Signed-off-by: Jun Su <howard0su@gmail.com>

Merge pull request #33199 from tchaikov/wip-crimson-do-with

crimson/common: more friendly to seastar::do_with()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: more friendly to seastar::do_with()

so we don't need to do the box-unbox dance when callng
seastar::do_with() with multiple with'ed variables and
a continuation which returns `errator<...>::future`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33172 from tchaikov/wip-43582

ceph-monstore-tool: correct the key for storing mgr_command_descs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33177 from tchaikov/wip-clang-pybind

pybind: customize compiler before checking cflags

Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge PR #33135 into master

* refs/pull/33135/head:
mon: fix tell command registration
ceph.in: fix -h/help behavior for tell commands
ceph.in: remove '{osd,mds,mon} tell' deprecation warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33190 into master

* refs/pull/33190/head:
cephadm: report image name for stopped daemons
cephadm: record the image name to unit.image

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33186 from yuriw/wip-yuriw-crontab-master

qa/tests: removed all runs for luminous - EOL

Merge PR #33179 into master

* refs/pull/33179/head:
mgr/orchestrator: Use `pickle` to pass exceptions across sub-interpreters

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33184 into master

* refs/pull/33184/head:
qa/suites/rados/multimon/tasks/mon_clock_with_skews: whitelist MOST_DOWN
qa/suites/rados/multimon/tasks/mon_clock_with_skews: disable ntpd etc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33043 into master

* refs/pull/33043/head:
doc/releases: octopus draft notes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32965 from Dmole/patch-3

doc: quick-rbd.rst de-duplicate

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33181 into master

* refs/pull/33181/head:
qa/workunits/cephadm/test_cephadm: fix typo

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/releases: octopus draft notes

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix redeploy (etc) of crash containers

When deploying a crash container, the crash keyring is the crash keyring,
and there is not additional crash_keyring for the implicit crash
container.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not use special unit, naming for crash agent

Use the normal unit file, with crash.$hostname as the name.  This makes
the crash agent behave like all of the other services instead of being
special.  The exception is that the crash agent is still implicitly
deployed alongside other containers.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: 'crash' is a ceph daemon

- we can run ceph -v inside
- we need to mount the crash dir

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33131 into master

* refs/pull/33131/head:
mgr/orch: 'ceph orchestrator ...' -> 'ceph orch ...'

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33162 into master

* refs/pull/33162/head:
cephadm: KillMode=none in unit file
cephadm: clean up unit string syntax a bit

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: report image name for stopped daemons

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: record the image name to unit.image

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: delete test_erasure pool

Other parts of this script leave OSDs reweighted, which can make this test
fail to go fully clean.

 0   ssd 0.08789         osd.0          up  0.63213 1.00000
 1   ssd 0.08789         osd.1          up  0.63213 1.00000
 2   ssd 0.08789         osd.2          up  1.00000 1.00000

35.0 raw ([2,1,2147483647], p2) up ([2,1,2147483647], p2) acting ([2,1,2], p2)

Fix by just deleting this pool when we're done.

Fixes: https://tracker.ceph.com/issues/44067
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33185 from tchaikov/wip-44063

pybind/mgr: bump six to 1.14

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/workunits/rbd: override CEPH_ARGS when initializing the site name

The mirroring site name is stored in the MON config which requires
higher privledges than the standard "client.mirror" user.

Fixes: https://tracker.ceph.com/issues/44066
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: removed all runs for luminous - EOL

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: quick-rbd.rst de-duplicate

Remove code duplication from the dependency.

Signed-off-by: Tim <elatllat@gmail.com>

pybind/mgr: bump six to 1.14

otherwise some packages which uses six.ensure_text will fail, as it
was introduced by 1.12. and 1.14 is the latest stable version of
six. so let's use it instead.

Fixes: https://tracker.ceph.com/issues/44063
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: fix expectations for mirror snapshots

When snapshot-based mirroring is enabled, the primary image
should always have at least one mirror-based snapshot to help
distinguish it from an incomplete non-primary image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33137 into master

* refs/pull/33137/head:
mon/PGMap: fix digest summary of PGs by state
mon/PGMap: fix compat encoding of num_pg_by_state

Reviewed-by: yanjun <yan.jun8@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: fix tell command registration

Pass full cmddesc, not just prefix.

Broken by 3394b70ba18cffa49317141ff444ba9990d6170b

Fixes: https://tracker.ceph.com/issues/44050
Signed-off-by: Sage Weil <sage@redhat.com>

ceph.in: fix -h/help behavior for tell commands

- add "Tell $type commands" heading
- 'ceph tell mon.a -h' now works
- 'ceph tell mon.a prefix -h' also works

Signed-off-by: Sage Weil <sage@redhat.com>

pybind: check arguments also when searching for "clang"

if "ccache" is used, compiler[0] would be "ccache", and compiler[1]
would be "clang", in this case, we should search for "clang" until
an argument starting with "-" is found. it signals be the first
cflag passed to the compiler.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: convert filterfalse to list before returning

distutils expects a list not an iterator

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: customize compiler before checking cflags

otherwise we are always looking at the default compiler on 'unix' --
'cc'. and will fail to filter cflags not supported by clang, if we
are compiling the python bindings using clang.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: corrected error code from image_snapshot_create API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/orchestrator: Use `pickle` to pass exceptions across sub-interpreters

`copy.deepcopy` doesn't work to cross sub-interpreter boundaries.

Fixes: https://tracker.ceph.com/issues/43913
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add some more type annotations

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: unify StatelessServiceSpec and StatefulServiceSpec

Both classes are essentially equal. Keeping both just makes thigs
more compicated.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Fix error creating OSD's

Fixes: https://tracker.ceph.com/issues/43972
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #33035 from jdurgin/wip-target-ratio

mgr/pg_autoscaler: treat target ratios as weights

Reviewed-by: Sage Weil <sage@redhat.com>

doc: update mondb recovery script

to note that we also need to add mgr's key to monitor's keyring

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-monstore-tool: correct the key for storing mgr_command_descs

Fixes: https://tracker.ceph.com/issues/43582
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-monstore-tool: rename mon-ids in initial monmap

when ceph-mon starts, it checks to see if it's listed in the monmap, if
not it complains
```
no public_addr or public_network specified, and mon.a not present in
monmap or ceph.conf.
```
then bails out. normally, the monitor will try to rename its name in
monmap when performing "mkfs", but in our case, we are merely using the
"mkfs" monmap for passing the monmap built by ceph-monstore-tools, and
we don't actually go through the "mkfs" process. so, ceph-mon won't
rename when booting up.

in this change, user is allowed to specify the mon-ids in command line
when rebuilding mondb, the default mon-ids would be a,b,c,... if not
specified.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/pg_autoscaler: add unit tests for effective_target_ratio

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: remove unused imports

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: cleanup pool_id type

Keep it as an int so we don't have to cast back and forth.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: correct and simplify progress tracking

Reset the progress each time we make an adjustment, and track progress
from that initial state to that new target. Previously we were also
using the wrong target: the current pg_num_target, not the new value
(pg_num_final) that we set.

Look up the pool by name, not id, in _maybe_adjust(), since that is how it is
retrieved by osdmap.get_pools_by_name().

Dedupe some logic into PgAdjustmentProgress to simplify things.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: mention target_size_ratio change

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

doc: update autoscaler docs for target ratio

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: include effective target ratio in status

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: add warning when target bytes and ratio are both set

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: remove target ratio warning

Since the ratios are normalized, they cannot exceed 1.0 or overcommit
combined with target_bytes.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: use ratios >1 in pg_autoscaler test

Also check for pg_num_target being set correctly, rather than pg_num,
so the test doesn't depend on merging/splitting speed.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/pg_autoscaler: treat target ratios as weights

Normalize across pools so that it's simpler to use - this way you
don't have to adjust every other pool when you add one.

Handle pools with target_bytes by taking their capacity off the top,
and dividing the rest into the pools with a target_ratio.

If both target bytes and ratio are specified, ignore bytes. This
matches the docs and makes accounting simpler.

Fixes: https://tracker.ceph.com/issues/43947
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/orch: 'ceph orchestrator ...' -> 'ceph orch ...'

Less typing for all!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33126 into master

* refs/pull/33126/head:
doc/mgr/orchestrator_cli: update support table
mgr/deepsea: remove
mgr/ansible: remove

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33136 into master

* refs/pull/33136/head:
cephadm: fix ceph version probe

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: fix ceph version probe

docker returns '<no value>' if the label isn't present, in which case we
still need to run ceph -v.

Also, don't probe non-ceph (e.g., monitoring) containers.

Also, only probe each image id once.

Add a simple test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33133 into master

* refs/pull/33133/head:
qa/workunits/cephadm/test_cephadm.sh: make monitoring tests faster
qa/workunits/cephadm/test_cephadm: 2 OSDs is enough
cephadm: disable node-exporter cpu/memory limits for the time being

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/multimon/tasks/mon_clock_with_skews: whitelist MOST_DOWN

The skewed clock makes some mons miss elections.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33134 into master

* refs/pull/33134/head:
qa/workunits/cephadm/test_repos: don't try to use the refspec

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/multimon/tasks/mon_clock_with_skews: disable ntpd etc

Fixes: https://tracker.ceph.com/issues/43889
Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: update support table

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/deepsea: remove

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33117 into master

* refs/pull/33117/head:
qa/suites/upgrade/nautilus-x-singleton: ensure hit sets behave across upgrade
osd/PrimaryLogPG: use legacy timestamp rendering for hit_set objects
include/utime: allow legacy rendering of timestamp

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: fix digest summary of PGs by state

This fixes the PG state summary seen in 'ceph -s' to reflect PG state bits
beyond 32 bits.

Fixes: https://tracker.ceph.com/issues/44052
Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix compat encoding of num_pg_by_state

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32928 from ljishen/wip-rados-bench-latency

rados bench: fix the delayed checking of completed ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>