mgr/orch: combine 'orch apply <type> [<placement>]' into one command

Declutter the CLI!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/octopus' into octopus

Merge PR #33520 into octopus

* refs/pull/33520/head:
mgr/dashboard: Isolate each RBD component

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #33758 into octopus

* refs/pull/33758/head:
mgr/dashboard: Replace ng2-tree with angular-tree-component

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33941 into octopus

* refs/pull/33941/head:
mgr/dashboard: hide daemon table when orchestrator is disabled

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #33948 into octopus

* refs/pull/33948/head:
python-common: PlacementSpec.from_string: drop `host_pattern:` prefix
python-common: remove `all_hosts` from `PlacementSpec`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33777 into octopus

* refs/pull/33777/head:
rgw: svc.config_key_rados: get() warns if potentially insecure connection
rgw: svc.rados: add clog_warn()
mon: log monitor: add command level param
auth registry: helpers for checking secure method/mode

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33950 into octopus

* refs/pull/33950/head:
ceph.in: use os._exit when we don't shut down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33951 into octopus

* refs/pull/33951/head:
mgr/orch: dump service spec by name

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph.in: use os._exit when we don't shut down

If we experience a timeout, we don't bother shutting down rados, because
it can't currently handle one thread blocked (or running) library init
at the same time that another thread calls rados_shutdown().  This
behavior was recently introduced by c8f353c50bcd2a30be9e3600dba91912b8cd0429

However, sys.exit() runs all kinds of shutdown work that will also
interfere with the running librados threads.

Fix by using os._exit instead of sys.exit.

Fixes: https://tracker.ceph.com/issues/44566
Signed-off-by: Sage Weil <sage@redhat.com>

15.1.1

mgr/orch: dump service spec by name

add optional arg to allow a dump by name:
`ceph orch spec dump [svc_name]`

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #33953 into octopus

* refs/pull/33953/head:
doc: Move ceph-deploy docs to doc/install/ceph-deploy

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Move ceph-deploy docs to doc/install/ceph-deploy

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33924 into octopus

* refs/pull/33924/head:
doc/mgr: Enhance placement specs
doc/cephadm: Fix toc of Drive Groups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33891 into octopus

* refs/pull/33891/head:
mgr/cephadm: fix dump output by formatting to json first

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33905 into octopus

* refs/pull/33905/head:
install-deps.sh: remove the workaround to bootstrap a new venv
install-deps.sh: install additional repo for RHEL8
install-deps.sh: install rhel-7-server-optional-rpms on el7 only

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #33940 from tchaikov/octopus-release-notes-header

doc: use subsection for representing components in release notes

Reviewed-by: Sage Weil <sage@redhat.com>

doc/mgr: Enhance placement specs

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/releases: use h3 for rendering subsections in release notes

to highlight the components with subsection header, and allow readers to
navigate through them with TOC.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: overide the default margin for "ul.simple > li"

by default, the user agent's css sets `margin-block-end` of "ul ul" to
"0", which renders the unordered lists in release notes cluttered. as we
are using nested unordered list to present the changes in different
components.

in this change a customized css is added to reset the begin and end
margin of ul to 1em, which is the default value of top level ul.

Signed-off-by: Kefu Chai <kchai@redhat.com>

python-common: PlacementSpec.from_string: drop `host_pattern:` prefix

* `host_pattern` is works as any other host spec.
* Improved type annotations
* Added tests
* `mgr/cephadm/test_scheduling`: Remove tests that don't have a valid spec.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33867 from dillaman/wip-43590

librbd: optimize image copy state machine to use fast-diff

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: hide daemon table when orchestrator is disabled

The refresh time of services/daemons is also increased to 1 minute since
they are not updated so frequently.

Fixes: https://tracker.ceph.com/issues/44558
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #33930 into octopus

* refs/pull/33930/head:
mgr/cephadm: implement pause/resume to suspect non-monitoring background work
mgr/orch: add pause/resume hooks

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33868 into octopus

* refs/pull/33868/head:
mgr/rook: implement remove_daemons()
mgr/rook: fix mds/fs create/apply/update
mgr/rook: use ceph_daemon_id label consistently
mgr/rook: show filesystems in 'orch ls'
mgr/orch: 'age' -> 'ago' in 'orch ls' output
mgr/rook: initial pass at 'orch ls'
mgr/rook: include timestamps in 'orch ps'
mgr/orch: shorten container_id (hash) in cephadm, not orch ps
mgr/rook: include container_image_name in 'orch ps'
mgr/rook: fix list_daemons host arg

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33932 into octopus

* refs/pull/33932/head:
cephadm: fix datetime regexp to capture at most 6 digits

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Isolate each RBD component

Before when you accessed any RBD component, you were loading all 4 components
at the same time and each was doing frequent requests to the server.

Now it only loads 1 at a time.

Fixes: https://tracker.ceph.com/issues/44278
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33900 from dillaman/wip-44459

qa/suites/rbd: removed OpenStack tempest test cases

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33840 from trociny/wip-schedule-optimize-scan

mgr/rbd_support: don't scan pools that don't have schedules

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33914 into octopus

* refs/pull/33914/head:
mgr/orch: Cleanup python imports of the interface
mgr/orch: Properly handle NotImplementedError
pybind/mgr: Rm PersistentStoreDict

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rook: implement remove_daemons()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Replace ng2-tree with angular-tree-component

Fixes: https://tracker.ceph.com/issues/44450
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/rook: fix mds/fs create/apply/update

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: use ceph_daemon_id label consistently

In particular, for mds's, it's $fsname-$rank for mds.$fsname-$rank,
which is what we want.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: show filesystems in 'orch ls'

The count vs size discrepancy is weird since rook things in terms of
ranks (*2 if standby) vs just a count.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: 'age' -> 'ago' in 'orch ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: initial pass at 'orch ls'

This doesn't cover mds.* or rgw.* yet.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: include timestamps in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: shorten container_id (hash) in cephadm, not orch ps

This lets us present a longer, human-readable id for rook.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: include container_image_name in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: fix list_daemons host arg

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix datetime regexp to capture at most 6 digits

trailing `0` digits are not shown in the precision

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33821 from votdev/issue_43446_bucket_lock

mgr/dashboard: Create bucket with x-amz-bucket-object-lock-enabled

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #31997 from bk201/wip-43062

mgr/dashboard: support removing OSDs in OSDs page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33920 from LenzGr/update-404-image

mgr/dashboard: Updated octopus image on 404 page

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/cephadm: implement pause/resume to suspect non-monitoring background work

If the user does 'orch pause', suspend all background work that makes
actual changes.

Continue to do read-only operations, like checking host connectivity
and scraping daemon and device status.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33858 from tspmelo/wip-use-cd-modal

mgr/dashboard: Use ModalComponent in all modals

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/orch: add pause/resume hooks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33926 into octopus

* refs/pull/33926/head:
python: fixate mypy requirement to mypy==0.770

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33874 into octopus

* refs/pull/33874/head:
qa/suites/powercycle: install build deps for building xfstest

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #33813 into octopus

* refs/pull/33813/head:
osd/PGStateUtils: initialize NamedState::enter_time.
osd/PGStatueUtils: remove unused member variable

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33814 into octopus

* refs/pull/33814/head:
osd/OSDMap: stop encoding osd_state with >8 bits wide states only for old client

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: yanjun <yan.jun8@zte.com.cn>

Merge PR #33064 into octopus

* refs/pull/33064/head:
cephadm: add version to `command_ls` output
cephadm: add type checking to `update_filewalld`
cephadm: allow prepare-host to start an enabled service
cephadm: add type checking for `check_host` and `prepare_host`
cephadm: generalize logic for checking and enabling units
cephadm: add 'CEPH_CONF' to the NFS ganesha container envs
cephadm: trim nfs.json sample
qa/workunits/cephadm/test_cephadm.sh: systemctl stop nfs-server
qa/workunits/cephadm/test_cephadm.sh: make pgs available
cephadm: add some log lines
cephadm: check port in use
cephadm: add/remove nfs ganesha grace
cephadm: update firewalld with nfs service
qa/workunits/cephadm/test_cephadm.sh: add nfs-ganesha test
cephadm: add ganasha.conf
cephadm: add NFSGanesha deployment type
cephadm: consolidate list of supported daemons
cephadm: use keyword instead of positional args

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33855 into octopus

* refs/pull/33855/head:
mgr/cephadm: reconfigure ceph daemons when monmap changes

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33829 into octopus

* refs/pull/33829/head:
cephadm: log when we infer the image
cephadm: Avoid `inspect` call by using ceph label
cephadm: Infer ceph image

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33919 from LenzGr/octopus-documentation

documentation: Updated Dashboard Features, improved flow

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #33871 into octopus

* refs/pull/33871/head:
qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

python-common: remove `all_hosts` from `PlacementSpec`

replace it with `PlacementSpec(host_pattern='*')`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33890 into octopus

* refs/pull/33890/head:
doc/install: rethink docs install page (similar to PR#33613)
doc/install: Rethink doc/install

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/cephadm: Fix toc of Drive Groups

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python: fixate mypy requirement to mypy==0.770

I don't like suprises when upstream publishes a new mypy version.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Append "Modal" to all modal components names

Fixes: https://tracker.ceph.com/issues/44547
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use ModalComponent in all modals

Fixes: https://tracker.ceph.com/issues/44547
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33902 from dillaman/wip-test-rbd-integration

test/cli-integration: fixed spacing issue for RBD formatted tables

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/cephadm: reconfigure ceph daemons when monmap changes

This keeps ceph.conf fresh.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33898 from dillaman/wip-44567

qa/workunits/rbd: wait for nbd map to close after unmap

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/cephadm: fix dump output by formatting to json first

Signed-off-by: Joshua Schmid <jschmid@suse.de>

cephadm: add version to `command_ls` output

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add type checking to `update_filewalld`

fixes mypy errors:
cephadm:1682: error: Incompatible types in assignment (expression has type "str", variable has type "int")
cephadm:1683: error: List item 3 has incompatible type "int"; expected "str"
cephadm:1686: error: List item 3 has incompatible type "int"; expected "str"
Found 3 errors in 1 file (checked 1 source file)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: allow prepare-host to start an enabled service

the service might be enabled, but not running

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add type checking for `check_host` and `prepare_host`

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: generalize logic for checking and enabling units

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add 'CEPH_CONF' to the NFS ganesha container envs

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: trim nfs.json sample

we only need to specify the rados pool and config object url

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: systemctl stop nfs-server

stop any running nfs servers before attempting a deploy of nfs-ganesha

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: make pgs available

allow rados obj. create to succeed during a single node test

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add some log lines

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: check port in use

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add/remove nfs ganesha grace

container unit pre-start/post-stop for ganesha grace db

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: update firewalld with nfs service

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: add nfs-ganesha test

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add ganasha.conf

also allows for any arbitrary nfs conf to be deployed by adding items to
the `files` dict of the config-json

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add NFSGanesha deployment type

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: consolidate list of supported daemons

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: use keyword instead of positional args

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/rbd_support: fix recursive non-global level schedule listing

For the case when the non-global level does not have a schedule
and a higher level is used as the parent, it wrongly listed
schedules from all branches under the parent, instead of only the
interested one.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Updated octopus image on 404 page

Replaced the image of the Nautilus octopus with another octopus
in preparation for the "Octopus" release.

The image was taken from Museums Victoria
(https://collections.museumvictoria.com.au/species/8696) and is
licensed under the Creative Commons "Attribution 4.0 International"
(CC BY 4.0) license.

Deleted older, now obsolete images from the assets directory.

Fixes: https://tracker.ceph.com/issues/43581
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

doc: Improved Dashboard installation workflow

Moved the "Accessing the Dashboard" section in front of
the paragraphs that talk about enabling additional services.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

doc: Updated dashboard feature overview for Octopus

Added new Dashboard features added for the Octopus
release.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/orch: Cleanup python imports of the interface

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: Properly handle NotImplementedError

They are masked as RuntimeError by the mgr.

Fixes: https://tracker.ceph.com/issues/44569
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: Rm PersistentStoreDict

`PersistentStoreDict` works great, except that is
too expensive to be used.

In addition, no one makes use of it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: log when we infer the image

This is similar to how we log when we infer the fsid.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: Avoid `inspect` call by using ceph label

Signed-off-by: Ricardo Marques <rimarques@suse.com>

cephadm: Infer ceph image

Fixes: https://tracker.ceph.com/issues/44440
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #33897 into octopus

* refs/pull/33897/head:
mgr/PyModule: initialize options on standby class too
Revert "pybind/mgr/mgr_module: fix standby module logging options"

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

osd/PGStateUtils: initialize NamedState::enter_time.

This bug introduce by d96c907.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd/PGStatueUtils: remove unused member variable

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

qa/workunits/rbd: wait for schedule status updated

Signed-off-by: Mykola Golub <mgolub@suse.com>