mgr/orch,cephadm: elevate --force flag for daemons to interface

Apply this check in the generic orchestrator code so that it applies
consistently to all implementations.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: backup data when we remove stateful daemons (mon, osd, prometheus)

This avoids doing permanent damage when removing stateful daemons.

We include OSDs here even though in most cases no precious data lives in
the osd data directory because there might be some important metadata
there--particularly for adopted OSDs from older clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not zap disk after removing an OSD daemon

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33922 into octopus

* refs/pull/33922/head:
qa/tasks/mgr/test_orchestrator_cli: adapt
qa/suites/rados/cephadm/upgrade: start from compatible build
mgr/cephadm: refine log message
python-common: correctly dictify DriveGroupSpec objects when paths are used
mgr/cephadm: adapt test_cephadm.sh to new syntax
mgr/cephadm: adapt qa->test_cephadm.sh
qa/tasks/cephadm: use 'orch daemon osd add'
mgr/cephadm: adapt osd deployment to service_apply

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: adapt

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: start from compatible build

The osd create command is changing, so upgrade from something that the
latest cephadm.py task can use.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/octopus' into octopus

Merge PR #33520 into octopus

* refs/pull/33520/head:
mgr/dashboard: Isolate each RBD component

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #33758 into octopus

* refs/pull/33758/head:
mgr/dashboard: Replace ng2-tree with angular-tree-component

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33941 into octopus

* refs/pull/33941/head:
mgr/dashboard: hide daemon table when orchestrator is disabled

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #33948 into octopus

* refs/pull/33948/head:
python-common: PlacementSpec.from_string: drop `host_pattern:` prefix
python-common: remove `all_hosts` from `PlacementSpec`

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refine log message

Avoid a lot of these

2020-03-13T14:58:38.785360-0500 mgr.x [INF] No data_devices, skipping DriveGroup: osd.all

every time we consider the OSD spec and find nothing to do.  Instead, log
something like this each time we actually have work to do:

2020-03-13T15:04:22.803356-0500 mgr.x [INF] Applying osd.all on host gnit...
2020-03-13T15:04:33.723731-0500 mgr.x [INF] Deploying daemon osd.5 on gnit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33777 into octopus

* refs/pull/33777/head:
rgw: svc.config_key_rados: get() warns if potentially insecure connection
rgw: svc.rados: add clog_warn()
mon: log monitor: add command level param
auth registry: helpers for checking secure method/mode

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33950 into octopus

* refs/pull/33950/head:
ceph.in: use os._exit when we don't shut down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33951 into octopus

* refs/pull/33951/head:
mgr/orch: dump service spec by name

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph.in: use os._exit when we don't shut down

If we experience a timeout, we don't bother shutting down rados, because
it can't currently handle one thread blocked (or running) library init
at the same time that another thread calls rados_shutdown().  This
behavior was recently introduced by c8f353c50bcd2a30be9e3600dba91912b8cd0429

However, sys.exit() runs all kinds of shutdown work that will also
interfere with the running librados threads.

Fix by using os._exit instead of sys.exit.

Fixes: https://tracker.ceph.com/issues/44566
Signed-off-by: Sage Weil <sage@redhat.com>

15.1.1

mgr/orch: dump service spec by name

add optional arg to allow a dump by name:
`ceph orch spec dump [svc_name]`

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #33953 into octopus

* refs/pull/33953/head:
doc: Move ceph-deploy docs to doc/install/ceph-deploy

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Move ceph-deploy docs to doc/install/ceph-deploy

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33924 into octopus

* refs/pull/33924/head:
doc/mgr: Enhance placement specs
doc/cephadm: Fix toc of Drive Groups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33891 into octopus

* refs/pull/33891/head:
mgr/cephadm: fix dump output by formatting to json first

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33905 into octopus

* refs/pull/33905/head:
install-deps.sh: remove the workaround to bootstrap a new venv
install-deps.sh: install additional repo for RHEL8
install-deps.sh: install rhel-7-server-optional-rpms on el7 only

Reviewed-by: David Galloway <dgallowa@redhat.com>

python-common: correctly dictify DriveGroupSpec objects when paths are used

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33940 from tchaikov/octopus-release-notes-header

doc: use subsection for representing components in release notes

Reviewed-by: Sage Weil <sage@redhat.com>

doc/mgr: Enhance placement specs

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/releases: use h3 for rendering subsections in release notes

to highlight the components with subsection header, and allow readers to
navigate through them with TOC.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: overide the default margin for "ul.simple > li"

by default, the user agent's css sets `margin-block-end` of "ul ul" to
"0", which renders the unordered lists in release notes cluttered. as we
are using nested unordered list to present the changes in different
components.

in this change a customized css is added to reset the begin and end
margin of ul to 1em, which is the default value of top level ul.

Signed-off-by: Kefu Chai <kchai@redhat.com>

python-common: PlacementSpec.from_string: drop `host_pattern:` prefix

* `host_pattern` is works as any other host spec.
* Improved type annotations
* Added tests
* `mgr/cephadm/test_scheduling`: Remove tests that don't have a valid spec.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: adapt test_cephadm.sh to new syntax

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: adapt qa->test_cephadm.sh

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa/tasks/cephadm: use 'orch daemon osd add'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: adapt osd deployment to service_apply

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33867 from dillaman/wip-43590

librbd: optimize image copy state machine to use fast-diff

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: hide daemon table when orchestrator is disabled

The refresh time of services/daemons is also increased to 1 minute since
they are not updated so frequently.

Fixes: https://tracker.ceph.com/issues/44558
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #33930 into octopus

* refs/pull/33930/head:
mgr/cephadm: implement pause/resume to suspect non-monitoring background work
mgr/orch: add pause/resume hooks

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33868 into octopus

* refs/pull/33868/head:
mgr/rook: implement remove_daemons()
mgr/rook: fix mds/fs create/apply/update
mgr/rook: use ceph_daemon_id label consistently
mgr/rook: show filesystems in 'orch ls'
mgr/orch: 'age' -> 'ago' in 'orch ls' output
mgr/rook: initial pass at 'orch ls'
mgr/rook: include timestamps in 'orch ps'
mgr/orch: shorten container_id (hash) in cephadm, not orch ps
mgr/rook: include container_image_name in 'orch ps'
mgr/rook: fix list_daemons host arg

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33932 into octopus

* refs/pull/33932/head:
cephadm: fix datetime regexp to capture at most 6 digits

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Isolate each RBD component

Before when you accessed any RBD component, you were loading all 4 components
at the same time and each was doing frequent requests to the server.

Now it only loads 1 at a time.

Fixes: https://tracker.ceph.com/issues/44278
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33900 from dillaman/wip-44459

qa/suites/rbd: removed OpenStack tempest test cases

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33840 from trociny/wip-schedule-optimize-scan

mgr/rbd_support: don't scan pools that don't have schedules

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33914 into octopus

* refs/pull/33914/head:
mgr/orch: Cleanup python imports of the interface
mgr/orch: Properly handle NotImplementedError
pybind/mgr: Rm PersistentStoreDict

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rook: implement remove_daemons()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Replace ng2-tree with angular-tree-component

Fixes: https://tracker.ceph.com/issues/44450
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/rook: fix mds/fs create/apply/update

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: use ceph_daemon_id label consistently

In particular, for mds's, it's $fsname-$rank for mds.$fsname-$rank,
which is what we want.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: show filesystems in 'orch ls'

The count vs size discrepancy is weird since rook things in terms of
ranks (*2 if standby) vs just a count.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: 'age' -> 'ago' in 'orch ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: initial pass at 'orch ls'

This doesn't cover mds.* or rgw.* yet.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: include timestamps in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: shorten container_id (hash) in cephadm, not orch ps

This lets us present a longer, human-readable id for rook.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: include container_image_name in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: fix list_daemons host arg

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix datetime regexp to capture at most 6 digits

trailing `0` digits are not shown in the precision

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33821 from votdev/issue_43446_bucket_lock

mgr/dashboard: Create bucket with x-amz-bucket-object-lock-enabled

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #31997 from bk201/wip-43062

mgr/dashboard: support removing OSDs in OSDs page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33920 from LenzGr/update-404-image

mgr/dashboard: Updated octopus image on 404 page

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/cephadm: implement pause/resume to suspect non-monitoring background work

If the user does 'orch pause', suspend all background work that makes
actual changes.

Continue to do read-only operations, like checking host connectivity
and scraping daemon and device status.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33858 from tspmelo/wip-use-cd-modal

mgr/dashboard: Use ModalComponent in all modals

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/orch: add pause/resume hooks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33926 into octopus

* refs/pull/33926/head:
python: fixate mypy requirement to mypy==0.770

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33874 into octopus

* refs/pull/33874/head:
qa/suites/powercycle: install build deps for building xfstest

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #33813 into octopus

* refs/pull/33813/head:
osd/PGStateUtils: initialize NamedState::enter_time.
osd/PGStatueUtils: remove unused member variable

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33814 into octopus

* refs/pull/33814/head:
osd/OSDMap: stop encoding osd_state with >8 bits wide states only for old client

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: yanjun <yan.jun8@zte.com.cn>

Merge PR #33064 into octopus

* refs/pull/33064/head:
cephadm: add version to `command_ls` output
cephadm: add type checking to `update_filewalld`
cephadm: allow prepare-host to start an enabled service
cephadm: add type checking for `check_host` and `prepare_host`
cephadm: generalize logic for checking and enabling units
cephadm: add 'CEPH_CONF' to the NFS ganesha container envs
cephadm: trim nfs.json sample
qa/workunits/cephadm/test_cephadm.sh: systemctl stop nfs-server
qa/workunits/cephadm/test_cephadm.sh: make pgs available
cephadm: add some log lines
cephadm: check port in use
cephadm: add/remove nfs ganesha grace
cephadm: update firewalld with nfs service
qa/workunits/cephadm/test_cephadm.sh: add nfs-ganesha test
cephadm: add ganasha.conf
cephadm: add NFSGanesha deployment type
cephadm: consolidate list of supported daemons
cephadm: use keyword instead of positional args

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33855 into octopus

* refs/pull/33855/head:
mgr/cephadm: reconfigure ceph daemons when monmap changes

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33829 into octopus

* refs/pull/33829/head:
cephadm: log when we infer the image
cephadm: Avoid `inspect` call by using ceph label
cephadm: Infer ceph image

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33919 from LenzGr/octopus-documentation

documentation: Updated Dashboard Features, improved flow

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #33871 into octopus

* refs/pull/33871/head:
qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

python-common: remove `all_hosts` from `PlacementSpec`

replace it with `PlacementSpec(host_pattern='*')`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33890 into octopus

* refs/pull/33890/head:
doc/install: rethink docs install page (similar to PR#33613)
doc/install: Rethink doc/install

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/cephadm: Fix toc of Drive Groups

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python: fixate mypy requirement to mypy==0.770

I don't like suprises when upstream publishes a new mypy version.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Append "Modal" to all modal components names

Fixes: https://tracker.ceph.com/issues/44547
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use ModalComponent in all modals

Fixes: https://tracker.ceph.com/issues/44547
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33902 from dillaman/wip-test-rbd-integration

test/cli-integration: fixed spacing issue for RBD formatted tables

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/cephadm: reconfigure ceph daemons when monmap changes

This keeps ceph.conf fresh.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33898 from dillaman/wip-44567

qa/workunits/rbd: wait for nbd map to close after unmap

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/cephadm: fix dump output by formatting to json first

Signed-off-by: Joshua Schmid <jschmid@suse.de>

cephadm: add version to `command_ls` output

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add type checking to `update_filewalld`

fixes mypy errors:
cephadm:1682: error: Incompatible types in assignment (expression has type "str", variable has type "int")
cephadm:1683: error: List item 3 has incompatible type "int"; expected "str"
cephadm:1686: error: List item 3 has incompatible type "int"; expected "str"
Found 3 errors in 1 file (checked 1 source file)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: allow prepare-host to start an enabled service

the service might be enabled, but not running

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add type checking for `check_host` and `prepare_host`

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: generalize logic for checking and enabling units

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add 'CEPH_CONF' to the NFS ganesha container envs

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: trim nfs.json sample

we only need to specify the rados pool and config object url

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: systemctl stop nfs-server

stop any running nfs servers before attempting a deploy of nfs-ganesha

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: make pgs available

allow rados obj. create to succeed during a single node test

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add some log lines

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: check port in use

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add/remove nfs ganesha grace

container unit pre-start/post-stop for ganesha grace db

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: update firewalld with nfs service

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: add nfs-ganesha test

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add ganasha.conf

also allows for any arbitrary nfs conf to be deployed by adding items to
the `files` dict of the config-json

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add NFSGanesha deployment type

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: consolidate list of supported daemons

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: use keyword instead of positional args

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/rbd_support: fix recursive non-global level schedule listing

For the case when the non-global level does not have a schedule
and a higher level is used as the parent, it wrongly listed
schedules from all branches under the parent, instead of only the
interested one.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Updated octopus image on 404 page

Replaced the image of the Nautilus octopus with another octopus
in preparation for the "Octopus" release.

The image was taken from Museums Victoria
(https://collections.museumvictoria.com.au/species/8696) and is
licensed under the Creative Commons "Attribution 4.0 International"
(CC BY 4.0) license.

Deleted older, now obsolete images from the assets directory.

Fixes: https://tracker.ceph.com/issues/43581
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

doc: Improved Dashboard installation workflow

Moved the "Accessing the Dashboard" section in front of
the paragraphs that talk about enabling additional services.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>