qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: pass -v to bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: only try to place mons on hosts matching public_network

Only try to schedule new mons on hosts that match the configured
public_network, if any.  If we do not have one configured, then don't
try to place new mons at all.

Note that there are other restrictions that ceph-mon supports that we
aren't considering, here: public_network_interface in particular, which
might further limit which IPs we consider binding to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: keep track of host networks, ips

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: automatically infer mon public_network, if we can

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add list-networks command

Summarize CIDR networks and our IPs in them.  Result looks like this:

{
    "10.3.64.0/24": [
        "10.3.64.23",
        "10.3.64.27"
    ],
    "10.88.0.0/16": [
        "10.88.0.1"
    ],
    "192.168.122.0/24": [
        "192.168.122.1"
    ]
}

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33987 from tchaikov/wip-octopus-formatting

doc/releases/octopus.rst: formatting tweaks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33980 into octopus

* refs/pull/33980/head:
cephadm: bootstrap: allow --output-dir
cephadm: do not infer image for bootstrap
cephadm: write output files to /etc/ceph by default
cephadm: verify the output files' containing directory exists

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/releases/octopus.rst: use note directive for notes

so the note stands out in the text

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus.rst: add '$' as prompt

to help the BashSession lexer to render the text starting with "#" as comments

see https://pygments.org/docs/lexers/#lexers-for-various-shells

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus: decrease the indent level

for less and better horizontal spacing

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_static/css: fine tune the spacing

* `ul.simple > li` is also used in front page, where the items are too
  sparse now. so we need to restore the default spacing.
* `ul.simple > li > ul > li:last-child` is used to control the spacing
  between nested unordered list. we need have more spacing there.
* `div.section > ul > li > p` is added to decrease the spacing of nested
  list items.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33971 from tchaikov/octopus-doc-release

doc/releases/octopus.rst: format tweaks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33968 into octopus

* refs/pull/33968/head:
mgr/cephadm: do not include 'osd' service in 'orch ls' output

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33970 into octopus

* refs/pull/33970/head:
python-common: 'all:true' -> '*'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33973 into octopus

* refs/pull/33973/head:
mgr/orch,cephadm: elevate --force flag for daemons to interface
cephadm: backup data when we remove stateful daemons (mon, osd, prometheus)
cephadm: do not zap disk after removing an OSD daemon

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33935 into octopus

* refs/pull/33935/head:
mgr/cephadm: fix host connection exception
doc/cephadm: use ssh-copy-id

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc/releases/octopus.rst: reindent top-level items

reduce the indent level, so these items are better formated

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus.rst: formatting tweaks

* use console lexer for highlighting command lines
  otherwise they will be rendered as python code
* use bold font instead italic in subsection headers,
  they look more formal this way
* fix a typo: delineated

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: fix hyper link

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/release: fix "CephFS" subsection heading

it's a regression introduced by 26db06cf212be029e5628a40de3410632c0b86a0

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: bootstrap: allow --output-dir

This is easier than specifying each of output-config, output-keyring, and
output-pub-ssh-key.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not infer image for bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: write output files to /etc/ceph by default

This streamlines the install for most users by putting the config and
keyring files in /etc/ceph, where other ceph CLI commands will find them.

The cost is making isolated installs a bit more painful, by requiring
that users specify a local location for --output-config etc.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: verify the output files' containing directory exists

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33977 into octopus

* refs/pull/33977/head:
pybind/mgr: apply_drivegroups should return Sequence[Completion]

Reviewed-by: Sage Weil <sage@redhat.com>

pybind/mgr: apply_drivegroups should return Sequence[Completion]

* change annotation of `orchestrator.Orchestrator.apply_drivegroups()` so it returns
  `Sequence[Completion]`
  see
  https://mypy.readthedocs.io/en/latest/generics.html#variance-of-generic-types
* to follow the interface defined by `orchestrator.Orchestrator`. change
  all its implementations. see `orchestrator/_interface.py`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch,cephadm: elevate --force flag for daemons to interface

Apply this check in the generic orchestrator code so that it applies
consistently to all implementations.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: backup data when we remove stateful daemons (mon, osd, prometheus)

This avoids doing permanent damage when removing stateful daemons.

We include OSDs here even though in most cases no precious data lives in
the osd data directory because there might be some important metadata
there--particularly for adopted OSDs from older clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not zap disk after removing an OSD daemon

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33922 into octopus

* refs/pull/33922/head:
qa/tasks/mgr/test_orchestrator_cli: adapt
qa/suites/rados/cephadm/upgrade: start from compatible build
mgr/cephadm: refine log message
python-common: correctly dictify DriveGroupSpec objects when paths are used
mgr/cephadm: adapt test_cephadm.sh to new syntax
mgr/cephadm: adapt qa->test_cephadm.sh
qa/tasks/cephadm: use 'orch daemon osd add'
mgr/cephadm: adapt osd deployment to service_apply

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: adapt

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: start from compatible build

The osd create command is changing, so upgrade from something that the
latest cephadm.py task can use.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix host connection exception

1- catch exception thrown by _get_connection itself
2- make the error message more informative.  the old one was

Error ENOENT: New host dael (10.3.64.25) failed to connect: `ssh -F /tmp/cephadm-conf-9tvrgrls -i /tmp/cephadm-identity-sj66ado1 root@10.3.64.25`

which was both misleading and had irrelevant information.  Now,

Error ENOENT: Failed to connect to dael (10.3.64.25).  Check that the host is reachable and accepts connections using the cephadm SSH key

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: use ssh-copy-id

Signed-off-by: Sage Weil <sage@redhat.com>

python-common: 'all:true' -> '*'

'all:true' is no longer a valid PlacementSpec.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not include 'osd' service in 'orch ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/octopus' into octopus

Merge PR #33520 into octopus

* refs/pull/33520/head:
mgr/dashboard: Isolate each RBD component

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #33758 into octopus

* refs/pull/33758/head:
mgr/dashboard: Replace ng2-tree with angular-tree-component

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33941 into octopus

* refs/pull/33941/head:
mgr/dashboard: hide daemon table when orchestrator is disabled

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #33948 into octopus

* refs/pull/33948/head:
python-common: PlacementSpec.from_string: drop `host_pattern:` prefix
python-common: remove `all_hosts` from `PlacementSpec`

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refine log message

Avoid a lot of these

2020-03-13T14:58:38.785360-0500 mgr.x [INF] No data_devices, skipping DriveGroup: osd.all

every time we consider the OSD spec and find nothing to do.  Instead, log
something like this each time we actually have work to do:

2020-03-13T15:04:22.803356-0500 mgr.x [INF] Applying osd.all on host gnit...
2020-03-13T15:04:33.723731-0500 mgr.x [INF] Deploying daemon osd.5 on gnit

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33777 into octopus

* refs/pull/33777/head:
rgw: svc.config_key_rados: get() warns if potentially insecure connection
rgw: svc.rados: add clog_warn()
mon: log monitor: add command level param
auth registry: helpers for checking secure method/mode

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33950 into octopus

* refs/pull/33950/head:
ceph.in: use os._exit when we don't shut down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33951 into octopus

* refs/pull/33951/head:
mgr/orch: dump service spec by name

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph.in: use os._exit when we don't shut down

If we experience a timeout, we don't bother shutting down rados, because
it can't currently handle one thread blocked (or running) library init
at the same time that another thread calls rados_shutdown().  This
behavior was recently introduced by c8f353c50bcd2a30be9e3600dba91912b8cd0429

However, sys.exit() runs all kinds of shutdown work that will also
interfere with the running librados threads.

Fix by using os._exit instead of sys.exit.

Fixes: https://tracker.ceph.com/issues/44566
Signed-off-by: Sage Weil <sage@redhat.com>

15.1.1

mgr/orch: dump service spec by name

add optional arg to allow a dump by name:
`ceph orch spec dump [svc_name]`

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #33953 into octopus

* refs/pull/33953/head:
doc: Move ceph-deploy docs to doc/install/ceph-deploy

Reviewed-by: Sage Weil <sage@redhat.com>

doc: Move ceph-deploy docs to doc/install/ceph-deploy

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33924 into octopus

* refs/pull/33924/head:
doc/mgr: Enhance placement specs
doc/cephadm: Fix toc of Drive Groups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33891 into octopus

* refs/pull/33891/head:
mgr/cephadm: fix dump output by formatting to json first

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33905 into octopus

* refs/pull/33905/head:
install-deps.sh: remove the workaround to bootstrap a new venv
install-deps.sh: install additional repo for RHEL8
install-deps.sh: install rhel-7-server-optional-rpms on el7 only

Reviewed-by: David Galloway <dgallowa@redhat.com>

python-common: correctly dictify DriveGroupSpec objects when paths are used

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33940 from tchaikov/octopus-release-notes-header

doc: use subsection for representing components in release notes

Reviewed-by: Sage Weil <sage@redhat.com>

doc/mgr: Enhance placement specs

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/releases: use h3 for rendering subsections in release notes

to highlight the components with subsection header, and allow readers to
navigate through them with TOC.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: overide the default margin for "ul.simple > li"

by default, the user agent's css sets `margin-block-end` of "ul ul" to
"0", which renders the unordered lists in release notes cluttered. as we
are using nested unordered list to present the changes in different
components.

in this change a customized css is added to reset the begin and end
margin of ul to 1em, which is the default value of top level ul.

Signed-off-by: Kefu Chai <kchai@redhat.com>

python-common: PlacementSpec.from_string: drop `host_pattern:` prefix

* `host_pattern` is works as any other host spec.
* Improved type annotations
* Added tests
* `mgr/cephadm/test_scheduling`: Remove tests that don't have a valid spec.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: adapt test_cephadm.sh to new syntax

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: adapt qa->test_cephadm.sh

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa/tasks/cephadm: use 'orch daemon osd add'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: adapt osd deployment to service_apply

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33867 from dillaman/wip-43590

librbd: optimize image copy state machine to use fast-diff

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: hide daemon table when orchestrator is disabled

The refresh time of services/daemons is also increased to 1 minute since
they are not updated so frequently.

Fixes: https://tracker.ceph.com/issues/44558
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #33930 into octopus

* refs/pull/33930/head:
mgr/cephadm: implement pause/resume to suspect non-monitoring background work
mgr/orch: add pause/resume hooks

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33868 into octopus

* refs/pull/33868/head:
mgr/rook: implement remove_daemons()
mgr/rook: fix mds/fs create/apply/update
mgr/rook: use ceph_daemon_id label consistently
mgr/rook: show filesystems in 'orch ls'
mgr/orch: 'age' -> 'ago' in 'orch ls' output
mgr/rook: initial pass at 'orch ls'
mgr/rook: include timestamps in 'orch ps'
mgr/orch: shorten container_id (hash) in cephadm, not orch ps
mgr/rook: include container_image_name in 'orch ps'
mgr/rook: fix list_daemons host arg

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33932 into octopus

* refs/pull/33932/head:
cephadm: fix datetime regexp to capture at most 6 digits

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Isolate each RBD component

Before when you accessed any RBD component, you were loading all 4 components
at the same time and each was doing frequent requests to the server.

Now it only loads 1 at a time.

Fixes: https://tracker.ceph.com/issues/44278
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33900 from dillaman/wip-44459

qa/suites/rbd: removed OpenStack tempest test cases

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33840 from trociny/wip-schedule-optimize-scan

mgr/rbd_support: don't scan pools that don't have schedules

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33914 into octopus

* refs/pull/33914/head:
mgr/orch: Cleanup python imports of the interface
mgr/orch: Properly handle NotImplementedError
pybind/mgr: Rm PersistentStoreDict

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rook: implement remove_daemons()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Replace ng2-tree with angular-tree-component

Fixes: https://tracker.ceph.com/issues/44450
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/rook: fix mds/fs create/apply/update

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: use ceph_daemon_id label consistently

In particular, for mds's, it's $fsname-$rank for mds.$fsname-$rank,
which is what we want.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: show filesystems in 'orch ls'

The count vs size discrepancy is weird since rook things in terms of
ranks (*2 if standby) vs just a count.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: 'age' -> 'ago' in 'orch ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: initial pass at 'orch ls'

This doesn't cover mds.* or rgw.* yet.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: include timestamps in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: shorten container_id (hash) in cephadm, not orch ps

This lets us present a longer, human-readable id for rook.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: include container_image_name in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: fix list_daemons host arg

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix datetime regexp to capture at most 6 digits

trailing `0` digits are not shown in the precision

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33821 from votdev/issue_43446_bucket_lock

mgr/dashboard: Create bucket with x-amz-bucket-object-lock-enabled

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #31997 from bk201/wip-43062

mgr/dashboard: support removing OSDs in OSDs page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33920 from LenzGr/update-404-image

mgr/dashboard: Updated octopus image on 404 page

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/cephadm: implement pause/resume to suspect non-monitoring background work

If the user does 'orch pause', suspend all background work that makes
actual changes.

Continue to do read-only operations, like checking host connectivity
and scraping daemon and device status.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33858 from tspmelo/wip-use-cd-modal

mgr/dashboard: Use ModalComponent in all modals

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/orch: add pause/resume hooks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33926 into octopus

* refs/pull/33926/head:
python: fixate mypy requirement to mypy==0.770

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33874 into octopus

* refs/pull/33874/head:
qa/suites/powercycle: install build deps for building xfstest

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #33813 into octopus

* refs/pull/33813/head:
osd/PGStateUtils: initialize NamedState::enter_time.
osd/PGStatueUtils: remove unused member variable

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33814 into octopus

* refs/pull/33814/head:
osd/OSDMap: stop encoding osd_state with >8 bits wide states only for old client

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: yanjun <yan.jun8@zte.com.cn>

Merge PR #33064 into octopus

* refs/pull/33064/head:
cephadm: add version to `command_ls` output
cephadm: add type checking to `update_filewalld`
cephadm: allow prepare-host to start an enabled service
cephadm: add type checking for `check_host` and `prepare_host`
cephadm: generalize logic for checking and enabling units
cephadm: add 'CEPH_CONF' to the NFS ganesha container envs
cephadm: trim nfs.json sample
qa/workunits/cephadm/test_cephadm.sh: systemctl stop nfs-server
qa/workunits/cephadm/test_cephadm.sh: make pgs available
cephadm: add some log lines
cephadm: check port in use
cephadm: add/remove nfs ganesha grace
cephadm: update firewalld with nfs service
qa/workunits/cephadm/test_cephadm.sh: add nfs-ganesha test
cephadm: add ganasha.conf
cephadm: add NFSGanesha deployment type
cephadm: consolidate list of supported daemons
cephadm: use keyword instead of positional args

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33855 into octopus

* refs/pull/33855/head:
mgr/cephadm: reconfigure ceph daemons when monmap changes

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33829 into octopus

* refs/pull/33829/head:
cephadm: log when we infer the image
cephadm: Avoid `inspect` call by using ceph label
cephadm: Infer ceph image

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33919 from LenzGr/octopus-documentation

documentation: Updated Dashboard Features, improved flow

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>