mgr/cephadm: progress for upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31338 from ivancich/wip-clean-up-ordered-list

rgw: clean up ordered list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33338 from smithfarm/wip-qa-cephadm-smoke-opensuse

qa/cephadm/smoke: run on opensuse_15.1

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/cephadm: "ls -l" cephadm after downloading it

For better debugging when the download goes awry.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/cephadm.py: better check for cephadm download success

When the URL passed to "curl --silent ..." does not exist, the
resulting file will be populated with the string:

    404: Not Found

If that (or something similar) happens, the file size will be
suspiciously low, like < 1000 bytes. Fail the test immediately in this
case.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/cephadm.py: lop off .git more robustly

The line

    rest.replace('.git/', '/')

was added to accommodate weird folks who run teuthology-suite
with an option like this:

    --suite-repo https://github.com/ceph/ceph.git/

but they might just as well give the option like this:

    --suite-repo https://github.com/ceph/ceph.git

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #31994 from pritha-srivastava/wip-rgw-assume-role-web-id-doc

rgw: adding documentation for AssumeRoleWithWebIdentity.

Merge pull request #33166 from dillaman/wip-43936

rbd-mirror: initial snapshot replay state machine

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33425 from tchaikov/wip-crimson-watcher

crimson/osd: capture watcher when calling its member function

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33423 from tchaikov/wip-doc-search

doc: fix errors in "search" page and use relative address for releases.json

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/cephadm/smoke: run on opensuse_15.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/distros: add opensuse_15.2.yaml

We don't have a FOG image for this, yet, and Shaman is not building for
it (yet), either, but we might as well get this file in place now, since
we already know it will be needed at some point.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/distros: drop obsolete opensuse distros

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33371 from leseb/raw-list

ceph-volume: silence 'ceph-bluestore-tool' failures

Merge pull request #33407 from rhcs-dashboard/coverage-venv-python-version

mgr/dashboard: coverage venv python version same as mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

crimson/osd: capture watcher when calling its member function

otherwise the extracted node handler will be destroyed, and so does the
watcher enclosed in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not use absolute address

use relative address instead, to ease the pain when testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not add suffix for search result links

since sphinx 1.5, SOURCELINK_SUFFIX is always added to the source file,
and search page always tries to add it to the highlighted source result.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not try to patch #edit-on-github if not found

we add this element in _templates/page.html, which is only used for the
content pages, not in the "search" page. to avoid the js errors, just
disabled it if it's not found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33394 into master

* refs/pull/33394/head:
mgr/cephadm: trigger device refresh after osd creation
mgr/orch,cephadm: implement 'device zap'
mgr/cephadm: remove inventory_cache
mgr/cephadm: reimplement 'device ls' in terms of cache
mgr/cephadm: fix invalidate helpers
mgr/cephadm: scrape and cache devices, too
mgr/cephadm: rename DaemonCache methods etc to generalize

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33400 into master

* refs/pull/33400/head:
mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33402 into master

* refs/pull/33402/head:
mgr/pg_autoscaler: fix division by zero

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33352 into master

* refs/pull/33352/head:
ceph.in: catch KeyboardInterrupt from outer code
ceph.in: fix second exception on ^C

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33374 into master

* refs/pull/33374/head:
cephadm: remove distutils dependency
mgr/cephadm: add prepare-host command
cephadm: prepare-host during bootstrap
cephadm: add prepare-host command
cephadm: include docker.io in image name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: trigger device refresh after osd creation

Presumably the new OSD(s) consumed one or more devices.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch,cephadm: implement 'device zap'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove inventory_cache

No longer used.

Clean up _get_hosts now that there is only one user.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: reimplement 'device ls' in terms of cache

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix invalidate helpers

These should wake up the serve() thread.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: scrape and cache devices, too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: rename DaemonCache methods etc to generalize

The HostCache is going to store both daemon and device state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32677 into master

* refs/pull/32677/head:
build/ops: make/install osd_support package
mgr/osd_support: move docs to docs/mgr/osd_support.rst
mgr/osd_support: new module for osd utility calls

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33123 into master

* refs/pull/33123/head:
qa/tasks/cephadm: deploy node-exporter
mgr/orch: deploy node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>

rgw: clean up interenal shard logic in ordered bucket list

The logic of RGWRados::cls_bucket_list_ordered is complex as it needs
to query multiple bucket index shards and manage their returned lists,
and walk through them in a manner to produce an appropriately ordered
set of dir entries to the caller. The previous implementation used
three of parallel std::vectors (e.g., the related data was spread
across the vectors), which needed to each be maintained
appropriately. This clean-up changes this to a single vector of
structs, and the struct both consolidates the related data and
encapsulates much of the related logic.

Additionally, const correctness is expanded.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #33359 into master

* refs/pull/33359/head:
mgr/dashboard/tests/test_orchestrator: disable service test
mgr/cephadm: fix race between check_hosts and host rm
mgr/volumes: fix remove_mds call
mgr/rook: hackly describe_service -> list_daemons
mgr/tests/test_orchestrator: fix ServiceDescription test
mgr/test_orchestrator: describe_service -> list_daemons
mgr/orch: resurrect ServiceDescription, 'orch ls'
qa/tasks/mgr/dashboard/test_orchestrator: disable services test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard/tests/test_orchestrator: disable service test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix race between check_hosts and host rm

Fixes: https://tracker.ceph.com/issues/44188
Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: do not attempt to replay incomplete primary snapshots

In case we race with the creation of a primary mirror snapshot,
do not attempt to replay incomplete snapshots. Once the image
state has been successfully written, the snapshot will be flagged
as complete and an update notification will be sent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: catch KeyboardInterrupt from outer code

This avoids getting random tracebacks when you control-C early in the
process.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: delay marking primary mirror snapshot as completed

Until the image state object(s) are written, the snapshot should
be considered incomplete and not ready for replay.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/volumes: fix remove_mds call

Fixes: https://tracker.ceph.com/issues/44176
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: hackly describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: fix ServiceDescription test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: resurrect ServiceDescription, 'orch ls'

This is a meta-service description that maps to multiple daemons.  Remove
the old useless properties, and add in size and running counts.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_orchestrator: disable services test

We need to either fill out the list_daemons APIs in the dashboard and test
that, or redesign and reimplement the services abstractions in the
orchestration layer.  Until then, disable this test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33333 from sebastian-philipp/cephadm-mon-apply-host-no-network

mgr/cephadm: mon: Don't show traceback for user errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33310 from sebastian-philipp/cephadm-add-host-exception

mgr/cephadm: Convert HostNotFound to OrchestratorError

Reviewed-by: Sage Weil <sage@redhat.com>

rbd-mirror: snapshot replayer should only unlink peers for unneeded snaps

Once the snapshot will no longer be needed by the peer for delta sync,
it should be unlinked. This will ensure the primary image will not
remove snapshots that will be needed for the next sync point.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: initial snapshot-based mirroring replayer logic

This initial implementation will scan the remote image for new mirror
snapshots. If a new primary snapshot it detected, it will start syncing
it. If the remote was demoted, it will notify the image replayer to
shut down. If an in-progress sync was previously interrupted, it will
attempt to restart it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: pass pool meta cache to replayer state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: keep remote image open after bootstrap for snapshot mirroring

The snapshot-based mirroring requires an open remote image to watch for
snapshot creation and to handle the deep-copy from remote to local image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/pybind/rbd: demoted images cannot be deleted

Now that we don't delete the last mirror snapshot, the test is failing
because you cannot delete a demoted image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit last mirror peer to be removed from mirror snapshot

This applies only in the case where there are no additional mirror
snapshots in order to preserve the current state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: tweak create non-primary mirror snapshot request

The replayer will need to pass in the demotion state as well as the
snapshot sequences for the copied snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: no longer re-create full mirror snapshot namespace to get id

Only the namespace type is required to locate the snap id by name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only use snapshot namespace type in unique key sort

Don't use the internal snapshot namespace comparator since for
the mirror snapshot the data can change. The important fact is
that snapshot names should be unique within the namespace type.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop MirrorNonPrimarySnapshotNamespace

The required fields (and state enum) have already been added to the
MirrorSnapshotNamespace. This drops the namespace and tweaks existing
code to use it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: rename MirrorPrimarySnapshotNamespace to MirrorSnapshotNamespace

This is the first pass at merging primary and non-primary mirror snapshots
together into a unified namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: collapse API mirror primary/non-primary namespaces to single type

Eventually we will want to be able to chain multiple peers together.
In order to avoid special logic for syncing non-primary snapshots from
a peer, just treat all mirror snapshots the same.

This also clears some confusion for how to handle demoted snapshots
since otherwise it would involve creating a primary-snapshot on the
non-primary image before the image was fully synced.

Finally, the copied flag was renamed to completed since we should
eventually only set this to true on primary snapshots after the
image state has been created.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume: silence 'ceph-bluestore-tool' failures

If 'ceph-bluestore-tool' fails on a device, the json output of the list
command will be messed up. Ignoring stderr of that command fixes this.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge PR #33387 into master

* refs/pull/33387/head:
spec: ceph-mgr: add missing python3-pyyaml

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: deploy node-exporter

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: deploy node-exporter

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: remove distutils dependency

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: add prepare-host command

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: prepare-host during bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add prepare-host command

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include docker.io in image name

This removes a configuration dependency on the container environment.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32895 from tspmelo/wip-sidebar-notification-v2

mgr/dashboard: Improve Notification sidebar

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33410 from capri1989/docs-fix

doc/mgr/orchestrator.rst: updated current implementation status

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33249 into master

* refs/pull/33249/head:
mgr/cephadm: move cutoff calc inside helper
mgr/orch: clean up service_action, remove_service
mgr/cephadm: move into DaemonCache class
mgr/cephadm: fix tests
mgr/cephadm: implete hacky 'refresh=True' path
mgr/cephadm: raise health alert when scrape fails
mgr/cephadm: persist cached daemon state
mgr/orch: serialize DaemonDescription last_refresh
mgr/cephadm: move _get_daemons() impl into list_daemons
mgr/cephadm: replace remaining _get_daemons() with daemon cache
mgr/cephadm: use daemon map for service|daemon removal
mgr/cephadm: avoid _get_daemons for service|daemon actions
mgr/cephadm: replace daemon_cache with an explicit set of dicts
mgr/cephadm: move DaemonDescription construction into helper

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33408 from tchaikov/wip-crimson-editor-prop

crimson: add editor properties header

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/mgr/orchestrator.rst updated current implementation status

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #33060 from votdev/issue_39252_rgw_beast

mgr/dashboard: RGW port autodetection does not support "Beast" RGW frontend

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

build/ops: make/install osd_support package

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: move docs to docs/mgr/osd_support.rst

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: new module for osd utility calls

it lets you start/stop and monitor osd draining processes

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/dashboard: coverage venv python version same as mgr

As https://github.com/ceph/ceph/pull/31525 is merged,
coverage dep. in run-backend-api-tests.sh has to be installed in venv
with the same python version as ceph-mgr.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #33395 into master

* refs/pull/33395/head:
cephadm: error out on filestore OSDs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33396 from liewegas/fix-cephadm-upgrade-test

qa/suites/rados/cephadm/upgrade: fix initial version

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32949 into master

* refs/pull/32949/head:
cephadm: separate out require files in config-json
cephadm: add alertmanager deployment feature

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

spec: ceph-mgr: add missing python3-pyyaml

Remove stray req from the -k8events package, since the base ceph-mgr now
requires it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33370 into master

* refs/pull/33370/head:
mgr/orchestrator_cli: sort host list

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/pg_autoscaler: fix division by zero

Fixes: https://tracker.ceph.com/issues/44186
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move cutoff calc inside helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: clean up service_action, remove_service

Standardize on service_name argument that looks like 'mgr', 'mds.fsname',
'mds', or some other daemon name prefix.  Avoid the ambiguously-named
service_type+service_name combinations.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move into DaemonCache class

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implete hacky 'refresh=True' path

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: raise health alert when scrape fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: persist cached daemon state

- load cached state on startup
- persist state after a scrape only
- scrape everything after startup

Note that we modify our in-memory cache when we add or remove a service
and then immediately trigger a scrape, but we do not invalidate the
persisted state, since it's simpler (and presumably a good idea) to
simply re-scrape everything after a mgr restart.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: serialize DaemonDescription last_refresh

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move _get_daemons() impl into list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace remaining _get_daemons() with daemon cache

New function _get_daemons_by_type() returns immediately with a result
from the cache.

Remove the with_daemons decorator.

Push daemon list fetch into _add_new_daemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use daemon map for service|daemon removal

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: avoid _get_daemons for service|daemon actions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace daemon_cache with an explicit set of dicts

- Cache DaemonDescription explicitly
- explicit timestamp for the host
- serve() scrapes inventory based on that timestamp

For the moment, persistence is broken, and --refresh is broken.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move DaemonDescription construction into helper

Signed-off-by: Sage Weil <sage@redhat.com>