mgr/ActivePyModules: drop GIL to register/unregister clients

We need to drop the GIL since we are taking ActivePyModules::lock.

Fixes: df507cde8d71063d5873a42f668156e4c32e86f9
Fixes: https://tracker.ceph.com/issues/44241
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33430 from rosinL/wip-testosdscrub

test/TestOSDScrub: fix mktime() error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33444 into master

* refs/pull/33444/head:
mgr/cephadm: no gibberish suffix for some services
mgr/cephadm: prometheus: scrape all mgrs
mgr/cephadm: make prometheus scrape all node-exporters
cephadm: reset-failed on reconfig
cephadm: fix reconfig for monitoring daemons

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33447 into master

* refs/pull/33447/head:
mgr/cephadm: fix upgrade wait loop

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33435 from tchaikov/wip-viva-la-module

deb,rpm,doc: s/plugin/module/

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33422 from tchaikov/wip-ceph-mgr-essentials

rpm,deb: package always-enabled plugins in a separated package

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33088 from liewegas/wip-cephadm-doc

doc/bootstrap: add mds and rgw steps to bootstrap

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33453 from tchaikov/wip-seastar

seastar: pickup change to add pthread linkage

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33452 from tchaikov/wip-docker-centos8

test: remove Dockerfile for centos7 and add Dockerfile for centos8

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: drop Threads::Threads linkage from libcrimson

since seastar is now linked against pthread by itself.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup change to add pthread linkage

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/centos-8: change to adapt to centos8

for more context, see

https://github.com/CentOS/sig-cloud-instance-images/issues/22
https://wiki.centos.org/AdditionalResources/Repositories/CR
https://fedoraproject.org/wiki/EPEL

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/centos-8: add centos8 Dockerfile

and drop centos-7 Dockerfile, as we've moved over centos8 for better
python3 support

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33441 into master

* refs/pull/33441/head:
mgr/orchestrator: add ability to parse placementspec from strings
mgr/volumes: add arg to fs volume create for mds daemons placement

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33443 into master

* refs/pull/33443/head:
cephadm: prepare-host: do not create Packager unless we need it

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33445 into master

* refs/pull/33445/head:
cephadm: fix adoption safety check

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33446 into master

* refs/pull/33446/head:
cephadm: pull: strip newline from version string

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32882 from trociny/wip-41559-2

rbd: add 'mirror snapshot schedule' commands

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33450 into master

* refs/pull/33450/head:
doc: add CephFS Octopus release notes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33368 from trociny/wip-44161

rbd-mirror: don't expect image map is always initialized

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33411 from trociny/wip-44159

rbd-mirror: improve detection of blacklisted state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33426 into master

* refs/pull/33426/head:
mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add CephFS Octopus release notes

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: fix upgrade wait loop

This was succeeding after 4 tries even though it wasn't ok-to-stop.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix adoption safety check

The 'type' file is only required for *osds*.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: pull: strip newline from version string

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33437 into master

* refs/pull/33437/head:
cephadm: allow multiple get_parm() calls

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: no gibberish suffix for some services

Lots of services only need to appear once per node; no need to create a
unique name.  (In fact, most of these *won't* run more than once per
node because they bind to a standard port.)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: prometheus: scrape all mgrs

Assume that all standby mgrs are running on the same port as the
active.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make prometheus scrape all node-exporters

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: reset-failed on reconfig

Sometimes we had a bad config that reconfig will fix, so we need to
reset-failed before restart.  If there was not failure this is harmless.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix reconfig for monitoring daemons

We shouldn't check for port usage here.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add ability to parse placementspec from strings

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

mgr/volumes: add arg to fs volume create for mds daemons placement

add placement arg for mds daemons created by fs volume create

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

cephadm: prepare-host: do not create Packager unless we need it

Otherwise, bootstrap may fail on an unrecognized/unsupported distro even
though all the dependencies are present.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31338 from ivancich/wip-clean-up-ordered-list

rgw: clean up ordered list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33338 from smithfarm/wip-qa-cephadm-smoke-opensuse

qa/cephadm/smoke: run on opensuse_15.1

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/cephadm: "ls -l" cephadm after downloading it

For better debugging when the download goes awry.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd: add 'mirror snapshot schedule' commands

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/rbd_support: rename 'dump' command to 'list'

and remove the old 'list' command -- we can live with one command.

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/cephadm.py: better check for cephadm download success

When the URL passed to "curl --silent ..." does not exist, the
resulting file will be populated with the string:

    404: Not Found

If that (or something similar) happens, the file size will be
suspiciously low, like < 1000 bytes. Fail the test immediately in this
case.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/cephadm.py: lop off .git more robustly

The line

    rest.replace('.git/', '/')

was added to accommodate weird folks who run teuthology-suite
with an option like this:

    --suite-repo https://github.com/ceph/ceph.git/

but they might just as well give the option like this:

    --suite-repo https://github.com/ceph/ceph.git

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cephadm: allow multiple get_parm() calls

This fixes deployment of monitoring components when the config-json is
passed via stdin.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31994 from pritha-srivastava/wip-rgw-assume-role-web-id-doc

rgw: adding documentation for AssumeRoleWithWebIdentity.

mgr/rbd_support: return 'rbd mirror snapshot schedule status' as json

Signed-off-by: Mykola Golub <mgolub@suse.com>

doc/mgr/dashboard.rst: s/plugin/module/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: s/plugin/module/

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,deb: package always-enabled plugins in a separated package

ceph-mgr depends on it in the sense that these plugins are
a hard dependency of ceph-mgr. like cmake depends on cmake-data,
even if the later cannot function on its own without the former.

the reason why we need a separated package is that mgr modules
are written in python, and should be arch independent. this change
helps the downstream to understand this and help to minimize the
divergence between upstream and downstream, because distros like
debian and its derivatives encourage splitting architecture independent
data into its own package.
see
https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#architecture-independent-data

also, the python related runtime dependencies are also moved into
ceph-mgr-modules-core.

if a python dependency is shared by another ceph-mgr-* package and
ceph-mgr-modules-core. it's only added to ceph-mgr-modules-core for
simplicity and consistency. as the name of dependency might vary
on different distros, so it's tedious and error-prone to repeat
these conditons and checks.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33166 from dillaman/wip-43936

rbd-mirror: initial snapshot replay state machine

Reviewed-by: Mykola Golub <mgolub@suse.com>

ceph.spec.in: update to adapt to el7

make sure ceph-mgr installs on el7

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/TestOSDScrub: fix mktime() error

The var tm tm isn't initialized, when the tm.tm_isdst is a
positive value, mktime(&tm) return -1 result in test failed
in ubuntu 19.10 for aarch64 GLIBC2.30.

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #33425 from tchaikov/wip-crimson-watcher

crimson/osd: capture watcher when calling its member function

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33423 from tchaikov/wip-doc-search

doc: fix errors in "search" page and use relative address for releases.json

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/cephadm/smoke: run on opensuse_15.1

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/distros: add opensuse_15.2.yaml

We don't have a FOG image for this, yet, and Shaman is not building for
it (yet), either, but we might as well get this file in place now, since
we already know it will be needed at some point.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/distros: drop obsolete opensuse distros

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33371 from leseb/raw-list

ceph-volume: silence 'ceph-bluestore-tool' failures

Merge pull request #33407 from rhcs-dashboard/coverage-venv-python-version

mgr/dashboard: coverage venv python version same as mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Fixes: https://tracker.ceph.com/issues/44201
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

crimson/osd: capture watcher when calling its member function

otherwise the extracted node handler will be destroyed, and so does the
watcher enclosed in it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not use absolute address

use relative address instead, to ease the pain when testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not add suffix for search result links

since sphinx 1.5, SOURCELINK_SUFFIX is always added to the source file,
and search page always tries to add it to the highlighted source result.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: do not try to patch #edit-on-github if not found

we add this element in _templates/page.html, which is only used for the
content pages, not in the "search" page. to avoid the js errors, just
disabled it if it's not found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33394 into master

* refs/pull/33394/head:
mgr/cephadm: trigger device refresh after osd creation
mgr/orch,cephadm: implement 'device zap'
mgr/cephadm: remove inventory_cache
mgr/cephadm: reimplement 'device ls' in terms of cache
mgr/cephadm: fix invalidate helpers
mgr/cephadm: scrape and cache devices, too
mgr/cephadm: rename DaemonCache methods etc to generalize

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33400 into master

* refs/pull/33400/head:
mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33402 into master

* refs/pull/33402/head:
mgr/pg_autoscaler: fix division by zero

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33352 into master

* refs/pull/33352/head:
ceph.in: catch KeyboardInterrupt from outer code
ceph.in: fix second exception on ^C

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33374 into master

* refs/pull/33374/head:
cephadm: remove distutils dependency
mgr/cephadm: add prepare-host command
cephadm: prepare-host during bootstrap
cephadm: add prepare-host command
cephadm: include docker.io in image name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: trigger device refresh after osd creation

Presumably the new OSD(s) consumed one or more devices.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch,cephadm: implement 'device zap'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove inventory_cache

No longer used.

Clean up _get_hosts now that there is only one user.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: reimplement 'device ls' in terms of cache

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix invalidate helpers

These should wake up the serve() thread.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: scrape and cache devices, too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: rename DaemonCache methods etc to generalize

The HostCache is going to store both daemon and device state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32677 into master

* refs/pull/32677/head:
build/ops: make/install osd_support package
mgr/osd_support: move docs to docs/mgr/osd_support.rst
mgr/osd_support: new module for osd utility calls

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33123 into master

* refs/pull/33123/head:
qa/tasks/cephadm: deploy node-exporter
mgr/orch: deploy node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>

rgw: clean up interenal shard logic in ordered bucket list

The logic of RGWRados::cls_bucket_list_ordered is complex as it needs
to query multiple bucket index shards and manage their returned lists,
and walk through them in a manner to produce an appropriately ordered
set of dir entries to the caller. The previous implementation used
three of parallel std::vectors (e.g., the related data was spread
across the vectors), which needed to each be maintained
appropriately. This clean-up changes this to a single vector of
structs, and the struct both consolidates the related data and
encapsulates much of the related logic.

Additionally, const correctness is expanded.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #33359 into master

* refs/pull/33359/head:
mgr/dashboard/tests/test_orchestrator: disable service test
mgr/cephadm: fix race between check_hosts and host rm
mgr/volumes: fix remove_mds call
mgr/rook: hackly describe_service -> list_daemons
mgr/tests/test_orchestrator: fix ServiceDescription test
mgr/test_orchestrator: describe_service -> list_daemons
mgr/orch: resurrect ServiceDescription, 'orch ls'
qa/tasks/mgr/dashboard/test_orchestrator: disable services test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard/tests/test_orchestrator: disable service test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix race between check_hosts and host rm

Fixes: https://tracker.ceph.com/issues/44188
Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: do not attempt to replay incomplete primary snapshots

In case we race with the creation of a primary mirror snapshot,
do not attempt to replay incomplete snapshots. Once the image
state has been successfully written, the snapshot will be flagged
as complete and an update notification will be sent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: catch KeyboardInterrupt from outer code

This avoids getting random tracebacks when you control-C early in the
process.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: delay marking primary mirror snapshot as completed

Until the image state object(s) are written, the snapshot should
be considered incomplete and not ready for replay.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/volumes: fix remove_mds call

Fixes: https://tracker.ceph.com/issues/44176
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: hackly describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: fix ServiceDescription test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: resurrect ServiceDescription, 'orch ls'

This is a meta-service description that maps to multiple daemons.  Remove
the old useless properties, and add in size and running counts.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_orchestrator: disable services test

We need to either fill out the list_daemons APIs in the dashboard and test
that, or redesign and reimplement the services abstractions in the
orchestration layer.  Until then, disable this test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33333 from sebastian-philipp/cephadm-mon-apply-host-no-network

mgr/cephadm: mon: Don't show traceback for user errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33310 from sebastian-philipp/cephadm-add-host-exception

mgr/cephadm: Convert HostNotFound to OrchestratorError

Reviewed-by: Sage Weil <sage@redhat.com>

rbd-mirror: snapshot replayer should only unlink peers for unneeded snaps

Once the snapshot will no longer be needed by the peer for delta sync,
it should be unlinked. This will ensure the primary image will not
remove snapshots that will be needed for the next sync point.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: initial snapshot-based mirroring replayer logic

This initial implementation will scan the remote image for new mirror
snapshots. If a new primary snapshot it detected, it will start syncing
it. If the remote was demoted, it will notify the image replayer to
shut down. If an in-progress sync was previously interrupted, it will
attempt to restart it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: pass pool meta cache to replayer state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: keep remote image open after bootstrap for snapshot mirroring

The snapshot-based mirroring requires an open remote image to watch for
snapshot creation and to handle the deep-copy from remote to local image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/pybind/rbd: demoted images cannot be deleted

Now that we don't delete the last mirror snapshot, the test is failing
because you cannot delete a demoted image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit last mirror peer to be removed from mirror snapshot

This applies only in the case where there are no additional mirror
snapshots in order to preserve the current state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: tweak create non-primary mirror snapshot request

The replayer will need to pass in the demotion state as well as the
snapshot sequences for the copied snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: no longer re-create full mirror snapshot namespace to get id

Only the namespace type is required to locate the snap id by name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>