qa: add upgrade test for volume upgrade from legacy

This tests that volumes created using the ceph_volume_client.py library
continue to be accessible/function via the Nautilus/Octopus ceph-mgr
volumes plugin.

Fixes: https://tracker.ceph.com/issues/42723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32937 from rzarzynski/wip-bl-carry-_carriage

common/bl: carry the bufferlist::_carriage over std::moves

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32515 from majianpeng/osd-remove-unused-func3

osd: remove unused functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33573 from votdev/fix_mypy_issue

mgr/dashboard: make check mypy failure

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: make check mypy failure

Fixes: https://tracker.ceph.com/issues/44319
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #33079 from jan--f/c-v-scan-dmcrypt-osds

ceph-volume: strip _dmcrypt suffix in simple scan json output

Merge pull request #33562 from dillaman/wip-fix-doc-builds

Revert "doc: do not add suffix for search result links"

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32953 from votdev/issue_43665_smart_data

mgr/dashboard: smart: smart data read out on down osd causes error popup

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #33547 into master

* refs/pull/33547/head:
mgr/volumes: unregister job upon async threads exception

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: unregister job upon async threads exception

If the async threads hit a temporary exception the job is
never unregistered and therefore gets skipped by the async
threads on subsequent scans.

Patrick hit this in nautilus when one of the purge threads
hit an exception when trying to log a message. The trash
entry was never picked up again by the purge threads.

Fixes: http://tracker.ceph.com/issues/44293
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge PR #33552 into master

* refs/pull/33552/head:
mgr/dashboard: Enhance user create CLI command to force password change

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #33550 into master

* refs/pull/33550/head:
rpm,deb: poke Python OpenSSL dependencies
mgr: outsource creation of self-signed certificates

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33517 into master

* refs/pull/33517/head:
cephadm: bootstrap: nag about telemetry

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33519 into master

* refs/pull/33519/head:
cephadm: shell: make a root home dir (/root) in /var/log/ceph/$fsid/home

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33544 into master

* refs/pull/33544/head:
qa/workunits/cephadm/test_cephadm.sh: remove image param non-ceph tests
qa/workunits/cephadm/test_cephadm.sh: consolidate wait loop logic

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Revert "doc: do not add suffix for search result links"

This reverts commit 88d6d499e0ff8341f8dc4c73b7a7c3cdc41b63d5.

All doc builds are currently broken due to this change since
the HTML files are written without the necessary ".html"
extension.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephadm: shell: make a root home dir (/root) in /var/log/ceph/$fsid/home

This makes the shell much more friendly to humans.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33551 into master

* refs/pull/33551/head:
doc/cephadm: Rephrased introduction sentence
doc/cephadm: Rename section titles and fix introductions
doc/cephadm: Add CLI and OS recommendations to TOC
doc: move cephadm mgr doc to /doc/cephadm
doc: move bootstrap.rst to cephadm

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33321 from rhcs-dashboard/43801-purge-button

mgr/dashboard: Fixes rbd image 'purge trash' button & modal text

mgr/dashboard: Enhance user create CLI command to force password change

Fixes: https://tracker.ceph.com/issues/44301
Signed-off-by: Volker Theile <vtheile@suse.com>

rpm,deb: poke Python OpenSSL dependencies

to ceph-mgr-modules-core package.

Fixes: https://tracker.ceph.com/issues/44306
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr: outsource creation of self-signed certificates

Remove duplicated code from mgr/cephadm and mgr/dashboard and move it to
`mgr_util.py`.

Fixes: https://tracker.ceph.com/issues/44306
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #33537 from cbodley/wip-test-rgw-common-commit

test/rgw: fixes for test-rgw-multisite.sh

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

doc/cephadm: Rephrased introduction sentence

Signed-off-by: Alexandra Settle <asettle@suse.com>

Merge pull request #33463 from sebastian-philipp/cephadm-workerpool-10

mgr/cephadm: set thread pool size to 10

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31200 into master

* refs/pull/31200/head:
qa/cephfs: test case for auto reconnect after blacklisted

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Fixes rbd image 'purge trash' button & modal text

Fixes: https://tracker.ceph.com/issues/43801
Signed-off-by: anurag <abandhu@redhat.com>

Merge pull request #33478 from yehudasa/wip-44246

ceph-volume: lvm: get_device_vgs() filter by provided prefix

doc/cephadm: Rename section titles and fix introductions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Add CLI and OS recommendations to TOC

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: move cephadm mgr doc to /doc/cephadm

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: move bootstrap.rst to cephadm

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33171 from rhcs-dashboard/wip-42914-fix-ui-issues

mgr/dashboard: UI fixes

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33405 from rhcs-dashboard/wip-44198-fix-style-issues

mgr/dashboard: 'destroyed' view in CRUSH map viewer

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33459 from liewegas/doc-adopt

doc/mgr/cephadm: document adoption process

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #33454 from votdev/fix_mypy_issues

mgr/dashboard: Fix mypy issues and enable it by default

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33372 from pritha-srivastava/rgw-omap-offload-gc-fixes

Fixes related to omap offload and gc.

Merge pull request #33528 from cyx1231st/wip_crimson_osd_misc_fixes

crimson/osd: misc fixes and cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33543 from badone/wip-doc-min_size-applicable-to-ec-pools

doc/rados/operations: min_size is applicable to EC

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: remove image param non-ceph tests

non-ceph components have a default image
as of 97def7cb60de8c52db1cfca198bd2cd64585883f

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: consolidate wait loop logic

into an `is_available` function

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc/rados/operations: Detail for min_size on EC pools

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #32342 from dzafman/wip-43126

mon: Improvements to slow heartbeat health messages

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rados/operations: min_size is applicable to EC

min_size is *not* replicated pool specific.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #33379 from badone/wip-upgrade-ansible-version-for-ca

qa/ceph-ansible: Upgrade ansible version

Merge PR #33516 into master

* refs/pull/33516/head:
ceph.in: Fix name retval is not defined error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>

Merge PR #33525 into master

* refs/pull/33525/head:
osd/PeeringState.h: ignore RemoteBackfillReserved in WaitLocalBackfillReserved

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #33263 into master

* refs/pull/33263/head:
qa/vstart_runner.py: make run()'s interface same as teuthology's run

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33499 into master

* refs/pull/33499/head:
mount: fix the debug log when keyring getting secret failed

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #33515 into master

* refs/pull/33515/head:
mgr/cephadm: replace static SSL cert
mgr/cephadm: dynamically add Grafana data sources
mgr/cephadm: provide static default configs for grafana deployment
mgr/cephadm: provision grafana
mgr/orch: add grafana hooks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29816 from coder-hugo/feature/fix-ldap-auth-doc

rgw: Fix documentation for rgw_ldap_secret

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #33480 into master

* refs/pull/33480/head:
test: add new ceph_lseek test
client: fix error handling in lseek SEEK_HOLE/SEEK_DATA cases

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #33427 into master

* refs/pull/33427/head:
qa/cephfs: rewrite a bit of code xfstests_dev.py
qa/cephfs: update xfstests-dev deps for RHEL 8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32657 into master

* refs/pull/32657/head:
test: query using mds id, not rank
mgr: re-enable mds `scrub status` info in ceph status
mon: filter out ceph normal ceph entity types when dumping service metadata
mgr: filter out normal ceph services when processing service map
mgr: helper function to check if a service is a normal ceph service

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33449 into master

* refs/pull/33449/head:
mds: handle bad purge queue item encoding

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #33471 into master

* refs/pull/33471/head:
mds: relevel debug message levels for balancer/migrator
mds: inline mds_role_t stream operator

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #33535 from smithfarm/wip-rados-cephadm-smoke-152

qa: run cephadm/smoke on opensuse 15.2 instead of 15.1

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #33078 into master

* refs/pull/33078/head:
qa: avoid subtree rep in test_version_splitting

Reviewed-by: Rishabh Dave <ridave@redhat.com>

ceph-volume: don't create Volume/VolumeGroup objects with empty name

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: get_device_lvs filter by provided prefix

Fixes: https://tracker.ceph.com/issues/44124
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: lvm.get_device_vgs: don't create VolumeGroup with empty name

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33456 from smithfarm/wip-44182

ceph-backport.sh: fix setup routine

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #33510 from krig/mypy-cache-gitignore

.gitignore: Add .mypy_cache to .gitignore

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33504 into master

* refs/pull/33504/head:
cephadm: allow skipping prepare_host in bootstrap step

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: replace static SSL cert

with dynamically generated one.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

qa: run cephadm/smoke on opensuse 15.2 instead of 15.1

Companion PR to https://github.com/ceph/ceph-build/pull/1509

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #33495 into master

* refs/pull/33495/head:
mgr/cephadm: do not refresh device inventory on mgr restart
mgr/cephadm: make cache invalidate less racy
mgr/cephadm: fix last_device_update persistence

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>

Merge PR #33501 into master

* refs/pull/33501/head:
debian: modules-core replaces and breaks older ceph-mgr

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: dynamically add Grafana data sources

depending on which Prometheus services have been deployed.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: provide static default configs for grafana deployment

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: provision grafana

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: Fix mypy issues and enable it by default

The decorator @no_type_check is used:
* To prevent a mypy error like 'error: INTERNAL ERROR -- Please try using mypy master on Github:'
* '#type: ignore' does not work, e.g. in broken lines

Fixes: https://tracker.ceph.com/issues/44269
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #33513 from liewegas/fix-44271

mgr/dashboard: behave when pwdUpdateRequired key is missing

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/orch: add grafana hooks

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

crimson/osd: hide add_blocker() and clear_blocker() in Operation

Cleanup. Hide add_blocker() and clear_blocker() as private members
because no one is using them anymore. They are not exception-safe. Also
take the chance to reorder Operation class members.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: smart: smart data read out on down osd causes error popup

Fixes: https://tracker.ceph.com/issues/43665
Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/osd: use with_blocking_future in CompoundPeeringRequest

Cleanup. with_blocking_future() can cover all the use cases of
add_blocker() and clear_blocker(), and it is exception-safe and not
error-prone.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: fix osdmap update during osd start

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: fix min_peers related logic in heartbeat

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #33440 from dillaman/wip-44103

rbd-mirror: provide initial snapshot replay status

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #33497 into master

* refs/pull/33497/head:
qa/suites/rados/singleton/all/thrash-eio: whitelist 'slow request'
qa: Whitelist 'slow request' within a bunch of tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33138 into master

* refs/pull/33138/head:
common/TextTable: only pad between columns
mgr/status: align with ceph table style
mgr/osd_perf_query: make table match ceph style
mgr: adjust tables to have 2 space column separation
common/TextTable: default to 2 spaces separating columns

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: bootstrap: nag about telemetry

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState.h: ignore RemoteBackfillReserved in WaitLocalBackfillReserved

It is possible to dequeue an outstanding RemoteBackfillReserved, though we may have
already released reservations for that backfill target. Currently, if this happens
while we are in WaitLocalBackfillReserved, it can lead to a crash on the primary.
Prevent this by treating this condition as a no-op.

The longer term fix is to add a RELEASE_ACK mechanism, which prevents the primary
from scheduling a backfill retry until all the RELEASE_ACKs have been received.

Fixes: https://tracker.ceph.com/issues/44248
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #33413 into master

* refs/pull/33413/head:
test: verify purge queue w/ large number of subvolumes
test: pass timeout argument to mount::wait_for_dir_empty()
mgr/volumes: access volume in lockless mode when fetching async job

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/TextTable: only pad between columns

No need to pad on the right-hand side if there are no additional columns
that follow.

Signed-off-by: Sage Weil <sage@redhat.com>

test/rgw: fixes for test-rgw-common.sh

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33113 into master

* refs/pull/33113/head:
doc/mgr/telemetry: added device channel details

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33439 into master

* refs/pull/33439/head:
common/options.cc: make rocksdb_delete_range_threshold very high

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd-mirror: implement basic status feedback for snapshot mirroring

The feedback includes the newest remote mirror snapshot timestamp,
the newest completely synced local mirror snapshot timestamp,
and optionally the in-progress sync snapshot timestamp and
percent complete.

Fixes: https://tracker.ceph.com/issues/44103
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved local to remote snapshot lookup to common function

This will be needed by the status formatter to lookup the remote
snapshot timestamp since the associated local snapshot timestamp
will be the time the snapshot was created on the local side.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: expand lock scope for snapshot replayer

The status callback will need access to the current state, so the
lock should be held while the snapshot scan is in-progress.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: periodically update sync position for snapshot-based images

This will allow the sync to restart at the last known successfully copied
object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33487 into master

* refs/pull/33487/head:
mgr/cephadm: upgrade: fix handling when target container id changes
mgr/cephadm: upgrade: handle stopped daemons

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33496 into master

* refs/pull/33496/head:
mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service
mgr/cephadm: remove apply_mon support
mgr/cephadm: use generics for add_mon
mgr/cephadm: use _apply_service for mgrs
mgr/cephadm: refactor most daemon add methods
mgr/cephadm: refactor _update_service and all apply methods
mgr/cephadm: fix get_unique_name when name in use

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33465 into master

* refs/pull/33465/head:
mgr/cephadm: implement placement by label
mgr/cephadm: fix self.log call in get_unique_name
mgr/cephadm: fix NodePlacement host names
mgr/tests/test_orchestrator: test PlacementSpec parsing
mgr/orch: remove label: prefix when parsing placementspec
mgr/orch: PlacementSpec: add all_hosts property

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33514 into master

* refs/pull/33514/head:
mgr/cephadm: drop mixin parent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: avoid subtree rep in test_version_splitting

Root subtree may be replicated which would open client sessions early.

Fixes: https://tracker.ceph.com/issues/43796
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33477 into master

* refs/pull/33477/head:
qa/tasks/mgr/test_orchestrator_cli: fix test_host_ls

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33474 into master

* refs/pull/33474/head:
mgr/orch: make arg hostname, not host

Reviewed-by: Sebastian Wagner <swagner@suse.com>