qa: test mgr cephfs mount blacklist

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: forward RADOS client instances for potential blacklist

The mgr creates a per-module RADOS client connection for modules which
interact with RADOS (e.g. the volumes module). These clients should also
be blacklisted when the active mgr is failed; we don't want the former
active mgr to continue interacting with RADOS when the new one takes
over. This is particularly impactful for avoiding extraneous
"unresponsive client" warnings from the MDS when the mgr switches
(especially in testing). The MDS will pickup the new OSD blacklists
which include's the old mgr's libcephfs instance and blacklist/evict
that session quietly.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MgrMonitor: blacklist previous instance

This wasn't realized as necessary early on in the ceph-mgr development
because the mgr didn't interact with RADOS. However, now it is becoming
common for plugins to store data there. It's important that the previous
instance can no longer interact with RADOS while the new mgr takes over.

In particular, this means that the mgr's client sessions with the MDS
are automatically evicted once the MDS receives the new OSDMap. This
avoids a pesky "unresponsive client" warning in the cluster logs.

Fixes: https://tracker.ceph.com/issues/42939
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon: set RUNTIME flag on mon_mds_blacklist_interval

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: use more efficient data structure

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31891 into master

* refs/pull/31891/head:
qa/suites/rados/ssh: only install ceph-daemon for packaged mode
tasks/ceph2: add support for packaged ceph-daemon
qa/tasks/ceph2: make image cluster-private
qa/tasks/ceph2: add 'shell' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31984 into master

* refs/pull/31984/head:
mgr/telemetry: fix log typo

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31869 into master

* refs/pull/31869/head:
ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()
qa/standalone/test_ceph_daemon.sh: more $SUDO
ceph-daemon: configure firewalld for new daemon deploys
ceph-daemon: name mgr the same way mgr/ssh does

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31930 into master

* refs/pull/31930/head:
mgr/orchestrator: add optional "format" param for "orchestrator host ls"

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31968 from idryomov/wip-krbd-python3

qa: update krbd tests for python3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/telemetry: fix log typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31983 into master

* refs/pull/31983/head:
mgr/ssh/tests: fix RGWSpec test

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/ssh/tests: fix RGWSpec test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31973 into master

* refs/pull/31973/head:
mgr/crash: fix 'crash ls[-new]' sorting

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: grafana charts match time picker selection. (#31964)

mgr/dashboard: grafana charts match time picker selection.

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge PR #31890 into master

* refs/pull/31890/head:
mgr/orchestrator: name rgw by client.rgw.$realm.$zone[.$id]

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #31952 into master

* refs/pull/31952/head:
mgr/ssh: fix remove_osds
mgr/ssh: fix service_action

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: more $SUDO

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: configure firewalld for new daemon deploys

Note that we only open firewalld ports; we never close them.

We could conceivably do that from rm-cluster, but that might also interfere
with something else on the host...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31202 from athanatos/sjust/wip-crimson-obc

crimson, common: Add ephemeral ObjectContext state to crimson

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31473 from ifed01/wip-ifed-smplify-fsck

os/bluestore: simplify multithreaded shallow fsck.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31969 from sebastian-philipp/mgr-fix-mocking

pybind/mgr: Make it easier to create a Module instance without the mgr

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard,grafana: remove shortcut menu (#31951)

mgr/dashboard,grafana: remove shortcut menu

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: grafana charts match time picker selection.

Fixes: https://tracker.ceph.com/issues/43097
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/crash: fix 'crash ls[-new]' sorting

Broken by c691f2f8763a9db081a0b61294c5eedf4e9c2cee

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31960 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0093 unless it is supported

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: do not use CMP0093 unless it is supported

CMake CMP0093 was introduced by CMake 3.15, but we only require CMake
v3.10.2. so use the old `Boost_VERSION_MACRO` variable by default,
unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: Make it easier to create a Module instance without the mgr

Caused by conflict between #30961 and #30262

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31872 from smithfarm/wip-mimic-v13.2.7-doc2

doc/releases: add mimic v13.2.7 to releases timeline

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Increase max-width of popovers (#31743)

mgr/dashboard: Increase max-width of popovers

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30961 from rjfd/wip-mgr-module-logging

mgr: module logging infrastructure

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard,grafana: remove shortcut menu

Remove shortcut menu (links) and add check in grafana CI script.

Fixes: https://tracker.ceph.com/issues/43091
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31962 from bk201/wip-43095

vstart: fix error when getting CMake variables with the same prefix

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: fix error when getting CMake variables with the same prefix

The function fails to work when there are multiple variables starts with
the same prefix. e.g.

- WITH_RBD
- WITH_RBD_RGW

Fixes: https://tracker.ceph.com/issues/43095
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

crimson/osd: convert PG::wait_for_active to a blocking_future

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: use ObjectContext and take obc locks

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: factor out load_object_state/snapset

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: add ObjectContext/Registry

ObjectContext will hold the ephemeral state and lock status
for an object as in classic osd.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd: factor out RWState, move SnapSetContext, ObjectState, RWState

We'll want to reuse this logic in crimson, but the actual operation
wake/wait mechanics will be different.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/: introduce intrusive_lru

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: factor OSD::init_op_flags into seperate class

We'll want to reuse this in crimson, extract the logic
for setting flags from MOSDOp into osd_types.h.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator.h: add finally method for resource release

This way we can add things like resource handlers which are otherwise
transparent to passing values and errors.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: add --debug for crimson if debug is enabled

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #31947 from zdover23/wip-doc-ubuntu-1804-python-minimal

doc: add Python 2 to Ubuntu 18.04 installations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31958 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0074 unless it is supported

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/suites/rados/ssh: only install ceph-daemon for packaged mode

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: do not use CMP0074 unless it is supported

CMake CMP0074 was introduced by CMake 3.12, but we only require CMake
v3.10.2. so default to check `Boost_ROOT` unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: name mgr the same way mgr/ssh does

(with a random 6-char id, not the hostname)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31913 into master

* refs/pull/31913/head:
ceph-daemon: Allow env var for setting the used image

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

tasks/ceph2: add support for packaged ceph-daemon

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix remove_osds

Broken by the new completions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix service_action

Broken by the new completions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31919 from idryomov/wip-krbd-udev-improvements

krbd: do away with explicit memory management and other cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: update krbd tests for python3

There is no /usr/bin/python symlink in CentOS 8 even if python2 is
installed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #31933 into master

* refs/pull/31933/head:
mgr/ssh: `test_mon_update` needs to set a mon name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31274 into master

* refs/pull/31274/head:
osd/OSD: enhance osd numa affinity compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31348 into master

* refs/pull/31348/head:
tools/rados: update advisory lock break usage with --lock-cookie required

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31669 into master

* refs/pull/31669/head:
bluestore: tiny tracepoints improvement

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #31943 from tchaikov/wip-31224

python-common: move pytest integration from setup.py to tox.ini

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc: add Python 2 to Ubuntu 18.04 installations

This commit updates the "Create a Cluster" section of the
"Storage Cluster Quick Start" document. Ubuntu 18.04 does
not have Python 2 installed by default. This commit instructs
users of Python 18.04 to install the package "python-minimal"
to provide Python 2 prior to running "ceph-deploy".

This commit fixes Bug number 9 in the list here:
https://pad.ceph.com/p/Report_Documentation_Bugs

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #31361 from hairesis/kms_transit_backend

rgw: extend SSE-KMS with Vault using transit secrets engine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

python-common: move pytest integration from setup.py to tox.ini

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31279 from lixiaoy1/rwl-image-writeback-cache-seq1

rbd/cache: Replicated Write Log core codes part 1

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Display the aggregated number of request (#31369)

mgr/dashboard: Display the aggregated number of request

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Resolve some npm audit issues (#31845)

mgr/dashboard: Resolve some npm audit issues

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Trim IQN on iSCSI target form (#31888)

mgr/dashboard: Trim IQN on iSCSI target form

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/orchestrator: add optional "format" param for "orchestrator host ls"

and update the test accordingly

Fixes: https://tracker.ceph.com/issues/43078
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/ssh: `test_mon_update` needs to set a mon name

... in order to exercise a code path to make use of `_get_services()`

Fixes: https://tracker.ceph.com/issues/43075
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Eye candy updates to datatable pages (#31764)

mgr/dashboard: Eye candy updates to datatable pages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31903 from majianpeng/os-bluestore-ceph-assert

os/bluestore: don't round_up_to in apply_for_bitset_range.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #31774 from dzafman/wip-balancer3

osd,tools: Balancer fixes without all of the calc_pg_upmaps() rewrites

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31317 from wjwithagen/wjw-update-FindBoost.cmake

cmake: update FindBoost.cmake for 1.71

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31841 from rosinL/wip-support-dpdk-numa

cmake: Add dpdk numa support

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31915 from majianpeng/osd-remove-useless-ceph-assert

osd: remove useless ceph_assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31723 from rosinL/wip-add-bench-log-usage

test/bench_log: add usage function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31840 from rosinL/wip-fix-dpdk

msg/async/dpdk: fix compilation errors when WITH_DPDK=on

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30631 from xxhdx1985126/wip-bluefs-sync-leak-fix

BlueFS: prevent BlueFS::dirty_files from being leaked when syncing metadata

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31900 into master

* refs/pull/31900/head:
mgr: fix debug typo

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31442 into master

* refs/pull/31442/head:
kv/RocksDBStore: Add minimum key limit before invoking DeleteRange.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async/dpdk: Porting seastar's init_port_start to fix to resolve compilation errors

Ceph currently uses dpdk19.05, seastar dpdk_device::init_port_start has
been upgraded to DPDK19.05, so porting changes to fix compiler error.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

Merge pull request #31165 from votdev/improve_py_unittests

mgr/dashboard: Refactor Python unittests and controller

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31856 from rzarzynski/wip-crimson-errorator-handle_error

crimson/common/errorator.h: add handle_error() method.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31853 from liewegas/fix-rn-typo

PendingReleaseNotes: fix typo

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31918 from rzarzynski/wip-crimson-errorator-simplify_safe_then

crimson/common/errorator.h: simplify the compound safe_then() variant.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31838 into master

* refs/pull/31838/head:
ceph-daemon: chown uid/gid files during `adopt`
ceph-daemon: use 'iglob' instead of 'glob'
ceph-daemon: Fix handling for symlinks on python2

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

krbd: misc cleanups

Introduce get_devnode() and append_unmap_options(); make some functions
static.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: make wait_for_udev_add() clearer and a bit more efficient

Collect only /dev/rbd* block events and dispose of them as soon as
possible; match on devnode and assert on major/minor instead of the
other way around.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: do away with explicit memory management

Wrap udev_monitor, udev_enumerate and udev_device with std::unique_ptr.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: Eye candy updates to datatable pages

- Introduce boolean and array pipes
- Introduce configurable 'badge' datatable column
- Refactor the Inventory, OSD, RBD, iSCSI, ... datatable pages

Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/common/errorator.h: simplify the compound safe_then() variant.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common/errorator.h: add handle_error() method.

In contrast to `safe_then` it deals only with errors
and, on return, leaves the value type `future<ValueT>`
embodies unchanged.
While being basically a specialization of `safe_then`,
it's also a convenience method to squeeze boilerplate.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #31914 into master

* refs/pull/31914/head:
mgr/orchestrator: set node labels to empty list if none specified

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: support creating OSDs on spare devices (#30921)

mgr/dashboard: support creating OSDs on spare devices

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cmake: Add shared library as dpdk dependency

librte_eal.a refers to libpthread.so and libdl.so, and libceph-common.so
refers to librte_eal.a. Many test application link libceph-common.so
without linking pthread, it will cause link error.
Such as unittest_interval_set, target_link_libraries (unittest_interval_set
ceph-common)

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

cmake: Add dpdk numa support

Find and link the numa library for DPDK.

Fixes: https://tracker.ceph.com/issues/42275
Signed-off-by: Hu Ye <yehu5@huawei.com>
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

Merge pull request #31785 from votdev/cleanup_remove_useless_var

mgr/dashboard: Remove unused variable

Reviewed-by: Kiefer Chang kiefer.chang@suse.com
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #31720 from rhcs-dashboard/fix-42708-master

mgr/dashboard: remove traceback/version assertions

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31843 from votdev/rename_list_pipe

mgr/dashboard: Rename pipe list -> join

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

rgw: improvements to SSE-KMS with Vault

* Minor improvements to Vault documentation
* Add teuthology tests for Transit secrets engine
* Add unit tests for KV secrets engine, minor improvements to Transit
  secrets engine
* use string_view::npos instead of string::npos

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>