Merge pull request #31950 from dillaman/wip-mirror-peer-fsid

rbd-mirror: peer_ping should send the local fsid to the remote

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #31908 into master

* refs/pull/31908/head:
ceph-daemon: use uid/gid 0 when creating tempfiles
ceph-daemon: drop unneeded `mode` during tmpfile create
ceph-daemon: consolidate NamedTemporaryFile logic
ceph-daemon: chown uid/gid `ceph-volume` tmpfiles

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31940 into master

* refs/pull/31940/head:
mgr/ssh: Port raising exceptions from completion handlers to Py2

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #31990 from dzafman/wip-43084

tools/osdmaptool.cc: do not use deprecated std::random_shuffle()

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

tools/osdmaptool.cc: do not use deprecated std::random_shuffle()
the use of `std::random_shuffle()` was introduced by
b946308 .

in this change, it is replaced using `std::shuffle()`.

Fixes: https://tracker.ceph.com/issues/43084
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31891 into master

* refs/pull/31891/head:
qa/suites/rados/ssh: only install ceph-daemon for packaged mode
tasks/ceph2: add support for packaged ceph-daemon
qa/tasks/ceph2: make image cluster-private
qa/tasks/ceph2: add 'shell' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31984 into master

* refs/pull/31984/head:
mgr/telemetry: fix log typo

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31869 into master

* refs/pull/31869/head:
ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()
qa/standalone/test_ceph_daemon.sh: more $SUDO
ceph-daemon: configure firewalld for new daemon deploys
ceph-daemon: name mgr the same way mgr/ssh does

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31930 into master

* refs/pull/31930/head:
mgr/orchestrator: add optional "format" param for "orchestrator host ls"

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31968 from idryomov/wip-krbd-python3

qa: update krbd tests for python3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/telemetry: fix log typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31983 into master

* refs/pull/31983/head:
mgr/ssh/tests: fix RGWSpec test

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/ssh/tests: fix RGWSpec test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31973 into master

* refs/pull/31973/head:
mgr/crash: fix 'crash ls[-new]' sorting

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: grafana charts match time picker selection. (#31964)

mgr/dashboard: grafana charts match time picker selection.

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge PR #31890 into master

* refs/pull/31890/head:
mgr/orchestrator: name rgw by client.rgw.$realm.$zone[.$id]

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #31952 into master

* refs/pull/31952/head:
mgr/ssh: fix remove_osds
mgr/ssh: fix service_action

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: more $SUDO

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: configure firewalld for new daemon deploys

Note that we only open firewalld ports; we never close them.

We could conceivably do that from rm-cluster, but that might also interfere
with something else on the host...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31202 from athanatos/sjust/wip-crimson-obc

crimson, common: Add ephemeral ObjectContext state to crimson

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: use uid/gid 0 when creating tempfiles

skip extraction of the uid/gid since `ceph-volume` runs as root

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31473 from ifed01/wip-ifed-smplify-fsck

os/bluestore: simplify multithreaded shallow fsck.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31969 from sebastian-philipp/mgr-fix-mocking

pybind/mgr: Make it easier to create a Module instance without the mgr

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard,grafana: remove shortcut menu (#31951)

mgr/dashboard,grafana: remove shortcut menu

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: grafana charts match time picker selection.

Fixes: https://tracker.ceph.com/issues/43097
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/crash: fix 'crash ls[-new]' sorting

Broken by c691f2f8763a9db081a0b61294c5eedf4e9c2cee

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31960 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0093 unless it is supported

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: do not use CMP0093 unless it is supported

CMake CMP0093 was introduced by CMake 3.15, but we only require CMake
v3.10.2. so use the old `Boost_VERSION_MACRO` variable by default,
unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: Make it easier to create a Module instance without the mgr

Caused by conflict between #30961 and #30262

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31872 from smithfarm/wip-mimic-v13.2.7-doc2

doc/releases: add mimic v13.2.7 to releases timeline

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Increase max-width of popovers (#31743)

mgr/dashboard: Increase max-width of popovers

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30961 from rjfd/wip-mgr-module-logging

mgr: module logging infrastructure

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard,grafana: remove shortcut menu

Remove shortcut menu (links) and add check in grafana CI script.

Fixes: https://tracker.ceph.com/issues/43091
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31962 from bk201/wip-43095

vstart: fix error when getting CMake variables with the same prefix

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: fix error when getting CMake variables with the same prefix

The function fails to work when there are multiple variables starts with
the same prefix. e.g.

- WITH_RBD
- WITH_RBD_RGW

Fixes: https://tracker.ceph.com/issues/43095
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

crimson/osd: convert PG::wait_for_active to a blocking_future

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: use ObjectContext and take obc locks

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: factor out load_object_state/snapset

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: add ObjectContext/Registry

ObjectContext will hold the ephemeral state and lock status
for an object as in classic osd.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd: factor out RWState, move SnapSetContext, ObjectState, RWState

We'll want to reuse this logic in crimson, but the actual operation
wake/wait mechanics will be different.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/: introduce intrusive_lru

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: factor OSD::init_op_flags into seperate class

We'll want to reuse this in crimson, extract the logic
for setting flags from MOSDOp into osd_types.h.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator.h: add finally method for resource release

This way we can add things like resource handlers which are otherwise
transparent to passing values and errors.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: add --debug for crimson if debug is enabled

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #31947 from zdover23/wip-doc-ubuntu-1804-python-minimal

doc: add Python 2 to Ubuntu 18.04 installations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31958 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0074 unless it is supported

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/suites/rados/ssh: only install ceph-daemon for packaged mode

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: do not use CMP0074 unless it is supported

CMake CMP0074 was introduced by CMake 3.12, but we only require CMake
v3.10.2. so default to check `Boost_ROOT` unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: drop unneeded `mode` during tmpfile create

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: name mgr the same way mgr/ssh does

(with a random 6-char id, not the hostname)

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: consolidate NamedTemporaryFile logic

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: chown uid/gid `ceph-volume` tmpfiles

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31913 into master

* refs/pull/31913/head:
ceph-daemon: Allow env var for setting the used image

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

tasks/ceph2: add support for packaged ceph-daemon

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix remove_osds

Broken by the new completions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix service_action

Broken by the new completions

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: peer_ping should send the local fsid to the remote

It was previously incorrectly sending the remote cluster's
fsid to the remote cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31919 from idryomov/wip-krbd-udev-improvements

krbd: do away with explicit memory management and other cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: update krbd tests for python3

There is no /usr/bin/python symlink in CentOS 8 even if python2 is
installed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #31933 into master

* refs/pull/31933/head:
mgr/ssh: `test_mon_update` needs to set a mon name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31274 into master

* refs/pull/31274/head:
osd/OSD: enhance osd numa affinity compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31348 into master

* refs/pull/31348/head:
tools/rados: update advisory lock break usage with --lock-cookie required

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31669 into master

* refs/pull/31669/head:
bluestore: tiny tracepoints improvement

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #31943 from tchaikov/wip-31224

python-common: move pytest integration from setup.py to tox.ini

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc: add Python 2 to Ubuntu 18.04 installations

This commit updates the "Create a Cluster" section of the
"Storage Cluster Quick Start" document. Ubuntu 18.04 does
not have Python 2 installed by default. This commit instructs
users of Python 18.04 to install the package "python-minimal"
to provide Python 2 prior to running "ceph-deploy".

This commit fixes Bug number 9 in the list here:
https://pad.ceph.com/p/Report_Documentation_Bugs

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #31361 from hairesis/kms_transit_backend

rgw: extend SSE-KMS with Vault using transit secrets engine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

python-common: move pytest integration from setup.py to tox.ini

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31279 from lixiaoy1/rwl-image-writeback-cache-seq1

rbd/cache: Replicated Write Log core codes part 1

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/ssh: Port raising exceptions from completion handlers to Py2

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Display the aggregated number of request (#31369)

mgr/dashboard: Display the aggregated number of request

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Resolve some npm audit issues (#31845)

mgr/dashboard: Resolve some npm audit issues

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Trim IQN on iSCSI target form (#31888)

mgr/dashboard: Trim IQN on iSCSI target form

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/orchestrator: add optional "format" param for "orchestrator host ls"

and update the test accordingly

Fixes: https://tracker.ceph.com/issues/43078
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/ssh: `test_mon_update` needs to set a mon name

... in order to exercise a code path to make use of `_get_services()`

Fixes: https://tracker.ceph.com/issues/43075
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Eye candy updates to datatable pages (#31764)

mgr/dashboard: Eye candy updates to datatable pages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31903 from majianpeng/os-bluestore-ceph-assert

os/bluestore: don't round_up_to in apply_for_bitset_range.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #31774 from dzafman/wip-balancer3

osd,tools: Balancer fixes without all of the calc_pg_upmaps() rewrites

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31317 from wjwithagen/wjw-update-FindBoost.cmake

cmake: update FindBoost.cmake for 1.71

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31841 from rosinL/wip-support-dpdk-numa

cmake: Add dpdk numa support

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31915 from majianpeng/osd-remove-useless-ceph-assert

osd: remove useless ceph_assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31723 from rosinL/wip-add-bench-log-usage

test/bench_log: add usage function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31840 from rosinL/wip-fix-dpdk

msg/async/dpdk: fix compilation errors when WITH_DPDK=on

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30631 from xxhdx1985126/wip-bluefs-sync-leak-fix

BlueFS: prevent BlueFS::dirty_files from being leaked when syncing metadata

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31900 into master

* refs/pull/31900/head:
mgr: fix debug typo

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31442 into master

* refs/pull/31442/head:
kv/RocksDBStore: Add minimum key limit before invoking DeleteRange.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async/dpdk: Porting seastar's init_port_start to fix to resolve compilation errors

Ceph currently uses dpdk19.05, seastar dpdk_device::init_port_start has
been upgraded to DPDK19.05, so porting changes to fix compiler error.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

Merge pull request #31165 from votdev/improve_py_unittests

mgr/dashboard: Refactor Python unittests and controller

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31856 from rzarzynski/wip-crimson-errorator-handle_error

crimson/common/errorator.h: add handle_error() method.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31853 from liewegas/fix-rn-typo

PendingReleaseNotes: fix typo

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31918 from rzarzynski/wip-crimson-errorator-simplify_safe_then

crimson/common/errorator.h: simplify the compound safe_then() variant.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31838 into master

* refs/pull/31838/head:
ceph-daemon: chown uid/gid files during `adopt`
ceph-daemon: use 'iglob' instead of 'glob'
ceph-daemon: Fix handling for symlinks on python2

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

krbd: misc cleanups

Introduce get_devnode() and append_unmap_options(); make some functions
static.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: make wait_for_udev_add() clearer and a bit more efficient

Collect only /dev/rbd* block events and dispose of them as soon as
possible; match on devnode and assert on major/minor instead of the
other way around.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: do away with explicit memory management

Wrap udev_monitor, udev_enumerate and udev_device with std::unique_ptr.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: Eye candy updates to datatable pages

- Introduce boolean and array pipes
- Introduce configurable 'badge' datatable column
- Refactor the Inventory, OSD, RBD, iSCSI, ... datatable pages

Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/common/errorator.h: simplify the compound safe_then() variant.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common/errorator.h: add handle_error() method.

In contrast to `safe_then` it deals only with errors
and, on return, leaves the value type `future<ValueT>`
embodies unchanged.
While being basically a specialization of `safe_then`,
it's also a convenience method to squeeze boilerplate.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #31914 into master

* refs/pull/31914/head:
mgr/orchestrator: set node labels to empty list if none specified

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: support creating OSDs on spare devices (#30921)

mgr/dashboard: support creating OSDs on spare devices

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>