Merge pull request #31876 from rosinL/wip-fix-dpdk-zero-length

msg/async/dpdk: fix SEGV caused by zero length packet

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31932 from tchaikov/wip-pr-29610

os/bluestore: print error if spdk_nvme_ns_cmd_writev() fails

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31715 into master

* refs/pull/31715/head:
mds: initialize the monc later in init()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31428 into master

* refs/pull/31428/head:
qa/tasks: Fixed AttributeError: can't set attribute
qa/tasks: drop/update name from Thrasher

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31898 into master

* refs/pull/31898/head:
mds: remove unnecessary debug warning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31455 into master

* refs/pull/31455/head:
mds: tolerate no snaprealm encoded in on-disk root inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31680 into master

* refs/pull/31680/head:
tools/cephfs: make 'cephfs-data-scan scan_links' fix dentry's first

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31712 into master

* refs/pull/31712/head:
fs/ceph_mds: set ceph_mds cpu affinity

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31986 into master

* refs/pull/31986/head:
qa: test volumes plugin mount cleanup
pybind/mgr/volumes: use py3 items iterator
pybind/mgr/volumes: print errors in cleanup timer
qa: improve variable name

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #31998 from yuvalif/kafka_spins_on_cpu

rgw/pubsub: prevent kafka thread from spinning when there are no messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31975 into master

* refs/pull/31975/head:
doc/install/containers: note vX.Y.Z[-YYYYMMDD] tags

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge PR #31955 into master

* refs/pull/31955/head:
doc/foundation: add ceph foundation info here

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31893 from fghaas/doc-rbd-exclusive-locks

doc: RBD exclusive locks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31902 into master

* refs/pull/31902/head:
doc: add description for fuse_disable_pagecache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31948 from zdover23/wip-doc-ntp-ubuntu-chrony2

doc: add "chrony" to preflight checklist for Ubuntu 18.04

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31967 from sebastian-philipp/ssh-understanding-async

mgr/ssh: HACKING.rst: Add Understanding `AsyncCompletion`

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: RBD exclusive locks

A discussion on the ceph-users list uncovered a bit of uncertainty
about how exclusive locking works:

https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56910.html

Add a bit of background information about exclusive locks, and
cross-reference the documentatation on OpenStack, and on CephX.

Signed-off-by: Florian Haas <florian@citynetwork.eu>

qa: test volumes plugin mount cleanup

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: use py3 items iterator

Otherwise this raises an exception.

Fixes: https://tracker.ceph.com/issues/43113
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: print errors in cleanup timer

The squelched error prevented us from knowing connection cleanup doesn't
work on py3.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: improve variable name

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/install/containers: note vX.Y.Z[-YYYYMMDD] tags

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add "chrony" to preflight checklist for Ubuntu 18.04

This commit updates the "Install NTP" section of the preflight
checklist, instructing readers who are using Ubuntu 18.04 to
use the package name "chrony" where elsewhere the package name
"ntp" would be used. "chrony" has replaced "ntp" in Ubnutu 18.04.

This commit fixes Bug number 8 in the list here:
https://pad.ceph.com/p/Report_Documentation_Bugs

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #31972 from clwluvw/patch-1

doc: rgw: Fixed bug on wrong name for user_id for OPA

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31906 from s0nea/wip-dashboard-fix-pg-num-warning

mgr/dashboard: fix tests in order to match pg num conventions

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31797 into master

* refs/pull/31797/head:
qa: test mgr cephfs mount blacklist
mgr: forward RADOS client instances for potential blacklist
mon/MgrMonitor: blacklist previous instance
mon: set RUNTIME flag on mon_mds_blacklist_interval
mgr: use more efficient data structure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31950 from dillaman/wip-mirror-peer-fsid

rbd-mirror: peer_ping should send the local fsid to the remote

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #31908 into master

* refs/pull/31908/head:
ceph-daemon: use uid/gid 0 when creating tempfiles
ceph-daemon: drop unneeded `mode` during tmpfile create
ceph-daemon: consolidate NamedTemporaryFile logic
ceph-daemon: chown uid/gid `ceph-volume` tmpfiles

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31940 into master

* refs/pull/31940/head:
mgr/ssh: Port raising exceptions from completion handlers to Py2

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #31990 from dzafman/wip-43084

tools/osdmaptool.cc: do not use deprecated std::random_shuffle()

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa: test mgr cephfs mount blacklist

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: forward RADOS client instances for potential blacklist

The mgr creates a per-module RADOS client connection for modules which
interact with RADOS (e.g. the volumes module). These clients should also
be blacklisted when the active mgr is failed; we don't want the former
active mgr to continue interacting with RADOS when the new one takes
over. This is particularly impactful for avoiding extraneous
"unresponsive client" warnings from the MDS when the mgr switches
(especially in testing). The MDS will pickup the new OSD blacklists
which include's the old mgr's libcephfs instance and blacklist/evict
that session quietly.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MgrMonitor: blacklist previous instance

This wasn't realized as necessary early on in the ceph-mgr development
because the mgr didn't interact with RADOS. However, now it is becoming
common for plugins to store data there. It's important that the previous
instance can no longer interact with RADOS while the new mgr takes over.

In particular, this means that the mgr's client sessions with the MDS
are automatically evicted once the MDS receives the new OSDMap. This
avoids a pesky "unresponsive client" warning in the cluster logs.

Fixes: https://tracker.ceph.com/issues/42939
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon: set RUNTIME flag on mon_mds_blacklist_interval

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: use more efficient data structure

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tools/osdmaptool.cc: do not use deprecated std::random_shuffle()
the use of `std::random_shuffle()` was introduced by
b946308 .

in this change, it is replaced using `std::shuffle()`.

Fixes: https://tracker.ceph.com/issues/43084
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31891 into master

* refs/pull/31891/head:
qa/suites/rados/ssh: only install ceph-daemon for packaged mode
tasks/ceph2: add support for packaged ceph-daemon
qa/tasks/ceph2: make image cluster-private
qa/tasks/ceph2: add 'shell' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31984 into master

* refs/pull/31984/head:
mgr/telemetry: fix log typo

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31869 into master

* refs/pull/31869/head:
ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()
qa/standalone/test_ceph_daemon.sh: more $SUDO
ceph-daemon: configure firewalld for new daemon deploys
ceph-daemon: name mgr the same way mgr/ssh does

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31930 into master

* refs/pull/31930/head:
mgr/orchestrator: add optional "format" param for "orchestrator host ls"

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31968 from idryomov/wip-krbd-python3

qa: update krbd tests for python3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/telemetry: fix log typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31983 into master

* refs/pull/31983/head:
mgr/ssh/tests: fix RGWSpec test

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw/pubsub: prevent kafka thread from spinning when there are no
messages

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/ssh/tests: fix RGWSpec test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31973 into master

* refs/pull/31973/head:
mgr/crash: fix 'crash ls[-new]' sorting

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: grafana charts match time picker selection. (#31964)

mgr/dashboard: grafana charts match time picker selection.

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

doc/foundation: add ceph foundation info here

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31890 into master

* refs/pull/31890/head:
mgr/orchestrator: name rgw by client.rgw.$realm.$zone[.$id]

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #31952 into master

* refs/pull/31952/head:
mgr/ssh: fix remove_osds
mgr/ssh: fix service_action

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: more $SUDO

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: configure firewalld for new daemon deploys

Note that we only open firewalld ports; we never close them.

We could conceivably do that from rm-cluster, but that might also interfere
with something else on the host...

Signed-off-by: Sage Weil <sage@redhat.com>

Fixed bug on wrong name for user_id

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #31202 from athanatos/sjust/wip-crimson-obc

crimson, common: Add ephemeral ObjectContext state to crimson

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/ssh: HACKING.rst: Add Understanding `AsyncCompletion`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-daemon: use uid/gid 0 when creating tempfiles

skip extraction of the uid/gid since `ceph-volume` runs as root

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31473 from ifed01/wip-ifed-smplify-fsck

os/bluestore: simplify multithreaded shallow fsck.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31969 from sebastian-philipp/mgr-fix-mocking

pybind/mgr: Make it easier to create a Module instance without the mgr

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard,grafana: remove shortcut menu (#31951)

mgr/dashboard,grafana: remove shortcut menu

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: grafana charts match time picker selection.

Fixes: https://tracker.ceph.com/issues/43097
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/crash: fix 'crash ls[-new]' sorting

Broken by c691f2f8763a9db081a0b61294c5eedf4e9c2cee

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31960 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0093 unless it is supported

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: do not use CMP0093 unless it is supported

CMake CMP0093 was introduced by CMake 3.15, but we only require CMake
v3.10.2. so use the old `Boost_VERSION_MACRO` variable by default,
unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: Make it easier to create a Module instance without the mgr

Caused by conflict between #30961 and #30262

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31872 from smithfarm/wip-mimic-v13.2.7-doc2

doc/releases: add mimic v13.2.7 to releases timeline

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Increase max-width of popovers (#31743)

mgr/dashboard: Increase max-width of popovers

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30961 from rjfd/wip-mgr-module-logging

mgr: module logging infrastructure

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard,grafana: remove shortcut menu

Remove shortcut menu (links) and add check in grafana CI script.

Fixes: https://tracker.ceph.com/issues/43091
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31962 from bk201/wip-43095

vstart: fix error when getting CMake variables with the same prefix

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart: fix error when getting CMake variables with the same prefix

The function fails to work when there are multiple variables starts with
the same prefix. e.g.

- WITH_RBD
- WITH_RBD_RGW

Fixes: https://tracker.ceph.com/issues/43095
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

crimson/osd: convert PG::wait_for_active to a blocking_future

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: use ObjectContext and take obc locks

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: factor out load_object_state/snapset

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: add ObjectContext/Registry

ObjectContext will hold the ephemeral state and lock status
for an object as in classic osd.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd: factor out RWState, move SnapSetContext, ObjectState, RWState

We'll want to reuse this logic in crimson, but the actual operation
wake/wait mechanics will be different.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/: introduce intrusive_lru

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: factor OSD::init_op_flags into seperate class

We'll want to reuse this in crimson, extract the logic
for setting flags from MOSDOp into osd_types.h.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator.h: add finally method for resource release

This way we can add things like resource handlers which are otherwise
transparent to passing values and errors.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: add --debug for crimson if debug is enabled

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #31947 from zdover23/wip-doc-ubuntu-1804-python-minimal

doc: add Python 2 to Ubuntu 18.04 installations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31958 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0074 unless it is supported

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/suites/rados/ssh: only install ceph-daemon for packaged mode

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: do not use CMP0074 unless it is supported

CMake CMP0074 was introduced by CMake 3.12, but we only require CMake
v3.10.2. so default to check `Boost_ROOT` unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: drop unneeded `mode` during tmpfile create

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: name mgr the same way mgr/ssh does

(with a random 6-char id, not the hostname)

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: consolidate NamedTemporaryFile logic

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: chown uid/gid `ceph-volume` tmpfiles

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31913 into master

* refs/pull/31913/head:
ceph-daemon: Allow env var for setting the used image

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

tasks/ceph2: add support for packaged ceph-daemon

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix remove_osds

Broken by the new completions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: fix service_action

Broken by the new completions

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: peer_ping should send the local fsid to the remote

It was previously incorrectly sending the remote cluster's
fsid to the remote cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31919 from idryomov/wip-krbd-udev-improvements

krbd: do away with explicit memory management and other cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: update krbd tests for python3

There is no /usr/bin/python symlink in CentOS 8 even if python2 is
installed.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #31933 into master

* refs/pull/31933/head:
mgr/ssh: `test_mon_update` needs to set a mon name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31274 into master

* refs/pull/31274/head:
osd/OSD: enhance osd numa affinity compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31348 into master

* refs/pull/31348/head:
tools/rados: update advisory lock break usage with --lock-cookie required

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31669 into master

* refs/pull/31669/head:
bluestore: tiny tracepoints improvement

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #31943 from tchaikov/wip-31224

python-common: move pytest integration from setup.py to tox.ini

Reviewed-by: Sebastian Wagner <swagner@suse.com>