ceph-daemon: trap on EXIT

tempfiles were not being removed after a standalone test failure

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31897 into master

* refs/pull/31897/head:
doc: obsolete entries for allow_standby_replay
wip-doc: entries obsoleted by allow_standby_replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31403 into master

* refs/pull/31403/head:
vstart.sh: highlight presence of stray conf

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29421 into master

* refs/pull/29421/head:
qa/cephfs: add tests for ACLs
qa/cephfs: allow running tests from xfstests-dev
qa/tasks: add methods to get monitor's sockets
qa/cephfs: don't crash if mountpoint dir is already deleted
vstart_runner.py: set omit_sudo's default value to False
qa/vstart_runner.py: fix get_keyring_path()
qa/cephfs: don't abort if mountpoint is already present
qa/cephfs: allow specifying mountpoint for kernel mounts
qa/cephfs: allow specifying mountpoints for FUSE mounts
qa/vstart_runner.py: allow specifying mountpoint for local FUSE mounts
qa/mount.py: allow setting mountpoint
qa/vstart_runner.py: add a method to create a temporary directory

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31926 from yuvalif/wip-yuval-fix-pubsub-record-format

rgw/pubsub: fix records/event json format to match documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32036 from zdover23/wip-doc-troubleshooting-pg-verb-fix

doc: s/achieve/achieves/ (Fixed a verb disagreement)

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: obsolete entries for allow_standby_replay

Nautilus release presented the allow_standby_replay fs setting that obsdoleted
several MDS config entries:  mds_standby_for_*, mon_force_standby_active, and
mds_standby_replay.

Removing entries instead of just marking them as "Obsolete' as per batrick
suggestion.

Signed-off-by: Rodrigo Severo <rodrigo@fabricadeideias.com>

doc: s/achieve/achieves (Fixed a verb disagreement)

This bug changes a clause of the form "When x remains in y status
and never achieve a z status" to a clause of the form "When x remains
in y status and never achieves a  z status".

The change is one of "achieve" to "achieves".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #31830 from joke-lee/rgw-sts-crash-token-not-base64

rgw: fix rgw crash when token is not base64 encode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31483 from zhangsw/fix-rgw-reshard-left

rgw: fix a bug that bucket instance obj can't be removed after resharding completed

Reviewed-by: Eric J. Ivancich <ivancich@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge PR #31989 into master

* refs/pull/31989/head:
mgr/MgrClient: fix 'tell mgr.x ...'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30878 from zhangsw/fix-rgw-lc-zerodays

rgw: Expiration days can't be zero and  transition days can be zero

Merge PR #31795 into master

* refs/pull/31795/head:
mgr/ssh: 'ssh {check,prepare}-host <host>'
mgr/ssh: return err from _run_ceph_daemon
ceph-daemon: add check-host and prepare-host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32004 into master

* refs/pull/32004/head:
mgr/restful: allow shutdown before we've fully started up

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30498 from jecluis/wip-mon-no-config

mon: allow running without a config file

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31988 into master

* refs/pull/31988/head:
common/config: less noise about configs from mon we can't apply

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: list configured prometheus alerts (#31937)

mgr/dashboard: list configured prometheus alerts

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Check password complexity in Dashboard CLI comm… (#31916)

mgr/dashboard: Check password complexity in Dashboard CLI commands

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31732 from aclamk/wip-vstart-osds-parallel

vstart: Now all OSDs are starting in parallel. Use --no-parallel to revert to sequential.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31765 from rjfd/wip-fix-mon-public-addr

ceph-mon: keep v1 address type when explicitly set

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #31876 from rosinL/wip-fix-dpdk-zero-length

msg/async/dpdk: fix SEGV caused by zero length packet

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31932 from tchaikov/wip-pr-29610

os/bluestore: print error if spdk_nvme_ns_cmd_writev() fails

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31715 into master

* refs/pull/31715/head:
mds: initialize the monc later in init()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31428 into master

* refs/pull/31428/head:
qa/tasks: Fixed AttributeError: can't set attribute
qa/tasks: drop/update name from Thrasher

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31898 into master

* refs/pull/31898/head:
mds: remove unnecessary debug warning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31455 into master

* refs/pull/31455/head:
mds: tolerate no snaprealm encoded in on-disk root inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31680 into master

* refs/pull/31680/head:
tools/cephfs: make 'cephfs-data-scan scan_links' fix dentry's first

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31712 into master

* refs/pull/31712/head:
fs/ceph_mds: set ceph_mds cpu affinity

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31986 into master

* refs/pull/31986/head:
qa: test volumes plugin mount cleanup
pybind/mgr/volumes: use py3 items iterator
pybind/mgr/volumes: print errors in cleanup timer
qa: improve variable name

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #31998 from yuvalif/kafka_spins_on_cpu

rgw/pubsub: prevent kafka thread from spinning when there are no messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31975 into master

* refs/pull/31975/head:
doc/install/containers: note vX.Y.Z[-YYYYMMDD] tags

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge PR #31955 into master

* refs/pull/31955/head:
doc/foundation: add ceph foundation info here

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31893 from fghaas/doc-rbd-exclusive-locks

doc: RBD exclusive locks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: show alert panel if prometheus/alertmanager is unconfigured

If the tabs under the "Monitoring" page aren't properly configured, a
notification is shown which explains the user which setting needs to be
enabled and also provides a link to the corresponding documentation.

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #31902 into master

* refs/pull/31902/head:
doc: add description for fuse_disable_pagecache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31948 from zdover23/wip-doc-ntp-ubuntu-chrony2

doc: add "chrony" to preflight checklist for Ubuntu 18.04

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/ssh: 'ssh {check,prepare}-host <host>'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: return err from _run_ceph_daemon

Sometimes callers will want to see this.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: add check-host and prepare-host

Check for (and/or install/configure):

- podman | docker
- systemctl
- LVM2
- chrony (or ntp or timesyncd)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31967 from sebastian-philipp/ssh-understanding-async

mgr/ssh: HACKING.rst: Add Understanding `AsyncCompletion`

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: RBD exclusive locks

A discussion on the ceph-users list uncovered a bit of uncertainty
about how exclusive locking works:

https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56910.html

Add a bit of background information about exclusive locks, and
cross-reference the documentatation on OpenStack, and on CephX.

Signed-off-by: Florian Haas <florian@citynetwork.eu>

qa: test volumes plugin mount cleanup

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: use py3 items iterator

Otherwise this raises an exception.

Fixes: https://tracker.ceph.com/issues/43113
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: print errors in cleanup timer

The squelched error prevented us from knowing connection cleanup doesn't
work on py3.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: improve variable name

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/install/containers: note vX.Y.Z[-YYYYMMDD] tags

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add "chrony" to preflight checklist for Ubuntu 18.04

This commit updates the "Install NTP" section of the preflight
checklist, instructing readers who are using Ubuntu 18.04 to
use the package name "chrony" where elsewhere the package name
"ntp" would be used. "chrony" has replaced "ntp" in Ubnutu 18.04.

This commit fixes Bug number 8 in the list here:
https://pad.ceph.com/p/Report_Documentation_Bugs

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #31972 from clwluvw/patch-1

doc: rgw: Fixed bug on wrong name for user_id for OPA

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31906 from s0nea/wip-dashboard-fix-pg-num-warning

mgr/dashboard: fix tests in order to match pg num conventions

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/restful: allow shutdown before we've fully started up

Sometimes we get shutdown() before self.server has been assigned.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31797 into master

* refs/pull/31797/head:
qa: test mgr cephfs mount blacklist
mgr: forward RADOS client instances for potential blacklist
mon/MgrMonitor: blacklist previous instance
mon: set RUNTIME flag on mon_mds_blacklist_interval
mgr: use more efficient data structure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31950 from dillaman/wip-mirror-peer-fsid

rbd-mirror: peer_ping should send the local fsid to the remote

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #31908 into master

* refs/pull/31908/head:
ceph-daemon: use uid/gid 0 when creating tempfiles
ceph-daemon: drop unneeded `mode` during tmpfile create
ceph-daemon: consolidate NamedTemporaryFile logic
ceph-daemon: chown uid/gid `ceph-volume` tmpfiles

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31940 into master

* refs/pull/31940/head:
mgr/ssh: Port raising exceptions from completion handlers to Py2

Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/dashboard: extend dashboard documentation

by adding the previously added monitoring related features as well as
the newest feature addition. Extends the documentation where necessary
to describe the Prometheus' alert configuration.

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #31990 from dzafman/wip-43084

tools/osdmaptool.cc: do not use deprecated std::random_shuffle()

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

qa: test mgr cephfs mount blacklist

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: forward RADOS client instances for potential blacklist

The mgr creates a per-module RADOS client connection for modules which
interact with RADOS (e.g. the volumes module). These clients should also
be blacklisted when the active mgr is failed; we don't want the former
active mgr to continue interacting with RADOS when the new one takes
over. This is particularly impactful for avoiding extraneous
"unresponsive client" warnings from the MDS when the mgr switches
(especially in testing). The MDS will pickup the new OSD blacklists
which include's the old mgr's libcephfs instance and blacklist/evict
that session quietly.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MgrMonitor: blacklist previous instance

This wasn't realized as necessary early on in the ceph-mgr development
because the mgr didn't interact with RADOS. However, now it is becoming
common for plugins to store data there. It's important that the previous
instance can no longer interact with RADOS while the new mgr takes over.

In particular, this means that the mgr's client sessions with the MDS
are automatically evicted once the MDS receives the new OSDMap. This
avoids a pesky "unresponsive client" warning in the cluster logs.

Fixes: https://tracker.ceph.com/issues/42939
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon: set RUNTIME flag on mon_mds_blacklist_interval

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: use more efficient data structure

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

tools/osdmaptool.cc: do not use deprecated std::random_shuffle()
the use of `std::random_shuffle()` was introduced by
b946308 .

in this change, it is replaced using `std::shuffle()`.

Fixes: https://tracker.ceph.com/issues/43084
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/MgrClient: fix 'tell mgr.x ...'

If we start the command before we get the mgrmap at the client, we
will send MCommand via a different code path; make sure it behaves
correctly too.

Missed this in 6f35d2835268eade059535b62378d6d407ef9e87

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: less noise about configs from mon we can't apply

It is normal to set things in teh mon config database that can't be absorbed
by clients/daemons at runtime.  Things like ms_type can/should be set there
(so that they, for instance, help produce the 'config minimal-conf') but
we don't need daemons to spit out error messages every time they get a config
update.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31891 into master

* refs/pull/31891/head:
qa/suites/rados/ssh: only install ceph-daemon for packaged mode
tasks/ceph2: add support for packaged ceph-daemon
qa/tasks/ceph2: make image cluster-private
qa/tasks/ceph2: add 'shell' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31984 into master

* refs/pull/31984/head:
mgr/telemetry: fix log typo

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31869 into master

* refs/pull/31869/head:
ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()
qa/standalone/test_ceph_daemon.sh: more $SUDO
ceph-daemon: configure firewalld for new daemon deploys
ceph-daemon: name mgr the same way mgr/ssh does

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31930 into master

* refs/pull/31930/head:
mgr/orchestrator: add optional "format" param for "orchestrator host ls"

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31968 from idryomov/wip-krbd-python3

qa: update krbd tests for python3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/telemetry: fix log typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31983 into master

* refs/pull/31983/head:
mgr/ssh/tests: fix RGWSpec test

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw/pubsub: support eventId in push mode

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rgw/pubsub: prevent kafka thread from spinning when there are no
messages

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/ssh/tests: fix RGWSpec test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31973 into master

* refs/pull/31973/head:
mgr/crash: fix 'crash ls[-new]' sorting

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: grafana charts match time picker selection. (#31964)

mgr/dashboard: grafana charts match time picker selection.

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

doc/foundation: add ceph foundation info here

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31890 into master

* refs/pull/31890/head:
mgr/orchestrator: name rgw by client.rgw.$realm.$zone[.$id]

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #31952 into master

* refs/pull/31952/head:
mgr/ssh: fix remove_osds
mgr/ssh: fix service_action

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: bootstrap: deploy initial mon via deploy_daemon()

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: more $SUDO

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: configure firewalld for new daemon deploys

Note that we only open firewalld ports; we never close them.

We could conceivably do that from rm-cluster, but that might also interfere
with something else on the host...

Signed-off-by: Sage Weil <sage@redhat.com>

Fixed bug on wrong name for user_id

Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #31202 from athanatos/sjust/wip-crimson-obc

crimson, common: Add ephemeral ObjectContext state to crimson

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: move monitoring tabs to a single page

with a tab for 'active alerts', 'all alerts' and 'silences'. Due to
ambiguity with existing names, `AlertListComponent` has been renamed to
`ActiveAlertListComponent`. Introduces `MonitoringListComponent` as
first page for monitoring concerns, using path `/monitoring`.

Keeps the activated tab open, independent of the way that's used to go
back to the previous page, be it the cancel button or submit button or
the link on the breadcrumb. Also keeps the active tab open even when the
page is reloaded.

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: refactor test of Prometheus alert service

Mocking the test the way it was removed the asynchronous nature of the
test. By using an Observable the test can stay asynchronous and be
tested as well.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: list configured Prometheus alerts

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: return Prometheus alert groups, too

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/ssh: HACKING.rst: Add Understanding `AsyncCompletion`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-daemon: use uid/gid 0 when creating tempfiles

skip extraction of the uid/gid since `ceph-volume` runs as root

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31473 from ifed01/wip-ifed-smplify-fsck

os/bluestore: simplify multithreaded shallow fsck.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31969 from sebastian-philipp/mgr-fix-mocking

pybind/mgr: Make it easier to create a Module instance without the mgr

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard,grafana: remove shortcut menu (#31951)

mgr/dashboard,grafana: remove shortcut menu

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: grafana charts match time picker selection.

Fixes: https://tracker.ceph.com/issues/43097
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/crash: fix 'crash ls[-new]' sorting

Broken by c691f2f8763a9db081a0b61294c5eedf4e9c2cee

Signed-off-by: Sage Weil <sage@redhat.com>

qa/cephfs: add tests for ACLs

Add code to run tests for ACLs from xfstests-dev against kernel and
FUSE CephFS mounts.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #31960 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0093 unless it is supported

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: do not use CMP0093 unless it is supported

CMake CMP0093 was introduced by CMake 3.15, but we only require CMake
v3.10.2. so use the old `Boost_VERSION_MACRO` variable by default,
unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: Make it easier to create a Module instance without the mgr

Caused by conflict between #30961 and #30262

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31872 from smithfarm/wip-mimic-v13.2.7-doc2

doc/releases: add mimic v13.2.7 to releases timeline

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>