cmake: silence messages when cppcheck/IWYU is not found

* remove messages spit when these tools are not found
  they are not necessary for building Ceph. and the messages in the cmake
  output are distracting.
* fix a typo

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29687 into master

* refs/pull/29687/head:
test/store_test: test coverage for anti-spillover framework.
os/bluestore: introduce bluestore_volume_selection_reserved cfg option.
os/bluestore: beautify RocksDBBlueFSVolumeSelector::dump output
os/bluestore: introduce 'bluestore bluefs stats' admin socket command
os/bluestore: implement more intelligent DB volume space managment for BlueFS.
os/kv: add static method to parse RocksDB options
os/bluestore: introduce volume selector abstraction to BlueFS

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32108 into master

* refs/pull/32108/head:
ceph-daemon: remove prepare-host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: remove prepare-host

I thought I took this out of the PR but somehow it got merged in... must
have repushed and old branch and not realized.  :/

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31959 into master

* refs/pull/31959/head:
os/bluestore/BlueStore.cc: set priorities for compression stats

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31995 into master

* refs/pull/31995/head:
msg/async: No need lock for func _filter_addrs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32101 into master

* refs/pull/32101/head:
mgr/orchestrator_cli: make rgw rm take realm+zone
qa/tasks/mgr/test_orchestrator_cli: fix rgw add test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/orchestrator_cli: make rgw rm take realm+zone

And adjust test accordingly

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32043 into master

* refs/pull/32043/head:
os/bluestore: default bluestore_block_size 1T -> 100G

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #32044 into master

* refs/pull/32044/head:
mon/OSDMonitor: make memory autotune disable itself if no rocksdb

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31581 into master

* refs/pull/31581/head:
mon: fix mon_sync_max_payload_size type
mon: cap keys in mon_sync messages
mon/MonitorDBStore: improve get_chunk_tx limits
mon/MonitorDBStore: better size estimation for Transaction

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32048 into master

* refs/pull/32048/head:
mon/PGMap: use NONE for 'pg ls[-*]' output too

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32039 into master

* refs/pull/32039/head:
test: Improve races by using kill_daemons which waits for OSDs terminate
test: run-standalone.sh: Only run execs in the subdirectories of qa/standalone
test: Use activate_osd() when restarting OSDs
test: osd-scrub-snaps.sh: Fix race with osd restart and doing a scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix rgw add test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32010 into master

* refs/pull/32010/head:
mgr/ssh: Add module to static type checking from tox.
mgr/orchestrator: Add mypy static type checking

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32023 into master

* refs/pull/32023/head:
qa/tasks/mgr/test_orchestrator_cli: test with valid network or IP
qa/orchestrator: do not test "mon update 3 host1"

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

test: Improve races by using kill_daemons which waits for OSDs terminate

osd-backfill-space.sh: More sleep time to make sure the backfill gets started

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: test with valid network or IP

Passing 'network' will fail with

Error EINVAL: Failed to parse host list: '['host1:network', 'host2:network', 'host3:network']': u'network' does not appear to be an IPv4 or IPv6 address

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32096 into master

* refs/pull/32096/head:
ceph-daemon: fix firewalld error case

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32097 into master

* refs/pull/32097/head:
ceph-daemon: fix bootstrap ownership of tmp monmap file

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32051 into master

* refs/pull/32051/head:
mgr: simply exit on SIGINT or SIGTERM

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: fix bootstrap ownership of tmp monmap file

The file is created with mode 0644 by the ceph:ceph user, but root
cannot write to that.  Instead, create the tmp file owned by root, write
to it, then chown it to user ceph.

Fixes e92929d86dce2297d3e02ec88c6830224aefc061

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: fix firewalld error case

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32021 from falcon78921/wip-doc-43130

doc: fixed broken reference link for Graphviz

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32017 from dzafman/wip-revert-verify-upmap

Revert "crush: remove invalid upmap items"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

test: run-standalone.sh: Only run execs in the subdirectories of qa/standalone

This will ignore scripts placed at the qa/standalone level, though
I'm not sure if we should be putting any tests there.  It does
allow support scripts present like ceph-helper.sh without modifying
run-standalone.sh to ignore it.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #31589 from zhengchengyao/mv_snapshot

librbd:move all snapshot API functions in internal.cc over to api/Snapshot.cc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31767 from pritha-srivastava/wip-rgw-mfa-set-versioning

rgw: adding mfa code validation when bucket versioning status is changed

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/dashboard: adapt to create_osds parameters change (#31928)

mgr/dashboard: adapt to create_osds parameters change

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31062 from dillaman/wip-42355

mgr/dashboard: support RBD mirroring bootstrap create/import

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Cleanup code (#32031)

mgr/dashboard: Cleanup code

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #31949 from dillaman/wip-42748

rbd: incorporate rbd-mirror daemon status in mirror pool status

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29964 into master

* refs/pull/29964/head:
qa/cephfs: test conf file reading for CephFS shell
qa/cephfs: allow passing conf to methods that runs shell command
qa/vstart_runner: add a method to create temporary file
test_cephfs_shell: update test since shell can now read ceph.conf
src/vstart.sh: add cephfs-shell section to ceph.conf
cephfs-shell: read shell variables from ceph.conf

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: test conf file reading for CephFS shell

Adds tests that checks -
* reading conf option
* reading conf option after setting it twice
* reading conf option after setting it after a reset

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: allow passing conf to methods that runs shell command

Add an argument to the methods that issues CephFS shell command to
accept path to the ceph.conf file.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/vstart_runner: add a method to create temporary file

The method's signature is kept identical to the method in teuthology.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: update test since shell can now read ceph.conf

In the test
test_cephfs_shell.TestMisc.test_issue_cephfs_shell_cmd_at_invocation,
don't set the shell option with in the command to be tested since
CephFS shell can now read it's value from ceph.conf.

Also, add methods to set config options for vstart as well teuthology
testing.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

src/vstart.sh: add cephfs-shell section to ceph.conf

And set "debug" to true, so that developers can see tracebacks in case
cephfs-shell crashes.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: read shell variables from ceph.conf

Read ceph.conf and initialize shell variables by reading options set
under section cephfs-shell. If there are sections/options that are
duplicated in ceph.conf, CephFS shell overrides the options with the
values last.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #31897 into master

* refs/pull/31897/head:
doc: obsolete entries for allow_standby_replay
wip-doc: entries obsoleted by allow_standby_replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31403 into master

* refs/pull/31403/head:
vstart.sh: highlight presence of stray conf

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: simply exit on SIGINT or SIGTERM

The python modules don't reliable shut down.  We've fixed a few shutdown
bugs chasing https://tracker.ceph.com/issues/42744.

This was compounded by https://tracker.ceph.com/issues/42981, which seems
to suggest something in the python module teardown is screwing with the
signal handlers.

Now that df507cde8d71063d5873a42f668156e4c32e86f9 is there, the mgr will
blacklist all rados clients the mgr instantiates, which means we can just
exit immediately and let the blacklisting clean things up.

Works-around: https://tracker.ceph.com/issues/42981
Fixes: https://tracker.ceph.com/issues/42744
Signed-off-by: Sage Weil <sage@redhat.com>

test: Use activate_osd() when restarting OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>

test: osd-scrub-snaps.sh: Fix race with osd restart and doing a scrub

Fixes: https://tracker.ceph.com/issues/43150
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #29421 into master

* refs/pull/29421/head:
qa/cephfs: add tests for ACLs
qa/cephfs: allow running tests from xfstests-dev
qa/tasks: add methods to get monitor's sockets
qa/cephfs: don't crash if mountpoint dir is already deleted
vstart_runner.py: set omit_sudo's default value to False
qa/vstart_runner.py: fix get_keyring_path()
qa/cephfs: don't abort if mountpoint is already present
qa/cephfs: allow specifying mountpoint for kernel mounts
qa/cephfs: allow specifying mountpoints for FUSE mounts
qa/vstart_runner.py: allow specifying mountpoint for local FUSE mounts
qa/mount.py: allow setting mountpoint
qa/vstart_runner.py: add a method to create a temporary directory

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon/PGMap: use NONE for 'pg ls[-*]' output too

We already show CRUSH_ITEM_NONE as NONE in the 'pg dump' output as of
57bac8ea88e5cdcb3854852d439f7e6cbf9df900; do the same for 'pg ls' (which
is the friendlier of the two and what users should be using anyway!).

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: make memory autotune disable itself if no rocksdb

Currently, we crash (failed assertion). Instead, just disable autotuning.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: default bluestore_block_size 1T -> 100G

This makes vstart *way* faster.  This option is only really relevant
for dev test environments.  We bumped it up back in dbdd1d9b6ec286982b5e86d4c51f831cc16afc12
from 10G just to make ENOSPC less common in dev/test.  Let's see if 100G
is a better balance.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31926 from yuvalif/wip-yuval-fix-pubsub-record-format

rgw/pubsub: fix records/event json format to match documentation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32036 from zdover23/wip-doc-troubleshooting-pg-verb-fix

doc: s/achieve/achieves/ (Fixed a verb disagreement)

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: obsolete entries for allow_standby_replay

Nautilus release presented the allow_standby_replay fs setting that obsdoleted
several MDS config entries:  mds_standby_for_*, mon_force_standby_active, and
mds_standby_replay.

Removing entries instead of just marking them as "Obsolete' as per batrick
suggestion.

Signed-off-by: Rodrigo Severo <rodrigo@fabricadeideias.com>

doc: s/achieve/achieves (Fixed a verb disagreement)

This bug changes a clause of the form "When x remains in y status
and never achieve a z status" to a clause of the form "When x remains
in y status and never achieves a  z status".

The change is one of "achieve" to "achieves".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #31830 from joke-lee/rgw-sts-crash-token-not-base64

rgw: fix rgw crash when token is not base64 encode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31483 from zhangsw/fix-rgw-reshard-left

rgw: fix a bug that bucket instance obj can't be removed after resharding completed

Reviewed-by: Eric J. Ivancich <ivancich@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Merge PR #31989 into master

* refs/pull/31989/head:
mgr/MgrClient: fix 'tell mgr.x ...'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30878 from zhangsw/fix-rgw-lc-zerodays

rgw: Expiration days can't be zero and  transition days can be zero

Merge PR #31795 into master

* refs/pull/31795/head:
mgr/ssh: 'ssh {check,prepare}-host <host>'
mgr/ssh: return err from _run_ceph_daemon
ceph-daemon: add check-host and prepare-host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32004 into master

* refs/pull/32004/head:
mgr/restful: allow shutdown before we've fully started up

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30498 from jecluis/wip-mon-no-config

mon: allow running without a config file

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: block mirroring bootstrap UI

Two new modal windows allow an admin to create a base64-encoded bootstrap token
which can then be imported into dashboard on another cluster. The bootstrap
token embeds all the necessary data required to connect to a peer cluster for
RBD mirroring.

Fixes: http://tracker.ceph.com/issues/42355
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: integrate site name into block mirroring overview

The site name is used to differentiate clusters for RBD mirroring.
The local site name is now displayed on the block mirroring
overview page. It also includes a new modal window to edit the site
name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: mirror site name and bootstrap service methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: add site name to mirroring summary collection

The summary is periodically polled by the UI so it can now
reflect changes to the site name w/o specifically needing to poll
just the site name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: new controller for RBD mirroring peer bootstrapping

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: new controller for RBD mirroring site name

This allows setting and retrieving the current site name for
RBD mirroring. This is a global property of the cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: fix broken rbd-mirror dashboard API test case

Fixes: https://tracker.ceph.com/issues/42512
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Cleanup code

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #31988 into master

* refs/pull/31988/head:
common/config: less noise about configs from mon we can't apply

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: list configured prometheus alerts (#31937)

mgr/dashboard: list configured prometheus alerts

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Check password complexity in Dashboard CLI comm… (#31916)

mgr/dashboard: Check password complexity in Dashboard CLI commands

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31732 from aclamk/wip-vstart-osds-parallel

vstart: Now all OSDs are starting in parallel. Use --no-parallel to revert to sequential.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31765 from rjfd/wip-fix-mon-public-addr

ceph-mon: keep v1 address type when explicitly set

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #31876 from rosinL/wip-fix-dpdk-zero-length

msg/async/dpdk: fix SEGV caused by zero length packet

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31932 from tchaikov/wip-pr-29610

os/bluestore: print error if spdk_nvme_ns_cmd_writev() fails

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/orchestrator: do not test "mon update 3 host1"

as the network in host spec is always required now

Fixes: https://tracker.ceph.com/issues/43132
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fixed broken reference link for Graphviz

Fixes: https://tracker.ceph.com/issues/43130
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

Revert "crush: remove invalid upmap items"

This reverts commit 712a39e5c9d9848f618ad55a768103d84c0a460f.

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #31715 into master

* refs/pull/31715/head:
mds: initialize the monc later in init()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31428 into master

* refs/pull/31428/head:
qa/tasks: Fixed AttributeError: can't set attribute
qa/tasks: drop/update name from Thrasher

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31898 into master

* refs/pull/31898/head:
mds: remove unnecessary debug warning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31455 into master

* refs/pull/31455/head:
mds: tolerate no snaprealm encoded in on-disk root inode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31680 into master

* refs/pull/31680/head:
tools/cephfs: make 'cephfs-data-scan scan_links' fix dentry's first

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31712 into master

* refs/pull/31712/head:
fs/ceph_mds: set ceph_mds cpu affinity

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/ssh: Add module to static type checking from tox.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #31986 into master

* refs/pull/31986/head:
qa: test volumes plugin mount cleanup
pybind/mgr/volumes: use py3 items iterator
pybind/mgr/volumes: print errors in cleanup timer
qa: improve variable name

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #31998 from yuvalif/kafka_spins_on_cpu

rgw/pubsub: prevent kafka thread from spinning when there are no messages

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31975 into master

* refs/pull/31975/head:
doc/install/containers: note vX.Y.Z[-YYYYMMDD] tags

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge PR #31955 into master

* refs/pull/31955/head:
doc/foundation: add ceph foundation info here

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/orchestrator: Add mypy static type checking

static type checking is a good way to quickly increase the code coverage.
without creating lots of unit tests.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31893 from fghaas/doc-rbd-exclusive-locks

doc: RBD exclusive locks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: show alert panel if prometheus/alertmanager is unconfigured

If the tabs under the "Monitoring" page aren't properly configured, a
notification is shown which explains the user which setting needs to be
enabled and also provides a link to the corresponding documentation.

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #31902 into master

* refs/pull/31902/head:
doc: add description for fuse_disable_pagecache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31948 from zdover23/wip-doc-ntp-ubuntu-chrony2

doc: add "chrony" to preflight checklist for Ubuntu 18.04

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/ssh: 'ssh {check,prepare}-host <host>'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: return err from _run_ceph_daemon

Sometimes callers will want to see this.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: add check-host and prepare-host

Check for (and/or install/configure):

- podman | docker
- systemctl
- LVM2
- chrony (or ntp or timesyncd)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31967 from sebastian-philipp/ssh-understanding-async

mgr/ssh: HACKING.rst: Add Understanding `AsyncCompletion`

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: RBD exclusive locks

A discussion on the ceph-users list uncovered a bit of uncertainty
about how exclusive locking works:

https://www.mail-archive.com/ceph-users@lists.ceph.com/msg56910.html

Add a bit of background information about exclusive locks, and
cross-reference the documentatation on OpenStack, and on CephX.

Signed-off-by: Florian Haas <florian@citynetwork.eu>

qa: test volumes plugin mount cleanup

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: use py3 items iterator

Otherwise this raises an exception.

Fixes: https://tracker.ceph.com/issues/43113
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>