qa: ignore MON_DOWN for volume-client testing

The test restarts the monitors.

Fixes: http://tracker.ceph.com/issues/38704
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26892 into master

* refs/pull/26892/head:
qa: stop testing simple messenger in CephFS suites

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26503 from ricardoasmarques/iscsi-support-dots-in-name

mgr/dashboard: iSCSI - Adds support for pool/image names with dots

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26891 from rjfd/wip-dashboard-iscsi-ssl

mgr/dashboard: iscsi: adds CLI command to enable/disable API SSL verification

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26658 from ivancich/wip-lc-prefix-fix

rgw: fix unordered bucket listing when object names are adorned

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26545 from ivancich/wip-rgw-gc-stall-min-fixes

rgw: resolve bug where marker was not advanced during garbage collection

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26601 from ivancich/wip-rgw-gc-stall-fixes

rgw: resolve bugs and clean up garbage collection code

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26599 from cbodley/wip-rgw-beast-default

rgw: make beast the default for rgw_frontends

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26896 from dillaman/wip-rbd-cls-warning

cls/rbd: fixed uninitialized variable compiler warning

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26854 from ricardoasmarques/fix-mutual-password-input-type

mgr/dashboard: Fix iSCSI mutual password input type

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix iSCSI mutual password input type

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #26906 into master

* refs/pull/26906/head:
Typo in Nautilus release notes: not -> now

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #26904 from SUSE/wip-orchestrator-deepsea-sharedsecret

mgr/deepsea: always use 'password' parameter for salt-api auth

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #26851 from rhcs-dashboard/36176-enable-api-tests-coverage

mgr/dashboard: enable coverage for API tests

Merge pull request #26669 from ricardoasmarques/iscsi-docs-disk-slash

doc/rbd/iscsi-target-cli: Update disk separator

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: tests: move helper classes to __init__.py

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: add iscsi CLI commands testing

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: update tests that mock mons kv store

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: add mixin classes to mock mons KV store

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc/mgr/dashboard: update iscsi documentation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: iscsi: use `ssl_verify` configuration option in restclient

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: iscsi: add CLI command to disable API SSL verification

Signed-off-by: Ricardo Dias <rdias@suse.com>

Adds support for pool/image names with dots

According to https://github.com/ceph/ceph-iscsi/pull/29

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Typo in Nautilus release notes: not -> now

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

mgr/deepsea: always use 'password' parameter for salt-api auth

Prior to https://github.com/saltstack/salt/commit/71d5601507, the
salt-api expected the password to be sent using the 'sharedsecret'
parameter if using shared secrets, and the 'password' parameter
for other authentication types.  The above commit unifies this so
that we always only need to use the 'password' parameter.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #26902 from tchaikov/wip-doc-erasure-clay

doc/rados/operations: add clay to erasure-code-profile

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/rados/operations: add clay to erasure-code-profile

so it's more visible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26763 into master

* refs/pull/26763/head:
mgr/diskprediction_cloud: Fixed cannot import string.maketrans in the python 3.x

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cls/rbd: fixed uninitialized variable compiler warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26885 into master

* refs/pull/26885/head:
doc/releases/nautilus: add msgr2 refs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26736 into master

* refs/pull/26736/head:
mgr: push localized config handling into BaseMgr*Module
mgr: Fix broken get_localized_module_option function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #26856 into master

* refs/pull/26856/head:
os/bluestore: make osd_memory_limit default to .8x the cgroup limit
os/bluestore: observe osd_memory_target config options
common/config: set osd_memory_target based on POD_MEMORY_REQUEST
common/util: include group limit in daemon metadata
common/util: get_cgroup_memory_limit() helper
common/config: pass entity_type down into parse_env()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #26879 into master

* refs/pull/26879/head:
vstart.sh: make 'db' volume 1G instead of 64M
os/bluestore: make bluefs spillover messages nicer

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #26878 from dillaman/wip-pool-init-validation

librbd: bypass pool validation if "rbd_validate_pool" is false

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26888 into master

* refs/pull/26888/head:
doc/releases/nautilus: add 'ceph-volume simple scan' to upgrade instructions

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

os/bluestore: make osd_memory_limit default to .8x the cgroup limit

We do this at runtime in _set_cache_sizes so that changes to these values
at runtime will be picked up.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: push localized config handling into BaseMgr*Module

We need both (1) the raw key name (in order to look up the correct option
name and transform the result into the correct type) and (2) the optional
prefix (in order to look up both possible keys).  This simplifies the
mgr_module implementation.

There is a weird mismatch between the way that BaseMgrModule and
BaseMgrStandbyModule implement this method.  I suspect they could be
unified to work the same, but I'll leave that for another day.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: Fix broken get_localized_module_option function

Fixes: https://tracker.ceph.com/issues/38560
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #26818 into master

* refs/pull/26818/head:
qa/cephfs: relax min_caps_per_client check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: stop testing simple messenger in CephFS suites

Simple messenger is on it's way out and it doesn't work with msgr2.

Fixes: http://tracker.ceph.com/issues/38676
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/nautilus: add msgr2 refs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26883 from rhcs-dashboard/wip-38657-master

mgr/dashboard: Change 'Client Recovery' title

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26721 from dillaman/wip-38538

librbd: improve object map performance under high IOPS workloads

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/releases/nautilus: add 'ceph-volume simple scan' to upgrade instructions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26799 from wjwithagen/wjw-fix-/src/test/rgw/CMakeLists.txt

build/rgw: unittest_rgw_dmclock_scheduler does not need Boost_LIBRARIES

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26877 into master

* refs/pull/26877/head:
mgr/MgrStandby: exit with 0 result code, as if we'd done an orderly shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: observe osd_memory_target config options

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: set osd_memory_target based on POD_MEMORY_REQUEST

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26858 into master

* refs/pull/26858/head:
qa: krbd deep-flatten test
qa/suites/krbd: enable deep-flatten feature

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/MgrStandby: exit with 0 result code, as if we'd done an orderly shutdown

This matches the prior behavior with shutdown().  It also makes
teuthology happy.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Change 'Client Recovery' title

Rename 'Client Recovery' info-card title to 'Recovery Throughput' as:
- 'Client' is a misleading concept there, as long as all recovery is
server-side driven.
- No indication of the kind of metric (e.g.: IOPS, throughput, etc).

Fixes: https://tracker.ceph.com/issues/38657
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #26825 from idryomov/wip-copyup-truncate

librbd: don't do create+truncate for discards with copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26867 from liewegas/wip-doc-msgr2

doc/rados/configuration/msgr2: some documentation about msgr2

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

vstart.sh: make 'db' volume 1G instead of 64M

64M is barely usable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make bluefs spillover messages nicer

Was

 osd:50 spilled over to slow device: 1.3 GiB

Now

 osd.0 spilled over 1.7 GiB metadata from 'db' device (1000 MiB used of 1024 MiB) to slow device

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: bypass pool validation if "rbd_validate_pool" is false

This config option is already used by the create image state
machine. This should address a broken test case when validating
pool vs self-managed snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26801 from wjwithagen/wjw-fix-use-binutils-ld

build: LLVM ld does not like the versioning scheme

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26859 into master

* refs/pull/26859/head:
qa: ignore slow metadata io wrn during osd thrash

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: improve object map performance under high IOPS workloads

Do not zero-fill the BitVector's bitset prior to decoding the data.
Additionally, only read-update-modify the portions of the footer
that are potentially affected by the updated state.

Fixes: http://tracker.ceph.com/issues/38538
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26876 from wjwithagen/wjw-fix-src/test/librbd/operation/test_mock_TrimRequest.cc

librbd: avoid aggregate-initializing any static_visitor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid aggregate-initializing any static_visitor

This will result in an error complaining about a protected dtor.
also see:
    https://github.com/ceph/ceph/pull/26838

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/rados/configuration/msgr2: some documentation about msgr2

This doesn't integrate very well into network-config.rst, mostly because
that document is horribly out of data and I don't know where to start.
:(

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26849 into master

* refs/pull/26849/head:
Revert "msg/async: msgr2: expected tags validation"
msg/async/ProtocolV2: state machine verifications

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26869 into master

* refs/pull/26869/head:
qa/standalone/erasure-code/test-erasure-eio.sh: still need to create rbd pool
qa/standalone/scrub/osd-scrub-repair: fix unfound grep
qa/standalone/ceph-helpers: fix test_wait_for_clean
qa/standalone/ceph-helpers.sh: fix test_run_mon
qa/standalone/osd/osd-rep-recov-eio: fix better

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26713 into master

* refs/pull/26713/head:
mds: wait for client to release shared cap when re-acquiring xlock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26828 into master

* refs/pull/26828/head:
src/tools/kvstore_tool.cc: Move bluestore start-up code

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/erasure-code/test-erasure-eio.sh: still need to create rbd pool

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26660 into master

* refs/pull/26660/head:
include/intarith: enforce the same type for p2*() arguments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26831 into master

* refs/pull/26831/head:
mon/OSDMonitor: add newline
mgr/pg_autoscaler: drop undefined variable 'cr_name'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26779 into master

* refs/pull/26779/head:
mgr/status: fix 'ceph fs status' AttributeError

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

src/tools/kvstore_tool.cc: Move bluestore start-up code

Move this code into a separate function to avoid ICE.

Fixes: http://tracker.ceph.com/issues/38633
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/standalone/scrub/osd-scrub-repair: fix unfound grep

It's now "1/2 unfound":

             1/2 objects unfound (50.000%)

..presumably due to the rbd pool init creating the rbd_directory.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/ceph-helpers: fix test_wait_for_clean

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/ceph-helpers.sh: fix test_run_mon

- Only create each osd once
- forget the first osdmap dump test; it's pointless

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/osd-rep-recov-eio: fix better

- no need for the default pool size
- no initial osds or it will collide with setup_osds later
- no need for rbd pool at all

Signed-off-by: Sage Weil <sage@redhat.com>

common/util: include group limit in daemon metadata

Signed-off-by: Sage Weil <sage@redhat.com>

common/util: get_cgroup_memory_limit() helper

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: pass entity_type down into parse_env()

No functional change, but we'll need this shortly.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: fix unordered bucket listing when object names are adorned

Namespaces, instances, and multipart metadata adorn object names and
are not used in the computation of an object's bucket index shard. The
logic for unordered bucket listing ignored these complexities, which
yielded incorrect results when dealing with such adorned object names.
Markers, for example, failed to continue an unordered bucket listing
where it left off and could sometimes cause listing loops.

The lifecycle rule dealing with multipart expiration exposed these
issues.

This commit makes unordered bucket listing handle these complexities.

Fixes: http://tracker.ceph.com/issues/38486
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: code-cleanup -- add const when possible and helpful

Found a number of opportunities to add const to parameters and
to member functions.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

qa: ignore slow metadata io wrn during osd thrash

Fixes: http://tracker.ceph.com/issues/38651
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26838 into master

* refs/pull/26838/head:
librbd: avoid aggregate-initializing IsWriteOpVisitor

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build/rgw: unittest_rgw_dmclock_scheduler does not need Boost_LIBRARIES

Otherwise linking could error like:
```
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyUnicodeUCS4_FromEncodedObject'
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyNumber_InPlaceDivide'
........
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyStaticMethod_Type'
/usr/local/bin/ld: /usr/local/lib/libboost_python27.so: undefined reference to `PyTuple_Size'
c++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [src/test/rgw/CMakeFiles/unittest_rgw_dmclock_scheduler.dir/build.make:147: bin/unittest_rgw_dmclock_scheduler] Error 1
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa: krbd deep-flatten test

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26826 from ceph/wip-simple-scan-all

ceph-volume: `simple scan` will now scan all running ceph-disk OSDs

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: enable coverage for API tests

* Fixed: install in system the 'coverage' version set in
  requirements.txt as dashboard/module.py uses system version.
* Updated 'coverage' to latest stable version.
* Added teuthology missing dependency:
  backports.ssl-match-hostname
* More precise coverage measurement through CherryPy Bus API.
* Coverage report shown when no ceph-mgr running
  (cleanup_teuthology).

Fixes: https://tracker.ceph.com/issues/36176
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

ceph-volume: make systemctl.get_running_osds resilient to garbage output

Makes systemd.systemctl.get_running_osds more resilient to output
that doesn't match what we're expecting to parse. This also handles
the case of a nonzero exit status by retuning an empty list.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: remove obsoleted simple unit test

This test is no longer valid because `simple scan`
without any arguments will now attempt to scan all
running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: tox tests should ignore the tests/functional directory

Without this the functional testinfra tests try to run when we
are intending to only run the unit tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use `simple scan` and `simple activate --all` in tests

In a couple of the functional tests for simple we should run `simple
scan` without any arguments and `simple activate --all` to increase
the test coverage. We want to test pointing at a diretory and scanning
all running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update docs and help menu for `simple scan`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds unit tests for systemd.systemctl.get_running_osd_ids

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update `simple scan` to scan all running OSDs

If no argument is passed to `ceph-volume simple scan` it will
inspect any running osds and scan them if they were created by
ceph-disk.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add systemd.systemctl.get_running_osd_ids

This is used to retrieve a list of osd ids from the currently
running ceph-osd systemd units.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26815 from tchaikov/wip-38584

doc/releases/nautilus: dashboard package notes

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #26823 into master

* refs/pull/26823/head:
qa/suites: disable valgrind leak checks on ceph-mgr
mgr: skip shutdown and exit

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: avoid aggregate-initializing IsWriteOpVisitor

Clang 8 complains about missing dtor, and this circumvents this problem.
```
/home/jenkins/workspace/ceph-master/src/librbd/io/ImageDispatchSpec.cc:136:48:
error: temporary of type 'boost::static_visitor<bool>' has protected
destructor
   return boost::apply_visitor(IsWriteOpVisitor{}, m_request);
                                                ^
/home/jenkins/workspace/ceph-master/src/librbd/io/ImageDispatchSpec.cc:152:28:
note: in instantiation of member function
'librbd::io::ImageDispatchSpec<librbd::ImageCtx>::is_write_op' requested
here
template class librbd::io::ImageDispatchSpec<librbd::ImageCtx>;
                            ^
/usr/local/include/boost/variant/static_visitor.hpp:53:5: note: declared
protected here
     ~static_visitor() = default;
     ^
1 error generated.
gmake[2]: *** [src/librbd/CMakeFiles/rbd_internal.dir/build.make:895:
src/librbd/CMakeFiles/rbd_internal.dir/io/ImageDispatchSpec.cc.o] Error 1
```

The details of the problem are explained in more detail in:
https://github.com/ceph/ceph/pull/26301/commits/09870d7a083913b5a3830ba8b0be04d0b0996aca

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Revert "msg/async: msgr2: expected tags validation"

This reverts commit e9ac3e0530b73ebb43db291d6562712b28c1e653.

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async/ProtocolV2: state machine verifications

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge PR #26796 into master

* refs/pull/26796/head:
common/options: osd_memory_target et al are TYPE_SIZE

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>