doc: add CephFS notes for nautilus

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26790 from ricardoasmarques/iscsi-passwords-with-slash

mgr/dashboard: iSCSI - Support iSCSI passwords with '/'

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Always get iSCSI passwords decrypted

Fixes: https://tracker.ceph.com/issues/38630
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Support iSCSI password with '/'

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26765 from p-na/wip-pna-rbd-conf-table-search

mgr/dashboard: Fix search in `Source` column of RBD configuration list

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26766 from p-na/wip-pna-fix-table-search

mgr/dashboard: Search broken for entries with null values

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26800 from linuxbox2/wip-rgw-limitcheck-trunc

rgw: bucket limit check misbehaves for > max-entries buckets (usually…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #26920 into master

* refs/pull/26920/head:
qa/tasks/mgr/test_module_selftest: fix localized value test
mgr/BaseMgrStandbyModule: parse prefix properly

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #26756 from badone/wip-f28-broken-Cython-dep

rpm: Install python2-Cython on f28

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

qa/tasks/mgr/test_module_selftest: fix localized value test

When mgr/selftest/testkey = foo and mgr/selftest/x/testkey is not set,
then get_localized() should return foo.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26886 into master

* refs/pull/26886/head:
crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26917 into master

* refs/pull/26917/head:
osd/osd_types: include omap and hit_set_archive bytes in pool usage

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26788 from ricardoasmarques/iscsi-docs-password

doc/rbd/iscsi-target-cli: Update auth command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rpm: Install python2-Cython in python2 environment

do_cmake.sh tests for Cython module version in python2

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #26880 from pcuzner/prom-add-recovery-metrics

mgr/prometheus: Add recovery metrics

Reviewed-by: Boris Ranto <branto@redhat.com>

mgr/BaseMgrStandbyModule: parse prefix properly

Fixes: http://tracker.ceph.com/issues/38705
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26712 from badone/wip-b64decode-decorator-traceback

pybind/mgr/restful: Decode the output of b64decode

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #26899 from dzafman/wip-38678

Minor cleanups in tests and log output

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26890 into master

* refs/pull/26890/head:
ceph.spec.in: add libstoragemgmt to mgr package

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26893 into master

* refs/pull/26893/head:
qa: unmount clients prior to marking fs down

Reviewed-by: Zheng Yan <zyan@redhat.com>

osd/osd_types: include omap and hit_set_archive bytes in pool usage

- Include the hit_set_archive bytes if we're in legacy mode.

- Add in (estimated) omap bytes for both nautilus+ and legacy mode, since
we didn't break out omap usage by pool (oops!).

Fixes: http://tracker.ceph.com/issues/38703
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26892 into master

* refs/pull/26892/head:
qa: stop testing simple messenger in CephFS suites

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26503 from ricardoasmarques/iscsi-support-dots-in-name

mgr/dashboard: iSCSI - Adds support for pool/image names with dots

Reviewed-by: Ricardo Dias <rdias@suse.com>

crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets

The crush/builder.c crush_add_bucket method resizes the max_buckets array
but a power of 2 when it has to expand, but the code in CrushWrapper was
assuming that if the array grew the pos for the new bucket would be the
last position in the new array.  This led to a situation where the
crush_choose_arg_map args array size didn't match max_buckets, and
eventually caused a crash.

Fixes: http://tracker.ceph.com/issues/38664
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26891 from rjfd/wip-dashboard-iscsi-ssl

mgr/dashboard: iscsi: adds CLI command to enable/disable API SSL verification

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #26658 from ivancich/wip-lc-prefix-fix

rgw: fix unordered bucket listing when object names are adorned

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26545 from ivancich/wip-rgw-gc-stall-min-fixes

rgw: resolve bug where marker was not advanced during garbage collection

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26601 from ivancich/wip-rgw-gc-stall-fixes

rgw: resolve bugs and clean up garbage collection code

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26599 from cbodley/wip-rgw-beast-default

rgw: make beast the default for rgw_frontends

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26896 from dillaman/wip-rbd-cls-warning

cls/rbd: fixed uninitialized variable compiler warning

Reviewed-by: Mykola Golub <mgolub@suse.com>

ceph.spec.in: add libstoragemgmt to mgr package

The libstoragemgmt package is needed so we can activate the drive
blink-led feature so we should pull that package.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #26854 from ricardoasmarques/fix-mutual-password-input-type

mgr/dashboard: Fix iSCSI mutual password input type

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix iSCSI mutual password input type

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #26906 into master

* refs/pull/26906/head:
Typo in Nautilus release notes: not -> now

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #26904 from SUSE/wip-orchestrator-deepsea-sharedsecret

mgr/deepsea: always use 'password' parameter for salt-api auth

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #26851 from rhcs-dashboard/36176-enable-api-tests-coverage

mgr/dashboard: enable coverage for API tests

Merge pull request #26669 from ricardoasmarques/iscsi-docs-disk-slash

doc/rbd/iscsi-target-cli: Update disk separator

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: tests: move helper classes to __init__.py

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: add iscsi CLI commands testing

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: update tests that mock mons kv store

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: add mixin classes to mock mons KV store

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc/mgr/dashboard: update iscsi documentation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: iscsi: use `ssl_verify` configuration option in restclient

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: iscsi: add CLI command to disable API SSL verification

Signed-off-by: Ricardo Dias <rdias@suse.com>

Adds support for pool/image names with dots

According to https://github.com/ceph/ceph-iscsi/pull/29

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Typo in Nautilus release notes: not -> now

Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

mgr/deepsea: always use 'password' parameter for salt-api auth

Prior to https://github.com/saltstack/salt/commit/71d5601507, the
salt-api expected the password to be sent using the 'sharedsecret'
parameter if using shared secrets, and the 'password' parameter
for other authentication types.  The above commit unifies this so
that we always only need to use the 'password' parameter.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #26902 from tchaikov/wip-doc-erasure-clay

doc/rados/operations: add clay to erasure-code-profile

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc/rados/operations: add clay to erasure-code-profile

so it's more visible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/test-erasure-code.sh: Don't grep entire bluestore directory

Bluestore caused grep crash with "grep: memory exhausted" due to
size of "block" storage.

Fixes: http://tracker.ceph.com/issues/38678
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Don't create rbd pool because it creates an object

This also reverts commit 10b9626ea7b09e7c124067a2ce08a76eea073c9c.

Fixes: http://tracker.ceph.com/issues/38631
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Can't wait for clean when there aren't any pools/PGs.

Fixes: http://tracker.ceph.com/issues/38678
Signed-off-by: David Zafman <dzafman@redhat.com>

mon, osd: Fix log messages (missing space, convert int to string)

Binary data was ending up in the log file which could cause
terminals to produce garbage output.

Fixes: http://tracker.ceph.com/issues/38678
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #26763 into master

* refs/pull/26763/head:
mgr/diskprediction_cloud: Fixed cannot import string.maketrans in the python 3.x

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cls/rbd: fixed uninitialized variable compiler warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26885 into master

* refs/pull/26885/head:
doc/releases/nautilus: add msgr2 refs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26736 into master

* refs/pull/26736/head:
mgr: push localized config handling into BaseMgr*Module
mgr: Fix broken get_localized_module_option function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #26856 into master

* refs/pull/26856/head:
os/bluestore: make osd_memory_limit default to .8x the cgroup limit
os/bluestore: observe osd_memory_target config options
common/config: set osd_memory_target based on POD_MEMORY_REQUEST
common/util: include group limit in daemon metadata
common/util: get_cgroup_memory_limit() helper
common/config: pass entity_type down into parse_env()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #26879 into master

* refs/pull/26879/head:
vstart.sh: make 'db' volume 1G instead of 64M
os/bluestore: make bluefs spillover messages nicer

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #26878 from dillaman/wip-pool-init-validation

librbd: bypass pool validation if "rbd_validate_pool" is false

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26888 into master

* refs/pull/26888/head:
doc/releases/nautilus: add 'ceph-volume simple scan' to upgrade instructions

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

os/bluestore: make osd_memory_limit default to .8x the cgroup limit

We do this at runtime in _set_cache_sizes so that changes to these values
at runtime will be picked up.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: push localized config handling into BaseMgr*Module

We need both (1) the raw key name (in order to look up the correct option
name and transform the result into the correct type) and (2) the optional
prefix (in order to look up both possible keys).  This simplifies the
mgr_module implementation.

There is a weird mismatch between the way that BaseMgrModule and
BaseMgrStandbyModule implement this method.  I suspect they could be
unified to work the same, but I'll leave that for another day.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: Fix broken get_localized_module_option function

Fixes: https://tracker.ceph.com/issues/38560
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #26818 into master

* refs/pull/26818/head:
qa/cephfs: relax min_caps_per_client check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: unmount clients prior to marking fs down

Evicted RHEL7.5 clients may hang.

Fixes: http://tracker.ceph.com/issues/38677
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: stop testing simple messenger in CephFS suites

Simple messenger is on it's way out and it doesn't work with msgr2.

Fixes: http://tracker.ceph.com/issues/38676
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/releases/nautilus: add msgr2 refs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26883 from rhcs-dashboard/wip-38657-master

mgr/dashboard: Change 'Client Recovery' title

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26721 from dillaman/wip-38538

librbd: improve object map performance under high IOPS workloads

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/releases/nautilus: add 'ceph-volume simple scan' to upgrade instructions

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26799 from wjwithagen/wjw-fix-/src/test/rgw/CMakeLists.txt

build/rgw: unittest_rgw_dmclock_scheduler does not need Boost_LIBRARIES

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26877 into master

* refs/pull/26877/head:
mgr/MgrStandby: exit with 0 result code, as if we'd done an orderly shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: observe osd_memory_target config options

Signed-off-by: Sage Weil <sage@redhat.com>

common/config: set osd_memory_target based on POD_MEMORY_REQUEST

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26858 into master

* refs/pull/26858/head:
qa: krbd deep-flatten test
qa/suites/krbd: enable deep-flatten feature

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/MgrStandby: exit with 0 result code, as if we'd done an orderly shutdown

This matches the prior behavior with shutdown().  It also makes
teuthology happy.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Change 'Client Recovery' title

Rename 'Client Recovery' info-card title to 'Recovery Throughput' as:
- 'Client' is a misleading concept there, as long as all recovery is
server-side driven.
- No indication of the kind of metric (e.g.: IOPS, throughput, etc).

Fixes: https://tracker.ceph.com/issues/38657
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #26825 from idryomov/wip-copyup-truncate

librbd: don't do create+truncate for discards with copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Add recovery metrics to prometheus exporter

Recovery information was missing from the exporter
stats. The source of the info is osd pool stats,
which was moved to mgr in the Mimic release. This
change adds recovery rates by pool, and also
includes a scrape_duration_seconds metric to track
latency of the exporter

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #26867 from liewegas/wip-doc-msgr2

doc/rados/configuration/msgr2: some documentation about msgr2

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

vstart.sh: make 'db' volume 1G instead of 64M

64M is barely usable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make bluefs spillover messages nicer

Was

 osd:50 spilled over to slow device: 1.3 GiB

Now

 osd.0 spilled over 1.7 GiB metadata from 'db' device (1000 MiB used of 1024 MiB) to slow device

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: bypass pool validation if "rbd_validate_pool" is false

This config option is already used by the create image state
machine. This should address a broken test case when validating
pool vs self-managed snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26801 from wjwithagen/wjw-fix-use-binutils-ld

build: LLVM ld does not like the versioning scheme

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26859 into master

* refs/pull/26859/head:
qa: ignore slow metadata io wrn during osd thrash

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: improve object map performance under high IOPS workloads

Do not zero-fill the BitVector's bitset prior to decoding the data.
Additionally, only read-update-modify the portions of the footer
that are potentially affected by the updated state.

Fixes: http://tracker.ceph.com/issues/38538
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26876 from wjwithagen/wjw-fix-src/test/librbd/operation/test_mock_TrimRequest.cc

librbd: avoid aggregate-initializing any static_visitor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid aggregate-initializing any static_visitor

This will result in an error complaining about a protected dtor.
also see:
    https://github.com/ceph/ceph/pull/26838

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/rados/configuration/msgr2: some documentation about msgr2

This doesn't integrate very well into network-config.rst, mostly because
that document is horribly out of data and I don't know where to start.
:(

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26849 into master

* refs/pull/26849/head:
Revert "msg/async: msgr2: expected tags validation"
msg/async/ProtocolV2: state machine verifications

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26869 into master

* refs/pull/26869/head:
qa/standalone/erasure-code/test-erasure-eio.sh: still need to create rbd pool
qa/standalone/scrub/osd-scrub-repair: fix unfound grep
qa/standalone/ceph-helpers: fix test_wait_for_clean
qa/standalone/ceph-helpers.sh: fix test_run_mon
qa/standalone/osd/osd-rep-recov-eio: fix better

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26713 into master

* refs/pull/26713/head:
mds: wait for client to release shared cap when re-acquiring xlock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26828 into master

* refs/pull/26828/head:
src/tools/kvstore_tool.cc: Move bluestore start-up code

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/erasure-code/test-erasure-eio.sh: still need to create rbd pool

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26660 into master

* refs/pull/26660/head:
include/intarith: enforce the same type for p2*() arguments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26831 into master

* refs/pull/26831/head:
mon/OSDMonitor: add newline
mgr/pg_autoscaler: drop undefined variable 'cr_name'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26779 into master

* refs/pull/26779/head:
mgr/status: fix 'ceph fs status' AttributeError

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

src/tools/kvstore_tool.cc: Move bluestore start-up code

Move this code into a separate function to avoid ICE.

Fixes: http://tracker.ceph.com/issues/38633
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/standalone/scrub/osd-scrub-repair: fix unfound grep

It's now "1/2 unfound":

             1/2 objects unfound (50.000%)

..presumably due to the rbd pool init creating the rbd_directory.

Signed-off-by: Sage Weil <sage@redhat.com>