Merge pull request #30245 from smithfarm/wip-41711-nautilus

nautilus: doc/ceph-kvstore-tool: add description for 'stats' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30664 from smithfarm/wip-42105-nautilus

nautilus: build/ops: admin/build-doc: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30004 from smithfarm/wip-41568-nautilus

nautilus: doc: pg_num should always be a power of two

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30003 from smithfarm/wip-41529-nautilus

nautilus: doc: default values for mon_health_to_clog_* were flipped

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29731 from pdvian/wip-41081-nautilus

nautilus: doc: Fix rbd namespace documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

admin/build-doc: use python3

to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ace8cb15a600daf5261c0ab349b43010a238a1d9)

Merge PR #30025 into nautilus

* refs/pull/30025/head:
doc: cephfs: add section on fsync error reporting to posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30649 from lordcirth/wip-doc-misplaced-ratio

doc: max_misplaced option was renamed in Nautilus

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: max_misplaced was renamed in Nautilus

mgr/balancer/max_misplaced was changed to target_max_misplaced_ratio
in 14.2.0.

Signed-off-by: Nathan Fish <lordcirth@gmail.com>

Merge pull request #30554 from alfredodeza/nautilus-guits-41392

nautilus ceph-volume: do not fail when trying to remove crypt mapper

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@redhat.com>

Merge pull request #30049 from smithfarm/wip-41333-nautilus

nautilus: rpm: always build ceph-test package

Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

ceph-volume: do not fail when trying to remove crypt mapper

In a containerized context, at some point, need to run `simple scan` on a device
from a separate container (not the existing and running corresponding container
to that device), but this can't work because when it tries to remove the
mapper which is still in use by the corresponding running osd container,
it fails.
This can be a bit more permissive and simply throw a warning.

Closes: https://tracker.ceph.com/issues/41392
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 80d3e4ca054b4d1334c2fb99fede46ab54ca4305)

Merge pull request #30304 from alfredodeza/nautilus-ceph-volume-zap-fix

nautilus ceph-volume lvm.zap fix cleanup for db partitions

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30520 from smithfarm/wip-41956-nautilus

nautilus: ceph-volume: systemd fix typo in log message

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: systemd fix typo in log message

Fixes: https://tracker.ceph.com/issues/41942
Signed-off-by: Manu Zurmühl <m.zurmuehl@heinlein-support.de>
(cherry picked from commit 8d9780ce50ef5d986c4994ed6ba5629be0bbfb74)

Merge PR #29926 into nautilus

* refs/pull/29926/head:
mgr/volumes: return string type to ceph-manager
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size
mgr/volumes: drop unnecessary size
mgr/volumes: cleanup FS subvolume or subvolume group path
mgr/volumes: give useful error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: return string type to ceph-manager

The 'volumes' ceph-mgr module returns a byte type instead of a string
type as the stdout result of commands, `fs subvolume getpath` and
`fs subvolumegroup getpath` to the ceph-mgr. This causes the commands
to hang and the ceph-mgr to utilize 100% CPU. Fix the 'volumes'
ceph-mgr module to return string type to ceph-mgr as the stdout
result of the commands.

Introduced-by: 3d63cd947f55a8508f972ee249ea6a04836429ae
Fixes: https://tracker.ceph.com/issues/41903
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 88aa0576a63c5b2e8d4478951b4f594fb5047b52)

qa/tasks: test for prevent negative subvolume size

Fixes: https://tracker.ceph.com/issues/41617
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 9c6be9586166c3092aabefbfcd3f48e81e3d32af)

mgr/volumes: prevent negative subvolume size

Fixes: https://tracker.ceph.com/issues/41617
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 8a1249626b56f345e20f1a7058aa70da9fb82576)

mgr/volumes: drop unnecessary size

Fixes: https://tracker.ceph.com/issues/41752
Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit 6c4154b1bbdb9deeb0420a604def43e9f52aa4a0)

mgr/volumes: cleanup FS subvolume or subvolume group path

... on best effort basis when FS subvolume or subvolumegroup create
command fails.

Fixes: https://tracker.ceph.com/issues/41371
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit ecf53a3527ed7561ddceeed75ed4aef1daf8230d)

mgr/volumes: give useful error message

... when creating FS subvolume or subvolume group with invalid
data pool layout.

Fixes: https://tracker.ceph.com/issues/41337
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit cbe48308e9e6b7e1b673e11101fd7e1403f52019)

Merge pull request #29487 from rhcs-dashboard/nautilus-run-dashboard-api-tests

nautilus: mgr/dashboard: run-backend-api-tests.sh improvements

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge tag 'v14.2.4' into nautilus

v14.2.4

This was required because the release was done with ceph-volume changes
only, and omitting the other code changes that were untested.

mgr/dashboard: run-backend-api-tests.sh CI improvements

As there is now a jenkins job to run this script
(see https://github.com/ceph/ceph-build/pull/1351),
this refactoring adapt the script to be run in a jenkins job as well as locally.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 7563f8b6d661886b7b3c7a259b70433a8962f55a)

pybind/mgr: install setuptools >= 12

otherwise we have
```
  Running setup.py egg_info for package apache-libcloud

    libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
      warnings.warn("Missing backports.ssl_match_hostname package")
    Traceback (most recent call last):
...
      File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated

...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```

and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2094c2f2fd6ac0f2028fcaf22f2b51a306e60172)

mgr/dashboard: set python binary for teuthology venv

* Added env. var. to set python binary when creating teuthology venv.
* venv/bin/activate: run after setting PATH (as it does the last PATH export).

Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 522cbe8a731cf01a48dd57b6169c66e4965ba272)

14.2.4

14.2.4

ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29769 from alfredodeza/nautilus-bz-1738379

nautilus ceph-volume: use the OSD identifier when reporting success

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30294 from alfredodeza/nautilus-rm41378-2

nautilus ceph-volume: pre-install python-apt and its variants before test runs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30307 from alfredodeza/nautilus-rm40664

nautilus ceph-volume  zap always skips block.db, leaves them around

Merge pull request #30300 from alfredodeza/nautilus-rm41660

nautilus: ceph-volume: fix stderr failure to decode/encode when redirected

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume tests: verify that wipefs tries several times

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 845e62cb4ba2636325304db2c2d65a185ace25cf)

ceph-volume lvm.zap: retry wipefs several times to prevent race condition failures

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b7b8b106fa485d3b8345b583fbc7f74e30cbc03e)

ceph-volume lvm.zap fix cleanup for db partitions

this uses the correct type 'db' for db type partitions, else
a block.db parition does not get cleaned up by ceph-volume zap

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
(cherry picked from commit 140af8a7eee8d230ac9d94ba100320a20c137a12)

ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 79196354aca3b3d30c7de1e2ac8b5a3c06ee8330)

ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 595e492783283359b613d199a75f3e7495bfb9ed)

ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ee18ebc078b2d380b88f3726737184f34345e2f3)

ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe66e01f6a4b19bb15695537713a629a951b28ce)

ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfb8422e262dcce49c50610a3977bfe1edc2a435)

ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 95b16b516c0736839fe50f12ecbeb7cd62c67335)

ceph-volume create a new tox.ini for shell-based tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4612ab3ae62a3e8e1cc9e51489feab0c938296d3)

ceph-volume tests pre-instrall python-apt to prevent auto-install failing later

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 40cac7b9510f7904c2f9dfb1294d809e04bdcf7e)

Merge pull request #30283 from tchaikov/wip-nautilus-c++17

nautilus: cmake: enforce C++17 instead of relying on cmake-compile-features

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: enforce C++17 instead of relying on cmake-compile-features

this change is a follow-up of #30114

in FindBoost.cmake, we try to deduce the C++ flags by checking the C++
features offered by different C++ standard like C++98, C++11 and C++14
against the ones required by different boost component. then choose the
C++ dialect based on the `CXX_STANDARD` and `COMPILE_FEATURES`.

in the case of nautilus, since cmake 3.5.1 does not offer
CXX_STANDARD=17 support by then, we have to add `-std=c++17` to
`CMAKE_CXX_FLAGS` manually. but unfortunately, the
`cmake-compile-features` machinary does not check `CMAKE_CXX_FLAGS` for
the default C++ standard required by the project. so it falls back to
the minimal C++ standard required by the target.

since Boost::fiber and Boost::context share the same C++ feature set
dependencies which can be fulfilled by C++11, and by default,
`CXX_EXTENSIONS` is enabled, `--std=gnu++11` is used as we are using gcc
in our building hosts. so we have CXX_FLAGS like:

-std=c++17 ... -std=gnu++11

when compiling targets linked against Boost::context. and it seems the
last one takes effect. that's why some targets failed to compile if
the makefiles were generated with cmake 3.5.1.

in this change, `INTERFACE_COMPILE_FEATURES <features>` is replaced with
`CXX_STANDARD 17`. this renders this setting a no-op, as CMake 3.5.1
does not translate this setting to any CXX_FLAGS. but this does not
hurt, as we use C++17 project-wide, and C++17 is able to offer all
compile features required by Boost::context.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts: this change is not cherry-picked from master, because in
  master, we use cmake 3.10 and up which offers the support of
  `CXX_STANDARD=17`, hence cmake is able to bump up the C++ standard
  to C++17 instead of falling back to C++11.

doc/man/ceph-kvstore-tool: fix typo

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 70621ae1a32672b4f13737911241a59d55db1634)

doc/ceph-kvsore-tool: add description for 'stats' command

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 00a5633af896a126aa481544bee74a63c43c688b)

Merge pull request #30114 from tchaikov/wip-nautilus-CXX_FLAGS

nautilus: seastar,dmclock: use CXX_FLAGS from parent project

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge branch 'nautilus' of github.com:ceph/ceph into nautilus

dmclock: pick up change to use specified C++ settings if any

see https://github.com/ceph/dmclock/pull/73

so dmclock won't pass `-std=c++11` even if the parent project already
have `-std=c++17` in CXX_FLAGS.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit fb83b331de9636e5de3d756e490748d5bfb9c4dc)

Conflicts: dmclock
  this change is not cherry-picked from master, because master is using
  a different dmclock version. so in this change, only the minimum fix
  is included.

seastar: pick up change in fmt submodule

see https://github.com/ceph/fmt/pull/1

so fmt won't fallback to C++11 if `CMAKE_CXX_STANDARD` is not set

Signed-off-by: Kefu Chai <kchai@redhat.com>
this change is not cherry-picked from master, because

- the seastar submodule in nautilus is not in sync with that in
  master
- master is not using fmt submodule included by seastar anymore
- master is using cmake 3.10, so there is no need to address
  this issue in master on fmt front.

Merge pull request #30093 from jan--f/wip-41614-nautilus

nautilus: ceph-volume: lvm list is O(n^2)

Reviewed-by: Alfredo Deza <adeza@redhat.com>

14.2.3

ceph-volume: test number of times LVs list was created

Add a test that cofirm that the LVs list was created only once when
calling devices.lvm.listing.list()

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 20d6264538dab444c0ef13f40c6c21ec3d2057f8)

ceph-volume: reuse list of LVs

Avoid creating of list of LVs in different methods of the same module
and resue them.

Fixes: https://tracker.ceph.com/issues/37490
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit ab3a58e78124a984d162b2f63992d01c5e59e2a7)

rpm: always build ceph-test package

Fixes: https://tracker.ceph.com/issues/41296
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit b9c744301051aa637f948be074be7b53c26cb504)

doc: cephfs: add section on fsync error reporting to posix.rst

Fixes: http://tracker.ceph.com/issues/24641
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit b52d85c2a71950db222e3649d5265e5891368ba1)

Merge pull request #29945 from trociny/wip-41475-nautilus

nautilus: pybind/mgr/rbd_support: ignore missing support for RBD namespaces

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29975 from theanalyst/nautilus-beast-endpoint-fix

nautilus: rgw: asio: check the remote endpoint before processing requests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: Address further comments on choosing pg_num

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
(cherry picked from commit 3a7e171df6b8a01197ff02a190f9f6e97e0d6627)

doc: adjust examples to use 2^n pg_num

The examples used pg_num set to 300 or 1000. This misled some users.

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>
(cherry picked from commit bd20d692bb38d206ebe268c4a2da35e35622f285)

doc: pg_num should always be a power of two

Signed-off-by: Kai Wagner <kwagner@suse.com>
(cherry picked from commit 3660e17e4b6a681fdca113740ff8b63b17b393d5)

doc: default values for mon_health_to_clog_* were flipped

On a freshly installed nautilus cluster (i.e. 14.2.2), the default values are:

mon_health_to_clog_interval = 3600
mon_health_to_clog_tick_interval = 60.000000

Fixes: https://tracker.ceph.com/issues/41403
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>
(cherry picked from commit 820ada1e58c3713b7de8a4ce669185d26860b84b)

Merge pull request #29977 from yuriw/wip-yuriw-41513-nautilus

qa/tests: adding mgr.x into the restart/upgrade sequence before monitors

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: adding mgr.x into the restart/upgrade sequence before monitors

Fixes: https://tracker.ceph.com/issues/41513
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: asio: check the remote endpoint before processing requests

`socket.remote_endpoint()` can throw exceptions corresponding to errors in the
`getpeername` syscall, make sure these are handled.

Fixes: CVE-2019-10222, https://tracker.ceph.com/issues/40018
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit caa653196856ecdf50519a9a33195d5c4e3372af)

Conflicts:
src/rgw/rgw_asio_frontend.cc
conflicts due to missing yield ctx in nautilus

pybind/mgr/rbd_support: ignore missing OSD support for RBD namespaces

During upgrades, the OSDs might not yet have been upgraded yet and
therefore might be missing namespace support.

Fixes: https://tracker.ceph.com/issues/41029
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 10c0c806819741012e1c440c38fac17682125d9a)

pybind/rbd: new OperationNotSupported exception

This will help catch errors where the backing OSDs do not support
a given RBD feature.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 97c1a120b7e7d934dd5d2456fd1fb4c1d0e891ec)

Merge pull request #29918 from badone/wip-nautilus-tracker-41518-grafana-server

nautilus: qa/ceph-ansible: Disable dashboard

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #29899 from alfredodeza/nautilus-rm41378

nautilus ceph-volume tests set the noninteractive flag for Debian

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

nautilus: qa/ceph-ansible: Disable dashboard

Recently a dashboard playbook was added which will fail without a
grafana server so disabling the dashboard

Fixes: https://tracker.ceph.com/issues/41521
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 131f5a145adf21d402e39ac2c151664eb8ccec76)

ceph-volume tests set the noninteractive flag for Debian, to avoid prompts in apt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89231c9a60a46ba0691f787efbfe5ab35f05c679)

Merge pull request #29805 from yuriw/wip-yuriw-41384-nautilus

qa/tests: changed running rbd tests test_librbd_python.sh from…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: changed running rbd tests test_librbd_python.sh from tag: v14.2.2

Fixes: https://tracker.ceph.com/issues/41384
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #29801 from smithfarm/wip-41263-nautilus

nautilus: rgw_file:  advance_mtime() takes RGWFileHandle::mutex unconditionally

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

rgw_file: dont deadlock in advance_mtime()

Fixes: https://tracker.ceph.com/issues/41173
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 0d3338c2d2271bbf6f4c7774ad175f7ee42d07f6)

Merge pull request #28862 from liewegas/wip-bluefs-extents-nautilus

nautilus: os/bluestore/bluefs_types: consolidate contiguous extents

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29191 from mynaramana/patch-1

doc/rados: Correcting some typos in the clay code documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados: Correcting some typos in the clay code documentation
Signed-off-by: Myna <mynaramana@gmail.com>

Merge pull request #29551 from pdvian/wip-40882-nautilus

nautilus: rbd: Reduce log level for cls/journal and cls/rbd expected errors

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29722 from dillaman/wip-39499-nautilus

nautilus: librbd: snapshot object maps can go inconsistent during copyup

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29723 from dillaman/wip-40511-nautilus

nautilus: librbd: improve journal performance to match expected degredation

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29725 from dillaman/wip-41078-nautilus

nautilus: mgr/rbd_support: support scheduling long-running background operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-volume devices.lvm zap use the identifier to report success

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d0d5750c1ba508152c1d5111f2003a039a27145f)

ceph-volume tests check success message when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 88c7429a60fdab6222d14132bea93b9bf773c720)

Merge pull request #29745 from liewegas/wip-bluestore-no-cgroup-nautilus

nautilus: os/bluestore: do not set osd_memory_target default from cgroup limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

os/bluestore/bluefs_types: consolidate contiguous extents

If we allocate a new extent that is contiguous with the last extent,
just extend it.  This avoids having long vectors of continguous extents
when a single large extent would suffice--especially with log files.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4f3ce9bc201c20d10d739d42e3f7018408f110a6)

os/bluestore: do not set osd_memory_target default from cgroup limit

On the aarch64 box I'm testing, this gives us a value of
7378697629483768832, which is not what we want.

I think we are better off relying on this limit being explicitly set via
environment variables (POD_* by kuberentes/rook) or via the command line.

This partially reverts 5c6b533697814af8acfd9e731a2599b2294687ef, but not
all of it, since we wan to keep the option itself, as it is now used by
common/config.cc when dealing with the POD_MEMORY_LIMIT env var.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9346d3c3bcde7d0a68e21a554eb7fc836df78ba9)

Merge pull request #29650 from pdvian/wip-40945-nautilus

nautilus: mgr/dashboard: RGW User quota validation is not working correctly

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #29678 from pdvian/wip-40948-nautilus

nautilus: core: Better default value for osd_snap_trim_sleep

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: tweaks to improve throughput for journaled IO

Increase the per-object append flush bytes to 1MiB to mimic the 16MiB
in-memory writeback cache. Finally, default enable the new journal
writethrough until flush option.

Performance results comparing no journal, pre-change, and post-change:

no journal original modified
4K 40310.84 7223.73 27427.27
32K 21530.37 3256.39 10284.05
256K 3159.67 564.73 1627.16
4M 223.03 48.03 91.91

Fixes: http://tracker.ceph.com/issues/40072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5f3b043896762fa97eb35648f8639004c45f5978)

librbd: new rbd_journal_object_writethrough_until_flush option

When set to true, the journal will not attempt to batch appends until
after it receives the the first flush request from the user.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 172ad649fb8a2410bdc078a5752b2ca09dc03ac2)

Conflicts:
src/librbd/io/ImageRequest.cc: trivial resolution

journal: support dynamically updating recorder flush options

Default to disabling writeback-style append flushes unless overridden
by a call to 'set_append_batch_options'.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c0322a13c83f590067a120212620ebba15fc8661)