crimson/osd: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add_recovery(oid) before recover_object(oid)

in 7736bddc534e5df2daf704c4bbc92a0927b0db4d, we assumed that the object
to be recovered did not exist in `recovering` before
`recover_object(oid)` was called. but this turns out not true. so, in
this change, `add_object(oid)` is called before `recover_object(oid)`
gets called.

Fixes: https://tracker.ceph.com/issues/47593
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use recoverying.at() when appropriate

this helps to avoid the confusion of "where the recovery is added" and
"are we adding a new instance of recovery here".

should call add_recovery() explicitly when we need to add a new recovery
instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add RecoveryBackend:add_recovering()

before this change, get_recovery() can also be used for adding a
recovery instance to `recovering`. this behavior is error-prone and
confusing.

after this change, add_recovery() is used in the place where we
want to add a new instance of recovery instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37106 from sebastian-philipp/cephadm-overwritable-templates

mgr/cephadm: make j2 teamplates overwritable

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #37121 from Songweibin/wip-rbd-metadata

librbd: async version of metadata_set/remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37083 from orozery/librbd-data-cryptor

librbd: add data encryptor/decryptor using openssl

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37110 from CongMinYin/qa_immutable_object_cache_check

qa: check immutable object cache content

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37237 from MahatiC/wip-fix-rwl-path

librbd/cache: Fix pwl_path read issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37206 from bk201/wip-47387

mgr/rook: fix listing daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #37043 from s0nea/wip-dashboard-47331

mgr/dashboard: log in non-admin users successfully if the telemetry notification is shown

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37293 from changchengx/osd_comment

osd: fix wrong comment about setting auth_client for messenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: correct comment about setting auth_client for messenger

client_messenger's auth_client has not been set until call
monc->init();

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37152 from rhcs-dashboard/bottom-links-component

mgr/dashboard: Creating a new Login Page Legal Links Component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37291 from badone/wip-docker-test-helper-fixes

test/docker-test: helper fixes and support more distro releases

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Add support for fedora 32, 33 and Ubuntu 20.04

Enable creation and use of these OS images in docker-test.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

tests: Make sure install-deps is run noninteractively

This gets past things like tzconfig stopping for user input.
Remove redundant install of python-virtualenv.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37200 from tchaikov/wip-crimson-with-lock

crimson/common: add helpers for seastar::with_lock()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37211 from tchaikov/wip-zbd

rpm,install-deps,run-make-check: enable WITH_ZBD build on demand

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/centos-8: Install git before running install-deps

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

test/docker-test: Fix permissions issue when using podman

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #36776 into master

* refs/pull/36776/head:
systemd: Support Graceful Reboot for AIO Node

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge PR #37227 into master

* refs/pull/37227/head:
qa/cephfs: don't pass args to destroy() in recreate()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37233 into master

* refs/pull/37233/head:
qa/mgr: revert a patch from commit 04ed58f

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37271 from cyx1231st/wip-seastore-fix-non-repeatable-read

crimson/seastore: fix potential non-repeatable-read from RootBlock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #36927 from idryomov/wip-krbd-noudev

krbd: optionally skip waiting for udev events

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #37213 into master

* refs/pull/37213/head:
mgr/rook: Pass pod namespace to list_namespaced_pod()

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #37101 from LenzGr/master-documentation

doc: Updated `HACKING.rst` and `README.rst`

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37137 from tspmelo/wip-fix-iscsi-tests

mgr/dashboard: Fix iSCSI backend unit-test

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37262 from trociny/wip-rbd-nbd-quiesce-hook

rbd-nbd: fix typo in mini help

Reviewed-by: Jason Dillaman <dillaman@rehdat.com>

Merge pull request #35785 from rhcs-dashboard/wip-45957-consolidate_Osd_Endpoints

mgr/dashboard: Consolidate Osd mark endpoints

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa: add test for mapping and unmapping from a network namespace

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37234 from guits/guits-quick-fix

ceph-volume: fix wrong type passed in terminal.warning()

Merge pull request #37222 from dillaman/wip-librbd-image-dispatch

librbd: bug fixes and cleanup for IO dispatch path

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37215 from uweigand/fix-librados-test-endian

test/librados: fix endian bugs in checksum test cases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36900 from wjwithagen/wjw-enhance-mgr_module.py

mgr/dashboard: Report the missing path in error message

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37087 from tspmelo/wip-iscsi-logged-in

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #37261 from tchaikov/wip-47552

common/BackTrace: do not use len for length of demangled symbol

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37185 from david-z/wip-fix-osdmaptool

tools/osdmaptool.cc: fix inaccurate pg map result when simulating osd out

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37210 from changchengx/no_tune_message

messenger: remove unused variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37183 from rhcs-dashboard/fix-47434-master

mgr/dashboard: table detail rows overflow

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

ceph-volume: fix wrong type passed in terminal.warning()

`terminal.warning()` excepts a `str`.
Passing `e` means we pass a type `exceptions.RuntimeError`

Changing to `terminal.warning(e.message)` fixes the issue.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1877672
Resolves: rhbz#1877672

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #37268 from anthonyeleven/anthonyeleven/doc-fixes

doc/man: Add optional reweight-by-utilization args

Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man: Add optional reweight-by-utilization args
doc/mgr: Grammar and wording for Prometheus labels
doc/rados: Spelling and clarity

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

crimson/seastore: fix potential non-repeatable-read from RootBlock

Load root block into the transaction when read it.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

librbd: async version of metadata_set/remove

In iSCSI scenario (via tcmu-runner), after a client has acquired exclusive
lock on an image, users will no longer be able to set metadata on the image.
This commit try to fix it by forward the request to the lock owner.

Steps to reproduce:

1. Client A:
```
>>> import rbd, rados;client=rados.Rados(conffile='');client.connect();ioctx=client.open_ioctx('rbd');rbd_inst=rbd.RBD();image=rbd.Image(ioctx, 'img1')
>>> from rbd import RBD_LOCK_MODE_EXCLUSIVE
>>> image.lock_acquire(RBD_LOCK_MODE_EXCLUSIVE)
>>>
```

2. Client B:
```
$ rbd image-meta set img1 conf_rbd_qos_iops_limit 10000
2020-09-12T15:19:58.325+0800 7f161affd700 -1 librbd::ManagedLock: 0x7f15f4001d48 handle_acquire_lock: failed to acquire exclusive lock:(30) Read-only file system
failed to set metadata conf_rbd_qos_iops_limit of image : (30) Read-only file system
rbd: setting metadata failed: (30) Read-only file system
$
```

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #37176 from badone/wip-enable-mgr-client-debug

qa: Enable debug_client for mgr tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

messenger: remove unused variable

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

librbd: add data encryptor/decryptor using openssl

This commit adds the first CryptoInterface implementation
which translates bufferlist encryption to the standard api of crypto libraries,
i.e. using block ciphers operating on C-style arrays.

It defines an abstract DataCryptor interface to allow different crypto libraries implementations.
With this commit, we add an OpenSSL based implementation.
Lastly, we implement a context-pool wrapper implementation to optimize performance.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa: check immutable object cache content

fix check immutable cache does not work

Signed-off-by: Yin Congmin <congmin.yin@intel.com>

rbd-nbd: fix typo in mini help

Signed-off-by: Mykola Golub <mgolub@suse.com>

common/BackTrace: do not use len for length of demangled symbol

it turns out `len` is longer than the length of demangled symbol,
let's rely on the `\0` sentry in the returned char* string instead.

in this change, use `status` to tell if the demangle is successful or
not.

Fixes: https://tracker.ceph.com/issues/47552
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37072 from idryomov/wip-kcephfs-blacklisted-string

mds: add " (blacklisted)" to session reject error string

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd: extract PG::load_head_obc() out

so it can be reused in later refactory

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: split PG::get_oid_and_lock() into two methods

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/object_context: add helper for acquiring locks

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common/tri_mutex: add helpers for seastar::with_lock()

so we don't need to worry about unlocking the object context.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: discard failure of finally() block

instead of assuming that the function passed to finally() returns an
erroratorized future, in this change:

* s/safe_then/then_wrapped/ to handle the exception thrown by
  the finally function.
* specialize for the case where the finally function does not return
  a future, and just call it. note, in seastar's implementation of
  finally, `finally_body` is used for specializing these two cases.
* rename "future" to "result", for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37207 from tchaikov/wip-doc-dev-osx

doc/dev/macos.rst: disable features not supported on osx

Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37252 from pponnuvel/spellcheck-docs

doc: Fixed a number of typos in documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37216 from tchaikov/wip-doc-cephadmin-codeblock

doc/cephadm: use appropriate directive for formatting codeblocks

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #37202 into master

* refs/pull/37202/head:
mon: allow overriding the initial mon_host

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37224 from tchaikov/wip-cmake-boost-MPL-list-size

cmake: introduce Boost::MPL interface library for increasing BOOST_MPL_LIMIT_LIST_SIZE

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37214 into master

* refs/pull/37214/head:
mgr/volumes/nfs: Check if orchestrator spec service_id is valid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37190 into master

* refs/pull/37190/head:
mon/MonCap: check profile_grants too while checking caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37148 into master

* refs/pull/37148/head:
mds/FSMap: do not set legacy_client_fscid after filtering

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #37037 into master

* refs/pull/37037/head:
mds: fix purge_queue's _calculate_ops is inaccurate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37218 into master

* refs/pull/37218/head:
qa: spawn MDS daemons before creating fs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35906 from gregsfortytwo/wip-stretch-mode

Add a new stretch mode for 2-site Ceph clusters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #36957 into master

* refs/pull/36957/head:
mds: convert stringstream to CachedStackStringStream

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

Merge pull request #37245 from mgfritch/cephadm-extra-ceph-conf-test

mgr/cephadm: fixup expected extra ceph conf test result

Reviewed-by: Adam King <adking@redhat.com>

doc: Fixed a number of typos in documentation

Signed-off-by: Ponnuvel Palaniyappan <pponnuvel@gmail.com>

mgr/cephadm: fixup expected extra ceph conf test result

fix test failure introduced by:
ff7e76348e5457fa6acb23545fcef56d6640c50a

```
E           AssertionError: expected call not found.
E           Expected: _run_cephadm('test', 'mon.test', 'deploy', ['--name', 'mon.test', '--reconfig', '--config-json', '-'], stdin='{"config": "\\n\\n[mon]\\nk=v\\n", "keyring": ""}')
E           Actual: _run_cephadm('test', 'mon.test', 'deploy',
['--name', 'mon.test', '--reconfig', '--config-json', '-'],
stdin='{"config": "\\n\\n[mon]\\nk=v\\n", "keyring": ""}', image='')
```

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc: Updated `HACKING.rst` and `README.rst`

Replaced the content of `HACKING.rst` in the dashboard source code
directory with a pointer to the new location in the developer guide.

Updated references in `README.rst` to also point to the online versions
of these files.

Fixes: tracker.ceph.com/issues/47396
Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

qa/mgr: revert a patch from commit 04ed58f

mds_cluster.mds_fail() runs command "mds fail" not "fs fail". The reason
for failure was PR #32581 which accidentally changed the return code
from 0 to EINVAL. Since this was reversed in PR #37159, the change
introduced by 04ed58f is not only incorrect but also redundant.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34545 from rhcs-dashboard/read_only

mgr/dashboard: Disabling the form inputs for the read_only modals

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37023 from p-se/grafana-many-to-many

mgr/dashboard: Fix many-to-many issue in host-details Grafana dashboard

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #37059 from guits/guits-cephadm-shell-multiple-mounts

cephadm: support multiple mounts when running interactive shell

Merge pull request #36890 from sebastian-philipp/cephadm-extend-ceph.conf

mgr/cephadm: Add extra-ceph-conf

Merge pull request #37135 from sebastian-philipp/cephadm-race-add-host-vs-apply

mgr/cephadm: Fix race between host_add and _apply_all_specs

Merge pull request #36969 from votdev/issue_46666_container_spec

cephadm: Introduce 'container' specification to deploy custom containers

qa/cephfs: don't pass args to destroy() in recreate()

In filesystem.py, don't set value of reset_obj_attrs to False.

Fixes: https://tracker.ceph.com/issues/47526
Signed-off-by: Rishabh Dave <ridave@redhat.com>

librbd/cache: Fix pwl_path read issue

... in ImageCacheState

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

mon/MonCap: check profile_grants too while checking caps

When checking if a certain fs subcommand can and should be executed in
FSCommands.cc, check permissions in "profile_grants" too when the caps
for that entity contains a cap profile.

Fixes: https://tracker.ceph.com/issues/47423
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #37158 from tchaikov/wip-no-more-assertDictContainsSubset

mgr/dashboard: replace assertDictContainsSubset() with assertLessEqual()

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37170 from yaarith/add-smartctl-nvme-dependencies

ceph.spec, debian: add smartmontools, nvme-cli dependencies

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: introduce Boost::MPL interface library

Signed-off-by: Kefu Chai <kchai@redhat.com>

src: Revert "Fix to raise BOOST_MPL_LIMIT_LIST_SIZE from 20 to 30"

This reverts commit 3f4e9a4526b8f174888828078e610769b80e48ec.

will fix the FTBFS by introducing a interface library in CMake script

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract admin/CMakeLists.txt

for better modularity

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: pass IOContext to image-extent IO dispatch methods

This allows a specific IOContext to be used regardless of the image's
current read and write snapshot state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: pass IOContext to object-extent IO dispatch methods

This allows a specific IOContext to be used regardless of the image's
current read and write snapshot state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: helper method to create new data pool IOContext

Deep-copy will require the ability to issue IOs against arbitrary
IOContexts via the image-extent IO dispatcher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: image dispatch spec tids are assigned by dispatcher

This was a legacy implementation where it was assigned by the ImageRequestWQ
and therefore needs to be part of the factory methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify in-flight IO tracking for write-block image dispatch

Now that we don't need to worry about read requests issuing a finish
callback, we can use a simple counter to track in-flight writes.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop ImageDispatchInterface::handle_finished virtual method

Any dispatch layer can now directly place themselves in the finish
callback handler chain without the use of the generic callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use an overridable finish handler for the image dispatcher

This mimics the design from the object dispatcher and will allow
for simplified in-flight IO tracking.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: drop flush tracker from exclusive lock image dispatch

We can now pass the flush through the exclusive-lock dispatch layer
to ensure all in-flight IOs have been processed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update refresh image dispatch layer flush exclusions

Only flush requests coming from the refresh state machine or from the
exclusive-lock dispatch layer initializationshould be ignored. This is
because both can be initiated from the refresh state machine and
therefore deadlock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reorder exclusive-lock pre-release state steps

The exclusive-lock dispatch layer should be locked and flushed to
ensure no IO is waiting for a refresh. Once that is complete, interlock
with the refresh state machine and re-flush one last time w/ the
refresh dispatch layer skipped.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid blocking writes when initializing exclusive-lock

The exclusive-lock dispatch layer will already block IOs as required
so this second layer of blocking just increases the complexity and
the potential for deadlocks when attempting to flush.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>