librbd: crypto format api semantics change

This commit alters the semantics of the encryption format api
to also load the encryption after format completes.
Additionally, several other small changes in librbd crypto are included,
in preparation of supporting clone formatting.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39473 from sebastian-philipp/doc-cephamd-trouble-events

doc/cephadm: Per-service and per-daemon events

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #39465 from sebastian-philipp/install-deps-rm-libboost1.72.list

install-deps: Cope with nonexistent /etc/apt/sources.list.d/ceph-libboost1.72.list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39249 from varshar16/wip-fix-vstart-ganesha-error

vstart: fix ganesha cluster id and rados url errors

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #39150 into master

* refs/pull/39150/head:
qa/suites/rados/cephadm/thrash: add cephadm thrashing test

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39482 into master

* refs/pull/39482/head:
Revert "Merge pull request #37764 from mgfritch/cephadm-no-container-init"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39424 into master

* refs/pull/39424/head:
cephadm: Mounting <empty> folder for selinux only if it is needed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #39185 into master

* refs/pull/39185/head:
os/bluestore: Add option to check BlueFS reads

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #39281 into master

* refs/pull/39281/head:
mgr/MgrClient; less copying of mgr report struct

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39282 into master

* refs/pull/39282/head:
osdc/Objecter: remove pool_full_try interface
librados: set FULL_TRY flag on ioctx, not objecter
librados/IoCtxImpl: pass extra_op_flags everywhere

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #39283 into master

* refs/pull/39283/head:
qa/suites/rados/dashboard: whitelist TELEMETRY_CHANGED

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge PR #39381 into master

* refs/pull/39381/head:
mgr/cephadm: based sleep interval on configured intervals/timeouts

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: Mounting <empty> folder for selinux only if it is needed

There are OSs without </usr/share/empty> folder. And selinux can be
enabled or not.

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Revert "Merge pull request #37764 from mgfritch/cephadm-no-container-init"

This reverts commit f635555fe76322336a1e0e8cc3c199562642ad0e, reversing
changes made to d4d3d17b23fcf6d8808a96b992f56924f495b174.

This PR seems to be (indirectly?) responsible for
  https://tracker.ceph.com/issues/49237

Also, it was causing the rados.py task's follow-up step to wait
for snap trimming to fail: it would time out a 'ceph osd dump --format=json'
command.  :/

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39461 from yuvalif/wip-yuval-fix-49261

rgw/test: allow for duplicate events in pubsub

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/cephadm: Per-service and per-daemon events

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39386 from zdover23/wip-doc-dev-triggering-tests-rewrite-10-Feb-2021

doc/dev: teuthology: improve triggering tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: teuthology: improve triggering tests

- This PR cleans up the language in the "Triggering
  Tests" section of the "Workflow" chapter of the
  "Tests: Integration Tests" (Teuthology) guide.
- Added bash prompts.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39407 from zdover23/wip-doc-dev-testing-qa-changes-rewrite-11-Feb-2021

doc/dev: rewrite 'testing qa changes'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev: rewrite 'testing qa changes'

This PR rewrites the "Testing QA Changes (Without
Re-Building Binaries)" section of the Developer
Guide. I also tried to unbury the lede.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #38562 into master

* refs/pull/38562/head:
Update batch.py

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

install-deps: Cope with nonexistent /etc/apt/sources.list.d/ceph-libboost1.72.list

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39458 from sebastian-philipp/cephadm-spec-store-fix-race

mgr/cephadm: SpecStore: use d-under accessor methods

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: SpecStore: use d-under accessor methods

This fixes a conflict between:

* 1fcdad17685a8884135a2cda0cdfe3525e325474
* 2c7f899124207d0d4866e16d9eb5609777ea22e8

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw/test: allow for duplicate events in pubsub

when tested via radosgw-admin.
also, add verification for the content of the events

Fixes: https://tracker.ceph.com/issues/49261
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #39379 from multi-arch/master

ceph.spec.in: Enable tcmalloc and lttng on IBM Power and Z

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Luke Browning <lbrownin@redhat.com>

Merge pull request #39389 from tchaikov/wip-cmake-ccache

cmake: s/CCACHE_FOUND/CCACHE_EXECUTABLE/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39411 from tchaikov/wip-rtd-theme

doc: cleanups and group tell commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39454 from rzarzynski/wip-crimson-crc-for-temp-msgr

crimson: configure CRC for the temp_mon_client's messenger.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39453 from rzarzynski/wip-crimson-monc-pick-from-addrs

crimson/monc: consider v1 addresses when connecting to a monitor.

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: define command properties using field list

the rendering is more compacted this way, and the parameters can be
aligned.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: silence warnings from openapi sphinx extension

this is a follow-up of 1debd9856565905439ae56f2cfdc16b0549055ec,
we should use the logging from sphinx.util instead from Python standard
library's logging.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: group commands by prefix

as there are lots commands, the toc in sidebar is clutterred with them,
so would be better to group them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: split the argdesc pair at the first '='

there is chance that the value of of the argdesc's kv pair contains
"=". for instance, the goodchars regexp could contain '='. so we should
stop at the first '='.

this change fixes the parsing of "osd pool application set" command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: drop ceph.js completely

these are leftovers of 7a62303b57150814ff465eb611b69b3b2e9f03f1

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: decrease padding in the "docubetter" div

for better UX

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39385 from adk3798/error-redeploy

cephadm: allow redeploy of daemons in error state if container running

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38549 from adk3798/prepare-redeploy

mgr/cephadm: make generate_config internal, prepare before redeploy

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson: configure CRC for the temp_mon_client's messenger.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: improve debugs around Connection::reply.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: consider v1 addresses when connecting to a monitor.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39450 from cbodley/wip-rgw-notify-double-dpp

rgw: remove duplicated ldout in rgw_notify.cc

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: remove duplicated ldout in rgw_notify.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39373 from varshar16/wip-fix-rook-timezone-error

mgr/rook: Add timezone info

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #39362 from rhcs-dashboard/49216-delete-eof-read-pwd-from-file

mgr/dashboard: delete EOF when reading passwords from file

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sebastian-philipp <swagner@suse.com>

Merge pull request #39434 from zdover23/wip-doc-dev-about-suites-and-filters-rewrite-2021-Feb-12

doc/dev: rewrite "about suites and filters"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39327 from kotreshhr/qa-fix-mgr-vol-test-cases

qa: Fix a few mgr/volume test cases

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #39409 from trociny/wip-49245

librbd: don't log error if get mirror status fails due to mirroring disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38883 from sebastian-philipp/cephadm-keep-deleted-specs

mgr/cephadm: Purge deleted services

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #39058 from sebastian-philipp/cephadm-use_repo_digest-default-true

mgr/cephadm: enable use_repo_digest by default for pacific

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39364 from mgfritch/cephadm-log-errs

cephadm: add error messages to log file

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

doc/dev: rewrite "about suites and filters"

This improves the English in the section
"About Suites and Filters" in the new
Teuthology Guide in the Developer Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #39404 into master

* refs/pull/39404/head:
qa/suites/upgrade/octopus-x/stress-split: cephadm-based stress-split upgrade

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #39375 from idryomov/wip-rbd-rm-exclusive

librbd: refuse to release exclusive lock when removing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39368 from MahatiC/wip-ssd-bug-fix

librbd/cache/pwl: Fix user request completion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>

qa: Fix a few mgr/volume test cases

Recovering dirty auth metadata file might not retain the order,
fixed the comparison in 'test_recover_auth_metadata_during_authorize'
and 'test_recover_auth_metadata_during_deauthorize'.

Fixes: https://tracker.ceph.com/issues/49192
Signed-off-by: Kotresh HR <khiremat@redhat.com>

librbd: don't log error if get mirror status fails due to mirroring disabled

Fixes: https://tracker.ceph.com/issues/49245
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39037 from liu-chunmei/seastore_collection

crimson/seastore: add collection implementation

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #39398 from ktdreyer/cephadm-sys-fs-selinux

cephadm: make /sys/fs/selinux empty

qa/suites/upgrade/octopus-x/stress-split: cephadm-based stress-split upgrade

Apply stress during upgrade, and also pause partway and do some
thrashing.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39384 from neha-ojha/wip-cephadm-compress

Revert "qa/tasks/cephadm: use cluster.sh and remote.sh for better readibility"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39155 from dillaman/wip-49037

librbd: correct incremental deep-copy object-map inconsistencies

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm: make /sys/fs/selinux empty

When the following conditions are true:

  1) A host has selinux-policy-targeted,
  2) We mount the host's /sys into a privileged container,
  3) The container has SELINUXTYPE=targeted in /etc/selinux/config,
  4) The container does not have an selinux-policy-targeted package,

then SELinux-enabled applications like restorecon or DNF do not work inside
the container.

Resolve this by making /sys/fs/selinux an empty directory.

Fixes: https://tracker.ceph.com/issues/49239
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

cephadm: allow redeploy of daemons in error state if container running

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #38963 from rhcs-dashboard/refresh-crushmap

mgr/dashboard: Automatically refresh crush map metadata table

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39355 from sebastian-philipp/doc-install-rm-deepsea

doc/install: cephadm is the successor of DeepSea

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38505 from sebastian-philipp/doc-dev-cephadm-scalability-notes

doc/dev/cephadm: scalability notes

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39287 from david-z/wip-fix-join-fscid

mon/MDSMonitor.cc: fix join fscid not applied with pending fsmap at boot

Merge PR #38318 into master

* refs/pull/38318/head:
common: Add Windows ETW support

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/seastore: add collection manager implementation

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>
Signed-off-by: Samuel Just <sjust@redhat.com>

cmake: s/CCACHE_FOUND/CCACHE_EXECUTABLE/

to be more consistent with other find_program() calls.
and more correct this way, as ${CCACHE_FOUND} is not a boolean, it
is a path pointing to the found cache executable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39387 from xxhdx1985126/wip-client-request-pg-ref

crimson/osd: capture pgref by value if it is to be referenced asynchronously

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39388 from ivzhh/master

cmake: don't pass ccache argument to RocksDB build

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: don't pass ccache argument to RocksDB build

RocksDB's cmake has a similar logic to turn on ccache as the main cmake.
Currently BuildRocksDB.cmake explicitly passes ccache to RocksDB cmake
arguments as CMAKE_CXX_COMPILER_LAUNCHER. This causes a compilation error
on ccache 4.0, especially in make -j mode.

ccache: error: Recursive invocation (the name of the ccache binary must
be "ccache")

This commit lets cmake from RocksDB decide RULE_LAUNCH_COMPILE by itself.

Signed-off-by: Sheng Mao <shngmao@gmail.com>

crimson/osd: capture pgref by value if it is to be referenced asynchronously

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39351 from zdover23/wip-doc-dev-getting-binaries-rewrite-08-Feb-2021

doc/dev: rewrite "getting binaries" section

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39383 from badone/wip-unmask-deep-scrub-stdout-lazy_omap_stats_test

test/lazy-omap-stats: Immediate deep-scrub output

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39333 from cyx1231st/wip-seastore-onode-tree-empty-oid-ns

crimson/onode-staged-tree: support empty ns and oid

Reviewed-by: Samuel Just <sjust@redhat.com>

Revert "qa/tasks/cephadm: use cluster.sh and remote.sh for better readibility"

This reverts commit 6d7ba1180e66df2d6dee3a01fe3e12cafaa6fe8c.

Fixes: https://tracker.ceph.com/issues/49228
Signed-off-by: Neha Ojha <nojha@redhat.com>

test/lazy-omap-stats: Immediate deep-scrub output

Show stdout from the child process as it happens.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38871 from cbodley/wip-qa-rgw-valgrind-on

qa/rgw: reenable valgrind in rgw/verify suite

Reviewed-by: Ali Maredia <amaredia@redhat.com>

doc/dev: rewrite "getting binaries" section

This rewrites the "Getting Binaries" section
of the "Workflow" section of the "Tests:
Integration Tests" section of the Developer Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

qa/rgw: test_rgw_reshard.py just tries common ports

the 'sudo netstat | grep radosgw' command is causing failures:

  File "qa/workunits/rgw/test_rgw_reshard.py", line 53, in get_radosgw_port
    x = out.decode('utf8').split(" ")
AttributeError: 'bool' object has no attribute 'decode'

because when valgrind is enabled, the process name is 'valgrind' instead
of 'radosgw'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: re-enable valgrind in rgw verify suite

this only enables it for radosgw itself, not the other daemons

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: move valgrind config into valgrind.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: based sleep interval on configured intervals/timeouts

Set a semi-arbitrary floor of 15 seconds.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/dashboard: Automatically refresh the crush map metadata table

If we make any change to the osd crush map like do an osd crush reweight from cli, for that change to be reflected on metadata table we need to reload the entire page. Instead this PR takes care of auto refreshing the tree view.

Fixes: https://tracker.ceph.com/issues/48922
Signed-off-by: Nizamudeen A <nia@redhat.com>
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

cephadm: add error messages to log file

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph.spec.in: bump gperftools requirement for ppc64le

gperftools' libprofiler did not build on ppc64le until 2.7.90.
The EPEL 8 package is being updated accordingly.

Signed-off-by: Yaakov Selkowitz <yselkowi@redhat.com>

ceph.spec.in: enable tcmalloc and lttng on s390x

The necessary prerequisites are already in RHEL+EPEL 8.

Signed-off-by: Yaakov Selkowitz <yselkowi@redhat.com>

librbd: journal::StandardPolicy template defaults to ImageCtx

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't hold owner_lock for validate_image_removal()

handle_exclusive_lock() and handle_shut_down_exclusive_lock() call
validate_image_removal() without owner_lock held, so holding it in
shut_down_exclusive_lock() appears to be redundant.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: treat EROFS as expected in handle_acquire_lock()

If the peer refuses to release exclusive lock (e.g. in case automatic
exclusive lock transitions are disabled), EROFS is retured.  Suppress
a rather confusing "Read-only file system" error message -- this case
is no different from EBUSY or EAGAIN.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: refuse to release exclusive lock when removing

Commit 25c2ffe145be ("librbd: acquire exclusive lock from peer when
removing") changed PreRemoveRequest to request exclusive lock from the
peer instead of giving up and proceeding without exclusive lock.  This
caused one of the test cases that sometimes runs concurrent "rbd rm"
against the same image to fail intermittently, most often on assert

  ceph_assert(image_ctx.exclusive_lock == nullptr ||
              image_ctx.exclusive_lock->is_lock_owner());

because exclusive lock is now automatically transitioned to another
"rbd rm" on its request.

The root cause is older and probably goes back to when synchronous
librbd::remove() which held owner_lock across all operations including
trim_image() was converted to a set of state machines.  Since then, any
peer that requests exclusive lock (instead of trying once and backing
off) is able to mess with image removal.

Install StandardPolicy to disable automatic exclusive lock transitions
during image removal.

Fixes: https://tracker.ceph.com/issues/49226
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: templatize exclusive_lock::StandardPolicy

This will be used by expect_set_exclusive_lock_policy() in
test_mock_PreRemoveRequest.cc.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #39211 from cbodley/wip-rgw-admin-formatter

radosgw-admin: fix leaks with make_unique()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39301 from cbodley/wip-49169

rgw: fix snprintf() truncate warning in ETagVerifier

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

doc/dev/cephadm: scalability notes

Add a few notes about possible future scalability improvements

Co-authored-by: Joshua Schmid <jschmid@suse.de>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/install: cephadm is the successor of DeepSea

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: enable use_repo_digest by default for pacific

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>