librbd: journal::StandardPolicy template defaults to ImageCtx

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't hold owner_lock for validate_image_removal()

handle_exclusive_lock() and handle_shut_down_exclusive_lock() call
validate_image_removal() without owner_lock held, so holding it in
shut_down_exclusive_lock() appears to be redundant.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: treat EROFS as expected in handle_acquire_lock()

If the peer refuses to release exclusive lock (e.g. in case automatic
exclusive lock transitions are disabled), EROFS is retured.  Suppress
a rather confusing "Read-only file system" error message -- this case
is no different from EBUSY or EAGAIN.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: refuse to release exclusive lock when removing

Commit 25c2ffe145be ("librbd: acquire exclusive lock from peer when
removing") changed PreRemoveRequest to request exclusive lock from the
peer instead of giving up and proceeding without exclusive lock.  This
caused one of the test cases that sometimes runs concurrent "rbd rm"
against the same image to fail intermittently, most often on assert

  ceph_assert(image_ctx.exclusive_lock == nullptr ||
              image_ctx.exclusive_lock->is_lock_owner());

because exclusive lock is now automatically transitioned to another
"rbd rm" on its request.

The root cause is older and probably goes back to when synchronous
librbd::remove() which held owner_lock across all operations including
trim_image() was converted to a set of state machines.  Since then, any
peer that requests exclusive lock (instead of trying once and backing
off) is able to mess with image removal.

Install StandardPolicy to disable automatic exclusive lock transitions
during image removal.

Fixes: https://tracker.ceph.com/issues/49226
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: templatize exclusive_lock::StandardPolicy

This will be used by expect_set_exclusive_lock_policy() in
test_mock_PreRemoveRequest.cc.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #39332 from cyx1231st/wip-seastore-range-query

crimson/onode-staged-tree: implement Cursor::get_next() and comparators for range query

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37729 from mgfritch/cephadm-podman-logdriver

cephadm: configure journald as the logdriver

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #37764 from mgfritch/cephadm-no-container-init

cephadm: run containers using `--init` by default

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39153 from mgfritch/cephadm-flake8

mgr/cephadm: introduce flake8

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39331 from ceph/revert-37954-wip-F46226-log-dq-throttle-limit-new

Revert "msg,mon,common: log when DispatchQueue throttle limit is reached"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39359 from tchaikov/wip-crimson-clang-build

crimson/os: fix build with clang

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: capture labels using a new variable

structured-binding does not define variables, so we cannot use "[=]" to
implicitly capture them. define variables in the capture list instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: mark unused variable [[maybe_unused]]

to silence warning like

key_layout.h:516:10: warning: unused variable 'nspace_masked'
[-Wunused-variable]
    auto nspace_masked = string_view_masked_t::decode(nspace, delta);
         ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not define assignment operator for Value

we cannot assign a reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: mark ExitBarrier final

to silence warnings like:

crimson/common/operation.h:394:20: warning: class with destructor marked
'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
    ~ExitBarrier() final {
                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39350 from tchaikov/wip-cephadm-unicode

cephadm: do not cast subnet to unicode

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39266 from sebastian-philipp/cephadm-rm-keyring-on-failed-deploy

mgr/cephadm: cleanup daemon keyring on failed daemon deploy

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38695 from pcuzner/orch-status-changes

mgr/orchestrator: minor improvements to orch status

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39347 from xxhdx1985126/wip-crimson-pg-bug-fix

crimson/osd: make sure ox be removed after all continuations are finished in PG::do_osd_ops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Revert "msg,mon,common: log when DispatchQueue throttle limit is reached"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: do not cast subnet to unicode

this change addresses a regression introduced by
fe4f4402fbcd87667613640f2808d5d0e07e749d, which was tested before
abd9287db0e4f4f7873864119f5ce62519af1d48 got merged. and the former was
merged after abd9287db0e4f4f7873864119f5ce62519af1d48. hence the
regression.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make sure ox be removed after all continuations are finished in PG::do_osd_ops

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39193 from tchaikov/wip-cephadm-py3

cephadm: drop support to python<3.3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38897 from Daniel-Pivonka/mgroktostop

mgr/cephadm: ok-to-stop for mgr

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39082 from ifed01/wip-ifed-meta-in-objectstore-tool

tools/ceph-objectstore-tool: allow object operations on meta pgid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39273 from tchaikov/wip-timer-mono

common/Timer: use mono_clock for clock_t

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39295 from tchaikov/wip-mgr-balancer-ann

pybind/mgr/balancer: add type annotations

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #39164 from tchaikov/wip-crimson-push-mov

crimson/osd: consume the push payload

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39204 from TRYTOBE8TME/wip-rgw-create-bucket-dpp

src/rgw: Dpp addition to create bucket log

rgw: Dpp addition to create bucket log

This commit adds dpp to the create bucket log.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

pybind/mgr/balancer: flake8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define Plan.{dump,show}()

as they are called by the commands

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/alerts: flake8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define options using Option

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: add type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define commands using CLICommand

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39336 from AlexWangD/patch-4

doc: fix the spelling mistake of “Topicd”

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix the spelling mistake of “Topicd”

“Topicd” is a spelling mistake of "Topics".

Merge pull request #37566 from AmnonHanuhov/wip-ObjectStore_EIO_Handling

crimson/osd: Recover missing object on read error

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: cleanup -- switch to get_*_slot() interfaces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #39303 from dillaman/wip-49173

librbd/io: conditionally disable move optimization

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39312 from ivancich/wip-orphan-list-binary-data

rgw: allow rgw-orphan-list to handle intermediate files w/ binary data

Reviewed-by: Michael Kidd <linuxkidd@gmail.com>

mgr/cephadm: Redefinition of unused name from line n (F811)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Module imported but unused (F401)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Local variable name is assigned to but never used (F841)

add error handling when `CompletedProcess` fails during rgw relam,
zongroup create et al.

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Local variable name is assigned to but never used (F841)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Line break occurred after a binary operator (W504)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Trailing whitespace (W291)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: undefined name 'channel' (F821)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use ==/!= to compare constant literals (F632)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: f-string is missing placeholders (F541)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not use variables named 'I', 'O', or 'l' (E741)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not assign a lambda expression, use a def (E731)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not use bare except, specify exception instead (E722)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Comparison to true should be 'if cond is true:' or 'if cond:' (E712)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Multiple statements on one line (def) (E704)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: statement ends with a semicolon (E703)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Expected 1 blank line before a nested definition (E306)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Expected 2 blank lines, found 0 (E302)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: at least two spaces before inline comment (E261)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: multiple spaces after ':' (E241)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Missing whitespace after ',', ';', or ':' (E231)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Missing whitespace around arithmetic operator (E226)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Missing whitespace around operator (E225)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line under-indented for visual indent (E128)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line over-indented for hanging indent (E126)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: closing bracket does not match visual indentation (E124)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line missing indentation or outdented (E122)

also resolves:
continuation line unaligned for hanging indent (E131)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line under-indented for hanging indent (E121)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Indentation is not a multiple of four (comment) (E114)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: introduce flake8

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: configure journald as the logdriver

Introduced by e6792f306ab4d07251588fdca6ed3876ae3a092a

podman containers running in a detached `-d` state need to have a
configured logdriver for `cephadm logs` to be able to gather the daemon
output from journald

Fixes: https://tracker.ceph.com/issues/47916
Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: extend `--container-init` to the shell command

allow for the `--init` flag to be passed during `cephadm shell`

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: run containers using `--init` by default

generally all ceph containers need an init process to both reap any
zombie pids and/or perform signal handling (e.g. coredumps, etc.)

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc/man/cephadm: add missing `--container-init` flag

adopt command allows for the `--container-init` flag

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39261 from adk3798/applytest

mgr/orchestrator: useful message when no service type in "ceph orch apply"

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39238 from liewegas/cephadm-conditional-osd-apply

mgr/cephadm: only apply osd specs if the devices have changed

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39259 from sebastian-philipp/cephadm-execnet-chmod-600

mgr/cephadm: on ssh connection error, advice chmod 0600

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39318 from AlexWangD/patch-3

doc: fix the spelling mistake of “Rersult”

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix the spelling mistake of “Rersult”

“Rersult” is a spelling mistake of "Result".

Signed-off-by: Alex Wang wangdashuai@inspur.com

crimson/onode-staged-tree: cleanup -- implement and use get_next_slot()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #39258 from Rethan/doc-remove-ms-options

doc/rados: remove some invalid ms option

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: allow rgw-orphan-list to handle intermediate files w/ binary data

Updates the grep commands with the --text command-line option since
there are cases where rados object names contain NUL characters.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #38538 from chenerqi/qos-skip-discard

librbd: optionally exclude ops from qos

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm:update test_orchestrator func signature

The available signature has changed to 3 parms, so this
patch is needed to ensure it conforms.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #39176 from ivancich/wip-rgw-gap-list

rgw: tooling to locate rgw objects with missing rados components

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Kidd <linuxkidd@gmail.com>

Merge PR #39213 into master

* refs/pull/39213/head:
qa/suites/upgrade/octopus-x/parallel: include cephfs in upgrade cluster

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39135 from orozery/librbd-crypto-performance

librbd: crypto performance optimization

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39298 from dillaman/wip-rbd-suite-readahead

qa/suites/rbd: drop require-osd-release command

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #39147 into master

* refs/pull/39147/head:
qa/tasks/ceph_fuse: do not createfs
qa/tasks/cephfs/fuse_mount: pass admin_socket path
qa/suites/fs/cephadm/multivolume: add basic multivolume test
mgr/mds_autoscaler: some fixes and cleanup
mgr/volumes: deploy MDSs when creating fs

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #39262 from zdover23/wip-doc-removing-workbench-from-docs-2021-Feb-first

doc/dev: Remove workbench mentions

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/rbd: drop require-osd-release command

Teuthology already defaults to quincy now and results in a failure
when trying to set to pacific. Additionally, drop the LUKS readbalance
test since it's unnecessary to duplicate that test.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38715 from lxbsz/rest_api

qa: add REST API method support for ceph-iscsi

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39265 from zdover23/wip-doc-rados-realy-typo-2020-feb

doc/rados: s/realy/really/

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39263 from cbodley/wip-48963

cmake: partial revert of BOOST_USE_VALGRIND when ALLOCATOR=libc

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

librbd/io: conditionally disable move optimization

If the requested write length does not match the provided bufferlist
length, disable the move optimization and instead fallback to creating
a new sub-bufferlist for the object request.

Fixes: https://tracker.ceph.com/issues/49173
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39182 from cbodley/wip-49065

rgw multisite: fix RGWCoroutine error handling

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39241 from dillaman/wip-49117

test/rbd-mirror: fix broken ceph_test_rbd_mirror_random_write

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/suites/upgrade/octopus-x/parallel: include cephfs in upgrade cluster

Signed-off-by: Sage Weil <sage@newdream.net>