Merge pull request #39153 from mgfritch/cephadm-flake8

mgr/cephadm: introduce flake8

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39331 from ceph/revert-37954-wip-F46226-log-dq-throttle-limit-new

Revert "msg,mon,common: log when DispatchQueue throttle limit is reached"

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39359 from tchaikov/wip-crimson-clang-build

crimson/os: fix build with clang

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/os: capture labels using a new variable

structured-binding does not define variables, so we cannot use "[=]" to
implicitly capture them. define variables in the capture list instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: mark unused variable [[maybe_unused]]

to silence warning like

key_layout.h:516:10: warning: unused variable 'nspace_masked'
[-Wunused-variable]
    auto nspace_masked = string_view_masked_t::decode(nspace, delta);
         ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not define assignment operator for Value

we cannot assign a reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: mark ExitBarrier final

to silence warnings like:

crimson/common/operation.h:394:20: warning: class with destructor marked
'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
    ~ExitBarrier() final {
                   ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39350 from tchaikov/wip-cephadm-unicode

cephadm: do not cast subnet to unicode

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39266 from sebastian-philipp/cephadm-rm-keyring-on-failed-deploy

mgr/cephadm: cleanup daemon keyring on failed daemon deploy

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38695 from pcuzner/orch-status-changes

mgr/orchestrator: minor improvements to orch status

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39347 from xxhdx1985126/wip-crimson-pg-bug-fix

crimson/osd: make sure ox be removed after all continuations are finished in PG::do_osd_ops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Revert "msg,mon,common: log when DispatchQueue throttle limit is reached"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: do not cast subnet to unicode

this change addresses a regression introduced by
fe4f4402fbcd87667613640f2808d5d0e07e749d, which was tested before
abd9287db0e4f4f7873864119f5ce62519af1d48 got merged. and the former was
merged after abd9287db0e4f4f7873864119f5ce62519af1d48. hence the
regression.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: make sure ox be removed after all continuations are finished in PG::do_osd_ops

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39193 from tchaikov/wip-cephadm-py3

cephadm: drop support to python<3.3

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38897 from Daniel-Pivonka/mgroktostop

mgr/cephadm: ok-to-stop for mgr

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39082 from ifed01/wip-ifed-meta-in-objectstore-tool

tools/ceph-objectstore-tool: allow object operations on meta pgid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39273 from tchaikov/wip-timer-mono

common/Timer: use mono_clock for clock_t

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39295 from tchaikov/wip-mgr-balancer-ann

pybind/mgr/balancer: add type annotations

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #39164 from tchaikov/wip-crimson-push-mov

crimson/osd: consume the push payload

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39204 from TRYTOBE8TME/wip-rgw-create-bucket-dpp

src/rgw: Dpp addition to create bucket log

rgw: Dpp addition to create bucket log

This commit adds dpp to the create bucket log.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

pybind/mgr/balancer: flake8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define Plan.{dump,show}()

as they are called by the commands

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/alerts: flake8 cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define options using Option

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: add type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/balancer: define commands using CLICommand

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39336 from AlexWangD/patch-4

doc: fix the spelling mistake of “Topicd”

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix the spelling mistake of “Topicd”

“Topicd” is a spelling mistake of "Topics".

Merge pull request #37566 from AmnonHanuhov/wip-ObjectStore_EIO_Handling

crimson/osd: Recover missing object on read error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39303 from dillaman/wip-49173

librbd/io: conditionally disable move optimization

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39312 from ivancich/wip-orphan-list-binary-data

rgw: allow rgw-orphan-list to handle intermediate files w/ binary data

Reviewed-by: Michael Kidd <linuxkidd@gmail.com>

mgr/cephadm: Redefinition of unused name from line n (F811)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Module imported but unused (F401)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Local variable name is assigned to but never used (F841)

add error handling when `CompletedProcess` fails during rgw relam,
zongroup create et al.

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Local variable name is assigned to but never used (F841)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Line break occurred after a binary operator (W504)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Trailing whitespace (W291)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: undefined name 'channel' (F821)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use ==/!= to compare constant literals (F632)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: f-string is missing placeholders (F541)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not use variables named 'I', 'O', or 'l' (E741)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not assign a lambda expression, use a def (E731)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Do not use bare except, specify exception instead (E722)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Comparison to true should be 'if cond is true:' or 'if cond:' (E712)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Multiple statements on one line (def) (E704)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: statement ends with a semicolon (E703)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Expected 1 blank line before a nested definition (E306)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Expected 2 blank lines, found 0 (E302)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: at least two spaces before inline comment (E261)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: multiple spaces after ':' (E241)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Missing whitespace after ',', ';', or ':' (E231)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Missing whitespace around arithmetic operator (E226)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Missing whitespace around operator (E225)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line under-indented for visual indent (E128)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line over-indented for hanging indent (E126)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: closing bracket does not match visual indentation (E124)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line missing indentation or outdented (E122)

also resolves:
continuation line unaligned for hanging indent (E131)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Continuation line under-indented for hanging indent (E121)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Indentation is not a multiple of four (comment) (E114)

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: introduce flake8

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39261 from adk3798/applytest

mgr/orchestrator: useful message when no service type in "ceph orch apply"

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39238 from liewegas/cephadm-conditional-osd-apply

mgr/cephadm: only apply osd specs if the devices have changed

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39259 from sebastian-philipp/cephadm-execnet-chmod-600

mgr/cephadm: on ssh connection error, advice chmod 0600

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39318 from AlexWangD/patch-3

doc: fix the spelling mistake of “Rersult”

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix the spelling mistake of “Rersult”

“Rersult” is a spelling mistake of "Result".

Signed-off-by: Alex Wang wangdashuai@inspur.com

Merge pull request #39258 from Rethan/doc-remove-ms-options

doc/rados: remove some invalid ms option

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: allow rgw-orphan-list to handle intermediate files w/ binary data

Updates the grep commands with the --text command-line option since
there are cases where rados object names contain NUL characters.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #38538 from chenerqi/qos-skip-discard

librbd: optionally exclude ops from qos

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm:update test_orchestrator func signature

The available signature has changed to 3 parms, so this
patch is needed to ensure it conforms.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #39176 from ivancich/wip-rgw-gap-list

rgw: tooling to locate rgw objects with missing rados components

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Kidd <linuxkidd@gmail.com>

Merge PR #39213 into master

* refs/pull/39213/head:
qa/suites/upgrade/octopus-x/parallel: include cephfs in upgrade cluster

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39135 from orozery/librbd-crypto-performance

librbd: crypto performance optimization

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39298 from dillaman/wip-rbd-suite-readahead

qa/suites/rbd: drop require-osd-release command

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #39147 into master

* refs/pull/39147/head:
qa/tasks/ceph_fuse: do not createfs
qa/tasks/cephfs/fuse_mount: pass admin_socket path
qa/suites/fs/cephadm/multivolume: add basic multivolume test
mgr/mds_autoscaler: some fixes and cleanup
mgr/volumes: deploy MDSs when creating fs

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #39262 from zdover23/wip-doc-removing-workbench-from-docs-2021-Feb-first

doc/dev: Remove workbench mentions

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites/rbd: drop require-osd-release command

Teuthology already defaults to quincy now and results in a failure
when trying to set to pacific. Additionally, drop the LUKS readbalance
test since it's unnecessary to duplicate that test.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38715 from lxbsz/rest_api

qa: add REST API method support for ceph-iscsi

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39265 from zdover23/wip-doc-rados-realy-typo-2020-feb

doc/rados: s/realy/really/

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39263 from cbodley/wip-48963

cmake: partial revert of BOOST_USE_VALGRIND when ALLOCATOR=libc

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

librbd/io: conditionally disable move optimization

If the requested write length does not match the provided bufferlist
length, disable the move optimization and instead fallback to creating
a new sub-bufferlist for the object request.

Fixes: https://tracker.ceph.com/issues/49173
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39182 from cbodley/wip-49065

rgw multisite: fix RGWCoroutine error handling

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39241 from dillaman/wip-49117

test/rbd-mirror: fix broken ceph_test_rbd_mirror_random_write

Reviewed-by: Mykola Golub <mgolub@suse.com>

qa/suites/upgrade/octopus-x/parallel: include cephfs in upgrade cluster

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39210 into master

* refs/pull/39210/head:
qa/suites/rados: move thrash-old-clients out of cephadm

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39229 from rhcs-dashboard/label-badge

mgr/dashboard: Add badge to the Label column in Host List

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/MgrClient: use mono_clock for clock_t used by Timer

so we don't suffer from "missing event" when adjust time the system
clock

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: use mono_clock for clock_t

there is chance that the system clock is adjust by chrony or ntpd, so that
a timer scheduled at epoch time 12:34 would never get scheduled if
the system clock is changed to 12:35 when the system clock is still
12:33.

if we have event which schedules itself when it is fired, there is
chance that this event could be put in a black hole if the system clock
is adjusted in the way explained above. this could be a serious issue
if we rely on the timer to do critical things.

but we *might* suffer from this issue even if we use mono_clock for
clock_t in Timer, because in older libstdc++ and libc++,
condition_variable::wait_until() and condition_variable::wait_for() were
still using pthread_cond_timedwait(). this is fixed after glibc
introduced pthread_cond_clockwait() in v2.3.0. for more details, see
https://reviews.llvm.org/D65339 and
https://gcc.gnu.org/legacy-ml/gcc-patches/2019-09/msg00190.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/Timer: add SafeTimer::add_event_at(real_clock::time_point)

so we can schedule an event with real_clock::time_point. timer should
wait using a mono_time, but user might want to pass a
real_clock::time_point to it when, for instance, scheduling a task which
should be performed in future at a specified time. in this case, we need
to convert the real_clock::time_point to a "known" time_point.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/condition_variable_debug: support mono_clock

it's allowed by the C++ standard library, there is not reason that it's
forbidden by us.

and it's important to use mono_clock along with condition_variable to
address the issue when system lock is modified when we are waiting for
an event using condition_variable::wait_until() or
condition_variable::wait_for()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39269 from tchaikov/wip-crimson-seastore-cleanup

crimson/os/seastore: do not capture unused variables

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #39162 from sebastian-philipp/cephadm-find-orch-ls-assert

mgr/cephadm: Add strings to assert statements

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38927 from sebastian-philipp/cephadm-upgrade-retry-ceph-volume

mgr/cephadm: try again calling ceph-volume without --filter-for-batch

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #39242 from vshankar/wip-cephfs-mirroring-doc-cleanup

doc: cleanup cephfs-mirroring document

Merge pull request #39208 from sebastian-philipp/pyhton-common-more-typing

python-common: Add more type annotations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38685 from tchaikov/wip-gil

mgr/ActivePyModules.cc: always acquire locks without holding GIL

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39217 from myoungwon/wip-fix-seg-fault-manifest-ops

osd: fix to call nullptr when cancel_manifest_ops

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39224 from petrutlucian94/rbd_fixes

Windows RBD fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>