Merge PR #39354 into master

* refs/pull/39354/head:
mds/client: create cephfs errno aliases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38684 into master

* refs/pull/38684/head:
qa: add _check_scrub_status helper to simplify the code
qa: add run_scrub helper in filesystem class
qa: add get_scrub_status helper in filesystem class
qa: wait the scrub task to complete
qa: remove passed_validation check for test_damage
qa: move wait_until_scrub_complete helper to filesystem class
mds: simplify the C_MDS_EnqueueScrub finish code

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #39835 from yuriw/wip-yuriw-crontab-master

qa/tests: moved nautilus and octopus tests to run on gibba and -p 70 …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39832 into master

* refs/pull/39832/head:
mgr/DaemonServer: osd ok-to-stop: return json when there are unknown PGs

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #39726 into master

* refs/pull/39726/head:
mgr/cephadm: document ok_to_stop output argument for clarity
mgr/DaemonServer: make warning language a bit friendlier
mgr/cephadm/upgrade: improve language a bit
mgr/cephadm/upgrade: restart multiple osds at once
mgr/cephadm: gather other osds that are safe to stop
mgr/cephadm: optional pass 'known' through to ok_to_stop
mgr/cephadm/upgrade: log start/stop/pause/resume

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #38741 from BryceCao/wip-standardize-request

rgw : add the check for bucket handler if http args exist object_exclusive sub_resource

Merge pull request #39790 from yuvalif/wip-yuval-fix-49557

rgw/notification: fix persistent notification hang when ack-levl=none

Merge pull request #39782 from yuvalif/wip-yuval-fix-49552

rgw/notification: fixing the "persistent=false" flag

Merge pull request #39515 from CloudFerro/gc_max_objs

rgw: limit rgw_gc_max_objs to RGW_SHARDS_PRIME_1

Merge pull request #39380 from leseb/rgw-svc-map-name

rgw: register daemon in service map with more details

qa/tests: moved nautilus and octopus tests to run on gibba and -p 70 for smoke tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

mgr/cephadm: document ok_to_stop output argument for clarity

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39823 into master

* refs/pull/39823/head:
qa/tasks/cephadm: remove mirror code

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39822 from mgfritch/cephadm-custom-container-sample

cephadm: split custom container args into argv

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39699 from mgfritch/cephadm-bin-flake8-quotes

cephadm: introduce flake8 and flake8-quotes

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

mgr/DaemonServer: osd ok-to-stop: return json when there are unknown PGs

In 791952cc01201010f298033003ba52374cc0159f we switched to return JSON
both on success and fail to describe which PGs are affected or are blocking
the ability to stop/restart OSDs.  Do the same for the case where
some PG states are unknown (i.e., just after a mgr restart) so that
the cephadm upgrade process can unconditionally expect a JSON result.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: remove mirror code

This is being done from ansible now.  Also, it breaks when
the conf file has unqualified-search-registries but not 'registry'
entries.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39831 from xxhdx1985126/wip-crimson-io-sequencer-bug

crimson/osd: do not pass lvalue of the lambda to seastar::futurize_invoke

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39656 from pritha-srivastava/wip-rgw-sts-user-fix

rgw/sts: fix for encoding/decoding user namespace

Merge pull request #39452 from rhcs-dashboard/wip-49283-master

mgr/dashboard: report mgr fsid

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/osd: do not pass lvalue of the lambda to seastar::futurize_invoke

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39815 from sebastian-philipp/github-cephadm-minor

.github: Minor cephadm changes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39827 from xxhdx1985126/wip-crimson-minor-bugs

crimson/osd: capture error_code by value in PG::handle_failed_op

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39812 from rzarzynski/wip-crimson-admin-config-help

crimson/admin: add support for 'config help'

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: capture error_code by value in PG::handle_failed_op

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39801 from rhcs-dashboard/49574-fix-pyjwt

mgr/dashboard: fix issues related with PyJWT versions >=2.0.0

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39755 from smithfarm/wip-49550

rpm: use PMDK system libraries on SUSE

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #39799 into master

* refs/pull/39799/head:
doc: update affiliation for myoungwon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update affiliation for myoungwon

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

cephadm: split custom container args into argv

introduced by usage of `shlex` in 3ea514c5

Fixes: https://tracker.ceph.com/issues/49590
Signed-off-by: Michael Fritch <mfritch@suse.com>

rpm: use PMDK system libraries on SUSE

As of a49d1dbb32e2436ff2836a85b2fa84418f0a5fff, when the rbd_rwl_cache and
rbd_ssd_cache bconds are enabled and WITH_SYSTEM_PMDK is disabled (as it is by
default), the RPM build attempts to

    git clone https://github.com/ceph/pmdk.git

but of course that won't work in the OBS, where the build workers have no
Internet connectivity.

Fortunately, the openSUSE/SLE versions targeted by Ceph master and pacific ship
the necessary PMDK libraries as RPM packages.

Fixes: a49d1dbb32e2436ff2836a85b2fa84418f0a5fff
Fixes: https://tracker.ceph.com/issues/49550
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #39765 from smithfarm/wip-ncpus-for-obs

rpm: set build parallelism from memory on SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #39682 into master

* refs/pull/39682/head:
vstart_runner: remove duplicate methods from LocalCephManager

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39780 into master

* refs/pull/39780/head:
qa/vstart_runner: dont log "not Ceph bin" msg too often

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #39681 into master

* refs/pull/39681/head:
vstart_runner: define path to ceph binary and use it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #39382 from gauravsitlani/master

rgw: added missing documentation on "pubsub" in rgw_enable_apis

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39762 from smithfarm/wip-support-leap-15.3

rpm/luarocks: simplify conditional and support Leap 15.3

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.de>

cephadm: fixup `alrady` -> `already`

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: Change outer quotes to avoid escaping inner quotes (Q003)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: Remove bad quotes from multiline string (Q001)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: Remove bad quotes (Q000)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: introduce flake8-quotes

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: line break after binary operator (W504)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: blank line contains whitespace (W293)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: trailing whitespace (W291)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: local variable 'e' is assigned to but never used (F841)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: 'select' imported but unused (F401)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: ambiguous variable name 'l' (E741)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: do not use bare 'except' (E722)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: statement ends with a semicolon (E703)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: module level import not at top of file (E402)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: expected 1 blank line before a nested definition (E306)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: expected 2 blank lines after end of function or class (E305)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: too many blank lines (E303)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: expected 2 blank lines, found 1 (E302)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: expected 1 blank line, found 0 (E301)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: too many leading '#' for block comment (E266)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: block comment should start with '# ' (E265)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: at least two spaces before inline comment (E261)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: unexpected spaces around keyword / parameter equals (E251)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: multiple spaces after ',' (E241)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: missing whitespace after ':' (E231)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: missing whitespace around arithmetic operator (E226)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: missing whitespace around operator (E225)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: whitespace before ':' (E203)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: whitespace after '{' (E201)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: continuation line unaligned for hanging indent (E131)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: continuation line under-indented for visual indent (E128)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: continuation line over-indented for visual indent (E127)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: continuation line over-indented for hanging indent (E126)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: continuation line with same indent as next logical line (E125)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: closing bracket does not match visual indentation (E124)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: ... does not match indentation of opening bracket's line (E123)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: continuation line missing indentation or outdented (E122)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: continuation line under-indented for hanging indent (E121)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: over-indented (E117)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: introduce flake8

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39789 from zdover23/wip-doc-cephadm-convert--bash-shells-2021-Mar-03

doc/cephadm: add prompts to adoption.rst

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39794 from cbodley/wip-qa-rgw-valgrind-supp-librados-delete

valgrind: suppress operator delete[] for tcmalloc

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

.github: Minor cephadm changes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39558 from sebastian-philipp/doc-rados-rm-toctree-cephadm

doc/rados: avoid including cephadm into the toctree

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

rpm: drop old SUSE-specific OOM fix

This fix was needed some years ago, but no longer.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: limit build jobs by system memory on SUSE

43b441f9a3bc907c17d52385251001ffcd5d3ff9 removed a bunch of code which the SUSE
builds were relying on to avoid OOM. This commit brings back that code in
a much-streamlined form: the SUSE-specific %limit_build macro.

This also has the advantage of not breaking the build on older RPMs which only
know about %_smp_mflags, and not the newer %_smp_build_ncpus etc. macros.

Fixes: 43b441f9a3bc907c17d52385251001ffcd5d3ff9
Fixes: https://tracker.ceph.com/issues/49556
Signed-off-by: Nathan Cutler <ncutler@suse.com>

crimson/admin: add support for 'config help'

This commit solves a problem unveiled in teuthology testing
where `ceph --admin-daemon ... config help` were constantly
returning `EINVAL`:

```
2021-03-03T02:43:01.566 DEBUG:tasks.admin_socket:Testing config help with config {'branch': 'master'}
2021-03-03T02:43:01.567 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config help
2021-03-03T02:43:01.711 INFO:teuthology.orchestra.run.smithi083.stderr:no valid command found; 3 closest matches:
2021-03-03T02:43:01.712 INFO:teuthology.orchestra.run.smithi083.stderr:config get <var>
2021-03-03T02:43:01.712 INFO:teuthology.orchestra.run.smithi083.stderr:config set <var> <val>...
2021-03-03T02:43:01.712 INFO:teuthology.orchestra.run.smithi083.stderr:config show
2021-03-03T02:43:01.713 INFO:teuthology.orchestra.run.smithi083.stderr:admin_socket: invalid command
2021-03-03T02:43:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 22
2021-03-03T02:43:01.714 INFO:tasks.admin_socket:ceph cli returned an error, command not registered yet?
2021-03-03T02:43:01.715 INFO:tasks.admin_socket:sleeping and retrying ...
```

The root cause is no support for `config help` in crimson.
As shown below this command hasn't been registered at all:

```
DEBUG 2021-03-03 02:42:48,242 [shard 0] osd - start: asok socket path=/var/run/ceph/ceph-osd.0.asok
INFO  2021-03-03 02:42:48,242 [shard 0] osd - register_command(): mark_unfound_lost)
INFO  2021-03-03 02:42:48,242 [shard 0] osd - register_command(): query)
INFO  2021-03-03 02:42:48,242 [shard 0] osd - register_command(): perf dump_seastar)
INFO  2021-03-03 02:42:48,242 [shard 0] osd - register_command(): dump_pgstate_history)
INFO  2021-03-03 02:42:48,242 [shard 0] osd - register_command(): flush_pg_stats)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): send_beacon)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): status)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): injectargs)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): config show)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): config set)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): config get)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): get_command_descriptions)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): help)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): git_version)
INFO  2021-03-03 02:42:48,243 [shard 0] osd - register_command(): version)
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: fix issues related with PyJWT versions >=2.0.0

Fixes: https://tracker.ceph.com/issues/49574
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #39399 from dang/wip-dang-zipper-10

RGW - Zipper 10: The Great Zippening

Merge PR #38909 into master

* refs/pull/38909/head:
PendingReleaseNotes: document option osd_fast_shutdown_notify_mon
osd: add osd_fast_shutdown_notify_mon option (default false)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39654 into master

* refs/pull/39654/head:
common/options: drop ms_async_max_op_threads
msg/async: drop Stack::num_workers
msg/async: s/num_workers/workers.size()/
msg/async: use range-based loop in NetworkStack
msg/async: do not pass worker id to Stack::spawn_worker()
async/Stack: pass Worker* to NetworkStack::add_thread()
async/rdma: do not reference worker id in RDMAStack::spawn_worker()
async/dpdk: do not use worker id when creating worker
async/PosixStack: do not reference worker id in ctor
async/rdma: initialize worker in RDMAStack::create_worker()
async/rdma: move RDMAStack::create_worker() to .cc

Reviewed-by: luo runbing <luo.runbing@zte.com.cn>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge PR #39665 into master

* refs/pull/39665/head:
qa/rados/dashbord use "random" objectore

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39689 into master

* refs/pull/39689/head:
src/global/signal_handler.h: fix preprocessor logic for alpine

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39750 into master

* refs/pull/39750/head:
mgr: create client messenger using ms_public_type

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39770 from liewegas/bug-46745

cephadm: add docker.service dependency in systemd units

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39795 from dillaman/wip-migration-import-docs

doc/rbd: clarify QCOW2 migration support

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39555 from tchaikov/wip-drop-pre-nautilus

*: drop backward compatibility with pre-nautilus versions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39798 from tchaikov/wip-crimson-monc-copy-sent

crimson/mon: keep a copy of sent MMonCommand messages

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/mon: keep a copy of sent MMonCommand messages

as per Yingxin Cheng,

> The send process can be asynchronous (there is a conn.out_q, or if the
> underlying socket lives in a different core in the m:n model to be
> ordered there). If user really wants to reuse a message, they must be
> careful not to modify it because it may result in modifing the pending
> messages.
>
> I think the best way is to copy the message if user want to resend it,
> and keep the ceph_assert(!msg->get_seq()). It may looks good to reuse a
> message under lossy policy, but the correctness is now up to user not to
> modify it inplace.

see also https://github.com/ceph/ceph/pull/39786

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/rbd: clarify QCOW2 migration support

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

valgrind: suppress operator delete[] for tcmalloc

suppresses a mismatched operator delete[] from the librados client's
destructor:

<error>
  <unique>0xeef7eb</unique>
  <tid>1</tid>
  <kind>MismatchedFree</kind>
  <what>Mismatched free() / delete / delete []</what>
  <stack>
    <frame>
      <ip>0x4C32EA0</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator delete[](void*, unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.16.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>660</line>
    </frame>
    <frame>
      <ip>0x5FBEB86</ip>
      <obj>/usr/lib64/librados.so.2.0.0</obj>
    </frame>
    <frame>
      <ip>0x5FBEF2C</ip>
      <obj>/usr/lib64/librados.so.2.0.0</obj>
    </frame>
    <frame>
      <ip>0x5F7F7C5</ip>
      <obj>/usr/lib64/librados.so.2.0.0</obj>
      <fn>librados::v14_2_0::Rados::shutdown()</fn>
    </frame>

Fixes: https://tracker.ceph.com/issues/49501
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/DaemonServer: make warning language a bit friendlier

This message is shown during the upgrade process.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/upgrade: improve language a bit

Signed-off-by: Sage Weil <sage@newdream.net>