crimson/mon: do not use futurize_invoke

flatten the call in seastar::futurize_invoke().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: let reopen_session() return future<bool>

this change is for improve the readability. to emphasis that the
next steps are performed only if a connection to monitor is
established.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use a vector for mon_commands

for smaller memory foot print. as we don't have lots of mon_command in
flight, hence not likely to benefit from a O(log(n)) lookup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: resend mon command when session established

this behavior matches that of `MonClient::_resend_mon_commands()`. so
far the only user which sends mon command in crimson is
`OSD::_add_me_to_crush()`, but there is still (rare) chance that the connected
monitor cannot be reached when we send the command to it, in that case,
we should retry when the connection is re-established.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: pass a string to Client::run_command()

as the reason why MMonCommand uses a vector is but for legacy reasons,
there is no need to expose this via the interface.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: move keyring and sub renewals into on_session_opened()

both of them fall into the category of jobs which we should do after
the connection to monitor is established.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: let on_session_opened() return a future

so we can expand its responsibility to perform some async calls.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: s/send_pendings/on_session_opened/

will use this method also for sending requests which are not acked

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: pending_messages should not be empty if active_conn

we always send all pending_messages, and clear it when establishing a
connection to mon, so there is no need to check for it when calling
`send_message()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: check for active_con before calling send_pendings()

before this change, we guard the `send_pendings()` call only in
`Client::send_message()`, after this change, all of the
`send_pendings()` calls are guarded with this check.

more consistent this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39673 from rzarzynski/wip-crimson-drop-protocol-v1

crimson: drop the support of ProtocolV1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39768 from rzarzynski/wip-crimson-monc-renew-subs

crimson/monc: renew subscriptions when reopening a session.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39718 from zdover23/wip-doc-dev-t8y-label-in-toc-2021-Feb-26

doc/dev: add t8y label to index.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39601 from badone/wip-ceph_test_rados_api_watch_notify-enoent-failure

tests: ceph_test_rados_api_watch_notify: Allow for reconnect

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/monc: renew subscriptions when reopening a session.

Lack of this feature was the root cause of an issue in
teuthology testing in which a socket failure injection
happened exactly during `mon_subscribe`; after the OSD
reconnected, the message hasn't been resent and entire
boot process has frozen.

```
DEBUG 2021-02-25 11:42:53,757 [shard 0] ms - [osd.2(client) v2:172.21.15.204:6804/33459@57376 >> mon.0 v2:172.21.15.204:3300/0] --> #6 === mon_subscribe({osdmap=1}) v3
(15)
DEBUG 2021-02-25 11:42:53,757 [shard 0] ms - authenticated_encrypt_update plaintext.length()=80 buffer.length()=80
DEBUG 2021-02-25 11:42:53,757 [shard 0] ms - authenticated_encrypt_final buffer.length()=96 final_len=0
DEBUG 2021-02-25 11:42:53,757 [shard 0] ms - authenticated_encrypt_update plaintext.length()=48 buffer.length()=48
DEBUG 2021-02-25 11:42:53,757 [shard 0] ms - authenticated_encrypt_update plaintext.length()=16 buffer.length()=64
DEBUG 2021-02-25 11:42:53,757 [shard 0] ms - authenticated_encrypt_final buffer.length()=80 final_len=0
INFO  2021-02-25 11:42:53,758 [shard 0] ms - [osd.2(client) v2:172.21.15.204:6804/33459@57376 >> mon.0 v2:172.21.15.204:3300/0] execute_ready(): fault at READY on lossy
 channel, going to CLOSING -- std::system_error (error crimson::net:4, read eof)
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #39680 into master

* refs/pull/39680/head:
mds: allow `fs authorize` command to work

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39710 into master

* refs/pull/39710/head:
qa: run fs:verify on all distros

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #39725 into master

* refs/pull/39725/head:
qa: delete all fs during tearDown

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #39732 from mychoxin/resolve_compile_error

test/librbd: resolve compile error on centos

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39731 from mychoxin/use_unified_zero_func

rbd: use portable zero-ing memory function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/monc: drop misleading comment about Connection::reply.

Before the commit the `crimson::mon:Connection::reply`
was commented as being specific to the ProtocolV1.
However, the code suggests this member participates also
in V2-related paths like `Connection::renew_tickets()`
where `do_auth()` is called from. `do_auth()` generates
`MAuth` which causes a monitor to send `MAuthReply`.
The `Connection::reply` synchronizes sending the auth
request with response handling.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mon: drop the unnecessary inclusion of MAuthReply.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: drop unnecessary forward declaration from AuthClientHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: fix debugs in Connection::do_auth_single().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/monc: drop the support for ProtocolV1.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: drop the implementation of ProtocolV1.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: drop the v1 support in unittest-seastar-messenger.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: do not use ProtocolV1 anymore.

At the moment crimson offers only a partial and buggy
support for ProtocolV1. When it enters the production,
V1 will be long obsolete, so spending time on improving
it doesn't seem to be a sound investment while offering
half-baked feature can be worse that lacking it entirely.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39521 from yuvalif/wip-yuval-fix-49322

rgw/notification: add exception handling for persistent notification thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/librbd: resolve compile error on centos

Signed-off-by: YuanXin <yuanxin@didiglobal.com>

Merge pull request #39743 from tchaikov/wip-crimson-cleanup

crimson/osd: capturing this in a better way

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39652 from athanatos/sjust/wip-crimson-omap

crimson: btree omap_manager rework with other misc fixes

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39631 into master

* refs/pull/39631/head:
osd/OSDMap: don't warn on NEW osd ids
mon/OSDMonitor: behave if inc map sets weight on not-yet-existing OSD
mon/OSDMonitor: do not mark newly created OSDs OUT

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39696 into master

* refs/pull/39696/head:
qa/suites/rados/monthrash: crank up client side debugging w/ mon thrashing

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: don't warn on NEW osd ids

If we allocate a new OSD, don't raise a health alert about it.

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/osd: keep ClientRequest alive during `seastar::repeat()`

before this change, `this` is wrapped by a smart pointer which is
in turn captured by multiple continuations. the continuation with the
longest life cycle is the one passed to `seastar::repeat()`. where
the underlying `repeater` captures the continuation as its member variable.
and `repeater` is not destroyed before `seastar::repeat()` returns.

so, we only need to capture `opref` in the continuation passed to
`seastar::repeat()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use finally for capturing this

* no need to capture this, it is already enclosed by `ref`
* use finally for capturing `ref`, and for printing out the "complete"
  logging message. we could use a different branch for printing out
  the failure message if any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reduce the indent

instead of using a lambda, use a future variable for reducing the indent
level.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: use portable zero-ing memory function

Signed-off-by: YuanXin <yuanxin@didiglobal.com>

Merge PR #39455 into master

* refs/pull/39455/head:
doc/man/8/ceph: document --max option
src/test/osd/safe-to-destroy: adjust test
ceph: print command output to stdout even on error
mgr/DaemonServer: include details in 'osd ok-to-stop' output
mgr: add --max <n> to 'osd ok-to-stop' command
mgr: relax osd ok-to-stop condition on degraded pgs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39629 into master

* refs/pull/39629/head:
crush/CrushWrapper: rebuild shadow tree on 'osd crush reweight-subtree'
crush/CrushWrapper: update shadow trees on update_item()

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #39674 into master

* refs/pull/39674/head:
mon/OSDMonitor: fix safety/idempotency of {set,rm}-device-class

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #39728 into master

* refs/pull/39728/head:
mon/PGMap: include the per-pool usage breakdown in pg dump

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39705 from smithfarm/wip-fix-s390x-build-regression

rpm: re-disable SUSE lttng build on z390x

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39733 from mgfritch/cephadm-test-multiple-digest

cephadm: add multi-digest test

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: add multi-digest test

quick unit test for inspect of a container with multiple digests

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39648 from liewegas/cephadm-spec-config

mgr/cephadm: add config section to ServiceSpec

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/man/8/ceph: document --max option

Signed-off-by: Sage Weil <sage@newdream.net>

src/test/osd/safe-to-destroy: adjust test

Signed-off-by: Sage Weil <sage@newdream.net>

ceph: print command output to stdout even on error

Currently in the case where the mon returns a command error code, we print
the error stream and Error ... message but not the command output.  Usually
there isn't any, so we haven't noticed until now, but there is not reason
why shouldn't return both an error code and some output.

Restructure the code so that the error message goes *after* the JSON output,
where it will be a bit more obvious to the user (if the stdout scrolled
the terminal, for instance).  (This is not a change in behavior since
previously we weren't seeing the stdout at all.)

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #39686 into master

* refs/pull/39686/head:
qa/ceph_fuse: don't use createfs anymore while mounting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39664 into master

* refs/pull/39664/head:
pybind/cephfs: DT_REG and DT_LNK values are wrong

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #39574 into master

* refs/pull/39574/head:
client: wake up the front pos waiter

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39294 into master

* refs/pull/39294/head:
client: add testcase for ceph_ll_lookup_vino
client: add ceph_ll_lookup_vino
client: make _lookup_ino take a vinodeno_t
client: stop doing unnecessary work in ll_lookup_inode

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #39538 into master

* refs/pull/39538/head:
mgr/telemetry: fix 'channel_indent' typo
mgr/telemetry: check if 'ident' channel is active

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39429 into master

* refs/pull/39429/head:
qa/tasks/mgr/dashboard: identify cluster in telemetry test

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

qa: delete all fs during tearDown

During the ceph task Unwind, the MDS are stopped. If any file system
still exists, we will see failover messages in the cluster log.

Fixes: https://tracker.ceph.com/issues/49510
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #39709 into master

* refs/pull/39709/head:
qa: use tcmalloc with valgrind in fs:valgrind

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #39714 from zdover23/wip-doc-cephadm-typo-togeter-2021-Feb-26

doc/cephadm: s/togeter/together/

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: validate config options at apply time

Make sure config options are valid/exist.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: disallow managed options in ServiceSpec config section

If a service is managing a config option, prevent the user from specifying
it in the spec.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: add config section to ServiceSpec

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39644 from liewegas/cephadm-service-id

cephadm: include service_name in unit.meta file

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/dev: add t8y label to index.rst

I added "(Teuthology)" to the "Tests: Integration
Tests" link, because that section is all about Teuthology,
but you wouldn't have known it from the text in the link
up to now.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: s/togeter/together

Exactly as it says in the title; this is a typo
I noticed while reading over the cephadm guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39713 from tchaikov/wip-mgr-rbd-support-type

mgr/rbd_support: fix type annotation of mirror_info

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39551 from sebastian-philipp/doc-cephadm-reduce-orch-cli

doc/cephadm: Restructure orchestrator docs

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/cephadm: remove Orchestrator CLI from cephadm toc

not much useful information there anymore.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move host labels to host mgmt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group MDS sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Add iscsi

Fixes https://tracker.ceph.com/issues/49488

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group NFS sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: rename monitoring chapter title

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group MON sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: make custom containers its own chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group RGW mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move scheduler topic to service mgmt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: move unmanaged=true to service-mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group general service mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group OSD mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Move FQDN chapter to host mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Move SSH config from operations to host-mgmt.rst

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: group host mgmt sections into one chapter

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39692 from zdover23/wip-doc-dev-t8y-debugging-analyzing-rewrite-2021-Feb-25

doc/dev: t8y - rewrite "a. & d. a t8y job"

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev: t8y - rewrite "a. & d. a t8y job"

This PR rewrites the "Analyzing and Debugging a
Teuthology Job" section of the "Debugging Tips"
page of the Teuthology User Guide in the Developer
Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39614 from sebastian-philipp/cephadm-again-test-ubuntu_2004_podman

qa/cephadm: Test again podman on ubuntu 20.04

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/rbd_support: fix type annotation of mirror_info

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38604 from cybozu/doc-fix-the-meaning-of-min_size

doc: fix the meaning of default_min_size of pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39684 from xxhdx1985126/wip-alienstore-parallelism-improvement

crimson/os/alienstore: improve alienstore's write parallelism

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39602 from tchaikov/wip-mgr-flux-ann

mgr: add type annotations to BaseMgrModule and influx

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39371 from tchaikov/wip-mgr-rbd-support-ann

mgr/rbd_support: add type annotation and cleanups

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

crimson/os/alienstore: improve alienstore's write parallelism

replace the grand per store tp_mutex with a finer grained per-collection
lock for better concurrency

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/futurized_store: fix return comment on omap_get_values

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/omap_manager: rework to be null safe and adjust interfaces

Adjusts a few things in omap_manager/btree:
- fix to be null safe, simplify/rename iterator interface
- update interface methods to match futurized_store.h
- misc whitespace fixes

Signed-off-by: Samuel Just <sjust@redhat.com>

mon/PGMap: include the per-pool usage breakdown in pg dump

We started accounting for this ages ago, but we don't expose it anywhere.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39666 from zdover23/wip-doc-dev-t8y-rerunning-tests-rewrite-2021-Feb-24

doc/dev: rewrite t8y "re-running tests"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39694 from zdover23/wip-doc-dev-t8y-debugging-triaging-rewrite-2021-Feb-25

doc/dev: rewrite t8y "triaging" section

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39695 from zdover23/wip-doc-dev-t8y-debugging-adding-sections-2021-Feb-26

doc/dev: add t8y debug section names

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39697 from zdover23/wip-doc-dev-t8y-debug-reporting-the-issue-rewrite-2021-Feb-26

doc/dev: rewrite t8y "reporting the issue"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39707 from petrutlucian94/overlapped

rbd: fix mingw 5.0.3 compatibility issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>