msg/async, v2: introduce pre_auth exchanges with CRC32.

CRC32 is just scaffolding and will be replaced with
HMAC-SHA256 or other msg auth code.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: introduce pre_auth buffers.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: rectify the encapsulation of rx_segments_{desc,data}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: rework decoding of MessageFrame.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: limit the num_segments to non-empty segments.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop the bl onwire space optimization in ControlFrames.

BEWARE: this changes the V2 onwire format without preserving backward
compatibility!

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: clean up ret handling in ProtocolV2::write().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop next_payload_len as we don't need anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop temp_buffer and limitations driven by it.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: switch to rx_buffer_t entirely.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: rx continuations use buffer::ptr_node.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: use bptr continuation for segment reading.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: introduce bptr-carrying continuations.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: replace CONTINUATION_PARAM() with specialized types.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: ::_banner_exchange() takes CtRef instead of CtPtr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: avoid extra pointers in continuation definitions.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: dissect setting stream handlers into ::finish_auth().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop ceph_msg_header2 handling from ControlFrames.

This is not needed anymore as MessageFrame class has been split
from the ControlFrame hierarchy - now it directly inherits from
the Frame root.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop the SignedEncryptedFrame entirely.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: reintroduce segment aligment. It's compile-time now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: generalize Frame about number of segments.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: rework and generalize Frame encryption.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: rework the class hierarchy - introduce MessageFrame.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: rework the class hierarchy - introduce ControlFrame.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async/ProtocolV2: remove obsolete AuthFlags

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 95e50c32d69daabc30bb850a58a26c84bb9c419a)

Merge PR #26849 into master

* refs/pull/26849/head:
Revert "msg/async: msgr2: expected tags validation"
msg/async/ProtocolV2: state machine verifications

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26869 into master

* refs/pull/26869/head:
qa/standalone/erasure-code/test-erasure-eio.sh: still need to create rbd pool
qa/standalone/scrub/osd-scrub-repair: fix unfound grep
qa/standalone/ceph-helpers: fix test_wait_for_clean
qa/standalone/ceph-helpers.sh: fix test_run_mon
qa/standalone/osd/osd-rep-recov-eio: fix better

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26713 into master

* refs/pull/26713/head:
mds: wait for client to release shared cap when re-acquiring xlock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26828 into master

* refs/pull/26828/head:
src/tools/kvstore_tool.cc: Move bluestore start-up code

Reviewed-by: Sage Weil <sage@redhat.com>

qa/standalone/erasure-code/test-erasure-eio.sh: still need to create rbd pool

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26660 into master

* refs/pull/26660/head:
include/intarith: enforce the same type for p2*() arguments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26831 into master

* refs/pull/26831/head:
mon/OSDMonitor: add newline
mgr/pg_autoscaler: drop undefined variable 'cr_name'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26779 into master

* refs/pull/26779/head:
mgr/status: fix 'ceph fs status' AttributeError

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

src/tools/kvstore_tool.cc: Move bluestore start-up code

Move this code into a separate function to avoid ICE.

Fixes: http://tracker.ceph.com/issues/38633
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/standalone/scrub/osd-scrub-repair: fix unfound grep

It's now "1/2 unfound":

             1/2 objects unfound (50.000%)

..presumably due to the rbd pool init creating the rbd_directory.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/ceph-helpers: fix test_wait_for_clean

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/ceph-helpers.sh: fix test_run_mon

- Only create each osd once
- forget the first osdmap dump test; it's pointless

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/osd-rep-recov-eio: fix better

- no need for the default pool size
- no initial osds or it will collide with setup_osds later
- no need for rbd pool at all

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26838 into master

* refs/pull/26838/head:
librbd: avoid aggregate-initializing IsWriteOpVisitor

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26826 from ceph/wip-simple-scan-all

ceph-volume: `simple scan` will now scan all running ceph-disk OSDs

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: make systemctl.get_running_osds resilient to garbage output

Makes systemd.systemctl.get_running_osds more resilient to output
that doesn't match what we're expecting to parse. This also handles
the case of a nonzero exit status by retuning an empty list.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: remove obsoleted simple unit test

This test is no longer valid because `simple scan`
without any arguments will now attempt to scan all
running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: tox tests should ignore the tests/functional directory

Without this the functional testinfra tests try to run when we
are intending to only run the unit tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use `simple scan` and `simple activate --all` in tests

In a couple of the functional tests for simple we should run `simple
scan` without any arguments and `simple activate --all` to increase
the test coverage. We want to test pointing at a diretory and scanning
all running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update docs and help menu for `simple scan`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds unit tests for systemd.systemctl.get_running_osd_ids

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: update `simple scan` to scan all running OSDs

If no argument is passed to `ceph-volume simple scan` it will
inspect any running osds and scan them if they were created by
ceph-disk.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add systemd.systemctl.get_running_osd_ids

This is used to retrieve a list of osd ids from the currently
running ceph-osd systemd units.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26815 from tchaikov/wip-38584

doc/releases/nautilus: dashboard package notes

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #26823 into master

* refs/pull/26823/head:
qa/suites: disable valgrind leak checks on ceph-mgr
mgr: skip shutdown and exit

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: avoid aggregate-initializing IsWriteOpVisitor

Clang 8 complains about missing dtor, and this circumvents this problem.
```
/home/jenkins/workspace/ceph-master/src/librbd/io/ImageDispatchSpec.cc:136:48:
error: temporary of type 'boost::static_visitor<bool>' has protected
destructor
   return boost::apply_visitor(IsWriteOpVisitor{}, m_request);
                                                ^
/home/jenkins/workspace/ceph-master/src/librbd/io/ImageDispatchSpec.cc:152:28:
note: in instantiation of member function
'librbd::io::ImageDispatchSpec<librbd::ImageCtx>::is_write_op' requested
here
template class librbd::io::ImageDispatchSpec<librbd::ImageCtx>;
                            ^
/usr/local/include/boost/variant/static_visitor.hpp:53:5: note: declared
protected here
     ~static_visitor() = default;
     ^
1 error generated.
gmake[2]: *** [src/librbd/CMakeFiles/rbd_internal.dir/build.make:895:
src/librbd/CMakeFiles/rbd_internal.dir/io/ImageDispatchSpec.cc.o] Error 1
```

The details of the problem are explained in more detail in:
https://github.com/ceph/ceph/pull/26301/commits/09870d7a083913b5a3830ba8b0be04d0b0996aca

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Revert "msg/async: msgr2: expected tags validation"

This reverts commit e9ac3e0530b73ebb43db291d6562712b28c1e653.

Signed-off-by: Ricardo Dias <rdias@suse.com>

msg/async/ProtocolV2: state machine verifications

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge PR #26796 into master

* refs/pull/26796/head:
common/options: osd_memory_target et al are TYPE_SIZE

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26827 from adamemerson/wip-doc-s3-op-acl

Document mappings of S3 Operations to ACL grants

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #26832 into master

* refs/pull/26832/head:
msg/async/ProtocolV2: end scope of std::hex properly

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add newline

To be in concert with other pool attributes..

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/pg_autoscaler: drop undefined variable 'cr_name'

```
a/pg_autoscaler/module.py:359: error: Name 'cr_name' is not defined
```

Actually _get_pool_status should have prevented any pool
with a zero-ed capacity from going this far, so it is safe
to simply drop the obviously redundant check here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26794 into master

* refs/pull/26794/head:
mon/MgrMonitor: only try to update always_on_modules if >= NAUTILUS
qa/standalone/mon/msgr-v2-transition: add some tests for enabling msgr v2
mon/MonmapMonitor: add 'ceph mon set-addrs <name> <addrvec>' command
Revert "mon/MonClient: disable ms_bind_msgr2 if NAUTILUS feature not set"
mon/OSDMonitor: use legacy_equals to compare osd addrs
msg/msg_types: make legacy_equals() symmetrical
mon/MDSMonitor: stop using get_orig_source_inst()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26770 into master

* refs/pull/26770/head:
qa/standalone/osd/osd-force-create-pg: create more pgs
qa/standalone: make sure an osd is running before create_rbd_pool

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26504 into master

* refs/pull/26504/head:
osd: fixup OpTracker destruct assert, waiting_for_osdmap take ref

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/status: fix 'ceph fs status' AttributeError

Fixes: http://tracker.ceph.com/issues/38122
Signed-off-by: Yanhu Cao <gmayyyha@gmail.com>

Merge PR #26803 into master

* refs/pull/26803/head:
msg/async/ProtocolV1: fix locking around authorizer_buf

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #26781 into master

* refs/pull/26781/head:
mds: avoid calling clientreplay_done() prematurely

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26773 into master

* refs/pull/26773/head:
mds: refactor subtree accessors

Reviewed-by: Zheng Yan <zyan@redhat.com>

mon/MgrMonitor: only try to update always_on_modules if >= NAUTILUS

Pre-nautilus mgrmap doesn't encode have this field, so we'll just loop.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/mon/msgr-v2-transition: add some tests for enabling msgr v2

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonmapMonitor: add 'ceph mon set-addrs <name> <addrvec>' command

This lets us explicitly adjust the IPs/ports a monitor binds to.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "mon/MonClient: disable ms_bind_msgr2 if NAUTILUS feature not set"

This reverts commit f3ddb1c9b88067b1a4b54d23d825ca4d62e10586.

That behavior complicated the process of moving to v1 -> v2+v1 (extra
daemon restart, for example).  Also, it wasn't sufficient: there are lots
of other reasons besides the mon version that could lead to a mismatch
between what we bind to locally and what is published in the cluster
maps.  Better to embrace that reality and address those other issues.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use legacy_equals to compare osd addrs

The OSDs may bind to v2+v1 due to ms_bind_msgr2, but we may only use the
v1 address in the OSDMap due to

 - the osd->mon connection being v1 (mon isn't using v2 yet)
 - the osd require_osd_release < nautilus
 - ???

Conversely, the OSD may end up with a v2+v1 address in the osdmap, but
may connect to the current monitor via v1 only, due to the mon only being
bound to a v1 address.

If/when this happens, make sure we are doing a loose comparison of the
osdmap addrs vs the osd's addr they are connecting from.

Fixes: http://tracker.ceph.com/issues/38598
Signed-off-by: Sage Weil <sage@redhat.com>

msg/msg_types: make legacy_equals() symmetrical

The prior version would return true for

 [v2,v1] vs [v2,v1]
 [v2,v1] vs [v1]

but not

 [v1] vs [v2,v1]

Now that final case is also true, which means a.legacy_equals(b) ==
b.legacy_equals(a) for any combination.  This is what you would expect
for an equality operator... even a fuzzy one.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MDSMonitor: stop using get_orig_source_inst()

Use get_orig_source() + get_orig_source_addrs()

Signed-off-by: Sage Weil <sage@redhat.com>

doc/radosgw: Document mappings of S3 Operations to ACL grants

Since the ACL documentation is a bit sparse, let people know exactly
what operations they can expect to enable with a given ACL grant.

Fixes: https://tracker.ceph.com/issues/38523
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #26802 into master

* refs/pull/26802/head:
qa/suites/upgrade/mimic-x/parallel: run master rados/test.sh

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/suites: disable valgrind leak checks on ceph-mgr

We've disabled the "clean" shutdown in ceph-mgr due to
https://tracker.ceph.com/issues/38621

Until then, no valgrind leak checks!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: skip shutdown and exit

The python subinterpreter teardown is broken; see
https://tracker.ceph.com/issues/38621

Instead, just exit with the appropriate error code.  That way we don't
crash and leave a core and ugly stack trace behind.

Users won't care, although our leak checking will.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26821 into master

* refs/pull/26821/head:
mds,mgr,mon: warn future respawn() copy-pasters to set thread name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26764 into master

* refs/pull/26764/head:
mgr: 'osd df' by specified class or (crush) name
mon/OSDMonitor: add 'osd crush get-device-class' command

mds,mgr,mon: warn future respawn() copy-pasters to set thread name

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: osd_memory_target et al are TYPE_SIZE

This makes us interpret "2G" and 2 GiB, not 2 billion bytes.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/dashboard: note dashboard as a separate package

Fixes: https://tracker.ceph.com/issues/38584
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/nautilus: dashboard package notes

Fixes: https://tracker.ceph.com/issues/38584
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26743 from votdev/cleanup

mgr/dashboard: Cleanup Python code

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #26797 into master

* refs/pull/26797/head:
ceph-mon: reset process name to 'ceph-mon' on startup

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #26795 into master

* refs/pull/26795/head:
mon/MgrStatMonitor: fix typo

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26027 into master

* refs/pull/26027/head:
osd,mon,mds: support get|set tcmalloc release rate

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26674 into master

* refs/pull/26674/head:
os/bluestore: log long aio operations in KernelDevice
os/bluestore: treat bdev_debug_aio_suicide_timeout==0 as no suicide
os/bluestore: track omap iterator latencies.
os/bluestore: log slow operations/long lasting states

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26718 into master

* refs/pull/26718/head:
rpm: put numpy and scipy deps in distro conditionals

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #26725 into master

* refs/pull/26725/head:
doc/releases/nautilus: ask users to opt in to telemetry
doc/mgr/telemtry: update docs
mgr/telemetry: drop config-set and config-show; add just show
mgr/telemetry: make 'telemetry show' readable by a human
mgr/telemetry: add 'telemetry on' and 'telemetry off' commands
mgr/telemetry: off by default

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #26716 from rjfd/wip-dashboard-ganesha-docs

mgr/dashboard: nfs-ganesha: controller API documentation

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

msg/async/ProtocolV2: end scope of std::hex properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26775 into master

* refs/pull/26775/head:
qa/suites/krbd/wac: bluestore snippet is placed incorrectly

Reviewed-by: Mike Christie <mchristi@redhat.com>

Merge pull request #26778 from xiexingguo/wip-fix-wrong-lpr

osd/PrimaryLogPG: fix last_peering_reset checking on manifest flushing

Reviewed-by: Myoungwon Oh <ommw@sk.com>

msg/async/ProtocolV1: fix locking around authorizer_buf

Fix two problems:

- we are accessing authorizer_buf without the connection lock, and
  under the lock we are modifying it (in connect()).
- if we receive two connect_msg's with a different length, we won't
  have a buffer that's large enough.

Fixes: http://tracker.ceph.com/issues/38524
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/mimic-x/parallel: run master rados/test.sh

We rename ceph_test_rados_api_tier to add _pp, so the mimic version doesn't
work.  And in any case, at this stage the client host has master installed.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/osd/osd-force-create-pg: create more pgs

Avoid warnings about too few pgs.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone: make sure an osd is running before create_rbd_pool

'rbd pool init' now does IO.  Drop the pool, or change the pool size to 1.

Fixes: http://tracker.ceph.com/issues/38585
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-mon: reset process name to 'ceph-mon' on startup

After a respawn the process name ends up as 'exe' (because we do execve
on /proc/$pid/exe).  Reset to 'ceph-mon' so that killall works.

Fixes: http://tracker.ceph.com/issues/38604
Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26633 from jtlayton/wip-nfs-scale

mgr/orchestrator: Allow the orchestrator to scale the NFS server count

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>