rpm,cmake: additional fixes to make amqp support optional

Signed-off-by: Tim Serong <tserong@suse.com>

Merge PR #26466 into master

* refs/pull/26466/head:
msg/async, v2: fix wrong base for KeepAliveFrameAck.
msg/async, v2: frame decoding operates on bufferlist.
msg/async, v2: drop ceph_msg_header2 fields duplicating segment info.
msg/async, v2: drop the scaffolding in preamble parsing.
msg/async, v2: handle msg authentication failures.
msg/async, v2: drop depedency on uint128_t. Clean up onwire crypto.
msg/async, v2: fix cur_msg_size in ::reset_recv_state().
msg/async, v2: drop magic numbers for segments.
msg/async, v2: get rid of magic number in SignedEncryptedFrame.
msg/async, v2: get rid of the magic number for default alignment.
msg/async, v2: decouple onwire segment length from logical length.
msg/async, v2: follow the const bl& concept in authenticated_encrypt_update().
msg/async, v2: drop handling of extra segments in ::fill_preamble().
msg/async, v2: get rid of magic numbers for alignment.
msg/async, v2: drop reserve() from onwire crypto's TxHandler.
msg/async: add con_mode to debug lines
msg/DispatchQueue: include con_mode in <== line
common/ceph_strings: get_con_mode_name()
msg/Connection: add get_con_mode()
msg/async/ProtocolV2: clean up preamble comments
msg/async, v2: improve debug around sending client indent.
msg/async, v2: bring back the no-encryption ability.
msg/async, v2: workaround con_mode handling.
msg/async, v2: drop the throttles bypass.
msg/async, v2: READ_MESSAGE_FRONT -> THROTTLE_DONE.
msg/async: WaitFrame of V2 can be crypto processed now.
msg/async: initial multi-segment support for V2.
msg/async: V2 bypasses throttles just for development.
msg/async: rectify reseting security state in ProtocolV2::reset_recv_state().
msg/async: switch to CRC32 for V2 preamble blocks.
msg/async: bump up preamble block size to 32 bytes.
msg/async: get rid of the distiction on main and extra V2 preamble.
msg/async: add debug around empty ClientIdent::addrs.
msg/async: V2 uses segments instead of next_payload_len, part 1.
msg/async: perform V2 frame dispatch in dedicated method.
msg/async: implement crc checking for main preamble of V2.
msg/async: receive V2 messages with new preable format.
msg/async: transmit V2 messages with new preable format.
msg/async: reset crypto processors in ProtocolV2::reset_recv_state().
msg/async: preamble of V2 Frames is now encrypted and authenticated.
msg/async: slightly rework ProtocolV2 preamble crafting.
msg/async: reset the rx stream handler in ::handle_read_frame_length_and_tag.
auth, msg/async, v2: drop AuthStreamHandler and AES128GCM_StreamHandler.
msg/async: Messages in Protocol2 are crypto-processed only once.
msg/async: SignedEncryptedFrame uses ceph::crypto::onwire.
msg/async: expose message segmentation to ::write_message().
auth: implement ceph::crypto::onwire with OpenSSL EVP.
auth: introduce ceph::crypto::onwire interfaces.
msg/async: decouple MessageHeaderFrame from SignedEncryptedFrame.
msg/async: move Protocol* asserts in SignedEncryptedFrame to compile time.
msg/async: simplify encryption handling in the PayloadFrame class.
auth: drop AES128CBC_HMACSHA256_StreamHandler.
msg/async: ensure consistency between con_mode and session_security.
msg/async: drop MessageFrame. Use MessageHeaderFrame instead.
msg/async: set con_mode and session_security at both peers.
msg/async, auth: switch AuthStreamHandler::rxtx_t to std::unique_ptr.
crypto: AES128GCM_StreamHandler brings authenticated encryption with AES-GCM.
include: uint128_t -> ceph::uint128_t + using.
msg/async: move crypto handling from ProtocolV2 into AuthStreamHandler.
auth, msg: dissect AuthStreamHandler from AuthSessionHandler.
auth/cephx: make _calc_signature() of CephxSessionHandler private.
auth: drop {en,de}crypt_message() from AuthSessionHandler.
auth: introduce DummyAuthSessionHandler.
auth: make AuthSessionHandler purely abstract.
auth: drop no_security() from AuthSessionHandler.
auth: drop get_protocol() and get_key() from AuthSessionHandler.
auth: drop sign_bufferlist() from AuthSessionHandler.
msg/async: drop get_auth_meta() from Protocol.
msg/async: emphasize ProtocolV2 does authenticated encryption.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26560 from xiexingguo/wip-pool-force-recovery

mgr: add per pool force-recovery/backfill commands

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

mgr: add per pool force-recovery/backfill commands

For those with multiple storage pools sharing the same devices,
I think it would make much more sense to offer per-pool
commands to bring pools with high priority, e.g., because they
are hosting data of more importance than others, back to normal
quickly.

Fixes: http://tracker.ceph.com/issues/38456
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

msg/async, v2: fix wrong base for KeepAliveFrameAck.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #26602 into master

* refs/pull/26602/head:
Revert "pybind: set language_level for cythonize explicitly"
Revert "pybind: auto encode/decode cstr"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26436 from dzafman/wip-bionic-core

run-standalone.sh: Need double-quotes to handle | in core_pattern on all distributions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Revert "pybind: set language_level for cythonize explicitly"

This reverts commit ded3d43dbf8a36daa65d8efe406af4d12ffc7dd8.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "pybind: auto encode/decode cstr"

This reverts commit c36d0f1a7de4668eb81075e4a94846cf81fc30cd.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25438 from MahatiC/wip-move-to-trash

rbd: move image to trash as first step when removing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26506 from ricardoasmarques/multi-backstore-backend

mgr/dashboard: iSCSI - Infrastructure for multiple backstores (backend)

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26515 from theanalyst/civetweb-fixes-n

civetweb: pull up to ceph-master

Merge PR #26550 into master

* refs/pull/26550/head:
mon/AuthMonitor: provide auth_lock-safe _assign_global_id()
mon/AuthMonitor: provide AuthMonitor with mon count and rank *only* while active

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #26476 into master

* refs/pull/26476/head:
pybind: auto encode/decode cstr
pybind: set language_level for cythonize explicitly

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26552 into master

* refs/pull/26552/head:
osdc/Objecter: invalidate crcs on preallocated rx buffers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26568 into master

* refs/pull/26568/head:
unittest_auth: add unit tests for AuthRegistry con_mode selection
auth/AuthRegistry: register as an observer
auth/AuthRegistry: clear output vectors
common/options: add ms_mon_client_mode
auth: push con_mode selection into AuthRegistry helper

Reviewed-by: Ricardo Dias <rdias@suse.com>

civetweb: pull up to ceph-master

Contains the following fixes
- absolute url fixes
- ssl socket closing fixes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26584 from capri1989/doc-fixes

doc: Remove value 'mon_osd_max_split_count'

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Remove value 'mon_osd_max_split_count' from the docs

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #26510 from callithea/wip-add-ganesha-qa-tests

mgr/dashboard: Added NFS Ganesha suite to QA tests

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #26520 from ktdreyer/rpm-rhel8

rpm: RHEL 8 fixes

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26555 from tchaikov/wip-install-build-deps

cmake,rgw: make amqp support optional

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26582 from tchaikov/wip-cmake-silence-syntax-warning

cmake: fix syntax error of set()

Reviewed-by: Ricardo Dias <rdias@suse.com>

cmake: fix syntax error of set()

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps,do_freebsd: disable amqp support

per Willem's request.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: disable amqp endpoint on SLE/OpenSUSE

as librabbitmq-devel is not packaged on them yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove amqp_mock from UNITTEST_LIBS

this mock library is only used by rgw tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async, v2: frame decoding operates on bufferlist.

This change is driven by buggy buffer's life time management
polluting AuthAuthorizer::bl with dangling raw_static instances.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop ceph_msg_header2 fields duplicating segment info.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop the scaffolding in preamble parsing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: handle msg authentication failures.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop depedency on uint128_t. Clean up onwire crypto.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: fix cur_msg_size in ::reset_recv_state().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop magic numbers for segments.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: get rid of magic number in SignedEncryptedFrame.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: get rid of the magic number for default alignment.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: decouple onwire segment length from logical length.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: follow the const bl& concept in authenticated_encrypt_update().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop handling of extra segments in ::fill_preamble().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: get rid of magic numbers for alignment.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop reserve() from onwire crypto's TxHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: add con_mode to debug lines

Signed-off-by: Sage Weil <sage@redhat.com>

msg/DispatchQueue: include con_mode in <== line

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_strings: get_con_mode_name()

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: add get_con_mode()

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: clean up preamble comments

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async, v2: improve debug around sending client indent.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: bring back the no-encryption ability.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: workaround con_mode handling.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop the throttles bypass.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: READ_MESSAGE_FRONT -> THROTTLE_DONE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: WaitFrame of V2 can be crypto processed now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rpm: build __pycache__ directories on RHEL 8

In RHEL 8, rpmbuild does not automatically generate __pycache__
directories. Generate them explicitly with the %py_byte_compile macro.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rpm: fix pyOpenSSL requirement for RHEL 8

On RHEL 7, this package is named "pyOpenSSL", but on RHEL 8, Fedora, and
SUSE, it is named "python3-pyOpenSSL".

Prior to this change, the RHEL conditional was applying to both RHEL 7
and 8, so we would fail to resolve the "pyOpenSSL" package dependency on
RHEL 8.

Update the RPM conditionals to use "pyOpenSSL" on RHEL 7 and
"python3-pyOpenSSL" on all other distributions.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rpm: fix python dependencies on RHEL 8

On RHEL 8, the python34-* and Cython packages are named identically to
Fedora (eg "python3-Cython"). Move the Python 2 "Cython" and python34-*
BuildRequires lines inside a RHEL 7 conditional, so RHEL 8 falls through
to the Fedora package names.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #26566 from theanalyst/vstart-beast

vstart: make beast as the default frontend for rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async: initial multi-segment support for V2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: V2 bypasses throttles just for development.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: rectify reseting security state in ProtocolV2::reset_recv_state().

The reason for that is:

  2019-02-14 19:59:00.774 7f78aebf3700 20 --2- v2:127.0.0.1:40678/0 >>
    v2:127.0.0.1:40680/0 conn(0x5574ae75e480 0x5574ae696000 :-1 s=READY
    pgs=11 cs=0 l=0 rx=0x5574ae2af5f0 tx=0x5574ae79bc00).handle_read_frame_preamble_main
    r=-1
  2019-02-14 19:59:00.774 7f78aebf3700  1 --2- v2:127.0.0.1:40678/0 >>
    v2:127.0.0.1:40680/0 conn(0x5574ae75e480 0x5574ae696000 :-1 s=READY
    pgs=11 cs=0 l=0 rx=0x5574ae2af5f0 tx=0x5574ae79bc00).handle_read_frame_preamble_main
    read frame length and tag failed r=-1 ((1) Operation not permitted)
  2019-02-14 19:59:00.774 7f78aebf3700 10 --2- v2:127.0.0.1:40678/0 >>
    v2:127.0.0.1:40680/0 conn(0x5574ae75e480 0x5574ae696000 :-1 s=READY
    pgs=11 cs=0 l=0 rx=0x5574ae2af5f0 tx=0x5574ae79bc00)._fault

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: switch to CRC32 for V2 preamble blocks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: bump up preamble block size to 32 bytes.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: get rid of the distiction on main and extra V2 preamble.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: add debug around empty ClientIdent::addrs.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: V2 uses segments instead of next_payload_len, part 1.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: perform V2 frame dispatch in dedicated method.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: implement crc checking for main preamble of V2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: receive V2 messages with new preable format.

Currently we do support messages with single segment only.
Adding multi-segment and proper alignment handling is WIP.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: transmit V2 messages with new preable format.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: reset crypto processors in ProtocolV2::reset_recv_state().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: preamble of V2 Frames is now encrypted and authenticated.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: slightly rework ProtocolV2 preamble crafting.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: reset the rx stream handler in ::handle_read_frame_length_and_tag.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth, msg/async, v2: drop AuthStreamHandler and AES128GCM_StreamHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: Messages in Protocol2 are crypto-processed only once.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: SignedEncryptedFrame uses ceph::crypto::onwire.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: expose message segmentation to ::write_message().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: implement ceph::crypto::onwire with OpenSSL EVP.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: introduce ceph::crypto::onwire interfaces.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: decouple MessageHeaderFrame from SignedEncryptedFrame.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: move Protocol* asserts in SignedEncryptedFrame to compile time.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: simplify encryption handling in the PayloadFrame class.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: drop AES128CBC_HMACSHA256_StreamHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: ensure consistency between con_mode and session_security.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: drop MessageFrame. Use MessageHeaderFrame instead.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: set con_mode and session_security at both peers.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, auth: switch AuthStreamHandler::rxtx_t to std::unique_ptr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crypto: AES128GCM_StreamHandler brings authenticated encryption with AES-GCM.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

include: uint128_t -> ceph::uint128_t + using.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: move crypto handling from ProtocolV2 into AuthStreamHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth, msg: dissect AuthStreamHandler from AuthSessionHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth/cephx: make _calc_signature() of CephxSessionHandler private.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: drop {en,de}crypt_message() from AuthSessionHandler.

Yeah, this means wire encryption isn't supposed for V1 anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: introduce DummyAuthSessionHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: make AuthSessionHandler purely abstract.

This is possible because the dropped state is/was required only by:
 * CephxSessionHandler,
 * methods removed by the previous commits.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: drop no_security() from AuthSessionHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: drop get_protocol() and get_key() from AuthSessionHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: drop sign_bufferlist() from AuthSessionHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: drop get_auth_meta() from Protocol.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: emphasize ProtocolV2 does authenticated encryption.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

unittest_auth: add unit tests for AuthRegistry con_mode selection

Signed-off-by: Sage Weil <sage@redhat.com>