qa/suites/rbd: enable partial discard test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: align discard requests to default bluestore allocation size

This will match the new krbd behavior of aligning discards to the
default bluestore minimum allocation size. Requests that are too
small (or the misaligned head / tail of the request) will be
ignored since it will not result in freed space.

Fixes: http://tracker.ceph.com/issues/38146
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: skip partial discards by default

Avoid the IO expense of writing zeros mid-object when a discard
is attempting to free space.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26603 from dillaman/wip-38453

rbd-mirror: use active/active policy by default

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26632 into master

* refs/pull/26632/head:
14.1.0

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #26609 into master

* refs/pull/26609/head:
ceph_test_msgr: skip injection test for simplemessenger

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26600 from jtlayton/wip-dashboard-rook

mgr/dashboard: special casing for minikube in run-backend-rook-api-request.sh

rbd-mirror: use active/active policy by default

Fixes: http://tracker.ceph.com/issues/38453
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26522 from ashishkumsingh/wip-doc-38310

doc: Fix incorrect mention of 'osd_deep_mon_scrub_interval'

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #26598 into master

* refs/pull/26598/head:
doc/releases/nautilus: note about msgr2
doc/releases/nautilus: rgw highlights
doc/releases/nautilus: draft notes

Reviewed-by: Kai Wagner <kwagner@suse.com>

Merge PR #26280 into master

* refs/pull/26280/head:
mgr: Remove _ceph_(g|s)et_module_option_ex() method
mgr: remove _ceph_get_module_option_ex() method

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #26487 into master

* refs/pull/26487/head:
CLI: ability to change file ownership

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26591 into master

* refs/pull/26591/head:
osd/PrimaryLogPG: handle object !exists in handle_watch_timeout

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/releases/nautilus: note about msgr2

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/nautilus: rgw highlights

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/nautilus: draft notes

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_msgr: skip injection test for simplemessenger

The interceptor is only implemented in asyncmessenger.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26539 from tspmelo/wip-nfs-status

mgr/dashboard: Add NFS status endpoint

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26620 from cyx1231st/wip-seastar-async-echo

test/crimson: split async-msgr out of alien_echo

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: enable unauthenticated connections for async_echo

Enable co-operability with crimson-msgr in unittest_alien_echo.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

test/crimson: split async-msgr out of alien_echo

async-msgr is not compatible with custom-built libs for seastar, so
split it.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

Merge PR #26466 into master

* refs/pull/26466/head:
msg/async, v2: fix wrong base for KeepAliveFrameAck.
msg/async, v2: frame decoding operates on bufferlist.
msg/async, v2: drop ceph_msg_header2 fields duplicating segment info.
msg/async, v2: drop the scaffolding in preamble parsing.
msg/async, v2: handle msg authentication failures.
msg/async, v2: drop depedency on uint128_t. Clean up onwire crypto.
msg/async, v2: fix cur_msg_size in ::reset_recv_state().
msg/async, v2: drop magic numbers for segments.
msg/async, v2: get rid of magic number in SignedEncryptedFrame.
msg/async, v2: get rid of the magic number for default alignment.
msg/async, v2: decouple onwire segment length from logical length.
msg/async, v2: follow the const bl& concept in authenticated_encrypt_update().
msg/async, v2: drop handling of extra segments in ::fill_preamble().
msg/async, v2: get rid of magic numbers for alignment.
msg/async, v2: drop reserve() from onwire crypto's TxHandler.
msg/async: add con_mode to debug lines
msg/DispatchQueue: include con_mode in <== line
common/ceph_strings: get_con_mode_name()
msg/Connection: add get_con_mode()
msg/async/ProtocolV2: clean up preamble comments
msg/async, v2: improve debug around sending client indent.
msg/async, v2: bring back the no-encryption ability.
msg/async, v2: workaround con_mode handling.
msg/async, v2: drop the throttles bypass.
msg/async, v2: READ_MESSAGE_FRONT -> THROTTLE_DONE.
msg/async: WaitFrame of V2 can be crypto processed now.
msg/async: initial multi-segment support for V2.
msg/async: V2 bypasses throttles just for development.
msg/async: rectify reseting security state in ProtocolV2::reset_recv_state().
msg/async: switch to CRC32 for V2 preamble blocks.
msg/async: bump up preamble block size to 32 bytes.
msg/async: get rid of the distiction on main and extra V2 preamble.
msg/async: add debug around empty ClientIdent::addrs.
msg/async: V2 uses segments instead of next_payload_len, part 1.
msg/async: perform V2 frame dispatch in dedicated method.
msg/async: implement crc checking for main preamble of V2.
msg/async: receive V2 messages with new preable format.
msg/async: transmit V2 messages with new preable format.
msg/async: reset crypto processors in ProtocolV2::reset_recv_state().
msg/async: preamble of V2 Frames is now encrypted and authenticated.
msg/async: slightly rework ProtocolV2 preamble crafting.
msg/async: reset the rx stream handler in ::handle_read_frame_length_and_tag.
auth, msg/async, v2: drop AuthStreamHandler and AES128GCM_StreamHandler.
msg/async: Messages in Protocol2 are crypto-processed only once.
msg/async: SignedEncryptedFrame uses ceph::crypto::onwire.
msg/async: expose message segmentation to ::write_message().
auth: implement ceph::crypto::onwire with OpenSSL EVP.
auth: introduce ceph::crypto::onwire interfaces.
msg/async: decouple MessageHeaderFrame from SignedEncryptedFrame.
msg/async: move Protocol* asserts in SignedEncryptedFrame to compile time.
msg/async: simplify encryption handling in the PayloadFrame class.
auth: drop AES128CBC_HMACSHA256_StreamHandler.
msg/async: ensure consistency between con_mode and session_security.
msg/async: drop MessageFrame. Use MessageHeaderFrame instead.
msg/async: set con_mode and session_security at both peers.
msg/async, auth: switch AuthStreamHandler::rxtx_t to std::unique_ptr.
crypto: AES128GCM_StreamHandler brings authenticated encryption with AES-GCM.
include: uint128_t -> ceph::uint128_t + using.
msg/async: move crypto handling from ProtocolV2 into AuthStreamHandler.
auth, msg: dissect AuthStreamHandler from AuthSessionHandler.
auth/cephx: make _calc_signature() of CephxSessionHandler private.
auth: drop {en,de}crypt_message() from AuthSessionHandler.
auth: introduce DummyAuthSessionHandler.
auth: make AuthSessionHandler purely abstract.
auth: drop no_security() from AuthSessionHandler.
auth: drop get_protocol() and get_key() from AuthSessionHandler.
auth: drop sign_bufferlist() from AuthSessionHandler.
msg/async: drop get_auth_meta() from Protocol.
msg/async: emphasize ProtocolV2 does authenticated encryption.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26560 from xiexingguo/wip-pool-force-recovery

mgr: add per pool force-recovery/backfill commands

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

mgr: add per pool force-recovery/backfill commands

For those with multiple storage pools sharing the same devices,
I think it would make much more sense to offer per-pool
commands to bring pools with high priority, e.g., because they
are hosting data of more importance than others, back to normal
quickly.

Fixes: http://tracker.ceph.com/issues/38456
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

msg/async, v2: fix wrong base for KeepAliveFrameAck.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #26602 into master

* refs/pull/26602/head:
Revert "pybind: set language_level for cythonize explicitly"
Revert "pybind: auto encode/decode cstr"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26436 from dzafman/wip-bionic-core

run-standalone.sh: Need double-quotes to handle | in core_pattern on all distributions

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: special casing for minikube in run-backend-rook-api-request.sh

For people running minikube for testing, the hostname "minikube" is
often not resolvable. If that's what the hostname is, then get
the address from "minikube ip" instead.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Revert "pybind: set language_level for cythonize explicitly"

This reverts commit ded3d43dbf8a36daa65d8efe406af4d12ffc7dd8.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "pybind: auto encode/decode cstr"

This reverts commit c36d0f1a7de4668eb81075e4a94846cf81fc30cd.

Signed-off-by: Neha Ojha <nojha@redhat.com>

14.1.0

Merge pull request #25438 from MahatiC/wip-move-to-trash

rbd: move image to trash as first step when removing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26506 from ricardoasmarques/multi-backstore-backend

mgr/dashboard: iSCSI - Infrastructure for multiple backstores (backend)

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

osd/PrimaryLogPG: handle object !exists in handle_watch_timeout

- watch on object
- watch timeout queued
- rados op deletes object
- handle_watch_timeout tries to delete it again

Fixes: http://tracker.ceph.com/issues/38432
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26515 from theanalyst/civetweb-fixes-n

civetweb: pull up to ceph-master

Merge PR #26550 into master

* refs/pull/26550/head:
mon/AuthMonitor: provide auth_lock-safe _assign_global_id()
mon/AuthMonitor: provide AuthMonitor with mon count and rank *only* while active

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #26476 into master

* refs/pull/26476/head:
pybind: auto encode/decode cstr
pybind: set language_level for cythonize explicitly

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26552 into master

* refs/pull/26552/head:
osdc/Objecter: invalidate crcs on preallocated rx buffers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26568 into master

* refs/pull/26568/head:
unittest_auth: add unit tests for AuthRegistry con_mode selection
auth/AuthRegistry: register as an observer
auth/AuthRegistry: clear output vectors
common/options: add ms_mon_client_mode
auth: push con_mode selection into AuthRegistry helper

Reviewed-by: Ricardo Dias <rdias@suse.com>

civetweb: pull up to ceph-master

Contains the following fixes
- absolute url fixes
- ssl socket closing fixes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26584 from capri1989/doc-fixes

doc: Remove value 'mon_osd_max_split_count'

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Remove value 'mon_osd_max_split_count' from the docs

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #26510 from callithea/wip-add-ganesha-qa-tests

mgr/dashboard: Added NFS Ganesha suite to QA tests

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #26520 from ktdreyer/rpm-rhel8

rpm: RHEL 8 fixes

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26555 from tchaikov/wip-install-build-deps

cmake,rgw: make amqp support optional

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26582 from tchaikov/wip-cmake-silence-syntax-warning

cmake: fix syntax error of set()

Reviewed-by: Ricardo Dias <rdias@suse.com>

cmake: fix syntax error of set()

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps,do_freebsd: disable amqp support

per Willem's request.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: disable amqp endpoint on SLE/OpenSUSE

as librabbitmq-devel is not packaged on them yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove amqp_mock from UNITTEST_LIBS

this mock library is only used by rgw tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async, v2: frame decoding operates on bufferlist.

This change is driven by buggy buffer's life time management
polluting AuthAuthorizer::bl with dangling raw_static instances.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop ceph_msg_header2 fields duplicating segment info.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop the scaffolding in preamble parsing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: handle msg authentication failures.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop depedency on uint128_t. Clean up onwire crypto.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: fix cur_msg_size in ::reset_recv_state().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop magic numbers for segments.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: get rid of magic number in SignedEncryptedFrame.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: get rid of the magic number for default alignment.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: decouple onwire segment length from logical length.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: follow the const bl& concept in authenticated_encrypt_update().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop handling of extra segments in ::fill_preamble().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: get rid of magic numbers for alignment.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop reserve() from onwire crypto's TxHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: add con_mode to debug lines

Signed-off-by: Sage Weil <sage@redhat.com>

msg/DispatchQueue: include con_mode in <== line

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_strings: get_con_mode_name()

Signed-off-by: Sage Weil <sage@redhat.com>

msg/Connection: add get_con_mode()

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: clean up preamble comments

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async, v2: improve debug around sending client indent.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: bring back the no-encryption ability.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: workaround con_mode handling.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: drop the throttles bypass.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async, v2: READ_MESSAGE_FRONT -> THROTTLE_DONE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: WaitFrame of V2 can be crypto processed now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rpm: build __pycache__ directories on RHEL 8

In RHEL 8, rpmbuild does not automatically generate __pycache__
directories. Generate them explicitly with the %py_byte_compile macro.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rpm: fix pyOpenSSL requirement for RHEL 8

On RHEL 7, this package is named "pyOpenSSL", but on RHEL 8, Fedora, and
SUSE, it is named "python3-pyOpenSSL".

Prior to this change, the RHEL conditional was applying to both RHEL 7
and 8, so we would fail to resolve the "pyOpenSSL" package dependency on
RHEL 8.

Update the RPM conditionals to use "pyOpenSSL" on RHEL 7 and
"python3-pyOpenSSL" on all other distributions.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rpm: fix python dependencies on RHEL 8

On RHEL 8, the python34-* and Cython packages are named identically to
Fedora (eg "python3-Cython"). Move the Python 2 "Cython" and python34-*
BuildRequires lines inside a RHEL 7 conditional, so RHEL 8 falls through
to the Fedora package names.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #26566 from theanalyst/vstart-beast

vstart: make beast as the default frontend for rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async: initial multi-segment support for V2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: V2 bypasses throttles just for development.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: rectify reseting security state in ProtocolV2::reset_recv_state().

The reason for that is:

  2019-02-14 19:59:00.774 7f78aebf3700 20 --2- v2:127.0.0.1:40678/0 >>
    v2:127.0.0.1:40680/0 conn(0x5574ae75e480 0x5574ae696000 :-1 s=READY
    pgs=11 cs=0 l=0 rx=0x5574ae2af5f0 tx=0x5574ae79bc00).handle_read_frame_preamble_main
    r=-1
  2019-02-14 19:59:00.774 7f78aebf3700  1 --2- v2:127.0.0.1:40678/0 >>
    v2:127.0.0.1:40680/0 conn(0x5574ae75e480 0x5574ae696000 :-1 s=READY
    pgs=11 cs=0 l=0 rx=0x5574ae2af5f0 tx=0x5574ae79bc00).handle_read_frame_preamble_main
    read frame length and tag failed r=-1 ((1) Operation not permitted)
  2019-02-14 19:59:00.774 7f78aebf3700 10 --2- v2:127.0.0.1:40678/0 >>
    v2:127.0.0.1:40680/0 conn(0x5574ae75e480 0x5574ae696000 :-1 s=READY
    pgs=11 cs=0 l=0 rx=0x5574ae2af5f0 tx=0x5574ae79bc00)._fault

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: switch to CRC32 for V2 preamble blocks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: bump up preamble block size to 32 bytes.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: get rid of the distiction on main and extra V2 preamble.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: add debug around empty ClientIdent::addrs.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: V2 uses segments instead of next_payload_len, part 1.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: perform V2 frame dispatch in dedicated method.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: implement crc checking for main preamble of V2.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: receive V2 messages with new preable format.

Currently we do support messages with single segment only.
Adding multi-segment and proper alignment handling is WIP.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: transmit V2 messages with new preable format.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: reset crypto processors in ProtocolV2::reset_recv_state().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: preamble of V2 Frames is now encrypted and authenticated.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: slightly rework ProtocolV2 preamble crafting.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: reset the rx stream handler in ::handle_read_frame_length_and_tag.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth, msg/async, v2: drop AuthStreamHandler and AES128GCM_StreamHandler.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: Messages in Protocol2 are crypto-processed only once.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: SignedEncryptedFrame uses ceph::crypto::onwire.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

msg/async: expose message segmentation to ::write_message().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

auth: implement ceph::crypto::onwire with OpenSSL EVP.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>