14.1.0

Merge pull request #25438 from MahatiC/wip-move-to-trash

rbd: move image to trash as first step when removing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #26506 from ricardoasmarques/multi-backstore-backend

mgr/dashboard: iSCSI - Infrastructure for multiple backstores (backend)

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #26515 from theanalyst/civetweb-fixes-n

civetweb: pull up to ceph-master

Merge PR #26550 into master

* refs/pull/26550/head:
mon/AuthMonitor: provide auth_lock-safe _assign_global_id()
mon/AuthMonitor: provide AuthMonitor with mon count and rank *only* while active

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #26476 into master

* refs/pull/26476/head:
pybind: auto encode/decode cstr
pybind: set language_level for cythonize explicitly

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26552 into master

* refs/pull/26552/head:
osdc/Objecter: invalidate crcs on preallocated rx buffers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26568 into master

* refs/pull/26568/head:
unittest_auth: add unit tests for AuthRegistry con_mode selection
auth/AuthRegistry: register as an observer
auth/AuthRegistry: clear output vectors
common/options: add ms_mon_client_mode
auth: push con_mode selection into AuthRegistry helper

Reviewed-by: Ricardo Dias <rdias@suse.com>

civetweb: pull up to ceph-master

Contains the following fixes
- absolute url fixes
- ssl socket closing fixes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26584 from capri1989/doc-fixes

doc: Remove value 'mon_osd_max_split_count'

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Remove value 'mon_osd_max_split_count' from the docs

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #26510 from callithea/wip-add-ganesha-qa-tests

mgr/dashboard: Added NFS Ganesha suite to QA tests

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #26520 from ktdreyer/rpm-rhel8

rpm: RHEL 8 fixes

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26555 from tchaikov/wip-install-build-deps

cmake,rgw: make amqp support optional

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #26582 from tchaikov/wip-cmake-silence-syntax-warning

cmake: fix syntax error of set()

Reviewed-by: Ricardo Dias <rdias@suse.com>

cmake: fix syntax error of set()

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps,do_freebsd: disable amqp support

per Willem's request.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: disable amqp endpoint on SLE/OpenSUSE

as librabbitmq-devel is not packaged on them yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove amqp_mock from UNITTEST_LIBS

this mock library is only used by rgw tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: build __pycache__ directories on RHEL 8

In RHEL 8, rpmbuild does not automatically generate __pycache__
directories. Generate them explicitly with the %py_byte_compile macro.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rpm: fix pyOpenSSL requirement for RHEL 8

On RHEL 7, this package is named "pyOpenSSL", but on RHEL 8, Fedora, and
SUSE, it is named "python3-pyOpenSSL".

Prior to this change, the RHEL conditional was applying to both RHEL 7
and 8, so we would fail to resolve the "pyOpenSSL" package dependency on
RHEL 8.

Update the RPM conditionals to use "pyOpenSSL" on RHEL 7 and
"python3-pyOpenSSL" on all other distributions.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

rpm: fix python dependencies on RHEL 8

On RHEL 8, the python34-* and Cython packages are named identically to
Fedora (eg "python3-Cython"). Move the Python 2 "Cython" and python34-*
BuildRequires lines inside a RHEL 7 conditional, so RHEL 8 falls through
to the Fedora package names.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #26566 from theanalyst/vstart-beast

vstart: make beast as the default frontend for rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

unittest_auth: add unit tests for AuthRegistry con_mode selection

Signed-off-by: Sage Weil <sage@redhat.com>

auth/AuthRegistry: register as an observer

Signed-off-by: Sage Weil <sage@redhat.com>

auth/AuthRegistry: clear output vectors

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #26541 into master

* refs/pull/26541/head:
msg/async: guard protocol Interceptor with preprocessor variable
cmake: add UNIT_TESTS_BUILD preprocessor var based on WITH_TESTS cmake var
test/msgr: remove test_msgr2 unused test
msg/async: msgr2: resolve reconnect races
test/msgr: msgr2 unit tests using the protocol interceptor WIP
msg/async: Protocol interceptor infrastructure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26458 into master

* refs/pull/26458/head:
MDSMonitor: do not assign standby-replay when degraded

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26468 into master

* refs/pull/26468/head:
qa: config recall settings to test cache drop
qa: check cache dump works without timeout
mds: add 2nd order recall throttle
mds: drive log flush and cache trim during recall
mds: avoid gather assertion when subs exist
mds: output full details for recall threshold

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26422 into master

* refs/pull/26422/head:
mds: start to trim cache when mds enters clientreplay state
mds: avoid evaluating stray dentry multiple times
mds: simplfy stray manager activation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #26459 into master

* refs/pull/26459/head:
mds: do not show standby-replay as ranks

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #26486 into master

* refs/pull/26486/head:
mds: remove wrong puts on Message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #26237 into master

* refs/pull/26237/head:
Revert "qa: update test_envlibrados_for_rocksdb.sh for libradospp split"
doc/librados: explicitly mention that the C++ API is not stable
ceph.spec: force use of upgrade devtoolset-gcc under RHEL 7
librados: add symbol versioning to the C++ API
librados: add symbol versioning to the C API
librados: revert librados3/libradoscc back to librados2

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async: guard protocol Interceptor with preprocessor variable

Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: add UNIT_TESTS_BUILD preprocessor var based on WITH_TESTS cmake var

Signed-off-by: Ricardo Dias <rdias@suse.com>

pybind: auto encode/decode cstr

assuming all string in the cythonized string use the legacy python2
string type, instead of using the unicode, and assuming all c strings
are ASCII, so we don't need encode/decode them when coerce from/to C
strings.

this helps when generating py3 bindings

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: set language_level for cythonize explicitly

Compiling rbd.pyx because it changed.
[1/1] Cythonizing rbd.pyx
/usr/lib/python2.7/dist-packages/Cython/Compiler/Main.py:367:
FutureWarning: Cython directive 'language_level' not set, using 2 for
now (Py2). This will change in a later re
lease! File: /var/ssd/ceph/src/pybind/rbd/rbd.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

this warning is raised by cython 0.29.2

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart: make beast as the default frontend for rgw

mstart also features an overrideable $RGW_FRONTEND which defaults to beast

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26395 from ykcowrebbaj/patch-1

doc: corrected options of iscsiadm command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26521 from dillaman/wip-38385

qa/workunits: fixed mon address parsing for rbd-mirror

Merge pull request #26502 from tchaikov/wip-common-coroutine-ifdef

common: define BOOST_COROUTINES_NO_DEPRECATION_WARNING if not yet

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Revert "qa: update test_envlibrados_for_rocksdb.sh for libradospp split"

This reverts commit 0475dfdea315e5208abc201355523c8dab78469f.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/librados: explicitly mention that the C++ API is not stable

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph.spec: force use of upgrade devtoolset-gcc under RHEL 7

There is a known crash under C++17 mode that requires an updated
devtoolset to address.

See https://bugzilla.redhat.com/show_bug.cgi?id=1616223

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: add symbol versioning to the C++ API

The future goal would be to change the version for each Ceph major
release to ensure C++ applications will need to be recompiled against
the librados C++ API since we don't guarentee ABI stability.

Fixes: https://tracker.ceph.com/issues/38177
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: fix options of iscsiadm command

Signed-off-by: ZhuJieWen <154667715@qq.com>

Merge pull request #26241 from jmolmo/hosts_management_add_rm

mgr/ansible: Add/remove hosts

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

rgw,cmake: add option "WITH_RADOSGW_AMQP_ENDPOINT"

WITH_RADOSGW_AMQP_ENDPOINT is enabled by default.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/ansible: Add/remove hosts

- Implementation of operations add/rm hosts
- Improvement of error management in Ansible Runner Service Client

- Addressed @sebastian-philipp suggestions and findings
- Remove the <role> parameter of hosts operations.
Now all the hosts are placed by default in a specific Ansible Inventory group called <orchestrator>.
Discussion on-going about to keep Host management API endpoint or not.
If hosts management is removed, further modifications will be needed to:
- Keep updated the <host inventory> in the orchestrator
- Manage hosts groups when operations are executed
- Adapted new commands (host add/rm) to use to the new <orchestrator_cli> syntax

- Improved error management
- Added return information to Add/remove host methods docstring

- Removed redundant exceptions following @sebastian-philipp suggestion

- Removed duplicated code
- Removed extra space

- Fixed unit tests
- Improved error management in launch playbook

- removed unused param in docstring (orchestrator.py)

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

cmake: find_package(RabbitMQ) instead of just checking its header

rabbitmq-c is not a header-only library. so we need to ensure that the
library is also around before linking against it. in this change:

* move the detection down to where librabbitmq is used
* add FindRabbitMQ.cmake for find_package(RabbitMQ)
* s/rabbitmq/RabbitMQ::RabbitMQ/ as the latter is an alias library
target, which has more information attached to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26525 from liupengs/wip-msg-async-fix-rdma-exchage-port

msg/async/rdma: parse IBSYNMsg.lid as hex when receiving message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26532 from xiexingguo/wip-pool-scrub

mgr: add per pool scrub commands

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26533 from badone/wip-test-dup_index-trim

test: Verify a log trim trims the dup_index

Reviewed-By: Eric Ivanvich <ivanvich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: do not install librabbitmq-dev manually

it will be installed by `mk-build-deps --install`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26402 from tchaikov/wip-install-deps

 install-deps.sh: selectively install dependencies

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #26551 into master

* refs/pull/26551/head:
osd/PG: fix get debug refcount line

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26511 into master

* refs/pull/26511/head:
mgr: Selftest fails, dashboard/server_port not string

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #26513 into master

* refs/pull/26513/head:
osd/PG: fix comments in merge_from
osd/PG: align past_intervals and last_epoch_clean for fabricated merge target

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #26526 into master

* refs/pull/26526/head:
rpm,deb: remove btrfs-tools as a build-dependency
rpm,deb: remove bc as a build-dependency

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #26536 into master

* refs/pull/26536/head:
src/tools: only build dedup tool if WITH_TESTS is set

Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

librbd: execute image pre-removal validation before moving image to trash

Fixes: http://tracker.ceph.com/issues/38404
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: split image pre-removal logic to separate state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

msg/async/rdma: parse IBSYNMsg.lid as hex when receiving message

function send_msg encode struct IBSYNMsg to a string, and recv_msg parse the string.
In the function send_msg, im.lid (unsigned short int) is formatted into a string with format string %04x,
so, recv_msg should read im.lid from the string with format string %hx instead of %hu (unsigned short).

Fixes: https://tracker.ceph.com/issues/38391
Signed-off-by: Peng Liu <liupeng37@baidu.com>

mgr: add per pool scrub commands

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osdc/Objecter: invalidate crcs on preallocated rx buffers

Both simple and async messengers use c_str() when copying the data from
the socket into the receive buffer, going behind bufferlist's back.  If
the receive buffer is preallocated, we need to invalidate its crc cache
by hand to avoid possible data crc mismatches on the client side.

Fixes: https://tracker.ceph.com/issues/38416
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon/AuthMonitor: provide auth_lock-safe _assign_global_id()

Break assign_global_id() into two parts:

- _assign_global_id(), called under auth_lock, that does nothing but
  assign an id (or fail).  This makes use of the num_mon and num_rank
  values, which means it only works while we are in quorum.
- increase_max_global_id(), if it looks like we should.

Also, adjust _should_increase_max_global_id() to also work under auth_lock
since we are looking at last_allocated_id.  All last_allocated_id users
are not protected by auth_lock.

Fixes: http://tracker.ceph.com/issues/38372
Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: fix get debug refcount line

Signed-off-by: Sage Weil <sage@redhat.com>

mon/AuthMonitor: provide AuthMonitor with mon count and rank *only* while active

Provide AuthMonitor with the mon count and rank under the mon->auth_lock.
Only set these values while the monitor is active (and the mon is allowed
to authenticate).  When we start an election or bootstrap, clear these
values.

Note that we aren't using them yet!

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: drop extra check for image watchers from trash move API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: attempt to restore deleted image from trash on known failures

If the image has watchers, snapshots, or group links, restore the image
from the trash back to the image directory since data removal hasn't yet
started.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: don't hide partially deleted images in long image list

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: include in-progress image removals in image list

Exclude them from the trash list to prevent double-counting
them. This mimics the previous behavior for handling interrupted
image removals.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fixed some edge cases when moving images to trash upon remove

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: skip mirror image verification when pruning rbd-mirror watchers

When an image is moved to the trash, mirroring is disabled which will remove
the image id from the mirroring directory. If you then quickly attempt to
remove the image from the trash, it might fail if the rbd-mirror daemon hasn't
closed the image since it fails to filter out rbd-mirror daemon watchers
against images with mirroring disabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: move image to trash as a first step

... while removing an Image

Fixes: http://tracker.ceph.com/issues/24226
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #26483 from idryomov/wip-send-copyup-cleanups

librbd: send_copyup() fixes and cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librados: add symbol versioning to the C API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: revert librados3/libradoscc back to librados2

For backwards compatibility and upgrade reasons, the librados2
API needs to be preserved and it needs to continue to be compatible
with dependent libraries like librbd1.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26465 from dzafman/wip-38344

osd: Prevent negative local num_bytes sent to peer for backfill reser…

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: CopyupRequest::m_pending_copyups doesn't need to be atomic

It's protected by CopyupRequest::m_lock mutex.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't clone IoCtx in send_copyup()

Calling aio_operate() with the explicit snap_seq and snaps doesn't
mutate IoCtx, so no reason to clone it.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: send copyup and hint only once

In the deep-copy case, don't send copyup and hint twice (once with the
blank and once with the current snapshot context).

Preserve the workaround for compare-and-write added in commit
f6db9b8027b6 ("librbd: copyup state machine needs to handle empty write
ops").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26519 from idryomov/wip-cmpext-bl

librbd: ensure compare-and-write doesn't skip compare after copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #26262 into master

* refs/pull/26262/head:
mgr/orch/ssh: add qa test
qa: install/exclude ceph-mgr-ssh package
debian: package ssh orchestrator
spec.ceph: package ssh orchestrator
ssh/orch: add ssh orchestrator

common/options: add ms_mon_client_mode

The server now has a list of *allowed* modes (no ordering) and the clients
have a list of modes in order of preference.  Since we want everything
connecting to the mon to be secure by default (think: ceph auth set ...),
we need a separate option to ensure we prefer secure mon connections from
the CLI etc.

Signed-off-by: Sage Weil <sage@redhat.com>

auth: push con_mode selection into AuthRegistry helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #26197 from wanghao72/master

rgw: bug in versioning concurrent, list and get have consistency issue

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #26494 from cbodley/wip-rgw-datalog-autotrim

rgw: fix rgw_data_sync_info::json_decode()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/orch/ssh: add qa test

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

qa: install/exclude ceph-mgr-ssh package

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

debian: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

spec.ceph: package ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

ssh/orch: add ssh orchestrator

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #26471 from tnaganawa/patch-3

ceph-deploy would not support --cluster option anymore

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd/PG: fix comments in merge_from

Some of these were stale.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Infrastructure for multiple backstores (backend)

Fixes: https://tracker.ceph.com/issues/38090
Signed-off-by: Ricardo Marques <rimarques@suse.com>

test/librbd: add test cases for compare-and-write with copyup

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: make get_temp_image_name() static

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

test/librados_test_stub: reimplement cmpext()

cmpext is a read operation.  As such, it shouldn't be creating objects
or extending them and it should accept snapshots.

The compare code is completely broken, resulting in false positives
(i.e. failures to detect mismatches) in many cases.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #26299 from yuvalif/add_pubsub_tests

rgw/pubsub: add pubsub tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>