Merge pull request #39044 from orozery/librbd-crypto-close-bug

librbd: do not share crypto image layers with ancestors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39006 from orozery/librbd-luks-cli-test

test/librbd: add luks encryption cli test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39005 from orozery/librbd-plain64-iv-fix

librbd: use 512 sector size for crypto IV

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39064 from idryomov/wip-doc-osdtimeout

doc: don't mention osdtimeout option in mount.ceph man page

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #39060 from idryomov/wip-krbd-msgr2

krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38929 from jdurgin/wip-cot-log

qa/tasks/ceph_manager: let c-o-t log errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38931 from trociny/wip-48821

osd: fix potential null pointer dereference when sending ping

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38901 from gmayyyha/io_uring-48872

blk/kernel: fix io_uring got (4) Interrupted system call

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38932 from tchaikov/wip-48905

mgr: update mon metadata when monmap is updated

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37250 from cbodley/wip-qa-librgw

qa/rgw: add librgw_file unit tests to rgw/verify task

Merge pull request #38852 from adamemerson/wip-thou-shalt-not-unwind

Thou Shalt Not Unwind

Merge pull request #38905 from cbodley/wip-48874

rgw: fix trailing null in object names of multipart reuploads

Merge pull request #39063 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific branch to the mix

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39009 from mdw-at-linuxbox/wip-master-keystonebindep

qa/tasks/keystone.py: centos-8 bindep fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38962 from aclamk/wip-aclamk-bluestore-tool-reshard-fix

Fix ceph-bluestore-tool reshard command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38965 from aclamk/wip-aclamk-bluestore-tool-sharding-show

Add command 'show-sharding' to ceph-bluestore-tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

test/rgw_file: bypass one bad-deletes test due to intermittent failure

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

test/rgw_file: override keyring if we gen keys

Run with ceph.keyring not that of client.${CEPH_ID}.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

test/rgw_file: elaborate test cycle

Ensure that all delete phases are run so that script can be
re-run when desired.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
 Please enter the commit message for your changes. Lines starting

rgw_file: delete test bucket when do_delete

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: add librgw unit test arguments

Also adjust some unit test files to avoid trivial failures.
Fixes missing commit after rgw_write(), as well as conditional
cleanup.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

qa/rgw: librgw workunit creates a test user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: install the ceph_test_librgw_file_* targets

these need to be installed in order to be included in packages for
testing in teuthology

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

qa/rgw: add librgw_file unit tests to rgw/verify task

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/suites/krbd: add msgr2 modes to most subsuites

basic, rbd and rbd-nomount subsuites are expanded to run with each
of ms_mode=legacy, ms_mode=crc and ms_mode=secure.  This increases
the total number of jobs in the suite from 100 to 220.

fsx, singleton and thrash subsuites choose ms_mode at random (from
the above plus ms_mode=prefer-crc).

unmap and wac subsuites remain msgr1-only.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: deprecate [no]cephx_require_signatures map options

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: add support for msgr2

Recognize ms_mode map option and filter initial monitor addresses
accordingly: if ms_mode is not given or ms_mode=legacy, discard v2
addresses, otherwise discard v1 addresses.

Note that nothing was discarded (i.e. v2 addresses were passed to
the kernel) previously.  The intent was to preserve that behaviour
in case ms_mode is not given, allowing to change the kernel default
in the future.  However, it turns out that mount.ceph helper has
been misguidedly discarding v2 addresses since commit eae01275134e
("mount.ceph: fork a child to get info from local configuration"),
so that ship has sailed.

Fixes: https://tracker.ceph.com/issues/48976
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37694 from BryceCao/wip-add-MD5-in-forward_request

rgw : add MD5 in forward_request

Merge pull request #38977 from adamemerson/wip-48929

rgw: Fix spurious error on empty datalog shard

Merge pull request #38958 from badone/wip-osd-df-tree-accounting-error

An empty bucket or OSD is not an error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38992 from ifed01/wip-ifed-fix-48776-2

os/bluestore: be more protective in _deferred_queue

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: don't mention osdtimeout option in mount.ceph man page

It was deprecated over 8 years ago and will be entirely removed in
kernel 5.12.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #39051 from tchaikov/wip-install-deps-libzbd

install-deps.sh: install libzbd-dev when WITH_ZBD is set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: added pacific branch to the mix

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

install-deps.sh: install libzbd-dev when WITH_ZBD is set

to avoid zbd backend bitrot, and for build test it in our "make check"
builds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39040 from liewegas/cephadm-qa-targets

qa/suites/rados/cephadm: test against ubuntu 20.04 (and update ubuntu_latest)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: do not share crypto image layers with ancestors

This commit creates a unique crypto image dispatch per image,
instead of sharing a single object with all ancestors.
This should fix a seg-fault caused when closing an encrypted cloned image.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: add luks encryption cli test

This commit adds a cli test for rbd encryption verifying LUKS compatbility with cryptsetup

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa/workunits/cephadm/test_repos: adjust target version

15.2.7 is available for 20.04; 15.1.1 is not

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39042 from tchaikov/wip-cram

test/run-cli-tests: use cram from github

Reviewed-by: Sage Weil <sage@redhat.com>

test/downloads: drop cram-0.5 tar ball

since we install cram from github, drop the tarball from repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/run-cli-tests: use cram from github

to address the failure of

  WARNING: Built wheel for cram is invalid: Metadata 1.2 mandates PEP 440 version, but '0.5.0ceph.2011-01-14' is not
Failed to build cram

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39035 from tchaikov/wip-cephadm-call

cephadm: rewrite call() with asyncio

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: rewrite call() with asyncio

for better readability, also return 124 when subprocess times out

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/cephadm: add 20.04 targets

Add 20.04 targets.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/cephadm: add 20.04 target

Link directly to the distro version... no need to use _latest here since
it obscures the podman vs docker difference.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/distros: change ubuntu_latest to 20.04

Signed-off-by: Sage Weil <sage@newdream.net>

qa/distros: add ubuntu 20.04 targets

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #38974 into master

* refs/pull/38974/head:
qa/suites/rados/cephadm/upgrade: avoid allow_ptrace with old cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39032 from adamemerson/wip-layout-initialize

rgw: Initialize Layout Logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39029 from dillaman/wip-48964

ocf: add support for mapping images within an RBD namespace

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39024 from dillaman/wip-librbd-q-release

librbd: bump minor version to match Q release

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39034 from liewegas/fix-vstart-init-ceph

init-ceph: fix python and library paths for vstart

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38957 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific, changed octopus number of runs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

init-ceph: fix python and library paths for vstart

The current version does not result in a functional mgr after restarting
from a vstart cluster.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: Initialize logs on layout initialization

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39013 from MahatiC/wip-ssd-doc

doc: update rwl doc to include ssd config

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38942 from smithfarm/wip-bci-pacific

script: make backport scripts know about pacific

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39003 into master

* refs/pull/39003/head:
cephadm: fix 2> syntax in unit.run

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

qa/tests: moved 'master' runs to 'gibba' machines

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38990 into master

* refs/pull/38990/head:
mgr/devicehealth: fix 'device monitoring on|off' return value

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

ocf: add support for mapping images within an RBD namespace

Support for RBD namespaces was not added to the OCF resource agent.
This commit fixes this oversight and also fixes a bug in handling
of the "rbd device list" output.

Fixes: https://tracker.ceph.com/issues/48964
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

script: make backport scripts know about pacific

Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: bump minor version to match Q release

This will assist layered projects like QEMU and libvirt
determine which features are supported.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38938 from rhcs-dashboard/rgw-user-validation

mgr/dashboard: Fix for incorrect validation in rgw user form

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #39016 from sebastian-philipp/cephadm-mypy-status-none

mgr/cephadm: mypy says: dd.status can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38999 from tchaikov/wip-crimson-bootstrap

crimson/osd: fetch_config() before mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cephadm: fix 2> syntax in unit.run

We need a space between the command (which ends with a container name)
and the 2> or else the 2 is considered part of the command.  E.g.,

! /usr/bin/podman rm -f ceph-a9a8c7ee-5b72-11eb-8f93-001a4aab830c-mon.a2> /dev/null

Fixes: 1bed46e4b0094863a119df59c6ae5f254c2e211d
Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: print out osd mkfs command

for better understanding the progress of vstart, and also allows
developer to repeat the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: do not include unused header

MGetConfig.h is not used anywhere in this source file, so no need to
include it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not use __func__ in lambda

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use switch case for checking return code

also, since seastar supports returning plain value instead a ready
future, let's return plain value. simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32027 into master

* refs/pull/32027/head:
librados: avoid symbol versioning on Windows

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: Minor pep8 fix

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: mypy says: dd.status can be None

`DaemonDescription.status` is annotated
as `Optional[int]` and thus can be `None`

This is a conflict between c95ba878c66aae44816b1451049449685d444f0a
and 01f60cf4e0a751c314120c02956d4ff941eb71b4

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38978 from liewegas/cephadm-daemon-health

mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38967 from liewegas/wip-cephadm-pacific

cephadm: latest stable release is now pacific

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38739 from jecluis/wip-cephadm-splits

cephadm: splits bootstrap function, add context, drop global variables

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: update rwl doc to include ssd config

... options.

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

crimson/osd: fetch_config() before mkfs

* fetch_config() before mkfs and starting osd
  for populating settings related to booting and transport layer
  before it starts.
* set fsid read from monitor before mkfs
  it's crucial to mkfs if osd is supposed to retrieve the fsid
  from monitor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: move implementation of dtor into .cc file

ceph_assert() is expanded into 5 lines of code. it'd help to speed up
the compiling a little bit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: add mon::Client::wait_for_config()

just for waiting for monmap and config from mon. crimson-osd needs this
for populating settings related to booting and transport layer before it
starts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: fallback to msgr v2 for unbound msgr

so, for instance, if we want to connect to monitor without bind to any
address, we can try to use the v2 addresses advertised in monmap or
local settings, instead of being unable to connect to mon because we are
using an `entity_addr_t::TYPE_NONE` address which is returned by
`entity_addrvec_t::front()` if the addrvec is empty.

see also AsyncMessenger::should_use_msgr2().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: print out entity addr type when non peer address matches

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39010 from mdw-at-linuxbox/wip-master-barbicany2021

qa/tasks/barbican.py: fix year2021 problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39008 from mdw-at-linuxbox/wip-master-vaultunzip

qa/tasks/vault.py: unzip: try harder to find a working unzip.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Since we aren't using these anyway

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38988 from orozery/librbd-crypto-empty-copyup

librbd: don't restart empty copyups in crypto layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38985 from orozery/rbd-nbd-luks2-blksize

tools/rbd-nbd: change block size to 4K for LUKS2 images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38983 from orozery/qa-qemu-rbd-encryption

qa/tasks/rbd: increase image size of encrypted disks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fix for incorrect validation in rgw user form

The rgw users create form doesnt validate the username correctly if the username is a tenated one. For eg. Consider there is a user called tenate$sample. Now I am trying to create another user and I entered tenate$sample as username. But it doesn't async validate the username as existing. Instead it just shows the green tick and once the submit button is clicked it'll show the user as existing.

Fixes: https://tracker.ceph.com/issues/48907
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/tasks/barbican.py: fix year2021 problem

The expiration timestamp was hard-coded as
2020-12-31T19:14:44.180394
which is now in the past.  Instead, use a timestamp
90 minutes in the future.

Fixes: https://tracker.ceph.com/issues/48919
Signed-off-by: Marcus Watts <mwatts@redhat.com>

qa/tasks/keystone.py: centos-8 bindep fix

The upstream git repository for keystone contains a dependency
for "python34-devel" which only exists in centos 7.
For centos 8, the correct package is "python36-devel".
So, patch bindep.txt before running bindep.

Fixes: https://tracker.ceph.com/issues/48920
Signed-off-by: Marcus Watts <mwatts@redhat.com>

qa/tasks/vault.py: unzip: try harder to find a working unzip.

The existing logic uses "python -m zipfile" to unzip files.
This will (most likely) fail on CentOS 8-Stream , where python
defaults to 'unset' (see man unversioned-python).

So: try harder: try unzip, python3, and python in that order,
to find something that can unzip files.

Fixes: https://tracker.ceph.com/issues/48921
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #38691 from soumyakoduri/lc_fixes

rgw/lc: Fix use-after-free in RGWLC::process

Merge pull request #38465 from IlsooByun/fix_lc_infinite_loop

rgw: lc: fix infinite loop in bucket_lc_prepare

librbd: use 512 sector size for crypto IV

This commit aligns with the LUKS2 format which requires plain-64 mode IV
to be derived from the 512 bytes sector number

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #36698 from vshankar/wip-cephfs-mirror-tests

cephfs-mirror: teuthology task and tests

Merge pull request #35993 from vshankar/wip-fs-mirror-interface

pybind/mgr/mirroring: directory snapshot mirror support

osd/OSDMap: An empty bucket or OSD is not an error

Fixes: https://tracker.ceph.com/issues/48884
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>