Merge pull request #38992 from ifed01/wip-ifed-fix-48776-2

os/bluestore: be more protective in _deferred_queue

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #39051 from tchaikov/wip-install-deps-libzbd

install-deps.sh: install libzbd-dev when WITH_ZBD is set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

install-deps.sh: install libzbd-dev when WITH_ZBD is set

to avoid zbd backend bitrot, and for build test it in our "make check"
builds.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39040 from liewegas/cephadm-qa-targets

qa/suites/rados/cephadm: test against ubuntu 20.04 (and update ubuntu_latest)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/cephadm/test_repos: adjust target version

15.2.7 is available for 20.04; 15.1.1 is not

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39042 from tchaikov/wip-cram

test/run-cli-tests: use cram from github

Reviewed-by: Sage Weil <sage@redhat.com>

test/downloads: drop cram-0.5 tar ball

since we install cram from github, drop the tarball from repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/run-cli-tests: use cram from github

to address the failure of

  WARNING: Built wheel for cram is invalid: Metadata 1.2 mandates PEP 440 version, but '0.5.0ceph.2011-01-14' is not
Failed to build cram

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39035 from tchaikov/wip-cephadm-call

cephadm: rewrite call() with asyncio

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: rewrite call() with asyncio

for better readability, also return 124 when subprocess times out

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/cephadm: add 20.04 targets

Add 20.04 targets.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/rados/cephadm: add 20.04 target

Link directly to the distro version... no need to use _latest here since
it obscures the podman vs docker difference.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/distros: change ubuntu_latest to 20.04

Signed-off-by: Sage Weil <sage@newdream.net>

qa/distros: add ubuntu 20.04 targets

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #38974 into master

* refs/pull/38974/head:
qa/suites/rados/cephadm/upgrade: avoid allow_ptrace with old cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39032 from adamemerson/wip-layout-initialize

rgw: Initialize Layout Logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39029 from dillaman/wip-48964

ocf: add support for mapping images within an RBD namespace

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39024 from dillaman/wip-librbd-q-release

librbd: bump minor version to match Q release

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39034 from liewegas/fix-vstart-init-ceph

init-ceph: fix python and library paths for vstart

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38957 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific, changed octopus number of runs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

init-ceph: fix python and library paths for vstart

The current version does not result in a functional mgr after restarting
from a vstart cluster.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: Initialize logs on layout initialization

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39013 from MahatiC/wip-ssd-doc

doc: update rwl doc to include ssd config

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38942 from smithfarm/wip-bci-pacific

script: make backport scripts know about pacific

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39003 into master

* refs/pull/39003/head:
cephadm: fix 2> syntax in unit.run

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

qa/tests: moved 'master' runs to 'gibba' machines

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38990 into master

* refs/pull/38990/head:
mgr/devicehealth: fix 'device monitoring on|off' return value

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

ocf: add support for mapping images within an RBD namespace

Support for RBD namespaces was not added to the OCF resource agent.
This commit fixes this oversight and also fixes a bug in handling
of the "rbd device list" output.

Fixes: https://tracker.ceph.com/issues/48964
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

script: make backport scripts know about pacific

Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: bump minor version to match Q release

This will assist layered projects like QEMU and libvirt
determine which features are supported.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38938 from rhcs-dashboard/rgw-user-validation

mgr/dashboard: Fix for incorrect validation in rgw user form

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #39016 from sebastian-philipp/cephadm-mypy-status-none

mgr/cephadm: mypy says: dd.status can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38999 from tchaikov/wip-crimson-bootstrap

crimson/osd: fetch_config() before mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cephadm: fix 2> syntax in unit.run

We need a space between the command (which ends with a container name)
and the 2> or else the 2 is considered part of the command.  E.g.,

! /usr/bin/podman rm -f ceph-a9a8c7ee-5b72-11eb-8f93-001a4aab830c-mon.a2> /dev/null

Fixes: 1bed46e4b0094863a119df59c6ae5f254c2e211d
Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: print out osd mkfs command

for better understanding the progress of vstart, and also allows
developer to repeat the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: do not include unused header

MGetConfig.h is not used anywhere in this source file, so no need to
include it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not use __func__ in lambda

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use switch case for checking return code

also, since seastar supports returning plain value instead a ready
future, let's return plain value. simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32027 into master

* refs/pull/32027/head:
librados: avoid symbol versioning on Windows

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: Minor pep8 fix

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: mypy says: dd.status can be None

`DaemonDescription.status` is annotated
as `Optional[int]` and thus can be `None`

This is a conflict between c95ba878c66aae44816b1451049449685d444f0a
and 01f60cf4e0a751c314120c02956d4ff941eb71b4

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38978 from liewegas/cephadm-daemon-health

mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38967 from liewegas/wip-cephadm-pacific

cephadm: latest stable release is now pacific

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38739 from jecluis/wip-cephadm-splits

cephadm: splits bootstrap function, add context, drop global variables

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: update rwl doc to include ssd config

... options.

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

crimson/osd: fetch_config() before mkfs

* fetch_config() before mkfs and starting osd
  for populating settings related to booting and transport layer
  before it starts.
* set fsid read from monitor before mkfs
  it's crucial to mkfs if osd is supposed to retrieve the fsid
  from monitor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: move implementation of dtor into .cc file

ceph_assert() is expanded into 5 lines of code. it'd help to speed up
the compiling a little bit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: add mon::Client::wait_for_config()

just for waiting for monmap and config from mon. crimson-osd needs this
for populating settings related to booting and transport layer before it
starts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: fallback to msgr v2 for unbound msgr

so, for instance, if we want to connect to monitor without bind to any
address, we can try to use the v2 addresses advertised in monmap or
local settings, instead of being unable to connect to mon because we are
using an `entity_addr_t::TYPE_NONE` address which is returned by
`entity_addrvec_t::front()` if the addrvec is empty.

see also AsyncMessenger::should_use_msgr2().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: print out entity addr type when non peer address matches

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39010 from mdw-at-linuxbox/wip-master-barbicany2021

qa/tasks/barbican.py: fix year2021 problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39008 from mdw-at-linuxbox/wip-master-vaultunzip

qa/tasks/vault.py: unzip: try harder to find a working unzip.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Since we aren't using these anyway

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38988 from orozery/librbd-crypto-empty-copyup

librbd: don't restart empty copyups in crypto layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38985 from orozery/rbd-nbd-luks2-blksize

tools/rbd-nbd: change block size to 4K for LUKS2 images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38983 from orozery/qa-qemu-rbd-encryption

qa/tasks/rbd: increase image size of encrypted disks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fix for incorrect validation in rgw user form

The rgw users create form doesnt validate the username correctly if the username is a tenated one. For eg. Consider there is a user called tenate$sample. Now I am trying to create another user and I entered tenate$sample as username. But it doesn't async validate the username as existing. Instead it just shows the green tick and once the submit button is clicked it'll show the user as existing.

Fixes: https://tracker.ceph.com/issues/48907
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/tasks/barbican.py: fix year2021 problem

The expiration timestamp was hard-coded as
2020-12-31T19:14:44.180394
which is now in the past.  Instead, use a timestamp
90 minutes in the future.

Fixes: https://tracker.ceph.com/issues/48919
Signed-off-by: Marcus Watts <mwatts@redhat.com>

qa/tasks/vault.py: unzip: try harder to find a working unzip.

The existing logic uses "python -m zipfile" to unzip files.
This will (most likely) fail on CentOS 8-Stream , where python
defaults to 'unset' (see man unversioned-python).

So: try harder: try unzip, python3, and python in that order,
to find something that can unzip files.

Fixes: https://tracker.ceph.com/issues/48921
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #38691 from soumyakoduri/lc_fixes

rgw/lc: Fix use-after-free in RGWLC::process

Merge pull request #38465 from IlsooByun/fix_lc_infinite_loop

rgw: lc: fix infinite loop in bucket_lc_prepare

Merge pull request #36698 from vshankar/wip-cephfs-mirror-tests

cephfs-mirror: teuthology task and tests

Merge pull request #35993 from vshankar/wip-fs-mirror-interface

pybind/mgr/mirroring: directory snapshot mirror support

qa/suites/rados/cephadm/upgrade: avoid allow_ptrace with old cephadm

Older cephadm is not smart enough to not combine --cap-add=SYS_PTRACE
and --privileged, which some version of podman cannot handle.

For upgrades, leave off the allow_ptrace behavior since we may be starting
on one of those old versions.

See also https://tracker.ceph.com/issues/46429

Fixes: https://tracker.ceph.com/issues/48142
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38991 from cbodley/wip-rgw-perm-state-valgrind

rgw: fix reference to temporary in perm_state ctor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38817 from ideepika/fix-interactive-error

qa/tasks/ceph: do not update info.yaml if ctx.archive is not set

Merge pull request #38941 from sseshasa/wip-48906-async-rec-min-cost-fix

osd: Remove override for osd_async_recovery_min_cost for mclock profiles

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: be more protective in _deferred_queue

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

rgw: avoid (rgw_file) illegal access to stack memory

Fixes: https://tracker.ceph.com/issues/48941
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/devicehealth: fix 'device monitoring on|off' return value

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38987 from cbodley/wip-qa-rgw-valgrind-supp-tls

valgrind: update suppression for _dl_allocate_tls

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: don't restart empty copyups in crypto layer

This commit fixes a bug where an empty parent copyup is restarted indefinitely.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38966 from rzarzynski/wip-qa-rgw-no-valgrind-early-exit

qa/task/rgw: instruct Valgrind to don't exit on first error.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

valgrind: update suppression for _dl_allocate_tls

in the rgw suite, there's an extra allocate_dtv() call in this stack.
add a ... to match it

  <stack>
    <frame>
      <ip>0x4C3321A</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>calloc</fn>
      <dir>/builddir/build/BUILD/valgrind-3.16.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>760</line>
    </frame>
    <frame>
      <ip>0x4012341</ip>
      <obj>/usr/lib64/ld-2.28.so</obj>
      <fn>allocate_dtv</fn>
    </frame>
    <frame>
      <ip>0x4012CD1</ip>
      <obj>/usr/lib64/ld-2.28.so</obj>
      <fn>_dl_allocate_tls</fn>
    </frame>
    <frame>
      <ip>0x101C7F32</ip>
      <obj>/usr/lib64/libpthread-2.28.so</obj>
      <fn>pthread_create@@GLIBC_2.2.5</fn>
    </frame>
    <frame>
      <ip>0x6F14E64</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>Thread::try_create(unsigned long)</fn>
    </frame>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephadm: fix stuff

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: annotate variable type for mypy

mypy was complaining about the type of the variable being None, later on
on assignment with an int.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/test: support context class

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: have context handle config options

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: drop ctx class from where we don't need it

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/bootstrap: move mon net discovery to function

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: don't forget ipv6 behind

Fixes a bug introduced earlier in the patchset. It's just too hard to go
back and fix it up with the right commit.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off config work on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off dashboard setup on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off ssh preparations on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off mgr creation

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split mon prepare and create

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: move configuration out of the way

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/bootstrap: spin-off waiting for mon

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split some of bootstrap to other functions

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: fix linting/types issues

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split main into an init function

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: introduce context, drop global variables

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

qa/tasks/rbd: increase image size of encrypted disks

This commit increases the rbd image sizes in used by qemu to support the storage requirements of the luks encryption header.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

tools/rbd-nbd: change block size to 4K for LUKS2 images

This commit sets the NBD device block size to 4K for images encrypted using LUKS2 (whose sector size is by default 4K(.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38945 from sebastian-philipp/qa-cephadm-smoke-yaml

qa/cephadm: Add yaml output to smoke test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38954 from sebastian-philipp/orchestrator-disallow-untyped

mgr/orchestrator: disallow_untyped_defs = True

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38946 from sebastian-philipp/cephadm-pyright-fix-node-exporter

cephadm: Fix node-exporter deployment.

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge pull request #38968 from dillaman/wip-rbd-error

rbd: deprecation warning should be printed to stderr

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38934 from ifed01/wip-ifed-fix-48776

os/bluestore: fix deferred_queue locking

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38959 from badone/wip-valgrind-tracker-48918

osd: initialise m_interval_start

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>