ocf: add support for mapping images within an RBD namespace

Support for RBD namespaces was not added to the OCF resource agent.
This commit fixes this oversight and also fixes a bug in handling
of the "rbd device list" output.

Fixes: https://tracker.ceph.com/issues/48964
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38938 from rhcs-dashboard/rgw-user-validation

mgr/dashboard: Fix for incorrect validation in rgw user form

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #39016 from sebastian-philipp/cephadm-mypy-status-none

mgr/cephadm: mypy says: dd.status can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38999 from tchaikov/wip-crimson-bootstrap

crimson/osd: fetch_config() before mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart.sh: print out osd mkfs command

for better understanding the progress of vstart, and also allows
developer to repeat the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: do not include unused header

MGetConfig.h is not used anywhere in this source file, so no need to
include it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not use __func__ in lambda

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use switch case for checking return code

also, since seastar supports returning plain value instead a ready
future, let's return plain value. simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32027 into master

* refs/pull/32027/head:
librados: avoid symbol versioning on Windows

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: Minor pep8 fix

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: mypy says: dd.status can be None

`DaemonDescription.status` is annotated
as `Optional[int]` and thus can be `None`

This is a conflict between c95ba878c66aae44816b1451049449685d444f0a
and 01f60cf4e0a751c314120c02956d4ff941eb71b4

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38978 from liewegas/cephadm-daemon-health

mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38967 from liewegas/wip-cephadm-pacific

cephadm: latest stable release is now pacific

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38739 from jecluis/wip-cephadm-splits

cephadm: splits bootstrap function, add context, drop global variables

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

crimson/osd: fetch_config() before mkfs

* fetch_config() before mkfs and starting osd
  for populating settings related to booting and transport layer
  before it starts.
* set fsid read from monitor before mkfs
  it's crucial to mkfs if osd is supposed to retrieve the fsid
  from monitor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: move implementation of dtor into .cc file

ceph_assert() is expanded into 5 lines of code. it'd help to speed up
the compiling a little bit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: add mon::Client::wait_for_config()

just for waiting for monmap and config from mon. crimson-osd needs this
for populating settings related to booting and transport layer before it
starts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: fallback to msgr v2 for unbound msgr

so, for instance, if we want to connect to monitor without bind to any
address, we can try to use the v2 addresses advertised in monmap or
local settings, instead of being unable to connect to mon because we are
using an `entity_addr_t::TYPE_NONE` address which is returned by
`entity_addrvec_t::front()` if the addrvec is empty.

see also AsyncMessenger::should_use_msgr2().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: print out entity addr type when non peer address matches

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39010 from mdw-at-linuxbox/wip-master-barbicany2021

qa/tasks/barbican.py: fix year2021 problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39008 from mdw-at-linuxbox/wip-master-vaultunzip

qa/tasks/vault.py: unzip: try harder to find a working unzip.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38988 from orozery/librbd-crypto-empty-copyup

librbd: don't restart empty copyups in crypto layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38985 from orozery/rbd-nbd-luks2-blksize

tools/rbd-nbd: change block size to 4K for LUKS2 images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38983 from orozery/qa-qemu-rbd-encryption

qa/tasks/rbd: increase image size of encrypted disks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fix for incorrect validation in rgw user form

The rgw users create form doesnt validate the username correctly if the username is a tenated one. For eg. Consider there is a user called tenate$sample. Now I am trying to create another user and I entered tenate$sample as username. But it doesn't async validate the username as existing. Instead it just shows the green tick and once the submit button is clicked it'll show the user as existing.

Fixes: https://tracker.ceph.com/issues/48907
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/tasks/barbican.py: fix year2021 problem

The expiration timestamp was hard-coded as
2020-12-31T19:14:44.180394
which is now in the past.  Instead, use a timestamp
90 minutes in the future.

Fixes: https://tracker.ceph.com/issues/48919
Signed-off-by: Marcus Watts <mwatts@redhat.com>

qa/tasks/vault.py: unzip: try harder to find a working unzip.

The existing logic uses "python -m zipfile" to unzip files.
This will (most likely) fail on CentOS 8-Stream , where python
defaults to 'unset' (see man unversioned-python).

So: try harder: try unzip, python3, and python in that order,
to find something that can unzip files.

Fixes: https://tracker.ceph.com/issues/48921
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #38691 from soumyakoduri/lc_fixes

rgw/lc: Fix use-after-free in RGWLC::process

Merge pull request #38465 from IlsooByun/fix_lc_infinite_loop

rgw: lc: fix infinite loop in bucket_lc_prepare

Merge pull request #36698 from vshankar/wip-cephfs-mirror-tests

cephfs-mirror: teuthology task and tests

Merge pull request #35993 from vshankar/wip-fs-mirror-interface

pybind/mgr/mirroring: directory snapshot mirror support

Merge pull request #38991 from cbodley/wip-rgw-perm-state-valgrind

rgw: fix reference to temporary in perm_state ctor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38817 from ideepika/fix-interactive-error

qa/tasks/ceph: do not update info.yaml if ctx.archive is not set

Merge pull request #38941 from sseshasa/wip-48906-async-rec-min-cost-fix

osd: Remove override for osd_async_recovery_min_cost for mclock profiles

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: avoid (rgw_file) illegal access to stack memory

Fixes: https://tracker.ceph.com/issues/48941
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #38987 from cbodley/wip-qa-rgw-valgrind-supp-tls

valgrind: update suppression for _dl_allocate_tls

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: don't restart empty copyups in crypto layer

This commit fixes a bug where an empty parent copyup is restarted indefinitely.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38966 from rzarzynski/wip-qa-rgw-no-valgrind-early-exit

qa/task/rgw: instruct Valgrind to don't exit on first error.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

valgrind: update suppression for _dl_allocate_tls

in the rgw suite, there's an extra allocate_dtv() call in this stack.
add a ... to match it

  <stack>
    <frame>
      <ip>0x4C3321A</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>calloc</fn>
      <dir>/builddir/build/BUILD/valgrind-3.16.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>760</line>
    </frame>
    <frame>
      <ip>0x4012341</ip>
      <obj>/usr/lib64/ld-2.28.so</obj>
      <fn>allocate_dtv</fn>
    </frame>
    <frame>
      <ip>0x4012CD1</ip>
      <obj>/usr/lib64/ld-2.28.so</obj>
      <fn>_dl_allocate_tls</fn>
    </frame>
    <frame>
      <ip>0x101C7F32</ip>
      <obj>/usr/lib64/libpthread-2.28.so</obj>
      <fn>pthread_create@@GLIBC_2.2.5</fn>
    </frame>
    <frame>
      <ip>0x6F14E64</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>Thread::try_create(unsigned long)</fn>
    </frame>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephadm: fix stuff

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: annotate variable type for mypy

mypy was complaining about the type of the variable being None, later on
on assignment with an int.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/test: support context class

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: have context handle config options

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: drop ctx class from where we don't need it

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/bootstrap: move mon net discovery to function

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: don't forget ipv6 behind

Fixes a bug introduced earlier in the patchset. It's just too hard to go
back and fix it up with the right commit.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off config work on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off dashboard setup on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off ssh preparations on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off mgr creation

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split mon prepare and create

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: move configuration out of the way

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/bootstrap: spin-off waiting for mon

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split some of bootstrap to other functions

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: fix linting/types issues

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split main into an init function

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: introduce context, drop global variables

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

qa/tasks/rbd: increase image size of encrypted disks

This commit increases the rbd image sizes in used by qemu to support the storage requirements of the luks encryption header.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

tools/rbd-nbd: change block size to 4K for LUKS2 images

This commit sets the NBD device block size to 4K for images encrypted using LUKS2 (whose sector size is by default 4K(.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38945 from sebastian-philipp/qa-cephadm-smoke-yaml

qa/cephadm: Add yaml output to smoke test

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38954 from sebastian-philipp/orchestrator-disallow-untyped

mgr/orchestrator: disallow_untyped_defs = True

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38946 from sebastian-philipp/cephadm-pyright-fix-node-exporter

cephadm: Fix node-exporter deployment.

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge pull request #38968 from dillaman/wip-rbd-error

rbd: deprecation warning should be printed to stderr

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38934 from ifed01/wip-ifed-fix-48776

os/bluestore: fix deferred_queue locking

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38959 from badone/wip-valgrind-tracker-48918

osd: initialise m_interval_start

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

If cephadm daemons are not happy we should raise a warning.  Aside from
being an important part of the user experience, this will also help us
catch teuthology test errors.

Fixes: https://tracker.ceph.com/issues/45628
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38944 from orozery/qa-qemu-rbd-encryption

qa/tasks/rbd: test qemu on top of rbd encryption

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: fix host maintenance health check update

We need to update the self.health_checks dict and use that or else we
will clobber any other exising checks.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38628 from ceph/wip-release-html

script/ceph-release-notes: --html flag to output PRs in HTML

:qa/tasks/rbd: test qemu on top of rbd encryption

This commit adds new qemu xfstests workloads that run on top of librbd luks1/luks2 encryption.
This is currently done via nbd, instead of the qemu rbd driver.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

rbd: deprecation warning should be printed to stderr

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephadm: latest stable release is now pacific

Signed-off-by: Sage Weil <sage@newdream.net>

qa/task/rgw: instruct Valgrind to don't exit on first error.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/orchestrator: ignore Liskov substitution principle violation

Right now, we're violating the Liskov substitution principle by deriving from `Orchestrator` but `process` takes a sub class of `Completion`:
See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

The idea is to make Orchestrator a type constructor with `CompletionT` as argument, but this is not supported by mypy: https://github.com/python/typing/issues/548

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: disallow_untyped_defs = True

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

test: cephfs-mirror teuthology task and test yamls

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add tests for mirroring module w/ daemon verification

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: optionally create a backup filesystem on startup

Also filter out client-id's starting with "mirror" when
cleaning leftover auth-ids since teuthology would be
configured to create client.mirror and client.mirror_remote
clients before executing mirroring tests.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mgr/mirroring: interface to mirror CephFS directory snapshots

Signed-off-by: Venky Shankar <vshankar@redhat.com>

pybind/mgr/mgr_util: make RTimer calss reusable

Signed-off-by: Venky Shankar <vshankar@redhat.com>

osd: initialise m_interval_start

Fixes: https://tracker.ceph.com/issues/48918
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37925 from liu-chunmei/seastore_omap_tree

crimson/seastore: add omap tree implementation

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38890 from cbodley/wip-qa-rgw-https-clients

qa/rgw: don't add a certificate for nonexistent rgw.client.1

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #38898 from pkulijiawei/bug-48866

librbd: remove the first if  at api::group::list

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38904 from sebastian-philipp/cephadm-sysctl-silent

cephadm: Don't make sysctl spam the log file

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38910 from batrick/pr38568-fix

cephadm: fix rgw osd cap tag

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38918 from adk3798/ha-rgw-fix

mgr/cephadm: fix ha-rgw removal

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

osd: Remove override for osd_async_recovery_min_cost for mclock profiles

Overriding osd_async_recovery_min_cost as part of enabling a built-in
mclock profile has the undesirable side effect of peers not choosing
the correct async recovery targets if osds are using mixed schedulers
(this could happen during upgrades or if "debug_random" is set for
osd_op_queue config option). Due to the above, osds get into a
"choose_acting" loop during peering.

The solution is to remove the override of osd_async_recovery_min_cost.

Fixes: https://tracker.ceph.com/issues/48906
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

cephadm: Fix node-exporter deployment.

Fixes: 2ce828d5f3682d3eee61e4a4a07a9eedb6a3d04e
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/cephadm: Add yaml output to smoke test

this will provide a more detailed output, like

```yaml
...snip...
service_type: node-exporter
service_name: node-exporter
placement:
  host_pattern: '*'
status:
  created: '2021-01-18T11:21:56.024810Z'
  last_refresh: '2021-01-18T11:23:24.477672Z'
  running: 0
  size: 1
events:
- "2021-01-18T11:23:09.602644Z service:node-exporter [ERROR] \"Failed while placing\
  \ node-exporter.ubuntuon ubuntu: cephadm exited with an error code: 1, stderr:Deploy\
  \ daemon node-exporter.ubuntu ...\nVerifying port 9100 ...\nTraceback (most recent\
  \ call last):\n  File \"<stdin>\", line 7274, in <module>\n  File \"<stdin>\", line\
  \ 1563, in _default_image\n  File \"<stdin>\", line 3698, in command_deploy\n  File\
  \ \"<stdin>\", line 2338, in deploy_daemon\n  File \"<stdin>\", line 1961, in create_daemon_dirs\n\
  AssertionError\""
...snip...
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38507 from sebastian-philipp/mypy-mgr_util

pybind/mgr: disallow_untyped_defs=True for mgr_util

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

rgw/lc: Fix use-after-free in RGWLC::process

Fixed use-after-free issue with 'rgw::sal::LCSerializer lock'
in RGWLC::process.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #38920 from sunnyku/wip-ceph-mclock-op

osd: handle ceph specific config changes for the mclock scheduler

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #38675 from sseshasa/wip-dmclock-config-sets

osd: Add mclock config sets that implement built-in and custom profiles

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #38933 from wjwithagen/wjw-fix-QCOW

librbd: remove inclusion of endian.h

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: Set recovery specific Ceph options for mclock profiles to work.

Set and disable relevant recovery specific Ceph options for mclock
profiles to work as expected. Broadly,
 - Set low value for recovery min cost
 - High values for max active recoveries and max backfills
 - Disable recovery sleep{_hdd, _ssd, _hybrid}

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

osd: Handle configuration changes to mclock config options

Handle configuration changes to the mclock cost per io, the max
capacity options and the mclock profile. Handle the case where the
profile is changed from a built-in profile to the custom profile.

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

osd: Add mclock profile infrastructure and implement mclock profiles

Define config options to specify the cost per io for an osd (hdd & ssd).
 - osd_mclock_cost_per_io_msec
 - osd_mclock_cost_per_io_msec_hdd
 - osd_mclock_cost_per_io_msec_ssd

Define config options to set max osd capacity (hdd & ssd) to be allocated
between clients of dmclock namely,
 - osd_mclock_max_capacity_iops
 - osd_mclock_max_capacity_iops_hdd
 - osd_mclock_max_capacity_iops_ssd

Define config option "osd_mclock_profile" to specify the built-in profile
to enable.

Also, Set the number of op shards being used in the osd within the mclock
scheduler as well. This is necessary to calculate the per shard limits
within the mclock scheduler.

With the above information, enable the specified mclock profile by
calling the appropriate method to set the profile specific mclock
parameters and Ceph options.

Prior to enqueuing an op, the scheduler performs a calculation to scale
up or down the cost associated for the OpSchedulerItem. This calculation
is done based on the existing item cost, the max osd capacity provided
and an additional cost factor based on underlying device type(hdd/ssd).

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

osd: handle ceph specific config changes for the mclock scheduler

The below ceph parameters are set automatically
while enabling the mclock scheduler with a built-in profile.
The user in this case will not be able to modify these
ceph specific config options during runtime.

  a. osd_async_recovery_min_cost
  b. osd_recovery_max_active{_hdd,_ssd}
  c. osd_max_backfills
  d. osd_recovery_sleep{_hdd,_ssd,_hybrid}

If the custom profile is enabled for the mclock scheduler,
the user can modify these parameters.

Signed-off-by: Sunny Kumar <sunkumar@redhat.com>

Merge PR #37297 into master

* refs/pull/37297/head:
qa: add new client tests
test: add client tests
client: wire up alternate_name
mds: fix alternate_name durability
mds: add alternate_name feature support for dentries
mds: add static encode/decode helpers for remote in CDentry
mds: add "fscrypt" flag support for inode_t
mds: add new CDentry tags support
include: cleanup filepath cons
mds: add comment on linkage durability
client: constify some RWRef methods
common: add strescape helper

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>