librbd: load existing cache

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34891 from sebastian-philipp/cephadm-cephadmservice

mgr/cephadm: Add CephadmService class

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34853 from yison/rwl-image-writeback-cache-seq9

rbd/cache: Replicated Write Log core codes - compare_and_write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34952 from cbodley/wip-44958

doc/rgw: document 'rgw gc max concurrent io'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #34929 from tspmelo/wip-e2e-script

mgr/dashboard: Improve run-frontend-e2e-tests.sh

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/rgw: document 'rgw gc max concurrent io'

Fixes: https://tracker.ceph.com/issues/44958
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34820 from sebastian-philipp/akamac-cephadm-fix-typo-in-postinst

debian: fix typo in cephadm user gecos

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Add IscsiService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CrashService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add NodeExporterService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add PrometheusService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add AlertmanagerService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add GrafanaService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add RbdMirrorService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add RgwService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add MdsService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add MgrService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add MonService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add NFSService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move OSD related code to OSDService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move nfs and osd to services/

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Improve run-frontend-e2e-tests.sh

Allow to configure Cypress using env vars.

Fixes: https://tracker.ceph.com/issues/45408
Add Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34855 from s0nea/wip-dashboard-troubleshooting

Reviewed-by: Enno Gotthold <egotthold@suse.de>
Reviewed-by: Volker Theile <vtheile@suse.com>

debian: fix typo in cephadm user gecos

Signed-off-by: Alexey Miasoedov <alexey.miasoedov@gmail.com>

Merge pull request #34591 from votdev/issue_45109

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

librbd: add aio_compare_and_write test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_compare_and_write

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add CompAndWriteRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34921 from tchaikov/wip-seastar

crimson: pick up a recent seastar version and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34851 from yison/rwl-image-writeback-cache-seq8

rbd/cache: Replicated Write Log core codes - retire entries and invalidate

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson: avoid using variadic future

it is deprecated by seastar. let's use future<tuple<...>> instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up a recent seastar

to alleviate the pain of keeping in sync with seastar upstream.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: friend with do_with_impl()

seastar changed the implementatino of do_with(), so it's do_with_impl()
who tries to up-cast erroratorized future to seastar::future<>.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: drop Dispatch::ms_verify_authorizer()

as it's not used at all. also, variadic future was deprecated by
seastar, so let's avoid using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: s/reactor::cpu_id/seastar::this_shard_id/

the former was deprecated since seastar
02956fd436b413d6bbe562cf91533ada36553e54

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: add invalidate test case

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add invalidate

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: retire entries

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34899 from smanjara/wip-dynamic-resharding

qa/rgw: add ssl option in reshard tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr/dashboard: Add troubleshooting guide

Fixes: https://tracker.ceph.com/issues/25196
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34887 from sebastian-philipp/cephadm-move-specstore

mgr/cephadm: move SpecStore and HostCache to inventoy.py

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34898 from matthewoliver/cephadm-iscsi-priv

cephadm: give ceph-iscsi permissions to configfs

Reviewed-by: Boris Ranto <branto@redhat.com>

crimson/common: make do_with() noexcept

to be consistent with seasetar change of
85aa178d9539b2c9301d4fb0815596a741fb82c4, which make
`seasetar::do_with()` noexcept

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: use invoke instead of apply

to be consistent with 9b69c7989da57e69ed1bfc7aa122876e67c3d1fd
in seasetar, in which, seastar::apply() is deprecated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34243 from trociny/wip-rbd-freeze-thaw

librbd: API for quiesce callbacks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34706 from yison/rwl-image-writeback-cache-seq6

rbd/cache: Replicated Write Log core codes - aio_writesame

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34868 from tspmelo/wip-fix-e2e-role

mgr/dashboard: E2E: Fix role-mgmt.po.ts

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34907 from tspmelo/wip-health

mgr/dashboard: Remove unnecessary call in health.component.ts

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

common/buffer_seastar: include used header

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: add aio_writesame test case

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_writesame

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add WriteSameRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add WriteSameLogOperation

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add WriteSameLogEntry

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34086 from ivancich/wip-deprecate-orphans-find

rgw: deprecate radosgw-admin orphans sub-commands

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #33546 from matthewoliver/rgw_swift_404_anon

rgw: anonomous swift to obj that dont exist should 401

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34896 from cbodley/wip-rgw-multisite-py3

test/rgw: py3 fixes for multisite tests

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: deprecate radosgw-admin orphans sub-commands

The orphans sub-commands have not been maintained for years and it's
unclear whether they're reliable. Additionally, they store logs on the
cluster, so if a user wants to run it to address a nearly full
cluster, it could push the user over the edge. As a result the
(currently experimental) `rgw-orphan-list` tool (separate executable
rather than a sub-command) has been created, which stores intermediate
results on the local filesystem.

Now all three orphans sub-commands produce a warning (on cerr) that
the command is being deprecated. Additionally all three now require
the --yes-i-really-mean-it command-line option to run.

This also documents the "bucket radoslist" sub-command to
`radosgw-admin` in its corresponding man page, a change that was
omitted from the prior commit that added "bucket radoslist"
documentation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #34911 into master

* refs/pull/34911/head:
qa: move fsx deps into common cephfs install yaml

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34287 from ofriedma/wip-ofriedma-anon

rgw: fix rgw tries to fetch anonymous user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: move fsx deps into common cephfs install yaml

The double install task causes errors during unwind.

Fixes: fe3961fc36e4d0af896fe140ba2450e1556ab563
Fixes: https://tracker.ceph.com/issues/45387
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

vstart: remove commented-out vault config options

we don't have any vstart automation for rgw encryption with vault. if
anyone wants to do local testing, they can follow instructions in
the documentation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: Cluster.admin() returns output as utf-8 str

fixes the TypeError:

  File "rgw_multi/tests.py", line 1107, in test_bucket_sync_disable
    check_buckets_sync_status_obj_not_exist(zone, buckets)
  File "rgw_multi/tests.py", line 438, in check_buckets_sync_status_obj_not_exist
    if log_list.find(':'+bucket+":") >= 0:
TypeError: argument should be integer or bytes-like object, not 'str'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: make "snapshot create" notification be "async"

After adding quiesce/unquiesce callbacks the "snapshot create" request
may run long time.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: API for quiesce callbacks

The callbacks are fired by librbd before/after creating a
snapshot. The callback users like QEMU could attempt to freeze
the FS before allowing librbd to actually perform the snapshot.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: don't use boost::variant for image watcher messages

We are reached 20 types limit and are not able to add new message
types.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34894 from yuriw/wip-yuriw-distro-master

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

mgr/dashboard: Remove unnecessary call in health.component.ts

Fixes: https://tracker.ceph.com/issues/45386
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34856 from jschmid1/cephadm_osdspec_affinity

mgr/cephadm: add support for proper osd daemon tracking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: add ssl option in reshard tests

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #34773 from ricardoasmarques/cephadm-shell-mount

cephadm: Add shell '--mount' option to mount host file or directory

Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34703 from SUSE/wip-fix-45129

cephadm: Add `chown` to unit.run for adoped simple OSDs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34778 from mgfritch/cephadm-dd-service-id

mgr/orch: service_id can contain a '.' char (mds, nfs, iscsi)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34814 from mgfritch/cephadm-iscsi-apply

mgr/orch: fix `orch apply iscsi`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34658 from matthewoliver/cephadm_iscsi_ssl

cephadm: Make ceph-iscsi SSL aware

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34745 from cybozu/cephadm-fix-the-hang-up-on-creating-encrypted-osd

cephadm: fix the hang-up of cryptsetup on creating encrypted OSD

Reviewed-by: Sage Weil <sage@redhat.com>

qa: mgr/dashboard: Replace Telemetry module in REST API test

Fixes: https://tracker.ceph.com/issues/45109
Signed-off-by: Volker Theile <vtheile@suse.com>

cephadm: give ceph-iscsi permissions to configfs

The cephadm container mounts the configfs and then bind mounts (-v) it
into the container. Currently the container is not a priviliaged
container which leads to 2 problems:

 1. The container can't insert the iscsi_target_mod kernel module; and
 2. The container can't write to the configfs as that's only writeable
    by root.

We _can_ get around 1, by preloading the kernel module. I.E add it the
systemd unit file. But that doesn't help with 2.

I've tried mounting the configfs with uid and gid options, but configfs
doesn't use them.

If we make the container a priviliged container then magically both 1
and 2 are solved. We don't need to preload the module so that's one less
workaround. But more importantly, configfs can be written to so we can
create ISCSI targets etc.

So that's what this patch does, it makes iscsi containers privileged
containers by setting the CephContainer --priviliged while creating it.

Fixes: https://tracker.ceph.com/issues/45252
Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #34660 from majianpeng/bluestore-pmemdevice

os/bluestore: avoid offset overflow.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34837 from sseshasa/wip-mon-memory-target-fix

mon/OSDMonitor: Always tune priority cache manager memory on all mons

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34788 from yuvalif/cls_q_has_empty_markers

cls/queue: fix empty markers when listing entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/orch: match last occurance of hostname in service_id

The RGW realm/zone might have a substr that matches the hostname

Fixes: https://tracker.ceph.com/issues/45294
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use keyword instead of positional args

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: get service_name from DaemonDescription

use the shared logic in the DaemonDescription class

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: allow '.' chars in the service_id

service_id can contain a '.' char (mds, nfs, iscsi)

Fixes: https://tracker.ceph.com/issues/45293
Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

... so all tests for `pacific` will be tested on the latest ubuntu version

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

test/rgw: SystemObject.json_command no longer trims json output

this was a workaround for 'period commit' commands that used to write
the "Sending period to new master zone" line to stdout

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: period commit writes messages to stderr

the period's json output is written to stdout. any other messages should
be written to stderr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34148 from ivancich/octopus-rados-list

rgw: add `rgw-orphan-list` tool & `radosgw-admin bucket radoslist ...`

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr/orchestrator: update iscsi

`apply iscsi` and `add iscsi` are implemented in cephadm

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: add iscsi apply cli

`$ ceph orch apply iscsi --pool <pool>`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: _apply_service: fix _create_iscsi invocation

Fixes: https://tracker.ceph.com/issues/45249
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: integration test for `rgw-orphan-list` & `radosgw-admin radoslist`

Add teuthology test for `rgw-orphan-list` in a new tool suite under
rgw. It only needs to be tested under one configuration. And the new
tool sub-suite can be used by other tooling int he
future. radosgw-admin `radoslist` is tested indirectly through
`rgw-orphan-list` and therefore does not need its own test.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

qa/rgw: allow the rgw teuthology task to capture/set dns names

A teuthology workunit might want to use the rgw task, setting the
rgw-dns-name and/or rgw-dns-s3website-name configuration options to
the fully-qualified domain name. Existing code implies that setting
these configuration options to the empty string will do that. However
the current logic does not support that given it has Python
conditionals that treat the empty string as false. This fixes that.

Now the following teuthology tasks YAML will work as expected:

  tasks:
  - rgw:
      client.0:
        dns-name: ''
        dns-s3website-name: ''

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

test/rgw: add static testing for `ceph-diff-sorted`

Tests are run during "make check".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/rgw: add docs for `rgw-orphan-list` and `ceph-diff-sorted`

Add man pages and documentation for both tools.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add `rgw-orphan-list` tool & `radosgw-admin bucket radoslist ...`

Adds a `radosgw-admin` subcommand and walks the associated bucket
indices and manifests to generate the list of rados objects that
represent the rgw objects in the bucket(s).

Also adds a tool named `rgw-orphan-list`, which uses the radoslist
subcommand, that produces a list in a local file of what appear to be
rgw orphans.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add const-correctness to rgw_obj_manifest and svc_tier_rados

Mark functions as "const" or function parameters as "const" in order
to make those functions more widely callable.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #34172 from cbodley/wip-44751

radosgw-admin: fix infinite loops in 'datalog list'

Merge pull request #34286 from cbodley/wip-qa-rgw-cls-2pc-queue

qa/rgw: run ceph_test_cls_2pc_queue in verify suite

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>