cls/rgw_gc: Fixing the iterator used to access urgent data map
in xattr in cls_rgw_gc. The incorrect iterator caused a crash
in osd.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #34950 from sebastian-philipp/cephadm-upgrade.py

mgr/cephadm: move upgrade code to upgrade.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34922 from SUSE/wip-fix-45393

mgr/cephadm: refresh osd config when mons change

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34903 from sebastian-philipp/cephadm-adopt-qa-grafana

qa/workunits/cephadm: test grafana adoption

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34949 from tspmelo/wip-i18n-fix

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34923 from ofriedma/wip-ofriedma-fcgi

rgw: unable to compile fcgi frontend

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #34709 from votdev/issue_45204_max_buckets

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34854 from BenoitKnecht/fix-alert-description

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #34843 from mgfritch/cephadm-cli-servicespec

mgr/orch: Clean-up ServiceSpec handling during add/remove (mds, rgw, iscsi, nfs)

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34909 from sebastian-philipp/cephadm-doc-ceph.conf

doc/cephadm: add section about setting config settings

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34891 from sebastian-philipp/cephadm-cephadmservice

mgr/cephadm: Add CephadmService class

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34853 from yison/rwl-image-writeback-cache-seq9

rbd/cache: Replicated Write Log core codes - compare_and_write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34952 from cbodley/wip-44958

doc/rgw: document 'rgw gc max concurrent io'

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #34929 from tspmelo/wip-e2e-script

mgr/dashboard: Improve run-frontend-e2e-tests.sh

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc/rgw: document 'rgw gc max concurrent io'

Fixes: https://tracker.ceph.com/issues/44958
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: move upgrade code to upgrade.py

Make it easier for everyone to read the upgrade algorithm.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix I18N errors

We were just checking if the translation sources were correct and ignore if the
extraction simply failed.

Fixes: https://tracker.ceph.com/issues/45428
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34820 from sebastian-philipp/akamac-cephadm-fix-typo-in-postinst

debian: fix typo in cephadm user gecos

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Add IscsiService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add CrashService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add NodeExporterService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add PrometheusService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add AlertmanagerService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add GrafanaService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add RbdMirrorService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add RgwService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add MdsService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add MgrService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add MonService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Add NFSService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move OSD related code to OSDService

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move nfs and osd to services/

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Improve run-frontend-e2e-tests.sh

Allow to configure Cypress using env vars.

Fixes: https://tracker.ceph.com/issues/45408
Add Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34855 from s0nea/wip-dashboard-troubleshooting

Reviewed-by: Enno Gotthold <egotthold@suse.de>
Reviewed-by: Volker Theile <vtheile@suse.com>

debian: fix typo in cephadm user gecos

Signed-off-by: Alexey Miasoedov <alexey.miasoedov@gmail.com>

Merge pull request #34591 from votdev/issue_45109

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

librbd: add aio_compare_and_write test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_compare_and_write

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add CompAndWriteRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34921 from tchaikov/wip-seastar

crimson: pick up a recent seastar version and cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34851 from yison/rwl-image-writeback-cache-seq8

rbd/cache: Replicated Write Log core codes - retire entries and invalidate

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

monitoring: Fix "10% OSDs down" alert description

The alert was triggered when less than 90% of OSDs were _up_, but then the
description took that value and described it as the percentage of OSDs being
_down_. So with 12% of OSDs down, the alert description would read:

```
88% or 88 of 100 OSDs are down (>=10%).
```

which can be panic-inducing.

This commit changes the alert expression to actually compute the ratio of OSDs
being down, which makes the correct value appear in the description.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

crimson: avoid using variadic future

it is deprecated by seastar. let's use future<tuple<...>> instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pick up a recent seastar

to alleviate the pain of keeping in sync with seastar upstream.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: friend with do_with_impl()

seastar changed the implementatino of do_with(), so it's do_with_impl()
who tries to up-cast erroratorized future to seastar::future<>.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: drop Dispatch::ms_verify_authorizer()

as it's not used at all. also, variadic future was deprecated by
seastar, so let's avoid using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: s/reactor::cpu_id/seastar::this_shard_id/

the former was deprecated since seastar
02956fd436b413d6bbe562cf91533ada36553e54

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: add section about setting config settings

Fixes: https://tracker.ceph.com/issues/44284
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: add invalidate test case

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add invalidate

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: retire entries

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34899 from smanjara/wip-dynamic-resharding

qa/rgw: add ssl option in reshard tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr/dashboard: Add troubleshooting guide

Fixes: https://tracker.ceph.com/issues/25196
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34887 from sebastian-philipp/cephadm-move-specstore

mgr/cephadm: move SpecStore and HostCache to inventoy.py

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34898 from matthewoliver/cephadm-iscsi-priv

cephadm: give ceph-iscsi permissions to configfs

Reviewed-by: Boris Ranto <branto@redhat.com>

crimson/common: make do_with() noexcept

to be consistent with seasetar change of
85aa178d9539b2c9301d4fb0815596a741fb82c4, which make
`seasetar::do_with()` noexcept

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: use invoke instead of apply

to be consistent with 9b69c7989da57e69ed1bfc7aa122876e67c3d1fd
in seasetar, in which, seastar::apply() is deprecated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34243 from trociny/wip-rbd-freeze-thaw

librbd: API for quiesce callbacks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34706 from yison/rwl-image-writeback-cache-seq6

rbd/cache: Replicated Write Log core codes - aio_writesame

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34868 from tspmelo/wip-fix-e2e-role

mgr/dashboard: E2E: Fix role-mgmt.po.ts

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34907 from tspmelo/wip-health

mgr/dashboard: Remove unnecessary call in health.component.ts

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

rgw: unable to compile fcgi frontend

Although we are not supporting this frontend it is still possible with cmake to compile it

This fix is intend to be able to compile fcgi

Signed-off-by: Or Friedmann <ofriedma@redhat.com>

mgr/cephadm: refresh osd config when mons change

OSDs don't have a service spec, so the previous implementation
was not reconfiguring them when the list of mons changed.

Fixes: https://tracker.ceph.com/issues/45393
Signed-off-by: Tim Serong <tserong@suse.com>

common/buffer_seastar: include used header

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: add aio_writesame test case

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_writesame

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add WriteSameRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add WriteSameLogOperation

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add WriteSameLogEntry

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34086 from ivancich/wip-deprecate-orphans-find

rgw: deprecate radosgw-admin orphans sub-commands

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #33546 from matthewoliver/rgw_swift_404_anon

rgw: anonomous swift to obj that dont exist should 401

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34896 from cbodley/wip-rgw-multisite-py3

test/rgw: py3 fixes for multisite tests

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/orch: fix mypy errors

orchestrator/module.py: note: In member "_daemon_add_misc" of class "OrchestratorCli":
orchestrator/module.py:655: error: Incompatible types in assignment (expression has type "PlacementSpec", variable has type "Optional[str]")
orchestrator/module.py:656: error: Argument 1 to "ServiceSpec" has incompatible type "Optional[str]"; expected "str"
orchestrator/module.py:656: error: Argument "placement" to "ServiceSpec" has incompatible type "Optional[str]"; expected "Optional[PlacementSpec]"

orchestrator/module.py: note: In member "_apply_misc" of class "OrchestratorCli":
orchestrator/module.py:881: error: Incompatible types in assignment (expression has type "PlacementSpec", variable has type "Optional[str]")
orchestrator/module.py:882: error: Argument 1 to "ServiceSpec" has incompatible type "Optional[str]"; expected "str"
orchestrator/module.py:882: error: Argument "placement" to "ServiceSpec" has incompatible type "Optional[str]"; expected "Optional[PlacementSpec

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: add missing `subcluster`, `port`, and `ssl`

subcluster, port, and ssl arguments are missing for `orch apply rgw`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: add type checking for add/apply commands

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: raise validation error for unexpected inbuf

ceph orch apply mds x -i nfs.yaml
Error ENOENT: unrecognized command -i; -h or --help for usage

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: remove inbuf logic during daemon add (rgw/iscsi)

Simplifies logic for handling the ServiceSpec

Allows for a ServiceSpec:
$ ceph orch daemon add -i iscsi.yaml

But not when combined with a service_type:
$ ceph orch daemon add iscsi <poolname> -i iscsi.yaml

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: remove extraneous PlacementSpec.validate()

PlacementSpec is being validated during `__init__()`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: add mds, rgw, nfs daemon types

$ ceph orch add -i nfs.yaml
Error ENOENT: unknown daemon type `nfs`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: use `service_type` as the `daemon_type`

$ ceph orch add -i mon.yaml
Error ENOENT: unknown daemon type `None`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: pass `inbuf` to add/apply (mds, rgw, nfs)

$ ceph orch apply mds a -i mds.yaml
TypeError: _apply_mds() got an unexpected keyword argument 'inbuf'

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: add misssing `raise_if_exception`

pickle exceptions during during `nfs apply`

Signed-off-by: Michael Fritch <mfritch@suse.com>

rgw: deprecate radosgw-admin orphans sub-commands

The orphans sub-commands have not been maintained for years and it's
unclear whether they're reliable. Additionally, they store logs on the
cluster, so if a user wants to run it to address a nearly full
cluster, it could push the user over the edge. As a result the
(currently experimental) `rgw-orphan-list` tool (separate executable
rather than a sub-command) has been created, which stores intermediate
results on the local filesystem.

Now all three orphans sub-commands produce a warning (on cerr) that
the command is being deprecated. Additionally all three now require
the --yes-i-really-mean-it command-line option to run.

This also documents the "bucket radoslist" sub-command to
`radosgw-admin` in its corresponding man page, a change that was
omitted from the prior commit that added "bucket radoslist"
documentation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #34911 into master

* refs/pull/34911/head:
qa: move fsx deps into common cephfs install yaml

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34287 from ofriedma/wip-ofriedma-anon

rgw: fix rgw tries to fetch anonymous user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: move fsx deps into common cephfs install yaml

The double install task causes errors during unwind.

Fixes: fe3961fc36e4d0af896fe140ba2450e1556ab563
Fixes: https://tracker.ceph.com/issues/45387
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

vstart: remove commented-out vault config options

we don't have any vstart automation for rgw encryption with vault. if
anyone wants to do local testing, they can follow instructions in
the documentation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: Cluster.admin() returns output as utf-8 str

fixes the TypeError:

  File "rgw_multi/tests.py", line 1107, in test_bucket_sync_disable
    check_buckets_sync_status_obj_not_exist(zone, buckets)
  File "rgw_multi/tests.py", line 438, in check_buckets_sync_status_obj_not_exist
    if log_list.find(':'+bucket+":") >= 0:
TypeError: argument should be integer or bytes-like object, not 'str'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: make "snapshot create" notification be "async"

After adding quiesce/unquiesce callbacks the "snapshot create" request
may run long time.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: API for quiesce callbacks

The callbacks are fired by librbd before/after creating a
snapshot. The callback users like QEMU could attempt to freeze
the FS before allowing librbd to actually perform the snapshot.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: don't use boost::variant for image watcher messages

We are reached 20 types limit and are not able to add new message
types.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34894 from yuriw/wip-yuriw-distro-master

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

mgr/dashboard: Remove unnecessary call in health.component.ts

Fixes: https://tracker.ceph.com/issues/45386
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34856 from jschmid1/cephadm_osdspec_affinity

mgr/cephadm: add support for proper osd daemon tracking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/cephadm: test grafana adoption

now includes https://github.com/ceph/cephadm-adoption-corpus/commit/af85d49307e1652cd0007bf3074030bf0a63c9b7

Fixes: https://github.com/ceph/cephadm-adoption-corpus/pull/3
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: add ssl option in reshard tests

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #34773 from ricardoasmarques/cephadm-shell-mount

cephadm: Add shell '--mount' option to mount host file or directory

Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34703 from SUSE/wip-fix-45129

cephadm: Add `chown` to unit.run for adoped simple OSDs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34778 from mgfritch/cephadm-dd-service-id

mgr/orch: service_id can contain a '.' char (mds, nfs, iscsi)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34814 from mgfritch/cephadm-iscsi-apply

mgr/orch: fix `orch apply iscsi`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>