crimson: s/reactor::cpu_id/seastar::this_shard_id/

the former was deprecated since seastar
02956fd436b413d6bbe562cf91533ada36553e54

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: make do_with() noexcept

to be consistent with seasetar change of
85aa178d9539b2c9301d4fb0815596a741fb82c4, which make
`seasetar::do_with()` noexcept

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: use invoke instead of apply

to be consistent with 9b69c7989da57e69ed1bfc7aa122876e67c3d1fd
in seasetar, in which, seastar::apply() is deprecated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/buffer_seastar: include used header

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34086 from ivancich/wip-deprecate-orphans-find

rgw: deprecate radosgw-admin orphans sub-commands

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #33546 from matthewoliver/rgw_swift_404_anon

rgw: anonomous swift to obj that dont exist should 401

Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34896 from cbodley/wip-rgw-multisite-py3

test/rgw: py3 fixes for multisite tests

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: deprecate radosgw-admin orphans sub-commands

The orphans sub-commands have not been maintained for years and it's
unclear whether they're reliable. Additionally, they store logs on the
cluster, so if a user wants to run it to address a nearly full
cluster, it could push the user over the edge. As a result the
(currently experimental) `rgw-orphan-list` tool (separate executable
rather than a sub-command) has been created, which stores intermediate
results on the local filesystem.

Now all three orphans sub-commands produce a warning (on cerr) that
the command is being deprecated. Additionally all three now require
the --yes-i-really-mean-it command-line option to run.

This also documents the "bucket radoslist" sub-command to
`radosgw-admin` in its corresponding man page, a change that was
omitted from the prior commit that added "bucket radoslist"
documentation.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #34911 into master

* refs/pull/34911/head:
qa: move fsx deps into common cephfs install yaml

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34287 from ofriedma/wip-ofriedma-anon

rgw: fix rgw tries to fetch anonymous user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: move fsx deps into common cephfs install yaml

The double install task causes errors during unwind.

Fixes: fe3961fc36e4d0af896fe140ba2450e1556ab563
Fixes: https://tracker.ceph.com/issues/45387
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

vstart: remove commented-out vault config options

we don't have any vstart automation for rgw encryption with vault. if
anyone wants to do local testing, they can follow instructions in
the documentation

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: Cluster.admin() returns output as utf-8 str

fixes the TypeError:

  File "rgw_multi/tests.py", line 1107, in test_bucket_sync_disable
    check_buckets_sync_status_obj_not_exist(zone, buckets)
  File "rgw_multi/tests.py", line 438, in check_buckets_sync_status_obj_not_exist
    if log_list.find(':'+bucket+":") >= 0:
TypeError: argument should be integer or bytes-like object, not 'str'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34894 from yuriw/wip-yuriw-distro-master

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

Merge pull request #34856 from jschmid1/cephadm_osdspec_affinity

mgr/cephadm: add support for proper osd daemon tracking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34773 from ricardoasmarques/cephadm-shell-mount

cephadm: Add shell '--mount' option to mount host file or directory

Reviewed-by: Matthew Oliver <moliver@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34703 from SUSE/wip-fix-45129

cephadm: Add `chown` to unit.run for adoped simple OSDs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34778 from mgfritch/cephadm-dd-service-id

mgr/orch: service_id can contain a '.' char (mds, nfs, iscsi)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34814 from mgfritch/cephadm-iscsi-apply

mgr/orch: fix `orch apply iscsi`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34658 from matthewoliver/cephadm_iscsi_ssl

cephadm: Make ceph-iscsi SSL aware

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34745 from cybozu/cephadm-fix-the-hang-up-on-creating-encrypted-osd

cephadm: fix the hang-up of cryptsetup on creating encrypted OSD

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34660 from majianpeng/bluestore-pmemdevice

os/bluestore: avoid offset overflow.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34837 from sseshasa/wip-mon-memory-target-fix

mon/OSDMonitor: Always tune priority cache manager memory on all mons

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34788 from yuvalif/cls_q_has_empty_markers

cls/queue: fix empty markers when listing entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/orch: match last occurance of hostname in service_id

The RGW realm/zone might have a substr that matches the hostname

Fixes: https://tracker.ceph.com/issues/45294
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: use keyword instead of positional args

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: get service_name from DaemonDescription

use the shared logic in the DaemonDescription class

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: allow '.' chars in the service_id

service_id can contain a '.' char (mds, nfs, iscsi)

Fixes: https://tracker.ceph.com/issues/45293
Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/tests: replaced ubuntu_latest.yaml with ubuntu 20.04

... so all tests for `pacific` will be tested on the latest ubuntu version

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

test/rgw: SystemObject.json_command no longer trims json output

this was a workaround for 'period commit' commands that used to write
the "Sending period to new master zone" line to stdout

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: period commit writes messages to stderr

the period's json output is written to stdout. any other messages should
be written to stderr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34148 from ivancich/octopus-rados-list

rgw: add `rgw-orphan-list` tool & `radosgw-admin bucket radoslist ...`

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr/orchestrator: update iscsi

`apply iscsi` and `add iscsi` are implemented in cephadm

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/orch: add iscsi apply cli

`$ ceph orch apply iscsi --pool <pool>`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: _apply_service: fix _create_iscsi invocation

Fixes: https://tracker.ceph.com/issues/45249
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: integration test for `rgw-orphan-list` & `radosgw-admin radoslist`

Add teuthology test for `rgw-orphan-list` in a new tool suite under
rgw. It only needs to be tested under one configuration. And the new
tool sub-suite can be used by other tooling int he
future. radosgw-admin `radoslist` is tested indirectly through
`rgw-orphan-list` and therefore does not need its own test.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

qa/rgw: allow the rgw teuthology task to capture/set dns names

A teuthology workunit might want to use the rgw task, setting the
rgw-dns-name and/or rgw-dns-s3website-name configuration options to
the fully-qualified domain name. Existing code implies that setting
these configuration options to the empty string will do that. However
the current logic does not support that given it has Python
conditionals that treat the empty string as false. This fixes that.

Now the following teuthology tasks YAML will work as expected:

  tasks:
  - rgw:
      client.0:
        dns-name: ''
        dns-s3website-name: ''

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

test/rgw: add static testing for `ceph-diff-sorted`

Tests are run during "make check".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/rgw: add docs for `rgw-orphan-list` and `ceph-diff-sorted`

Add man pages and documentation for both tools.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add `rgw-orphan-list` tool & `radosgw-admin bucket radoslist ...`

Adds a `radosgw-admin` subcommand and walks the associated bucket
indices and manifests to generate the list of rados objects that
represent the rgw objects in the bucket(s).

Also adds a tool named `rgw-orphan-list`, which uses the radoslist
subcommand, that produces a list in a local file of what appear to be
rgw orphans.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add const-correctness to rgw_obj_manifest and svc_tier_rados

Mark functions as "const" or function parameters as "const" in order
to make those functions more widely callable.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #34172 from cbodley/wip-44751

radosgw-admin: fix infinite loops in 'datalog list'

Merge pull request #34286 from cbodley/wip-qa-rgw-cls-2pc-queue

qa/rgw: run ceph_test_cls_2pc_queue in verify suite

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #29062 from zhangsw/rgw-reshard-stats

rgw: add check for index entry's existing when adding bucket stats during bucket reshard.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #34892 from ivancich/wip-fix-tests_ps

qa/rgw: fix issue error in tests_ps.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: fix issue error in tests_ps.py

The Python interpreter on teuthology complains about the lack of
parens in a print statement.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

cephadm: Add '--mount' option to mount host file or directory

Fixes: https://tracker.ceph.com/issues/45284
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #34088 into master

* refs/pull/34088/head:
doc: mailmap: changes for octopus release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #34436 from jschmid1/dg_affinity

ceph-volume: add ceph.osdspec_affinity tag

Merge pull request #34867 from tspmelo/wip-fix-table-filter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

python-common: do not pass affinity to ceph_volume when None

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: add support for proper osd daemon tracking

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34779 from mgfritch/cephadm-workunit-node-exporter-avail

qa/workunits/cephadm/test_cephadm.sh: wait longer for monitoring stack

Merge pull request #34728 from ricardoasmarques/infer-config

cephadm: Infer config on "cephadm shell"

Merge pull request #34723 from matthewoliver/cephadm_iscsi_remove_fqdn_enabled

cephadm: remove fqdn_enabled from ceph-iscsi

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34448 from Devp00l/wip-44621

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33991 from SchoolGuy/monitoring/rbd-image-details

mgr/dashboard/grafana: Add rbd-image details dashboard

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34874 from tchaikov/wip-crimson-thread-pool-test

test/crimson: minimize the lexical scope of thread pool

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #34880 from neha-ojha/wip-balancer-cleanup

qa/suites/rados/thrash*: remove upmap.yaml and off.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34536 from varshar16/wip-cephfs-shell-flake8-py3

cephfs-shell: Change tox testenv name to py3

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash*: add on.yaml

To test with balancer in upmap mode, which is the default.

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash*: remove upmap.yaml and off.yaml

The balancer is always on, so we don't need upmap.yaml.
off.yaml doesn't make sense anymore.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34765 from changchengx/doc

doc/rbd: explicit reminder for libvirt directory access permissions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34877 from vumrao/wip-vumrao-45355

rgw/rgw_bucket: clear tenant string for non tenanted buckets in rgw_bucket_parse_bucket_key function.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/rgw_bucket: clear tenant string for non tenanted buckets
in rgw_bucket_parse_bucket_key function.

Fixes: https://tracker.ceph.com/issues/45355
Signed-off-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #34787 from trociny/wip-45305

qa/workunits/rbd: fix list-mapped filter in unmap_device

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: explicit remind the directory access right

It needs to create admin socket to use perf counter for
debug purpose. Remind the access right to directory.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #34756 from yison/rwl-image-writeback-cache-seq7

rbd/cache: Replicated Write Log core codes - aio_discard

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34575 from hillpd/suicide_grace_fix

rados: prevent ShardedOpWQ suicide_grace drop when waiting for work.

Merge pull request #34569 from ifed01/wip-ifed-more-bluefs-stats

os/bluestore: improve bluefs stats reporting

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34376 from yuvalif/fix_amqp_routable_option

rgw/amqp: fix the "routable" delivery mode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/crimson: minimize the lexical scope of thread pool

to show the typical use case of thread pool in a better manner: thread
pool can be initialized *in* a seastar application.

also move `handle_exception()` out to catch all exceptions throwns in
the seastar app.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34657 from lxbsz/py3_com

qa/tasks/vstart_runner.py: to be python3 compatible

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34824 from sebastian-philipp/python-common-fix-device-selection-type-annotation

python-common: Fix wrong type annotation

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #34846 from tchaikov/wip-45147

mgr: do not load disabled modules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #34786 from aclamk/wip-common-pretty-binary

common: Add pretty_binary_string conversions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34772 from aclamk/wip-bs-faster-fsck-no-cache

kv/KeyValueDB, os/bluestore: Made fsck faster by not caching results

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34754 from ifed01/wip-ifed-fix-45195

os/bluestore: fix improper blob usage while handling deferred big write.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34738 from sebastian-philipp/insights-rm-tox

pybind/mgr: Remove insights/tox.ini

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34690 from sebastian-philipp/cephadm-crush-keyring-erro

mgr/cephadm: raise, if crash keyring fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34632 from kotreshhr/bash-completion

bash_completion: Do not auto complete obsolete and hidden cmds

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34503 from ifed01/wip-ifed-fix-huge-bluefs-reads

os/bluestore: fix huge (>4GB) bluefs reads

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34363 from majianpeng/osd-thread-per-shard-ssd

osd: wakeup all threads of shard rather than one thread.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: Fix error when filtering table without data

Fixes: https://tracker.ceph.com/issues/45351
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34641 from yuvalif/fix_zer_size_in_notifications

rgw/notifications: fix zero size in notifications

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34352 from smanjara/wip-dynamic-resharding

rgw: more multisite dynamic resharding pieces.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34743 from sebastian-philipp/doc-fix-getting-started

doc: fix "getting started" link

mgr: do not load disabled modules

an option named "mgr_disabled_modules" is added in this change to
prevent mgr from loading modules listed in this option. because mgr
loads *all* modules found in the configured path, and per
https://tracker.ceph.com/issues/45147, python subinterpreter could hang
when loading numpy, so this behavior practically creates a deadlock
in mgr.

this issue is found when mgr uses python3.8 runtime. in development
environment, it'd be inconvenient to disable the offending mgr module
without changing the source code, even if we can choose to not install
them, for instance, the enduser can workaround this issue by
uninstalling `ceph-mgr-diskprediction-local`.

an option would be useful in this case, so we can add the module to the
list before mgr tries to load it.

as this issue is found with python3.8 + diskprediction_local (numpy), so
this mgr module is disabled by default if mgr is compiled with python3.8
runtime.

Fixes: https://tracker.ceph.com/issues/45147
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/PyModuleRegistry: probe modules using std::filesystem

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: add osdspec_affinity flag to ceph-osd

Signed-off-by: Joshua Schmid <jschmid@suse.de>

ceph-volume: add ceph.osdspec_affinity tag

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34594 from tchaikov/wip-more-tolerant-perf-test

test/crimson/cbt: increase stddev variance to 2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #34603 from liu-chunmei/crimson_msgr_local_conf

crimson/net: remove hardcoded msgr configuration

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34707 from jschmid1/fix_orch_docs

doc/orchestrator: adapt to new syntax, add more examples

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

minor fix: run tests in python venv

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

cephadm: Make ceph-iscsi SSL aware

Ceph-iscsi's `rbd-target-api.py` supports listening over SSL if you
provide an SSL cert and key. Originally the script is opinionated and
requires these files to be named `/etc/ceph/iscsi-gateway.{crt,key}`.

When dealing with containers, having to place files inside a container to
enable SSL isn't very clean. To make things easier, like RGW, you can
now place the SSL cert and key data in the mon config-key store.

This will mean there are 2 ways to enable SSL in ceph-iscsi via orch/cephadm.

1. Push the SSL key and cert into the mon config-key under the keys, and
   then make sure api_secure is enabled (requires json):

  iscsi/{clientname}/iscsi-gateway.crt
  iscsi/{clientname}/iscsi-gateway.key

2. Provide the SSL key and cert in the json you pass the orchestrator and
   it'll push them up for you.

Also lockdown the caps so the container can only access iscsi ssl
key/certs.

Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #34616 from ifed01/wip-ifed-fix-upgrade-super

os/bluestore: invoke _prepare_ondisk_format_super as the last op

Reviewed-by: Greg Farnum <gfarnum@redhat

cephadm: Infer config on "cephadm shell"

Fixes: https://tracker.ceph.com/issues/44792
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #33462 from s0nea/wip-dashboard-telemetry-report

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add telemetry report component

Add a telemetry component in order to give the user the
possibility to configure the telemetry module in a more
guided fashion. The component offers broader explanations,
shows a preview of the generated report and asks the user
to accept the license before enabling the module.

Fixes: https://tracker.ceph.com/issues/43956
Signed-off-by: Tatjana Dehler <tdehler@suse.com>