cephadm: Make ceph-iscsi SSL aware

Ceph-iscsi's `rbd-target-api.py` supports listening over SSL if you
provide an SSL cert and key. Originally the script is opinionated and
requires these files to be named `/etc/ceph/iscsi-gateway.{crt,key}`.

When dealing with containers, having to place files inside a container to
enable SSL isn't very clean. To make things easier, like RGW, you can
now place the SSL cert and key data in the mon config-key store.

This will mean there are 2 ways to enable SSL in ceph-iscsi via orch/cephadm.

1. Push the SSL key and cert into the mon config-key under the keys, and
   then make sure api_secure is enabled (requires json):

  iscsi/{clientname}/iscsi-gateway.crt
  iscsi/{clientname}/iscsi-gateway.key

2. Provide the SSL key and cert in the json you pass the orchestrator and
   it'll push them up for you.

Also lockdown the caps so the container can only access iscsi ssl
key/certs.

Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #34616 from ifed01/wip-ifed-fix-upgrade-super

os/bluestore: invoke _prepare_ondisk_format_super as the last op

Reviewed-by: Greg Farnum <gfarnum@redhat

Merge pull request #33462 from s0nea/wip-dashboard-telemetry-report

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add telemetry report component

Add a telemetry component in order to give the user the
possibility to configure the telemetry module in a more
guided fashion. The component offers broader explanations,
shows a preview of the generated report and asks the user
to accept the license before enabling the module.

Fixes: https://tracker.ceph.com/issues/43956
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: handle `textarea` and add tooltip

Enhance `Copy2ClipboardButtonDirective` in order to
handle `textarea`s and copy a formatted input.
Add a tooltip to the button, too.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34818 from idryomov/wip-krbd-tempfile-fixes

qa/tasks/rbd{,_fio}: update NamedTemporaryFile usage for python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: refactor/enhance CLI commands

1. Move the implementations of the `on` and `off`
command into dedicated methods so that they can
be accessed by other mgr modules via `mgr.remote`.

2. Add a `show-all` command to return the report
and device report at once. Also add a method
`get_report` to make the reports accessible for
other mgr modules as mentioned above.

3. Add two  minor changes and a note where the
gathered data can be found to the Telemetry docu
file.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34580 from bk201/wip-45106

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34587 from rhcs-dashboard/async_user_name

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34733 from sebastian-philipp/cepahdm-15.2.2-release-notes

PendingReleaseNotes: Add Cephadm for >15.2.1

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #34682 from rhcs-dashboard/fix-45183-master

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34251 from JackSlateur/doc-os-recommendations

doc/start/os-recommendations.rst: add Octopus' platform

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/start/os-recommendations.rst: add opensuse, comment centos7

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

PendingReleaseNotes: Add Cephadm for >15.2.1

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34590 from zdover23/wip-doc-cidr-network-troubleshooting2

ceph/doc: Add guidance for cidr network troubleshooting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34729 from votdev/issue_43081_show_prom_alerts_in_navbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34747 from varshar16/wip-doc-remove-duplicate-ps

doc/mgr/orchestrator: Remove duplicate ps from support table

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34704 from jap/big-endian-day-operator

osd: Make the _day operator"" internals big-endian.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34812 from rzarzynski/wip-osd-drop-statechart-from-pgh

osd: drop unneeded inclusion of boost::statechart in PG.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34813 from rzarzynski/wip-osd-fix-typo-pglog

crimson/osd, osd: fix typos in PG Log-related comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34791 from rzarzynski/wip-crimson-backfill-pgtemp-ref

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: s/log.bottom/log.tail/ in osd_types.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: fix typo in PGLog.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/rbd_fio: update NamedTemporaryFile usage for python3

Open the file in text mode to avoid 'bytes' objects.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd: update NamedTemporaryFile usage for python3

- open the file in text mode to avoid 'bytes' objects

- 'bufsize' is now spelled 'buffering' and switching buffering off
  with 'buffering=0' is only allowed in binary mode.  As we need the
  file to be either in the page cache or on disk before we copy it
  to the remote, call flush().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #34753 from trociny/wip-44938-fixup

qa/workunits/rbd: retry the addition of a mirror pool peer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: drop unneeded inclusion of boost::statechart in PG.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34699 from yison/rwl-image-writeback-cache-seq5

rbd/cache: Replicated Write Log core codes - aio_read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rbd: fix peer_add helper

We need to temporary disable "exit on error" mode so it does not
abort when `rbd mirror pool peer add` returns "already exists"
error code.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34700 from lixiaoy1/early_finish

librbd: Return read request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34285 from tspmelo/wip-cypress

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30024 from vumrao/wip-vumrao-41061

radosgw-admin: add support for --bucket-id in bucket stats command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: E2E: Add navigateEdit method

This method should be use to navigate to the edit page.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace Protractor with Cypress

Fixes: https://tracker.ceph.com/issues/44812
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34429 from xxhdx1985126/wip-crimson-recovery-pr

crimson: pglog based recovery ---- part 1

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34656 from rhcs-dashboard/44935-custom-cherryPy-header

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34748 from neha-ojha/wip-jewel-cleanup

qa/suites/rados: jewel cleanup

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34602 from neha-ojha/wip-45075

qa/standalone/scrub/osd-scrub-repair.sh: fix race in TEST_auto_repair_bluestore_failed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34595 from rishabh-d-dave/fs-move-run_shell

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Reviewed-by: Jeff Layton <jlayton@redhat.com>

librbd: Return read request early

Finish read requests if the total length is 0.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #34705 from bk201/wip-45149

mgr/dashboard: fixing RBD purge error in backend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: monitoring menu entry should indicate firing alerts

Fixes: https://tracker.ceph.com/issues/43081
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34752 from tchaikov/fix-test-perf-counters-mgr-get

mgr/dashboard: fix API tests to be py3 compatible

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34292 from cbodley/wip-bufferlist-comparison

bufferlist: use iterators for comparison operators

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34090 from trociny/wip-ceph-erasure-code-tool

tools/erasure-code: new tool to encode/decode files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34683 from ifed01/wip-ifed-fix-freelist-label

os/bluestore: fix freelist's meta usage when bdev label is unsupported

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34588 from ifed01/wip-ifed-lower-spinner-mas

os/bluestore:  set bluestore_min_alloc_size to 4K for spinners

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34006 from aclamk/wip-rocksdb-sharding

kv/RocksDBStore,os/blustore: shard rocksdb database using column families

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33790 from bstillwell/add_conf_option_zstd_compression_level

compressor: Add a config option to specify Zstd compression level

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34635 from tchaikov/wip-mgr-py3

mgr: drop python2 support

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: displaying Service detail inside table

Adapt the changes in https://github.com/ceph/ceph/pull/32747.

Fixes: https://tracker.ceph.com/issues/45106
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

librbd: add aio_read test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

doc/cephadm/install.rst: Add troubleshooting for 'CIDR' error

add troubleshooting for 'failed to infer CIDR network...' error

See also
https://tracker.ceph.com/issues/44828

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson: prevent PG::schedule_event_after from creating daggling future

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osdmap_gate: sort the blockers in ascending order of map epoch

When the osdmap_gate got a new map, it's supposed to release blocked ops on
all previous maps section of which is represented by [waiting_peering.begin(),
waiting_peering.upper_bound(epoch)]. So it's essentail that waiting_peering is
ordered in ascending order of the map epoch

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: keep track of clean_regions/dirty_regions of modifications

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add osd interface for recovery messages

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add recovery operations oriented request ---- "RecoverySubRequest"

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add pg log based recovery machinery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/os: add readv method to futurized_store

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add background recovery request and necessary interfaces for recover/backfill

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

osd: add some clean ups

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operation: add support for AggregateBlocker

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: introduce scheduler implementations and operation throttler

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: wire up local and remote async reservers

Signed-off-by: Samuel Just <sjust@redhat.com>

common/AsyncReserver: template out the finisher type

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PeeringState: eliminate unused cct from PGPool

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: move earliest_backfill to PeeringState

We'll want this in crimson as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: add WriteLogMap test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add LogMap class

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add ReadRequest to handle read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34732 from lxbsz/fs_fix_new

qa: misc fixes to run for cephfs qa test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31912 from zjcmszh/cmpext

pybind/rados: add Ioctx::cmpext()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

test/pybind/test_rados.py: test Ioctx::cmpext()
Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

pybind/rados: add Ioctx:cmpext()
Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

Merge pull request #34578 from lxbsz/volume_client_py3

qa/volume_client: be python3 compatible

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: switch to StringIO to fix byte-like errors

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: switch to mount_wait() to avoid possible umounted assert

For the fuse mount, it sometime will need to wait to a moment to let
the client get ready.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/fuse_mount: fix possible chmod 1777 error

INFO:teuthology.orchestra.run.smithi13a2:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2)
INFO:teuthology.orchestra.run.smithi132.stderr:chmod: changing permissions of '/home/ubuntu/cephtest/mnt.2': Permission denied
DEBUG:teuthology.orchestra.run:got remote process result: 1

Here just wait and rety for 10 times.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/vstart_runner: remove the stdin str type check

For the "ceph-dencoder" it will use the byte type data.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: fix AssertionError: CommandFailedError not raised under py3

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/workunits/rbd: retry the addition of a mirror pool peer

fb4311f5 has fixed this for setup, but "remove mirroring pool"
test needs fixing too.

Fixes: https://tracker.ceph.com/issues/44938
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34697 from dillaman/wip-44596

rbd-mirror: skip snapshot image-sync for newly created/cloned images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34751 from tchaikov/wip-doc-fix

doc/releases/mimic.rst: add title underline

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34741 from tchaikov/wip-teuthology-cronjobs-mimic

qa/tests: pass "-t 2" to teuthology-suite when testing mimic

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: fix API tests

* test_perf_counters_mgr_get
* test_selftest_cluster_log

Fixes: https://tracker.ceph.com/issues/45170
Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

qa/tasks/vstart_runner.py: use StringIO for capturing text

to be consistent with 8bfe977854686149c74de3ebaed7b571d43404dd

Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/general.rst: add mising link targets

they are referenced by 60c825486916d0b0b43248a0966d2ecb90f352df

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: silence sphinx warnings

* mimic.rst: add title underline

to address sphinx warning like:
```
WARNING: Title underline too short.
```

* octopus.rst: use consistent underline

to address following sphinx warning:
```
doc/releases/octopus.rst:49: WARNING: Title level inconsistent:
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-old-clients/1-install: remove hammer

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash/workloads: use default mon osd initial require min compat client

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash-old-clients/1-install: remove jewel

Since we've changed mon_osd_initial_require_min_compat_client to
luminous in 986c271b756a6b3f472ec6848fc0d6680c45cf5b, we can remove jewel from
mix.

Fixes: https://tracker.ceph.com/issues/45242
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/mgr/orchestrator: Remove duplicate ps from support table

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #34698 from matthewoliver/cephadm_check_pool_exists

cephadm: Check for pool existence for iscsi And NFS

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34676 from sebastian-philipp/cephadm-inventory-class

mgr/cephadm: add Inventory class

Merge pull request #34695 from mgfritch/cephadm-iscsi-container-image

mgr/cephadm: allow config for an iscsi `container_image`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34718 from mgfritch/cephadm-mgr-is-available

cephadm: catch Exception during is_mgr_available

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34619 from sebastian-philipp/qa-adopt-specific-commit

qa/cephadm: make test_adoption checkout a specific commit