cephadm: give ceph-iscsi permissions to configfs

The cephadm container mounts the configfs and then bind mounts (-v) it
into the container. Currently the container is not a priviliaged
container which leads to 2 problems:

 1. The container can't insert the iscsi_target_mod kernel module; and
 2. The container can't write to the configfs as that's only writeable
    by root.

We _can_ get around 1, by preloading the kernel module. I.E add it the
systemd unit file. But that doesn't help with 2.

I've tried mounting the configfs with uid and gid options, but configfs
doesn't use them.

If we make the container a priviliged container then magically both 1
and 2 are solved. We don't need to preload the module so that's one less
workaround. But more importantly, configfs can be written to so we can
create ISCSI targets etc.

So that's what this patch does, it makes iscsi containers privileged
containers by setting the CephContainer --priviliged while creating it.

Fixes: https://tracker.ceph.com/issues/45252
Signed-off-by: Matthew Oliver <moliver@suse.com>

Merge pull request #34824 from sebastian-philipp/python-common-fix-device-selection-type-annotation

python-common: Fix wrong type annotation

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #34846 from tchaikov/wip-45147

mgr: do not load disabled modules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #34786 from aclamk/wip-common-pretty-binary

common: Add pretty_binary_string conversions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34772 from aclamk/wip-bs-faster-fsck-no-cache

kv/KeyValueDB, os/bluestore: Made fsck faster by not caching results

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34754 from ifed01/wip-ifed-fix-45195

os/bluestore: fix improper blob usage while handling deferred big write.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34738 from sebastian-philipp/insights-rm-tox

pybind/mgr: Remove insights/tox.ini

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34690 from sebastian-philipp/cephadm-crush-keyring-erro

mgr/cephadm: raise, if crash keyring fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34632 from kotreshhr/bash-completion

bash_completion: Do not auto complete obsolete and hidden cmds

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34503 from ifed01/wip-ifed-fix-huge-bluefs-reads

os/bluestore: fix huge (>4GB) bluefs reads

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34363 from majianpeng/osd-thread-per-shard-ssd

osd: wakeup all threads of shard rather than one thread.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #34641 from yuvalif/fix_zer_size_in_notifications

rgw/notifications: fix zero size in notifications

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34352 from smanjara/wip-dynamic-resharding

rgw: more multisite dynamic resharding pieces.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34743 from sebastian-philipp/doc-fix-getting-started

doc: fix "getting started" link

mgr: do not load disabled modules

an option named "mgr_disabled_modules" is added in this change to
prevent mgr from loading modules listed in this option. because mgr
loads *all* modules found in the configured path, and per
https://tracker.ceph.com/issues/45147, python subinterpreter could hang
when loading numpy, so this behavior practically creates a deadlock
in mgr.

this issue is found when mgr uses python3.8 runtime. in development
environment, it'd be inconvenient to disable the offending mgr module
without changing the source code, even if we can choose to not install
them, for instance, the enduser can workaround this issue by
uninstalling `ceph-mgr-diskprediction-local`.

an option would be useful in this case, so we can add the module to the
list before mgr tries to load it.

as this issue is found with python3.8 + diskprediction_local (numpy), so
this mgr module is disabled by default if mgr is compiled with python3.8
runtime.

Fixes: https://tracker.ceph.com/issues/45147
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/PyModuleRegistry: probe modules using std::filesystem

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34594 from tchaikov/wip-more-tolerant-perf-test

test/crimson/cbt: increase stddev variance to 2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #34603 from liu-chunmei/crimson_msgr_local_conf

crimson/net: remove hardcoded msgr configuration

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34707 from jschmid1/fix_orch_docs

doc/orchestrator: adapt to new syntax, add more examples

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

minor fix: run tests in python venv

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #34616 from ifed01/wip-ifed-fix-upgrade-super

os/bluestore: invoke _prepare_ondisk_format_super as the last op

Reviewed-by: Greg Farnum <gfarnum@redhat

Merge pull request #33462 from s0nea/wip-dashboard-telemetry-report

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add telemetry report component

Add a telemetry component in order to give the user the
possibility to configure the telemetry module in a more
guided fashion. The component offers broader explanations,
shows a preview of the generated report and asks the user
to accept the license before enabling the module.

Fixes: https://tracker.ceph.com/issues/43956
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: handle `textarea` and add tooltip

Enhance `Copy2ClipboardButtonDirective` in order to
handle `textarea`s and copy a formatted input.
Add a tooltip to the button, too.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34818 from idryomov/wip-krbd-tempfile-fixes

qa/tasks/rbd{,_fio}: update NamedTemporaryFile usage for python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: refactor/enhance CLI commands

1. Move the implementations of the `on` and `off`
command into dedicated methods so that they can
be accessed by other mgr modules via `mgr.remote`.

2. Add a `show-all` command to return the report
and device report at once. Also add a method
`get_report` to make the reports accessible for
other mgr modules as mentioned above.

3. Add two  minor changes and a note where the
gathered data can be found to the Telemetry docu
file.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34580 from bk201/wip-45106

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34587 from rhcs-dashboard/async_user_name

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

python-common: Fix wrong type annotation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34733 from sebastian-philipp/cepahdm-15.2.2-release-notes

PendingReleaseNotes: Add Cephadm for >15.2.1

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

pybind/mgr: Remove insights/tox.ini

All we need is part of mgr/tox.ini nowadays.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

kv/KeyValueDB, os/bluestore: Made fsck faster by not caching results

Added ability to pass extra options when creating iterators to KV.
Used that ability to perform fsck operations without caching results.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34682 from rhcs-dashboard/fix-45183-master

Reviewed-by: Stephan Müller <smueller@suse.com>

common: Add pretty_binary_string conversions

Moved all incarnations of pretty binary printing into one common place.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34251 from JackSlateur/doc-os-recommendations

doc/start/os-recommendations.rst: add Octopus' platform

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/start/os-recommendations.rst: add opensuse, comment centos7

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

PendingReleaseNotes: Add Cephadm for >15.2.1

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34590 from zdover23/wip-doc-cidr-network-troubleshooting2

ceph/doc: Add guidance for cidr network troubleshooting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34729 from votdev/issue_43081_show_prom_alerts_in_navbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34747 from varshar16/wip-doc-remove-duplicate-ps

doc/mgr/orchestrator: Remove duplicate ps from support table

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: update docs to new syntax and updated examples

Signed-off-by: Joshua Schmid <jschmid@suse.de>
Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>

test/crimson/cbt: increase stddev variance to 2.0

as per Mark requests. we've being observing

```
03:55:21 - WARNING  - cbt      - prefill/incerta04.front.sepia.ceph.com/1: iops_stddev: (or (less) (near 0.50)):: 434.037/69.2122  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.50)):: 3830.51/2473.17  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/1: iops_avg: (or (greater) (near 0.05)):: 53916.0/57375.0  => rejected
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34704 from jap/big-endian-day-operator

osd: Make the _day operator"" internals big-endian.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34812 from rzarzynski/wip-osd-drop-statechart-from-pgh

osd: drop unneeded inclusion of boost::statechart in PG.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34813 from rzarzynski/wip-osd-fix-typo-pglog

crimson/osd, osd: fix typos in PG Log-related comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34791 from rzarzynski/wip-crimson-backfill-pgtemp-ref

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: s/log.bottom/log.tail/ in osd_types.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: fix typo in PGLog.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/rbd_fio: update NamedTemporaryFile usage for python3

Open the file in text mode to avoid 'bytes' objects.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd: update NamedTemporaryFile usage for python3

- open the file in text mode to avoid 'bytes' objects

- 'bufsize' is now spelled 'buffering' and switching buffering off
  with 'buffering=0' is only allowed in binary mode.  As we need the
  file to be either in the page cache or on disk before we copy it
  to the remote, call flush().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #34753 from trociny/wip-44938-fixup

qa/workunits/rbd: retry the addition of a mirror pool peer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: drop unneeded inclusion of boost::statechart in PG.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34699 from yison/rwl-image-writeback-cache-seq5

rbd/cache: Replicated Write Log core codes - aio_read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: fix huge (>4GB) bluefs reads

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rbd: fix peer_add helper

We need to temporary disable "exit on error" mode so it does not
abort when `rbd mirror pool peer add` returns "already exists"
error code.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34700 from lixiaoy1/early_finish

librbd: Return read request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34285 from tspmelo/wip-cypress

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30024 from vumrao/wip-vumrao-41061

radosgw-admin: add support for --bucket-id in bucket stats command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: E2E: Add navigateEdit method

This method should be use to navigate to the edit page.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace Protractor with Cypress

Fixes: https://tracker.ceph.com/issues/44812
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34429 from xxhdx1985126/wip-crimson-recovery-pr

crimson: pglog based recovery ---- part 1

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34656 from rhcs-dashboard/44935-custom-cherryPy-header

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34748 from neha-ojha/wip-jewel-cleanup

qa/suites/rados: jewel cleanup

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34602 from neha-ojha/wip-45075

qa/standalone/scrub/osd-scrub-repair.sh: fix race in TEST_auto_repair_bluestore_failed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34595 from rishabh-d-dave/fs-move-run_shell

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Reviewed-by: Jeff Layton <jlayton@redhat.com>

librbd: Return read request early

Finish read requests if the total length is 0.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #34705 from bk201/wip-45149

mgr/dashboard: fixing RBD purge error in backend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/rgw: some basic reshard tests

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

minor fix: pass bucket_info as const ref in bi_list()

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: parameterized bucket_index_layout_generation in BucketShard::init()
for better flexibility.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: more multisite dynamic resharding pieces.

rados object names of bi shard to take a unique id called
generation number.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: monitoring menu entry should indicate firing alerts

Fixes: https://tracker.ceph.com/issues/43081
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34752 from tchaikov/fix-test-perf-counters-mgr-get

mgr/dashboard: fix API tests to be py3 compatible

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34292 from cbodley/wip-bufferlist-comparison

bufferlist: use iterators for comparison operators

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34090 from trociny/wip-ceph-erasure-code-tool

tools/erasure-code: new tool to encode/decode files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34683 from ifed01/wip-ifed-fix-freelist-label

os/bluestore: fix freelist's meta usage when bdev label is unsupported

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34588 from ifed01/wip-ifed-lower-spinner-mas

os/bluestore:  set bluestore_min_alloc_size to 4K for spinners

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34006 from aclamk/wip-rocksdb-sharding

kv/RocksDBStore,os/blustore: shard rocksdb database using column families

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33790 from bstillwell/add_conf_option_zstd_compression_level

compressor: Add a config option to specify Zstd compression level

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34635 from tchaikov/wip-mgr-py3

mgr: drop python2 support

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: displaying Service detail inside table

Adapt the changes in https://github.com/ceph/ceph/pull/32747.

Fixes: https://tracker.ceph.com/issues/45106
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

librbd: add aio_read test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

doc/cephadm/install.rst: Add troubleshooting for 'CIDR' error

add troubleshooting for 'failed to infer CIDR network...' error

See also
https://tracker.ceph.com/issues/44828

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson: prevent PG::schedule_event_after from creating daggling future

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osdmap_gate: sort the blockers in ascending order of map epoch

When the osdmap_gate got a new map, it's supposed to release blocked ops on
all previous maps section of which is represented by [waiting_peering.begin(),
waiting_peering.upper_bound(epoch)]. So it's essentail that waiting_peering is
ordered in ascending order of the map epoch

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: keep track of clean_regions/dirty_regions of modifications

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add osd interface for recovery messages

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add recovery operations oriented request ---- "RecoverySubRequest"

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add pg log based recovery machinery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/os: add readv method to futurized_store

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add background recovery request and necessary interfaces for recover/backfill

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

osd: add some clean ups

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operation: add support for AggregateBlocker

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: introduce scheduler implementations and operation throttler

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: wire up local and remote async reservers

Signed-off-by: Samuel Just <sjust@redhat.com>

common/AsyncReserver: template out the finisher type

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PeeringState: eliminate unused cct from PGPool

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: move earliest_backfill to PeeringState

We'll want this in crimson as well.

Signed-off-by: Samuel Just <sjust@redhat.com>