librbd: make "snapshot create" notification be "async"

After adding quiesce/unquiesce callbacks the "snapshot create" request
may run long time.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: API for quiesce callbacks

The callbacks are fired by librbd before/after creating a
snapshot. The callback users like QEMU could attempt to freeze
the FS before allowing librbd to actually perform the snapshot.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: don't use boost::variant for image watcher messages

We are reached 20 types limit and are not able to add new message
types.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: small improvements for TaskFinisher

1) cancel: make return success status
2) queue: allow to pass return value
3) add method to reschedule event

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34436 from jschmid1/dg_affinity

ceph-volume: add ceph.osdspec_affinity tag

Merge pull request #34867 from tspmelo/wip-fix-table-filter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34779 from mgfritch/cephadm-workunit-node-exporter-avail

qa/workunits/cephadm/test_cephadm.sh: wait longer for monitoring stack

Merge pull request #34728 from ricardoasmarques/infer-config

cephadm: Infer config on "cephadm shell"

Merge pull request #34723 from matthewoliver/cephadm_iscsi_remove_fqdn_enabled

cephadm: remove fqdn_enabled from ceph-iscsi

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34448 from Devp00l/wip-44621

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33991 from SchoolGuy/monitoring/rbd-image-details

mgr/dashboard/grafana: Add rbd-image details dashboard

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34874 from tchaikov/wip-crimson-thread-pool-test

test/crimson: minimize the lexical scope of thread pool

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #34880 from neha-ojha/wip-balancer-cleanup

qa/suites/rados/thrash*: remove upmap.yaml and off.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34536 from varshar16/wip-cephfs-shell-flake8-py3

cephfs-shell: Change tox testenv name to py3

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash*: add on.yaml

To test with balancer in upmap mode, which is the default.

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash*: remove upmap.yaml and off.yaml

The balancer is always on, so we don't need upmap.yaml.
off.yaml doesn't make sense anymore.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34765 from changchengx/doc

doc/rbd: explicit reminder for libvirt directory access permissions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34877 from vumrao/wip-vumrao-45355

rgw/rgw_bucket: clear tenant string for non tenanted buckets in rgw_bucket_parse_bucket_key function.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw/rgw_bucket: clear tenant string for non tenanted buckets
in rgw_bucket_parse_bucket_key function.

Fixes: https://tracker.ceph.com/issues/45355
Signed-off-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge pull request #34787 from trociny/wip-45305

qa/workunits/rbd: fix list-mapped filter in unmap_device

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: explicit remind the directory access right

It needs to create admin socket to use perf counter for
debug purpose. Remind the access right to directory.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #34756 from yison/rwl-image-writeback-cache-seq7

rbd/cache: Replicated Write Log core codes - aio_discard

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34575 from hillpd/suicide_grace_fix

rados: prevent ShardedOpWQ suicide_grace drop when waiting for work.

Merge pull request #34569 from ifed01/wip-ifed-more-bluefs-stats

os/bluestore: improve bluefs stats reporting

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34376 from yuvalif/fix_amqp_routable_option

rgw/amqp: fix the "routable" delivery mode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/crimson: minimize the lexical scope of thread pool

to show the typical use case of thread pool in a better manner: thread
pool can be initialized *in* a seastar application.

also move `handle_exception()` out to catch all exceptions throwns in
the seastar app.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34657 from lxbsz/py3_com

qa/tasks/vstart_runner.py: to be python3 compatible

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34824 from sebastian-philipp/python-common-fix-device-selection-type-annotation

python-common: Fix wrong type annotation

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #34846 from tchaikov/wip-45147

mgr: do not load disabled modules

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #34786 from aclamk/wip-common-pretty-binary

common: Add pretty_binary_string conversions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34772 from aclamk/wip-bs-faster-fsck-no-cache

kv/KeyValueDB, os/bluestore: Made fsck faster by not caching results

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34754 from ifed01/wip-ifed-fix-45195

os/bluestore: fix improper blob usage while handling deferred big write.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34738 from sebastian-philipp/insights-rm-tox

pybind/mgr: Remove insights/tox.ini

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34690 from sebastian-philipp/cephadm-crush-keyring-erro

mgr/cephadm: raise, if crash keyring fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34632 from kotreshhr/bash-completion

bash_completion: Do not auto complete obsolete and hidden cmds

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34503 from ifed01/wip-ifed-fix-huge-bluefs-reads

os/bluestore: fix huge (>4GB) bluefs reads

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34363 from majianpeng/osd-thread-per-shard-ssd

osd: wakeup all threads of shard rather than one thread.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: Fix error when filtering table without data

Fixes: https://tracker.ceph.com/issues/45351
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34641 from yuvalif/fix_zer_size_in_notifications

rgw/notifications: fix zero size in notifications

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34352 from smanjara/wip-dynamic-resharding

rgw: more multisite dynamic resharding pieces.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34743 from sebastian-philipp/doc-fix-getting-started

doc: fix "getting started" link

mgr: do not load disabled modules

an option named "mgr_disabled_modules" is added in this change to
prevent mgr from loading modules listed in this option. because mgr
loads *all* modules found in the configured path, and per
https://tracker.ceph.com/issues/45147, python subinterpreter could hang
when loading numpy, so this behavior practically creates a deadlock
in mgr.

this issue is found when mgr uses python3.8 runtime. in development
environment, it'd be inconvenient to disable the offending mgr module
without changing the source code, even if we can choose to not install
them, for instance, the enduser can workaround this issue by
uninstalling `ceph-mgr-diskprediction-local`.

an option would be useful in this case, so we can add the module to the
list before mgr tries to load it.

as this issue is found with python3.8 + diskprediction_local (numpy), so
this mgr module is disabled by default if mgr is compiled with python3.8
runtime.

Fixes: https://tracker.ceph.com/issues/45147
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/PyModuleRegistry: probe modules using std::filesystem

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: add osdspec_affinity flag to ceph-osd

Signed-off-by: Joshua Schmid <jschmid@suse.de>

ceph-volume: add ceph.osdspec_affinity tag

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34594 from tchaikov/wip-more-tolerant-perf-test

test/crimson/cbt: increase stddev variance to 2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #34603 from liu-chunmei/crimson_msgr_local_conf

crimson/net: remove hardcoded msgr configuration

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34707 from jschmid1/fix_orch_docs

doc/orchestrator: adapt to new syntax, add more examples

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

minor fix: run tests in python venv

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #34616 from ifed01/wip-ifed-fix-upgrade-super

os/bluestore: invoke _prepare_ondisk_format_super as the last op

Reviewed-by: Greg Farnum <gfarnum@redhat

cephadm: Infer config on "cephadm shell"

Fixes: https://tracker.ceph.com/issues/44792
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #33462 from s0nea/wip-dashboard-telemetry-report

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add telemetry report component

Add a telemetry component in order to give the user the
possibility to configure the telemetry module in a more
guided fashion. The component offers broader explanations,
shows a preview of the generated report and asks the user
to accept the license before enabling the module.

Fixes: https://tracker.ceph.com/issues/43956
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: handle `textarea` and add tooltip

Enhance `Copy2ClipboardButtonDirective` in order to
handle `textarea`s and copy a formatted input.
Add a tooltip to the button, too.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34818 from idryomov/wip-krbd-tempfile-fixes

qa/tasks/rbd{,_fio}: update NamedTemporaryFile usage for python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: refactor/enhance CLI commands

1. Move the implementations of the `on` and `off`
command into dedicated methods so that they can
be accessed by other mgr modules via `mgr.remote`.

2. Add a `show-all` command to return the report
and device report at once. Also add a method
`get_report` to make the reports accessible for
other mgr modules as mentioned above.

3. Add two  minor changes and a note where the
gathered data can be found to the Telemetry docu
file.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34580 from bk201/wip-45106

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34587 from rhcs-dashboard/async_user_name

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

python-common: Fix wrong type annotation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34733 from sebastian-philipp/cepahdm-15.2.2-release-notes

PendingReleaseNotes: Add Cephadm for >15.2.1

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

pybind/mgr: Remove insights/tox.ini

All we need is part of mgr/tox.ini nowadays.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

kv/KeyValueDB, os/bluestore: Made fsck faster by not caching results

Added ability to pass extra options when creating iterators to KV.
Used that ability to perform fsck operations without caching results.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34682 from rhcs-dashboard/fix-45183-master

Reviewed-by: Stephan Müller <smueller@suse.com>

common: Add pretty_binary_string conversions

Moved all incarnations of pretty binary printing into one common place.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34251 from JackSlateur/doc-os-recommendations

doc/start/os-recommendations.rst: add Octopus' platform

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/start/os-recommendations.rst: add opensuse, comment centos7

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

PendingReleaseNotes: Add Cephadm for >15.2.1

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34590 from zdover23/wip-doc-cidr-network-troubleshooting2

ceph/doc: Add guidance for cidr network troubleshooting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34729 from votdev/issue_43081_show_prom_alerts_in_navbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34747 from varshar16/wip-doc-remove-duplicate-ps

doc/mgr/orchestrator: Remove duplicate ps from support table

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: update docs to new syntax and updated examples

Signed-off-by: Joshua Schmid <jschmid@suse.de>
Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>

librbd: add aio_discard test case

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_discard

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add DiscardRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add DiscardLogOperation

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add DiscardLogEntry

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

test/crimson/cbt: increase stddev variance to 2.0

as per Mark requests. we've being observing

```
03:55:21 - WARNING  - cbt      - prefill/incerta04.front.sepia.ceph.com/1: iops_stddev: (or (less) (near 0.50)):: 434.037/69.2122  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.50)):: 3830.51/2473.17  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/1: iops_avg: (or (greater) (near 0.05)):: 53916.0/57375.0  => rejected
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34704 from jap/big-endian-day-operator

osd: Make the _day operator"" internals big-endian.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34812 from rzarzynski/wip-osd-drop-statechart-from-pgh

osd: drop unneeded inclusion of boost::statechart in PG.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34813 from rzarzynski/wip-osd-fix-typo-pglog

crimson/osd, osd: fix typos in PG Log-related comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34791 from rzarzynski/wip-crimson-backfill-pgtemp-ref

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: s/log.bottom/log.tail/ in osd_types.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: fix typo in PGLog.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/rbd_fio: update NamedTemporaryFile usage for python3

Open the file in text mode to avoid 'bytes' objects.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd: update NamedTemporaryFile usage for python3

- open the file in text mode to avoid 'bytes' objects

- 'bufsize' is now spelled 'buffering' and switching buffering off
  with 'buffering=0' is only allowed in binary mode.  As we need the
  file to be either in the page cache or on disk before we copy it
  to the remote, call flush().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #34753 from trociny/wip-44938-fixup

qa/workunits/rbd: retry the addition of a mirror pool peer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: drop unneeded inclusion of boost::statechart in PG.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34699 from yison/rwl-image-writeback-cache-seq5

rbd/cache: Replicated Write Log core codes - aio_read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: fix huge (>4GB) bluefs reads

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rbd: fix peer_add helper

We need to temporary disable "exit on error" mode so it does not
abort when `rbd mirror pool peer add` returns "already exists"
error code.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: ECP modal enhanced validation

New validations with meaningful error messages:
* For all plugins
  * k + m <= OSDs that are available
  * k > 1
* For LRC
  * Display calculation factor for k (always)
  * Show error calculation for k is not met (also shows calculation)
  * Display number of groups for l (always)
  * Show error if calculation for l is not met (also shows calculation)
* For SHEC
  * m <= k
  * c <= m

Fixes: https://tracker.ceph.com/issues/44621
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use of crush node class in ECP modal

Now the crush node class preselects root, failure domain and device
class in the erasure code profile modal.

Like for crush rule, now also if you try to delete an used ECP you can't
and the info box will show you, what pool is using the profile.

Fixes: https://tracker.ceph.com/issues/44621
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use of crush node class in crush rule modal

Fixes: https://tracker.ceph.com/issues/44621
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Crush node selection class

This class helps the erasure code profile modal and the crush rule modal
to determine which inputs can be made to create a decent rule or
profile.

Fixes: https://tracker.ceph.com/issues/44621
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #34700 from lixiaoy1/early_finish

librbd: Return read request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34285 from tspmelo/wip-cypress

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30024 from vumrao/wip-vumrao-41061

radosgw-admin: add support for --bucket-id in bucket stats command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Remove additional unneeded steps for the metrics calculation

Signed-off-by: Enno Gotthold <egotthold@suse.de>

qa/workunits/rbd: fix list-mapped filter in unmap_device

In recent versions `rbd list-mapped` does not print the white space
at the end of the line.

Fixes: https://tracker.ceph.com/issues/45305
Signed-off-by: Mykola Golub <mgolub@suse.com>