Merge pull request #34786 from aclamk/wip-common-pretty-binary

common: Add pretty_binary_string conversions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34772 from aclamk/wip-bs-faster-fsck-no-cache

kv/KeyValueDB, os/bluestore: Made fsck faster by not caching results

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34754 from ifed01/wip-ifed-fix-45195

os/bluestore: fix improper blob usage while handling deferred big write.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34738 from sebastian-philipp/insights-rm-tox

pybind/mgr: Remove insights/tox.ini

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34690 from sebastian-philipp/cephadm-crush-keyring-erro

mgr/cephadm: raise, if crash keyring fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34632 from kotreshhr/bash-completion

bash_completion: Do not auto complete obsolete and hidden cmds

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34503 from ifed01/wip-ifed-fix-huge-bluefs-reads

os/bluestore: fix huge (>4GB) bluefs reads

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34363 from majianpeng/osd-thread-per-shard-ssd

osd: wakeup all threads of shard rather than one thread.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #34641 from yuvalif/fix_zer_size_in_notifications

rgw/notifications: fix zero size in notifications

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34352 from smanjara/wip-dynamic-resharding

rgw: more multisite dynamic resharding pieces.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34743 from sebastian-philipp/doc-fix-getting-started

doc: fix "getting started" link

Merge pull request #34594 from tchaikov/wip-more-tolerant-perf-test

test/crimson/cbt: increase stddev variance to 2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #34603 from liu-chunmei/crimson_msgr_local_conf

crimson/net: remove hardcoded msgr configuration

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34707 from jschmid1/fix_orch_docs

doc/orchestrator: adapt to new syntax, add more examples

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

minor fix: run tests in python venv

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #34616 from ifed01/wip-ifed-fix-upgrade-super

os/bluestore: invoke _prepare_ondisk_format_super as the last op

Reviewed-by: Greg Farnum <gfarnum@redhat

Merge pull request #33462 from s0nea/wip-dashboard-telemetry-report

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add telemetry report component

Add a telemetry component in order to give the user the
possibility to configure the telemetry module in a more
guided fashion. The component offers broader explanations,
shows a preview of the generated report and asks the user
to accept the license before enabling the module.

Fixes: https://tracker.ceph.com/issues/43956
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: handle `textarea` and add tooltip

Enhance `Copy2ClipboardButtonDirective` in order to
handle `textarea`s and copy a formatted input.
Add a tooltip to the button, too.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34818 from idryomov/wip-krbd-tempfile-fixes

qa/tasks/rbd{,_fio}: update NamedTemporaryFile usage for python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: refactor/enhance CLI commands

1. Move the implementations of the `on` and `off`
command into dedicated methods so that they can
be accessed by other mgr modules via `mgr.remote`.

2. Add a `show-all` command to return the report
and device report at once. Also add a method
`get_report` to make the reports accessible for
other mgr modules as mentioned above.

3. Add two  minor changes and a note where the
gathered data can be found to the Telemetry docu
file.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34580 from bk201/wip-45106

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34587 from rhcs-dashboard/async_user_name

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34733 from sebastian-philipp/cepahdm-15.2.2-release-notes

PendingReleaseNotes: Add Cephadm for >15.2.1

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

pybind/mgr: Remove insights/tox.ini

All we need is part of mgr/tox.ini nowadays.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

kv/KeyValueDB, os/bluestore: Made fsck faster by not caching results

Added ability to pass extra options when creating iterators to KV.
Used that ability to perform fsck operations without caching results.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34682 from rhcs-dashboard/fix-45183-master

Reviewed-by: Stephan Müller <smueller@suse.com>

common: Add pretty_binary_string conversions

Moved all incarnations of pretty binary printing into one common place.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34251 from JackSlateur/doc-os-recommendations

doc/start/os-recommendations.rst: add Octopus' platform

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/start/os-recommendations.rst: add opensuse, comment centos7

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

PendingReleaseNotes: Add Cephadm for >15.2.1

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34590 from zdover23/wip-doc-cidr-network-troubleshooting2

ceph/doc: Add guidance for cidr network troubleshooting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34729 from votdev/issue_43081_show_prom_alerts_in_navbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34747 from varshar16/wip-doc-remove-duplicate-ps

doc/mgr/orchestrator: Remove duplicate ps from support table

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: update docs to new syntax and updated examples

Signed-off-by: Joshua Schmid <jschmid@suse.de>
Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>

test/crimson/cbt: increase stddev variance to 2.0

as per Mark requests. we've being observing

```
03:55:21 - WARNING  - cbt      - prefill/incerta04.front.sepia.ceph.com/1: iops_stddev: (or (less) (near 0.50)):: 434.037/69.2122  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.50)):: 3830.51/2473.17  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/1: iops_avg: (or (greater) (near 0.05)):: 53916.0/57375.0  => rejected
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34704 from jap/big-endian-day-operator

osd: Make the _day operator"" internals big-endian.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34812 from rzarzynski/wip-osd-drop-statechart-from-pgh

osd: drop unneeded inclusion of boost::statechart in PG.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34813 from rzarzynski/wip-osd-fix-typo-pglog

crimson/osd, osd: fix typos in PG Log-related comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34791 from rzarzynski/wip-crimson-backfill-pgtemp-ref

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: s/log.bottom/log.tail/ in osd_types.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: fix typo in PGLog.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/rbd_fio: update NamedTemporaryFile usage for python3

Open the file in text mode to avoid 'bytes' objects.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd: update NamedTemporaryFile usage for python3

- open the file in text mode to avoid 'bytes' objects

- 'bufsize' is now spelled 'buffering' and switching buffering off
  with 'buffering=0' is only allowed in binary mode.  As we need the
  file to be either in the page cache or on disk before we copy it
  to the remote, call flush().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #34753 from trociny/wip-44938-fixup

qa/workunits/rbd: retry the addition of a mirror pool peer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: drop unneeded inclusion of boost::statechart in PG.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34699 from yison/rwl-image-writeback-cache-seq5

rbd/cache: Replicated Write Log core codes - aio_read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: fix huge (>4GB) bluefs reads

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rbd: fix peer_add helper

We need to temporary disable "exit on error" mode so it does not
abort when `rbd mirror pool peer add` returns "already exists"
error code.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34700 from lixiaoy1/early_finish

librbd: Return read request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34285 from tspmelo/wip-cypress

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30024 from vumrao/wip-vumrao-41061

radosgw-admin: add support for --bucket-id in bucket stats command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: E2E: Add navigateEdit method

This method should be use to navigate to the edit page.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace Protractor with Cypress

Fixes: https://tracker.ceph.com/issues/44812
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34429 from xxhdx1985126/wip-crimson-recovery-pr

crimson: pglog based recovery ---- part 1

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34656 from rhcs-dashboard/44935-custom-cherryPy-header

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34748 from neha-ojha/wip-jewel-cleanup

qa/suites/rados: jewel cleanup

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34602 from neha-ojha/wip-45075

qa/standalone/scrub/osd-scrub-repair.sh: fix race in TEST_auto_repair_bluestore_failed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34595 from rishabh-d-dave/fs-move-run_shell

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Reviewed-by: Jeff Layton <jlayton@redhat.com>

librbd: Return read request early

Finish read requests if the total length is 0.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #34705 from bk201/wip-45149

mgr/dashboard: fixing RBD purge error in backend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/rgw: some basic reshard tests

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

minor fix: pass bucket_info as const ref in bi_list()

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: parameterized bucket_index_layout_generation in BucketShard::init()
for better flexibility.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: more multisite dynamic resharding pieces.

rados object names of bi shard to take a unique id called
generation number.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: monitoring menu entry should indicate firing alerts

Fixes: https://tracker.ceph.com/issues/43081
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34752 from tchaikov/fix-test-perf-counters-mgr-get

mgr/dashboard: fix API tests to be py3 compatible

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34292 from cbodley/wip-bufferlist-comparison

bufferlist: use iterators for comparison operators

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34090 from trociny/wip-ceph-erasure-code-tool

tools/erasure-code: new tool to encode/decode files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34683 from ifed01/wip-ifed-fix-freelist-label

os/bluestore: fix freelist's meta usage when bdev label is unsupported

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34588 from ifed01/wip-ifed-lower-spinner-mas

os/bluestore:  set bluestore_min_alloc_size to 4K for spinners

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34006 from aclamk/wip-rocksdb-sharding

kv/RocksDBStore,os/blustore: shard rocksdb database using column families

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33790 from bstillwell/add_conf_option_zstd_compression_level

compressor: Add a config option to specify Zstd compression level

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34635 from tchaikov/wip-mgr-py3

mgr: drop python2 support

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: displaying Service detail inside table

Adapt the changes in https://github.com/ceph/ceph/pull/32747.

Fixes: https://tracker.ceph.com/issues/45106
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

librbd: add aio_read test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

doc/cephadm/install.rst: Add troubleshooting for 'CIDR' error

add troubleshooting for 'failed to infer CIDR network...' error

See also
https://tracker.ceph.com/issues/44828

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson: prevent PG::schedule_event_after from creating daggling future

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osdmap_gate: sort the blockers in ascending order of map epoch

When the osdmap_gate got a new map, it's supposed to release blocked ops on
all previous maps section of which is represented by [waiting_peering.begin(),
waiting_peering.upper_bound(epoch)]. So it's essentail that waiting_peering is
ordered in ascending order of the map epoch

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: keep track of clean_regions/dirty_regions of modifications

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add osd interface for recovery messages

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add recovery operations oriented request ---- "RecoverySubRequest"

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add pg log based recovery machinery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/os: add readv method to futurized_store

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add background recovery request and necessary interfaces for recover/backfill

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

osd: add some clean ups

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operation: add support for AggregateBlocker

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: introduce scheduler implementations and operation throttler

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: wire up local and remote async reservers

Signed-off-by: Samuel Just <sjust@redhat.com>

common/AsyncReserver: template out the finisher type

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PeeringState: eliminate unused cct from PGPool

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: move earliest_backfill to PeeringState

We'll want this in crimson as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: add WriteLogMap test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add LogMap class

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add ReadRequest to handle read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34732 from lxbsz/fs_fix_new

qa: misc fixes to run for cephfs qa test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31912 from zjcmszh/cmpext

pybind/rados: add Ioctx::cmpext()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

test/pybind/test_rados.py: test Ioctx::cmpext()
Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>