ceph-volume: add osdspec_affinity flag to ceph-osd

Signed-off-by: Joshua Schmid <jschmid@suse.de>

ceph-volume: add ceph.osdspec_affinity tag

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34594 from tchaikov/wip-more-tolerant-perf-test

test/crimson/cbt: increase stddev variance to 2

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #34603 from liu-chunmei/crimson_msgr_local_conf

crimson/net: remove hardcoded msgr configuration

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34707 from jschmid1/fix_orch_docs

doc/orchestrator: adapt to new syntax, add more examples

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34616 from ifed01/wip-ifed-fix-upgrade-super

os/bluestore: invoke _prepare_ondisk_format_super as the last op

Reviewed-by: Greg Farnum <gfarnum@redhat

Merge pull request #33462 from s0nea/wip-dashboard-telemetry-report

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add telemetry report component

Add a telemetry component in order to give the user the
possibility to configure the telemetry module in a more
guided fashion. The component offers broader explanations,
shows a preview of the generated report and asks the user
to accept the license before enabling the module.

Fixes: https://tracker.ceph.com/issues/43956
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: handle `textarea` and add tooltip

Enhance `Copy2ClipboardButtonDirective` in order to
handle `textarea`s and copy a formatted input.
Add a tooltip to the button, too.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34818 from idryomov/wip-krbd-tempfile-fixes

qa/tasks/rbd{,_fio}: update NamedTemporaryFile usage for python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/telemetry: refactor/enhance CLI commands

1. Move the implementations of the `on` and `off`
command into dedicated methods so that they can
be accessed by other mgr modules via `mgr.remote`.

2. Add a `show-all` command to return the report
and device report at once. Also add a method
`get_report` to make the reports accessible for
other mgr modules as mentioned above.

3. Add two  minor changes and a note where the
gathered data can be found to the Telemetry docu
file.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34580 from bk201/wip-45106

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34587 from rhcs-dashboard/async_user_name

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34733 from sebastian-philipp/cepahdm-15.2.2-release-notes

PendingReleaseNotes: Add Cephadm for >15.2.1

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #34682 from rhcs-dashboard/fix-45183-master

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34251 from JackSlateur/doc-os-recommendations

doc/start/os-recommendations.rst: add Octopus' platform

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/start/os-recommendations.rst: add opensuse, comment centos7

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

PendingReleaseNotes: Add Cephadm for >15.2.1

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34590 from zdover23/wip-doc-cidr-network-troubleshooting2

ceph/doc: Add guidance for cidr network troubleshooting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34729 from votdev/issue_43081_show_prom_alerts_in_navbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34747 from varshar16/wip-doc-remove-duplicate-ps

doc/mgr/orchestrator: Remove duplicate ps from support table

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: update docs to new syntax and updated examples

Signed-off-by: Joshua Schmid <jschmid@suse.de>
Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>

test/crimson/cbt: increase stddev variance to 2.0

as per Mark requests. we've being observing

```
03:55:21 - WARNING  - cbt      - prefill/incerta04.front.sepia.ceph.com/1: iops_stddev: (or (less) (near 0.50)):: 434.037/69.2122  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/0: iops_stddev: (or (less) (near 0.50)):: 3830.51/2473.17  => rejected
...
04:37:52 - WARNING  - cbt      - rand/incerta04.front.sepia.ceph.com/1: iops_avg: (or (greater) (near 0.05)):: 53916.0/57375.0  => rejected
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34704 from jap/big-endian-day-operator

osd: Make the _day operator"" internals big-endian.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34812 from rzarzynski/wip-osd-drop-statechart-from-pgh

osd: drop unneeded inclusion of boost::statechart in PG.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34813 from rzarzynski/wip-osd-fix-typo-pglog

crimson/osd, osd: fix typos in PG Log-related comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34791 from rzarzynski/wip-crimson-backfill-pgtemp-ref

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: s/log.bottom/log.tail/ in osd_types.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: fix typo in PGLog.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/rbd_fio: update NamedTemporaryFile usage for python3

Open the file in text mode to avoid 'bytes' objects.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/rbd: update NamedTemporaryFile usage for python3

- open the file in text mode to avoid 'bytes' objects

- 'bufsize' is now spelled 'buffering' and switching buffering off
  with 'buffering=0' is only allowed in binary mode.  As we need the
  file to be either in the page cache or on disk before we copy it
  to the remote, call flush().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #34753 from trociny/wip-44938-fixup

qa/workunits/rbd: retry the addition of a mirror pool peer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: drop unneeded inclusion of boost::statechart in PG.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34699 from yison/rwl-image-writeback-cache-seq5

rbd/cache: Replicated Write Log core codes - aio_read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rbd: fix peer_add helper

We need to temporary disable "exit on error" mode so it does not
abort when `rbd mirror pool peer add` returns "already exists"
error code.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34700 from lixiaoy1/early_finish

librbd: Return read request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34285 from tspmelo/wip-cypress

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30024 from vumrao/wip-vumrao-41061

radosgw-admin: add support for --bucket-id in bucket stats command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: E2E: Add navigateEdit method

This method should be use to navigate to the edit page.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace Protractor with Cypress

Fixes: https://tracker.ceph.com/issues/44812
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34429 from xxhdx1985126/wip-crimson-recovery-pr

crimson: pglog based recovery ---- part 1

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34656 from rhcs-dashboard/44935-custom-cherryPy-header

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34748 from neha-ojha/wip-jewel-cleanup

qa/suites/rados: jewel cleanup

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34602 from neha-ojha/wip-45075

qa/standalone/scrub/osd-scrub-repair.sh: fix race in TEST_auto_repair_bluestore_failed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34595 from rishabh-d-dave/fs-move-run_shell

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Reviewed-by: Jeff Layton <jlayton@redhat.com>

librbd: Return read request early

Finish read requests if the total length is 0.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #34705 from bk201/wip-45149

mgr/dashboard: fixing RBD purge error in backend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: monitoring menu entry should indicate firing alerts

Fixes: https://tracker.ceph.com/issues/43081
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34752 from tchaikov/fix-test-perf-counters-mgr-get

mgr/dashboard: fix API tests to be py3 compatible

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34292 from cbodley/wip-bufferlist-comparison

bufferlist: use iterators for comparison operators

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34090 from trociny/wip-ceph-erasure-code-tool

tools/erasure-code: new tool to encode/decode files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34683 from ifed01/wip-ifed-fix-freelist-label

os/bluestore: fix freelist's meta usage when bdev label is unsupported

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34588 from ifed01/wip-ifed-lower-spinner-mas

os/bluestore:  set bluestore_min_alloc_size to 4K for spinners

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34006 from aclamk/wip-rocksdb-sharding

kv/RocksDBStore,os/blustore: shard rocksdb database using column families

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33790 from bstillwell/add_conf_option_zstd_compression_level

compressor: Add a config option to specify Zstd compression level

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34635 from tchaikov/wip-mgr-py3

mgr: drop python2 support

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: displaying Service detail inside table

Adapt the changes in https://github.com/ceph/ceph/pull/32747.

Fixes: https://tracker.ceph.com/issues/45106
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

librbd: add aio_read test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

doc/cephadm/install.rst: Add troubleshooting for 'CIDR' error

add troubleshooting for 'failed to infer CIDR network...' error

See also
https://tracker.ceph.com/issues/44828

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson: prevent PG::schedule_event_after from creating daggling future

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osdmap_gate: sort the blockers in ascending order of map epoch

When the osdmap_gate got a new map, it's supposed to release blocked ops on
all previous maps section of which is represented by [waiting_peering.begin(),
waiting_peering.upper_bound(epoch)]. So it's essentail that waiting_peering is
ordered in ascending order of the map epoch

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: keep track of clean_regions/dirty_regions of modifications

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add osd interface for recovery messages

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add recovery operations oriented request ---- "RecoverySubRequest"

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add pg log based recovery machinery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/os: add readv method to futurized_store

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add background recovery request and necessary interfaces for recover/backfill

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

osd: add some clean ups

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operation: add support for AggregateBlocker

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: introduce scheduler implementations and operation throttler

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: wire up local and remote async reservers

Signed-off-by: Samuel Just <sjust@redhat.com>

common/AsyncReserver: template out the finisher type

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PeeringState: eliminate unused cct from PGPool

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: move earliest_backfill to PeeringState

We'll want this in crimson as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: add WriteLogMap test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add LogMap class

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add ReadRequest to handle read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34732 from lxbsz/fs_fix_new

qa: misc fixes to run for cephfs qa test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31912 from zjcmszh/cmpext

pybind/rados: add Ioctx::cmpext()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

test/pybind/test_rados.py: test Ioctx::cmpext()
Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

pybind/rados: add Ioctx:cmpext()
Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

Merge pull request #34578 from lxbsz/volume_client_py3

qa/volume_client: be python3 compatible

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: switch to StringIO to fix byte-like errors

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: switch to mount_wait() to avoid possible umounted assert

For the fuse mount, it sometime will need to wait to a moment to let
the client get ready.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/fuse_mount: fix possible chmod 1777 error

INFO:teuthology.orchestra.run.smithi13a2:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2)
INFO:teuthology.orchestra.run.smithi132.stderr:chmod: changing permissions of '/home/ubuntu/cephtest/mnt.2': Permission denied
DEBUG:teuthology.orchestra.run:got remote process result: 1

Here just wait and rety for 10 times.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/vstart_runner: remove the stdin str type check

For the "ceph-dencoder" it will use the byte type data.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: fix AssertionError: CommandFailedError not raised under py3

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/workunits/rbd: retry the addition of a mirror pool peer

fb4311f5 has fixed this for setup, but "remove mirroring pool"
test needs fixing too.

Fixes: https://tracker.ceph.com/issues/44938
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34697 from dillaman/wip-44596

rbd-mirror: skip snapshot image-sync for newly created/cloned images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34751 from tchaikov/wip-doc-fix

doc/releases/mimic.rst: add title underline

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34741 from tchaikov/wip-teuthology-cronjobs-mimic

qa/tests: pass "-t 2" to teuthology-suite when testing mimic

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: fix API tests

* test_perf_counters_mgr_get
* test_selftest_cluster_log

Fixes: https://tracker.ceph.com/issues/45170
Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

qa/tasks/vstart_runner.py: use StringIO for capturing text

to be consistent with 8bfe977854686149c74de3ebaed7b571d43404dd

Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/general.rst: add mising link targets

they are referenced by 60c825486916d0b0b43248a0966d2ecb90f352df

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: silence sphinx warnings

* mimic.rst: add title underline

to address sphinx warning like:
```
WARNING: Title underline too short.
```

* octopus.rst: use consistent underline

to address following sphinx warning:
```
doc/releases/octopus.rst:49: WARNING: Title level inconsistent:
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-old-clients/1-install: remove hammer

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash/workloads: use default mon osd initial require min compat client

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash-old-clients/1-install: remove jewel

Since we've changed mon_osd_initial_require_min_compat_client to
luminous in 986c271b756a6b3f472ec6848fc0d6680c45cf5b, we can remove jewel from
mix.

Fixes: https://tracker.ceph.com/issues/45242
Signed-off-by: Neha Ojha <nojha@redhat.com>