librbd: add aio_discard test case

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_discard

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add DiscardRequest

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add DiscardLogOperation

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add DiscardLogEntry

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34813 from rzarzynski/wip-osd-fix-typo-pglog

crimson/osd, osd: fix typos in PG Log-related comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34791 from rzarzynski/wip-crimson-backfill-pgtemp-ref

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: s/log.bottom/log.tail/ in osd_types.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: fix typo in PGLog.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34753 from trociny/wip-44938-fixup

qa/workunits/rbd: retry the addition of a mirror pool peer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34699 from yison/rwl-image-writeback-cache-seq5

rbd/cache: Replicated Write Log core codes - aio_read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: fix missed ref in ShardServices::send_pg_temp().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rbd: fix peer_add helper

We need to temporary disable "exit on error" mode so it does not
abort when `rbd mirror pool peer add` returns "already exists"
error code.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34700 from lixiaoy1/early_finish

librbd: Return read request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34285 from tspmelo/wip-cypress

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #30024 from vumrao/wip-vumrao-41061

radosgw-admin: add support for --bucket-id in bucket stats command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: E2E: Add navigateEdit method

This method should be use to navigate to the edit page.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Replace Protractor with Cypress

Fixes: https://tracker.ceph.com/issues/44812
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34429 from xxhdx1985126/wip-crimson-recovery-pr

crimson: pglog based recovery ---- part 1

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34656 from rhcs-dashboard/44935-custom-cherryPy-header

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34748 from neha-ojha/wip-jewel-cleanup

qa/suites/rados: jewel cleanup

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34602 from neha-ojha/wip-45075

qa/standalone/scrub/osd-scrub-repair.sh: fix race in TEST_auto_repair_bluestore_failed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34595 from rishabh-d-dave/fs-move-run_shell

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Reviewed-by: Jeff Layton <jlayton@redhat.com>

librbd: Return read request early

Finish read requests if the total length is 0.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #34705 from bk201/wip-45149

mgr/dashboard: fixing RBD purge error in backend

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34752 from tchaikov/fix-test-perf-counters-mgr-get

mgr/dashboard: fix API tests to be py3 compatible

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34292 from cbodley/wip-bufferlist-comparison

bufferlist: use iterators for comparison operators

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34090 from trociny/wip-ceph-erasure-code-tool

tools/erasure-code: new tool to encode/decode files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34683 from ifed01/wip-ifed-fix-freelist-label

os/bluestore: fix freelist's meta usage when bdev label is unsupported

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #34588 from ifed01/wip-ifed-lower-spinner-mas

os/bluestore:  set bluestore_min_alloc_size to 4K for spinners

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34006 from aclamk/wip-rocksdb-sharding

kv/RocksDBStore,os/blustore: shard rocksdb database using column families

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33790 from bstillwell/add_conf_option_zstd_compression_level

compressor: Add a config option to specify Zstd compression level

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34635 from tchaikov/wip-mgr-py3

mgr: drop python2 support

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: add aio_read test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add aio_read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

crimson: prevent PG::schedule_event_after from creating daggling future

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osdmap_gate: sort the blockers in ascending order of map epoch

When the osdmap_gate got a new map, it's supposed to release blocked ops on
all previous maps section of which is represented by [waiting_peering.begin(),
waiting_peering.upper_bound(epoch)]. So it's essentail that waiting_peering is
ordered in ascending order of the map epoch

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: keep track of clean_regions/dirty_regions of modifications

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add osd interface for recovery messages

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add recovery operations oriented request ---- "RecoverySubRequest"

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add pg log based recovery machinery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/os: add readv method to futurized_store

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add background recovery request and necessary interfaces for recover/backfill

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

osd: add some clean ups

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/osd_operation: add support for AggregateBlocker

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd: introduce scheduler implementations and operation throttler

Signed-off-by: Samuel Just <sjust@redhat.com>
Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson/osd: wire up local and remote async reservers

Signed-off-by: Samuel Just <sjust@redhat.com>

common/AsyncReserver: template out the finisher type

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PeeringState: eliminate unused cct from PGPool

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PrimaryLogPG: move earliest_backfill to PeeringState

We'll want this in crimson as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: add WriteLogMap test cases

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add LogMap class

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

librbd: add ReadRequest to handle read

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34732 from lxbsz/fs_fix_new

qa: misc fixes to run for cephfs qa test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31912 from zjcmszh/cmpext

pybind/rados: add Ioctx::cmpext()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

test/pybind/test_rados.py: test Ioctx::cmpext()
Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

pybind/rados: add Ioctx:cmpext()
Signed-off-by: Zhang Jiao <zhangjiao@cmss.chinamobile.com>

Merge pull request #34578 from lxbsz/volume_client_py3

qa/volume_client: be python3 compatible

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: switch to StringIO to fix byte-like errors

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: switch to mount_wait() to avoid possible umounted assert

For the fuse mount, it sometime will need to wait to a moment to let
the client get ready.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/fuse_mount: fix possible chmod 1777 error

INFO:teuthology.orchestra.run.smithi13a2:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2)
INFO:teuthology.orchestra.run.smithi132.stderr:chmod: changing permissions of '/home/ubuntu/cephtest/mnt.2': Permission denied
DEBUG:teuthology.orchestra.run:got remote process result: 1

Here just wait and rety for 10 times.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/vstart_runner: remove the stdin str type check

For the "ceph-dencoder" it will use the byte type data.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/cephfs: fix AssertionError: CommandFailedError not raised under py3

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/workunits/rbd: retry the addition of a mirror pool peer

fb4311f5 has fixed this for setup, but "remove mirroring pool"
test needs fixing too.

Fixes: https://tracker.ceph.com/issues/44938
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34697 from dillaman/wip-44596

rbd-mirror: skip snapshot image-sync for newly created/cloned images

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34751 from tchaikov/wip-doc-fix

doc/releases/mimic.rst: add title underline

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34741 from tchaikov/wip-teuthology-cronjobs-mimic

qa/tests: pass "-t 2" to teuthology-suite when testing mimic

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: fix API tests

* test_perf_counters_mgr_get
* test_selftest_cluster_log

Fixes: https://tracker.ceph.com/issues/45170
Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

qa/tasks/vstart_runner.py: use StringIO for capturing text

to be consistent with 8bfe977854686149c74de3ebaed7b571d43404dd

Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/general.rst: add mising link targets

they are referenced by 60c825486916d0b0b43248a0966d2ecb90f352df

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: silence sphinx warnings

* mimic.rst: add title underline

to address sphinx warning like:
```
WARNING: Title underline too short.
```

* octopus.rst: use consistent underline

to address following sphinx warning:
```
doc/releases/octopus.rst:49: WARNING: Title level inconsistent:
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-old-clients/1-install: remove hammer

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash/workloads: use default mon osd initial require min compat client

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/suites/rados/thrash-old-clients/1-install: remove jewel

Since we've changed mon_osd_initial_require_min_compat_client to
luminous in 986c271b756a6b3f472ec6848fc0d6680c45cf5b, we can remove jewel from
mix.

Fixes: https://tracker.ceph.com/issues/45242
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34698 from matthewoliver/cephadm_check_pool_exists

cephadm: Check for pool existence for iscsi And NFS

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34676 from sebastian-philipp/cephadm-inventory-class

mgr/cephadm: add Inventory class

Merge pull request #34695 from mgfritch/cephadm-iscsi-container-image

mgr/cephadm: allow config for an iscsi `container_image`

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34718 from mgfritch/cephadm-mgr-is-available

cephadm: catch Exception during is_mgr_available

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34619 from sebastian-philipp/qa-adopt-specific-commit

qa/cephadm: make test_adoption checkout a specific commit

Merge pull request #34693 from sebastian-philipp/cephadm-warn-master-improvement

cephadm: improve warn message

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34675 from sebastian-philipp/cephadm-service-rm-feedback

mgr/cephadm: remove_service: Feedback, if spec not found

Reviewed-by: Joshua Schmid <jschmid@suse.de>

qa/tests: pass "-t 2" to teuthology-suite when testing mimic

it's an oversight in 21dacc823be184718584c5fd3b647e87b3eeba42

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34142 from rhcs-dashboard/wip-44667-fix-reload-issues

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34730 from theanalyst/docs/release-updates-04-20

docs: release updates for nautilus,mimic,octopus

Reviewed-By: Nathan Cutler <ncutler@suse.com>

os/bluestore: Enable sharding by default

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34673 from sebastian-philipp/cephadm-man

doc/man: add cephadm

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #34725 from lxbsz/_type_equality_funcs

qa/tasks/cephfs/test_scrub: fix self.assertEqual no attribute '_type_equality_funcs'

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: update release.yml with octopus, nautilus & mimic

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: releases: add release notes for v14.2.9

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #34726 from lxbsz/dict_key_error

qa/tasks/cephfs/cephfs_test_case: switch 'dict_keys' object to list

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: release notes for v13.2.10 mimic

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: releases: release notes for v13.2.9 mimic

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

doc: releases: v15.2.1 release notes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/tasks/cephfs/test_volumes: to make sure the size is int type

Fix "Error EINVAL: invalid size option '4915200.0'".

In pytho2 int/2 will get a int type result, but in python3 it maybe
a float type.

Fixes: https://tracker.ceph.com/issues/45247
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/mount: to make sure that the count/seek are int type

To fix "stderr:dd: invalid number: '5.0'".

Fixes: https://tracker.ceph.com/issues/45247
Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/test_scrub: fix self.assertEqual no attribute '_type_equality_funcs'

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/cephfs_test_case: switch 'dict_keys' object to list

Fix the "TypeError: 'dict_keys' object does not support indexing" under
py3.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #33769 from lxbsz/qa_vstart_runner_os

qa/vstart_runner.py: add os and arch properties support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner: make remote.sh return str as default

To address the issue in https://github.com/ceph/teuthology/pull/1459.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/vstart_runner.py: add os and arch properties support

Fixes: https://tracker.ceph.com/issues/44456
Signed-off-by: Xiubo Li <xiubli@redhat.com>