scripts/run-cbt.sh: correct script option usage

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge PR #34672 into master

* refs/pull/34672/head:
qa/tasks/cephfs: Enable multiple exports tests
mgr/nfs: Instead of 'auth del' use 'auth rm'
qa/tasks/cephfs: Don't enable cephadm in TestNFS
qa/tasks/cephfs: Add tests for nfs exports
mgr/volumes/nfs: Fix idempotency of cluster and export commands
mgr/volumes/nfs: Fix incorrect read only access_type value
mgr/fs/nfs: Use check_mon_command() instead of mon_command()
qa/cephfs: Add tests for nfs
mgr/volumes/nfs: Remove type option from export create interface
vstart: Instead of CACHEINODE use MDCACHE
mgr/volumes: Rearrange nfs export interface
mgr/volumes/nfs: Delete common config object on cluster deletion
mgr/volumes/nfs: Delete all exports on cluster deletion
mgr/volumes: Make nfs create export interface idempotent
vstart: Add watch url for conf-nfs object
mgr/volumes/nfs: Delete user on removing export
mgr/volumes: Create user for given path and fs
vstart: Ensure cephadm and NFS does not conflict
vstart: Update details about ganesha packages
mgr/volumes/nfs: Add delete cephfs export command
mgr/volumes/nfs: Add RADOS notify for common config object
mgr/volumes/nfs: Pass cluster_id directly to NFSCluster {create, update, delete} methods
mgr/volumes: Add nfs cluster delete interface
mgr/volumes: Add nfs cluster update interface
vstart: Enable test_orchestrator in start_ganesha()
mgr/volumes: Add placement option to create nfs cluster interface
mgr/volumes/nfs: Change common ganesha config object name to 'conf-nfs.ganesha-<cluster_id>'
mgr/volumes/nfs: Call orch nfs apply

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35334 from changchengx/rwl_code_clean

librbd/rwl: remove extra ';' after defining function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35336 from tchaikov/wip-clang-cache

pybind/*/setup.py: fix clang detection with ccache

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35332 from idryomov/wip-rbd-read-flags

librbd: make rbd_read_from_replica_policy actually work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind/*/setup.py: fix clang detection with ccache

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd/rwl: remove extra ';' after defining function

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

librbd: make rbd_read_from_replica_policy actually work

Commit d92be8b5856d ("librbd: Improved support for balanced and
localized reads") introduced read_flags, but used low-level Objecter
flags (CEPH_OSD_FLAG_*).  This is wrong because librbd sits on top of
librados, not Objecter.  "balance" ends up being translated into
IGNORE_REDIRECT and "localize" being ignored.

Fixes: https://tracker.ceph.com/issues/45798
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #35301 from dillaman/wip-45715

librbd: Watcher should not attempt to re-watch after detecting blacklisting

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35138 from maydaycc/wip-45313

librbd: support duration controllable continuous burst

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35173 from trociny/wip-group-quiesce

librbd: quiesce/unquiesce support for group_snap_create

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: support duration controllable continuous burst

Modify the design of token bucket to save more tokens than burst value,
so as to support continuous burst. The desired burst duration can be
controlled by rbd_qos_XYZ_burst_seconds.

Fixes: https://tracker.ceph.com/issues/45313
Signed-off-by: wencong wan <wanwc@chinatelecom.cn>

qa/tasks/cephfs: Enable multiple exports tests

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: Instead of 'auth del' use 'auth rm'

`mgr` profile allows 'auth rm'. Use it instead of 'auth del' which is not
allowed.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35303 from hansbogert/patch-3

doc: rbd-mirroring: Explain snapshot mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35287 from gekios/wip-cephadm-bootstrapped-fix

qa/tasks/cephadm: Add check for already set image

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35006 from zdover23/wip-doc-45411-contain-latest-tag

[doc/install]: Add note warning against the :latest tag.

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/tasks/cephfs: Don't enable cephadm in TestNFS

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/tasks/cephfs: Add tests for nfs exports

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Fix idempotency of cluster and export commands

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Fix incorrect read only access_type value

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/fs/nfs: Use check_mon_command() instead of mon_command()

check_mon_command() checks the return code of mon command.

Signed-off-by: Varsha Rao <varao@redhat.com>

qa/cephfs: Add tests for nfs

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Remove type option from export create interface

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Instead of CACHEINODE use MDCACHE

CACHEINODE will be deprecated soon. Instead use MDCACHE.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Rearrange nfs export interface

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Delete common config object on cluster deletion

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Delete all exports on cluster deletion

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Make nfs create export interface idempotent

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Add watch url for conf-nfs object

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Delete user on removing export

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Create user for given path and fs

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Ensure cephadm and NFS does not conflict

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Update details about ganesha packages

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Add delete cephfs export command

$ ceph nfs export delete <binding> <clusterid>

binding: It is the pseudo root name
clusterid: It is name of the cluster

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Add RADOS notify for common config object

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Pass cluster_id directly to NFSCluster {create, update, delete} methods

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add nfs cluster delete interface

$ ceph nfs cluster delete <clusterid>

This deletes the deployed cluster.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add nfs cluster update interface

$ ceph nfs cluster update <clusterid> <placement>

This updates the existing deployed cluster according to placement value.

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Enable test_orchestrator in start_ganesha()

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add placement option to create nfs cluster interface

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Change common ganesha config object name to 'conf-nfs.ganesha-<cluster_id>'

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Call orch nfs apply

Signed-off-by: Varsha Rao <varao@redhat.com>

Add note warning against the :latest tag.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Incorporating mgfritch's improvements.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/docker/podman/ -- as per mgfritch's suggestion

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/: latest/:latest/

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/image ids/image IDs/ -- as per Alex Settle's suggestion.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Add note warning against the :latest tag.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/image ids/image IDs/ -- as per Alex Settle's suggestion.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Linked to container page from cephadm install page.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Clarified the note directing readers to the container documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35302 from neha-ojha/wip-45619

qa/suites/rgw, qa/suites/rados: whitelist PG_AVAILABILITY

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35304 from neha-ojha/wip-45760

qa/standalone/scrub/osd-scrub-snaps.sh: fix grep pattern

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34622 from changchengx/libblkdrv

os/bluestore: extract common backend driver library to access device

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Review-by: Kefu Chai <kchai@redhat.com>

qa/standalone/scrub/osd-scrub-snaps.sh: fix grep pattern

The error looks like this:

2020-05-28T20:56:30.214+0000 7f66cdecf700 -1 log_channel(cluster) log [ERR] : scrub 1.0 1:ab946124:::obj15:head : can't decode 'snapset' attr void SnapSet::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 97: Malformed input

Fixes: https://tracker.ceph.com/issues/45760
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: rbd-mirroring: Explain snapshot mirroring

* Rectify that snapshot based mirroring is point-in-time consistent
* Reword slightly s.t. there is less repetition.

Based on the discussion in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/G6ZW5OP4IXGUBAJTKZDPABK7HCKYISQH/

Signed-off-by: Hans van den Bogert <hansbogert@gmail.com>

Merge pull request #35280 from neha-ojha/wip-45660

qa/standalone/scrub/osd-scrub-repair.sh: fix grep pattern to match decode exception

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

librbd: Watcher should not attempt to re-watch after detecting blacklisting

Currently, the Watcher state machine will spin as fast as it can sending
re-watch requests to the OSD and then retrying after it fails with the
EBLACKLISTED error. Treat a blacklisting similarly to how removal of the
object is treated: stop attempting to re-watch.

Fixes: https://tracker.ceph.com/issues/45715
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rgw/verify/tasks/cls.yaml: whitelist PG_AVAILABILITY

The balancer was turned on by default in
d4fbaf7ea959fd945857abd327271a97fb1da631, as a result of which we might see
PG_AVAILABILITY health warnings when pg-upmap-items are applied.

Fixes: https://tracker.ceph.com/issues/45619
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/*/tasks/rados_cls_all.yaml: whitelist PG_AVAILABILITY

The balancer was turned on by default in
d4fbaf7ea959fd945857abd327271a97fb1da631, as a result of which we might see
PG_AVAILABILITY health warnings when pg-upmap-items are applied.

Fixes: https://tracker.ceph.com/issues/45619
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/cephadm: Add check fox already set image

Adding a check for already bootstrapped clusters where the image is
already set to avoid overriding it.

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>

Merge pull request #35222 from smithfarm/wip-45606

osd: make "missing incremental map" a debug log message

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35194 from bk201/wip-45625

mgr/cephadm: config service_url in Dashboard for Prometheus and Alert…

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33872 from majianpeng/osd-remove-duplicated-assert

osd/OSD: remove useless ceph_assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35221 from tchaikov/wip-22052

qa/valgrind.supp: less specific when suppressing issues/22052

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34835 from jschmid1/osdspec_affinity

osd: add --osdspec-affinity flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35259 from tchaikov/wip-fmt-mingw

fmt: pickup change for adding MinGW support

Reviewed-by: Neha Ojha <nojha@redhat.com>

src: extract backend driver from bluestore to access device

1. Both bluestore or other component e.g. rbd could use the same
   driver to access storage device. It's better to use one library
   to integrate the driver.
2. os and crimson-alienstore are static libraries. Link blk library into
   them.

Main changes are below:
1. move backend driver into src/blk
   src/$ mkdir -p blk/{aio,kernel,spdk,pmem,zns}
   src/$ mv os/bluestore/{BlockDevice.h,BlockDevice.cc} blk
   src/$ mv os/bluestore/{ceph_aio.h} aio/aio.h
   src/$ mv os/bluestore/{aio.cc} aio/
   src/$ mv os/bluestore/{KernelDevice.h,KernelDevice.cc} kernel/
   src/$ mv os/bluestore/{ceph_io_uring.h} kernel/io_uring.h
   src/$ mv os/bluestore/{io_uring.cc} kernel_drv/
   src/$ mv os/bluestore/{NVMEDevice.h,NVMEDevice.cc} spdk/
   src/$ mv os/bluestore/{PMEMDevice.h,PMEMDevice.cc} pmem/
   src/$ mv os/bluestore/{HMSMRDevice.h,HMSMRDevice.cc} zns/
2. customize macro name in header file to remove bluestore specific text
3. adjust header file patch in source code
4. create cmake rule blk/CMakeLists.txt to build blk
5. modify src/CMakeLists.txt to integrate blk
6. modify other CMakeLists.txt to adapt to new file structure.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
Co-authored-by: Kefu Chai <kefu@redhat.com>

mgr: include retval in MonCommandFailed exception

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: config service_url in Dashboard for Prometheus and AlertManager

Calling Dashboard's CLI to set service URLs after deploying Prometheus
and AlertManager Daemons.

Fixes: https://tracker.ceph.com/issues/45625
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

qa/standalone/scrub/osd-scrub-repair.sh: fix grep pattern to match decode exception

We fail because the error message in the log looks like:

2020-05-27T21:02:48.447+0000 7fbfc4e60700 -1 log_channel(cluster) log [ERR] : scrub 3.0 3:5c7b2c47:::ROBJ16:head : can't decode 'snapset' attr void SnapSet::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 97: Malformed input

Fixes: https://tracker.ceph.com/issues/45660
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #35197 into master

* refs/pull/35197/head:
os/Transaction: do not use anonymous aggregate with member having ctor
include/interval_set: no need to check if an unsigned greater or equal to 0
include/ceph_fs: do not use anonymous aggregate with member having ctor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35229 from ricardoasmarques/wip-45696

cephadm: Validate bootstrap "--dashboard-{key|crt}" path

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #35106 from p-se/wip-pse-cephadm-custom-monitoring-images

mgr/cephadm: allow custom images for grafana, prometheus, alertmanager and node_exporter

Merge pull request #35159 from batrick/i45632

pybind/mgr/cephadm: remove mds auth cap for nfs recovery db cred

Merge pull request #35266 from jschmid1/fix_previews_for_multimatch

mgr/cephadm: fix host matching when multiple hosts match

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35195 from ricardoasmarques/allow-specify-ssh

cephadm: Allow users to provide ssh keys during bootstrap

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #35273 from sebastian-philipp/qa-cephadm-mirror-ubuntu

qa/cephadm: Docker doesn't ship a registries.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35267 into master

* refs/pull/35267/head:
vstart_runner: set mounted to True at the end of mount()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #34974 from dungdm93/patch-1

pybind/ceph_volume_client: Fix PEP-8 SyntaxWarning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35012 from tchaikov/wip-45147

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35033 from badone/wip-test-the-right-rocksdb-version

qa/workunits/rados/test_envlibrados_rocksdb: Build correct rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35032 from badone/wip-update-rocksdb-to-v6.8.1

rocksdb: Updated to v6.8.1

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35111 from agayev/hm-smr

os/bluestore: Enable writing data to HM-SMR hard drives.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35097 from matthewoliver/cephadm_iscsi_api_user_password

cephadm: Make ceph-iscsi api user and password mandatory

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35079 from liewegas/wip-dedup-tool

ceph-dedup-tool: add new FastCDC chunker, and make estimate test a range of chunk sizes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #35198 from varshar16/wip-cephadm-module-error

qa/tasks/mgr: Don't disable cephadm module

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35128 from yuvalif/fix-amqp-urls-with-vhosts

rgw/url: fix amqp urls with vhosts

common/rabin: remove

The implementation is buggy, and slower than FastCDC.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: default to fastcdc

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: better behavior if TARGET_WINDOW_BITS==0

Skip the middle scan call entirely in that case.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: add some comments

Map terms back to those in the paper.

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: make calc_chunks const

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: take const bufferlist

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: use bufferlist iterator

More complex, but we avoid a buffer copy/rebuild if it is non-contiguous.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: simplify initial fingerprint calc

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: refactor scan into a helper

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: generate fragmented buffers

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: add test with a specific chunking result

Canary for alg changes

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: show object count explosion too

Also show dedup ratio as dedup_size/original_size (1.0 = no dedup,
.9 = 10%, etc.)

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: show estimate progress

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: test a range of chunk sizes at once

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: add --max-seconds

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: timeout -> report_period

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: use CDC class for chunking

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: useful error messages

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: EstimateThread -> CrawlerThread

Signed-off-by: Sage Weil <sage@newdream.net>