qa/cephfs: Add tests for nfs

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Remove type option from export create interface

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Instead of CACHEINODE use MDCACHE

CACHEINODE will be deprecated soon. Instead use MDCACHE.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Rearrange nfs export interface

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Delete common config object on cluster deletion

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Delete all exports on cluster deletion

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Make nfs create export interface idempotent

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Add watch url for conf-nfs object

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Delete user on removing export

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Create user for given path and fs

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Ensure cephadm and NFS does not conflict

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Update details about ganesha packages

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Add delete cephfs export command

$ ceph nfs export delete <binding> <clusterid>

binding: It is the pseudo root name
clusterid: It is name of the cluster

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Add RADOS notify for common config object

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Pass cluster_id directly to NFSCluster {create, update, delete} methods

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add nfs cluster delete interface

$ ceph nfs cluster delete <clusterid>

This deletes the deployed cluster.

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add nfs cluster update interface

$ ceph nfs cluster update <clusterid> <placement>

This updates the existing deployed cluster according to placement value.

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Enable test_orchestrator in start_ganesha()

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes: Add placement option to create nfs cluster interface

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Change common ganesha config object name to 'conf-nfs.ganesha-<cluster_id>'

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: Call orch nfs apply

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #35302 from neha-ojha/wip-45619

qa/suites/rgw, qa/suites/rados: whitelist PG_AVAILABILITY

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35304 from neha-ojha/wip-45760

qa/standalone/scrub/osd-scrub-snaps.sh: fix grep pattern

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34622 from changchengx/libblkdrv

os/bluestore: extract common backend driver library to access device

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Review-by: Kefu Chai <kchai@redhat.com>

qa/standalone/scrub/osd-scrub-snaps.sh: fix grep pattern

The error looks like this:

2020-05-28T20:56:30.214+0000 7f66cdecf700 -1 log_channel(cluster) log [ERR] : scrub 1.0 1:ab946124:::obj15:head : can't decode 'snapset' attr void SnapSet::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 97: Malformed input

Fixes: https://tracker.ceph.com/issues/45760
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35280 from neha-ojha/wip-45660

qa/standalone/scrub/osd-scrub-repair.sh: fix grep pattern to match decode exception

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/suites/rgw/verify/tasks/cls.yaml: whitelist PG_AVAILABILITY

The balancer was turned on by default in
d4fbaf7ea959fd945857abd327271a97fb1da631, as a result of which we might see
PG_AVAILABILITY health warnings when pg-upmap-items are applied.

Fixes: https://tracker.ceph.com/issues/45619
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/*/tasks/rados_cls_all.yaml: whitelist PG_AVAILABILITY

The balancer was turned on by default in
d4fbaf7ea959fd945857abd327271a97fb1da631, as a result of which we might see
PG_AVAILABILITY health warnings when pg-upmap-items are applied.

Fixes: https://tracker.ceph.com/issues/45619
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35222 from smithfarm/wip-45606

osd: make "missing incremental map" a debug log message

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35194 from bk201/wip-45625

mgr/cephadm: config service_url in Dashboard for Prometheus and Alert…

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33872 from majianpeng/osd-remove-duplicated-assert

osd/OSD: remove useless ceph_assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35221 from tchaikov/wip-22052

qa/valgrind.supp: less specific when suppressing issues/22052

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34835 from jschmid1/osdspec_affinity

osd: add --osdspec-affinity flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35259 from tchaikov/wip-fmt-mingw

fmt: pickup change for adding MinGW support

Reviewed-by: Neha Ojha <nojha@redhat.com>

src: extract backend driver from bluestore to access device

1. Both bluestore or other component e.g. rbd could use the same
   driver to access storage device. It's better to use one library
   to integrate the driver.
2. os and crimson-alienstore are static libraries. Link blk library into
   them.

Main changes are below:
1. move backend driver into src/blk
   src/$ mkdir -p blk/{aio,kernel,spdk,pmem,zns}
   src/$ mv os/bluestore/{BlockDevice.h,BlockDevice.cc} blk
   src/$ mv os/bluestore/{ceph_aio.h} aio/aio.h
   src/$ mv os/bluestore/{aio.cc} aio/
   src/$ mv os/bluestore/{KernelDevice.h,KernelDevice.cc} kernel/
   src/$ mv os/bluestore/{ceph_io_uring.h} kernel/io_uring.h
   src/$ mv os/bluestore/{io_uring.cc} kernel_drv/
   src/$ mv os/bluestore/{NVMEDevice.h,NVMEDevice.cc} spdk/
   src/$ mv os/bluestore/{PMEMDevice.h,PMEMDevice.cc} pmem/
   src/$ mv os/bluestore/{HMSMRDevice.h,HMSMRDevice.cc} zns/
2. customize macro name in header file to remove bluestore specific text
3. adjust header file patch in source code
4. create cmake rule blk/CMakeLists.txt to build blk
5. modify src/CMakeLists.txt to integrate blk
6. modify other CMakeLists.txt to adapt to new file structure.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
Co-authored-by: Kefu Chai <kefu@redhat.com>

mgr: include retval in MonCommandFailed exception

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: config service_url in Dashboard for Prometheus and AlertManager

Calling Dashboard's CLI to set service URLs after deploying Prometheus
and AlertManager Daemons.

Fixes: https://tracker.ceph.com/issues/45625
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

qa/standalone/scrub/osd-scrub-repair.sh: fix grep pattern to match decode exception

We fail because the error message in the log looks like:

2020-05-27T21:02:48.447+0000 7fbfc4e60700 -1 log_channel(cluster) log [ERR] : scrub 3.0 3:5c7b2c47:::ROBJ16:head : can't decode 'snapset' attr void SnapSet::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 97: Malformed input

Fixes: https://tracker.ceph.com/issues/45660
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #35197 into master

* refs/pull/35197/head:
os/Transaction: do not use anonymous aggregate with member having ctor
include/interval_set: no need to check if an unsigned greater or equal to 0
include/ceph_fs: do not use anonymous aggregate with member having ctor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35229 from ricardoasmarques/wip-45696

cephadm: Validate bootstrap "--dashboard-{key|crt}" path

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #35106 from p-se/wip-pse-cephadm-custom-monitoring-images

mgr/cephadm: allow custom images for grafana, prometheus, alertmanager and node_exporter

Merge pull request #35159 from batrick/i45632

pybind/mgr/cephadm: remove mds auth cap for nfs recovery db cred

Merge pull request #35266 from jschmid1/fix_previews_for_multimatch

mgr/cephadm: fix host matching when multiple hosts match

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35195 from ricardoasmarques/allow-specify-ssh

cephadm: Allow users to provide ssh keys during bootstrap

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #35273 from sebastian-philipp/qa-cephadm-mirror-ubuntu

qa/cephadm: Docker doesn't ship a registries.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35267 into master

* refs/pull/35267/head:
vstart_runner: set mounted to True at the end of mount()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #34974 from dungdm93/patch-1

pybind/ceph_volume_client: Fix PEP-8 SyntaxWarning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35012 from tchaikov/wip-45147

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35033 from badone/wip-test-the-right-rocksdb-version

qa/workunits/rados/test_envlibrados_rocksdb: Build correct rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35032 from badone/wip-update-rocksdb-to-v6.8.1

rocksdb: Updated to v6.8.1

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35111 from agayev/hm-smr

os/bluestore: Enable writing data to HM-SMR hard drives.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35097 from matthewoliver/cephadm_iscsi_api_user_password

cephadm: Make ceph-iscsi api user and password mandatory

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35079 from liewegas/wip-dedup-tool

ceph-dedup-tool: add new FastCDC chunker, and make estimate test a range of chunk sizes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #35198 from varshar16/wip-cephadm-module-error

qa/tasks/mgr: Don't disable cephadm module

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35128 from yuvalif/fix-amqp-urls-with-vhosts

rgw/url: fix amqp urls with vhosts

common/rabin: remove

The implementation is buggy, and slower than FastCDC.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: default to fastcdc

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: better behavior if TARGET_WINDOW_BITS==0

Skip the middle scan call entirely in that case.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: add some comments

Map terms back to those in the paper.

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: make calc_chunks const

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: take const bufferlist

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: use bufferlist iterator

More complex, but we avoid a buffer copy/rebuild if it is non-contiguous.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: simplify initial fingerprint calc

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: refactor scan into a helper

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: generate fragmented buffers

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: add test with a specific chunking result

Canary for alg changes

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: show object count explosion too

Also show dedup ratio as dedup_size/original_size (1.0 = no dedup,
.9 = 10%, etc.)

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: show estimate progress

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: test a range of chunk sizes at once

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: add --max-seconds

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: timeout -> report_period

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: use CDC class for chunking

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: useful error messages

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: EstimateThread -> CrawlerThread

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: remove rabin-specific options

We don't want to fiddle with these.

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: adjust tests a bit

Seed random buffers!

Include average in histogram, and do the binning slightly differently.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FixedCDC: add 'fixed' chunker

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: initial implementation

Loosely based on the paper, but with a few changes.

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: add abstract interface

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: warn on hash collision

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_rabin_chunk: add unit test

This includes a "shift" test that shows that the rabin chunking misbehaves
when 5+ bytes are inserted at the front (many chunk boundaries shift!)

Signed-off-by: Sage Weil <sage@newdream.net>

common/rabin: add helper set_target_bits()

Sets key chunking parameters in one easy call.

Signed-off-by: Sage Weil <sage@newdream.net>

common/rabin: some cleanups

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: rename var

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: larger default op size

If we don't read the entire object the rabin calculation won't work.  Avoid
this by default by just using a sufficiently large chunk size.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: fix host matching when multiple hosts match for in osd
previews

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa/cephadm: Docker doesn't ship a registries.conf

I guess not using the mirror for docker based tests is ok for now.

For adding docker support, we need to:

1. change the docker config
2. restart the docker daemon

Note that Docker's config only supports to mirror the docker.io registry.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: Alow users to provide ssh config during bootstrap

Custom ssh config can be provided by using the '--ssh-config' option

Signed-off-by: Ricardo Marques <rimarques@suse.com>

cephadm: Allow users to provide ssh keys during bootstrap

Fixes: https://tracker.ceph.com/issues/45629
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34902 from p-se/wip-pse-doc-prom-security

mgr/dashboard: document Prometheus' security model

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

vstart_runner: set mounted to True at the end of mount()

Not doing so prevents cleanup methods from running.

Fixes: https://tracker.ceph.com/issues/45723
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34863 from rhcs-dashboard/45310-add-grafana-rgw-sync-info

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34321 from jan--f/c-v-manage-lv_tags-bulk

ceph-volume: add and delete lvm tags in a single lvchange call.

fmt: pickup change for adding MinGW support

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/Transaction: do not use anonymous aggregate with member having ctor

use a single `hint` to replace `hint_type` and `alloc_hint_flags` in
the anonymous union.

this helps to address the compile error reported by GCC-10

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: no need to check if an unsigned greater or equal to 0

this silences the warning of

../src/include/interval_set.h:512:23: warning: comparison of unsigned expression in ‘>= 0’ is always true [-Wtype-limits]
  512 |     ceph_assert(_size >= 0);

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35235 from sebastian-philipp/qa-cephadm-mirror

qa/cephadm: Add local dockerhub mirror

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #34931 from trociny/wip-45072-2

rbd-mirror: wait for in-flight start/stop/restart

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35245 from ivancich/wip-remove-unused-func

rgw: remove unused function obj_force_ns

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/cephadm: Add local registry mirror

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>