vstart_runner: don't use namespaces by default

And add option --usens to vstart_runner.py to use namespaces.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: run nsenter commands with superuser privileges

And add a method that sets self.fuse_daemon.subproc.pid to the PID of
the process that doesn't have sudo in its arguments. For example, when
"sudo ceph-fuse /mnt/cephfs" is run on the shell, it launches process
with arguments "ceph-fuse /mnt/cephfs". The added method gets PID of
latter/child process and sets that as the fuse daemon's PID. Not doing
so kills the former/parent process but the not the child process.

Also, since we are around cleanup this method a bit.

Fixes: https://tracker.ceph.com/issues/45339
Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: look for mountpoint in cmdline file

Look for self.mountpoint in the contents of /proc/<pid>/cmdline file
when finding asok file for the client so that vstart_runner.py won't end
up picking asok file for a client not created in current run.

This usually never happens so far because PID of newly created processes
is higher than that of previously created processes and list of asok
files returned by "glob.glob(asok_path)" in find_socket() is in
descending order of PIDs.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #35334 from changchengx/rwl_code_clean

librbd/rwl: remove extra ';' after defining function

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35336 from tchaikov/wip-clang-cache

pybind/*/setup.py: fix clang detection with ccache

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35332 from idryomov/wip-rbd-read-flags

librbd: make rbd_read_from_replica_policy actually work

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

pybind/*/setup.py: fix clang detection with ccache

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd/rwl: remove extra ';' after defining function

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

librbd: make rbd_read_from_replica_policy actually work

Commit d92be8b5856d ("librbd: Improved support for balanced and
localized reads") introduced read_flags, but used low-level Objecter
flags (CEPH_OSD_FLAG_*).  This is wrong because librbd sits on top of
librados, not Objecter.  "balance" ends up being translated into
IGNORE_REDIRECT and "localize" being ignored.

Fixes: https://tracker.ceph.com/issues/45798
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #35301 from dillaman/wip-45715

librbd: Watcher should not attempt to re-watch after detecting blacklisting

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35138 from maydaycc/wip-45313

librbd: support duration controllable continuous burst

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35173 from trociny/wip-group-quiesce

librbd: quiesce/unquiesce support for group_snap_create

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: support duration controllable continuous burst

Modify the design of token bucket to save more tokens than burst value,
so as to support continuous burst. The desired burst duration can be
controlled by rbd_qos_XYZ_burst_seconds.

Fixes: https://tracker.ceph.com/issues/45313
Signed-off-by: wencong wan <wanwc@chinatelecom.cn>

Merge pull request #35303 from hansbogert/patch-3

doc: rbd-mirroring: Explain snapshot mirroring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35287 from gekios/wip-cephadm-bootstrapped-fix

qa/tasks/cephadm: Add check for already set image

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35006 from zdover23/wip-doc-45411-contain-latest-tag

[doc/install]: Add note warning against the :latest tag.

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Add note warning against the :latest tag.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Incorporating mgfritch's improvements.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/docker/podman/ -- as per mgfritch's suggestion

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/: latest/:latest/

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/image ids/image IDs/ -- as per Alex Settle's suggestion.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Add note warning against the :latest tag.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
s/image ids/image IDs/ -- as per Alex Settle's suggestion.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Linked to container page from cephadm install page.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Clarified the note directing readers to the container documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35302 from neha-ojha/wip-45619

qa/suites/rgw, qa/suites/rados: whitelist PG_AVAILABILITY

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35304 from neha-ojha/wip-45760

qa/standalone/scrub/osd-scrub-snaps.sh: fix grep pattern

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34622 from changchengx/libblkdrv

os/bluestore: extract common backend driver library to access device

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Review-by: Kefu Chai <kchai@redhat.com>

qa/standalone/scrub/osd-scrub-snaps.sh: fix grep pattern

The error looks like this:

2020-05-28T20:56:30.214+0000 7f66cdecf700 -1 log_channel(cluster) log [ERR] : scrub 1.0 1:ab946124:::obj15:head : can't decode 'snapset' attr void SnapSet::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 97: Malformed input

Fixes: https://tracker.ceph.com/issues/45760
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: rbd-mirroring: Explain snapshot mirroring

* Rectify that snapshot based mirroring is point-in-time consistent
* Reword slightly s.t. there is less repetition.

Based on the discussion in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/G6ZW5OP4IXGUBAJTKZDPABK7HCKYISQH/

Signed-off-by: Hans van den Bogert <hansbogert@gmail.com>

Merge pull request #35280 from neha-ojha/wip-45660

qa/standalone/scrub/osd-scrub-repair.sh: fix grep pattern to match decode exception

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

librbd: Watcher should not attempt to re-watch after detecting blacklisting

Currently, the Watcher state machine will spin as fast as it can sending
re-watch requests to the OSD and then retrying after it fails with the
EBLACKLISTED error. Treat a blacklisting similarly to how removal of the
object is treated: stop attempting to re-watch.

Fixes: https://tracker.ceph.com/issues/45715
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rgw/verify/tasks/cls.yaml: whitelist PG_AVAILABILITY

The balancer was turned on by default in
d4fbaf7ea959fd945857abd327271a97fb1da631, as a result of which we might see
PG_AVAILABILITY health warnings when pg-upmap-items are applied.

Fixes: https://tracker.ceph.com/issues/45619
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/*/tasks/rados_cls_all.yaml: whitelist PG_AVAILABILITY

The balancer was turned on by default in
d4fbaf7ea959fd945857abd327271a97fb1da631, as a result of which we might see
PG_AVAILABILITY health warnings when pg-upmap-items are applied.

Fixes: https://tracker.ceph.com/issues/45619
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/cephadm: Add check fox already set image

Adding a check for already bootstrapped clusters where the image is
already set to avoid overriding it.

Signed-off-by: Georgios Kyratsas <gkyratsas@suse.com>

Merge pull request #35222 from smithfarm/wip-45606

osd: make "missing incremental map" a debug log message

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35194 from bk201/wip-45625

mgr/cephadm: config service_url in Dashboard for Prometheus and Alert…

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33872 from majianpeng/osd-remove-duplicated-assert

osd/OSD: remove useless ceph_assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35221 from tchaikov/wip-22052

qa/valgrind.supp: less specific when suppressing issues/22052

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34835 from jschmid1/osdspec_affinity

osd: add --osdspec-affinity flag

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35259 from tchaikov/wip-fmt-mingw

fmt: pickup change for adding MinGW support

Reviewed-by: Neha Ojha <nojha@redhat.com>

src: extract backend driver from bluestore to access device

1. Both bluestore or other component e.g. rbd could use the same
   driver to access storage device. It's better to use one library
   to integrate the driver.
2. os and crimson-alienstore are static libraries. Link blk library into
   them.

Main changes are below:
1. move backend driver into src/blk
   src/$ mkdir -p blk/{aio,kernel,spdk,pmem,zns}
   src/$ mv os/bluestore/{BlockDevice.h,BlockDevice.cc} blk
   src/$ mv os/bluestore/{ceph_aio.h} aio/aio.h
   src/$ mv os/bluestore/{aio.cc} aio/
   src/$ mv os/bluestore/{KernelDevice.h,KernelDevice.cc} kernel/
   src/$ mv os/bluestore/{ceph_io_uring.h} kernel/io_uring.h
   src/$ mv os/bluestore/{io_uring.cc} kernel_drv/
   src/$ mv os/bluestore/{NVMEDevice.h,NVMEDevice.cc} spdk/
   src/$ mv os/bluestore/{PMEMDevice.h,PMEMDevice.cc} pmem/
   src/$ mv os/bluestore/{HMSMRDevice.h,HMSMRDevice.cc} zns/
2. customize macro name in header file to remove bluestore specific text
3. adjust header file patch in source code
4. create cmake rule blk/CMakeLists.txt to build blk
5. modify src/CMakeLists.txt to integrate blk
6. modify other CMakeLists.txt to adapt to new file structure.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>
Co-authored-by: Kefu Chai <kefu@redhat.com>

mgr: include retval in MonCommandFailed exception

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: config service_url in Dashboard for Prometheus and AlertManager

Calling Dashboard's CLI to set service URLs after deploying Prometheus
and AlertManager Daemons.

Fixes: https://tracker.ceph.com/issues/45625
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

qa/standalone/scrub/osd-scrub-repair.sh: fix grep pattern to match decode exception

We fail because the error message in the log looks like:

2020-05-27T21:02:48.447+0000 7fbfc4e60700 -1 log_channel(cluster) log [ERR] : scrub 3.0 3:5c7b2c47:::ROBJ16:head : can't decode 'snapset' attr void SnapSet::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 97: Malformed input

Fixes: https://tracker.ceph.com/issues/45660
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #35197 into master

* refs/pull/35197/head:
os/Transaction: do not use anonymous aggregate with member having ctor
include/interval_set: no need to check if an unsigned greater or equal to 0
include/ceph_fs: do not use anonymous aggregate with member having ctor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35229 from ricardoasmarques/wip-45696

cephadm: Validate bootstrap "--dashboard-{key|crt}" path

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #35106 from p-se/wip-pse-cephadm-custom-monitoring-images

mgr/cephadm: allow custom images for grafana, prometheus, alertmanager and node_exporter

Merge pull request #35159 from batrick/i45632

pybind/mgr/cephadm: remove mds auth cap for nfs recovery db cred

Merge pull request #35266 from jschmid1/fix_previews_for_multimatch

mgr/cephadm: fix host matching when multiple hosts match

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35195 from ricardoasmarques/allow-specify-ssh

cephadm: Allow users to provide ssh keys during bootstrap

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #35273 from sebastian-philipp/qa-cephadm-mirror-ubuntu

qa/cephadm: Docker doesn't ship a registries.conf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #35267 into master

* refs/pull/35267/head:
vstart_runner: set mounted to True at the end of mount()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #34974 from dungdm93/patch-1

pybind/ceph_volume_client: Fix PEP-8 SyntaxWarning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35012 from tchaikov/wip-45147

qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Yuri Weinstein <yuriw@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35033 from badone/wip-test-the-right-rocksdb-version

qa/workunits/rados/test_envlibrados_rocksdb: Build correct rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35032 from badone/wip-update-rocksdb-to-v6.8.1

rocksdb: Updated to v6.8.1

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #35111 from agayev/hm-smr

os/bluestore: Enable writing data to HM-SMR hard drives.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35097 from matthewoliver/cephadm_iscsi_api_user_password

cephadm: Make ceph-iscsi api user and password mandatory

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35079 from liewegas/wip-dedup-tool

ceph-dedup-tool: add new FastCDC chunker, and make estimate test a range of chunk sizes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #35198 from varshar16/wip-cephadm-module-error

qa/tasks/mgr: Don't disable cephadm module

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35128 from yuvalif/fix-amqp-urls-with-vhosts

rgw/url: fix amqp urls with vhosts

common/rabin: remove

The implementation is buggy, and slower than FastCDC.

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: default to fastcdc

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: better behavior if TARGET_WINDOW_BITS==0

Skip the middle scan call entirely in that case.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: add some comments

Map terms back to those in the paper.

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: make calc_chunks const

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: take const bufferlist

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: use bufferlist iterator

More complex, but we avoid a buffer copy/rebuild if it is non-contiguous.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: simplify initial fingerprint calc

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: refactor scan into a helper

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: generate fragmented buffers

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: add test with a specific chunking result

Canary for alg changes

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: show object count explosion too

Also show dedup ratio as dedup_size/original_size (1.0 = no dedup,
.9 = 10%, etc.)

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: show estimate progress

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: test a range of chunk sizes at once

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: add --max-seconds

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: timeout -> report_period

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: use CDC class for chunking

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: useful error messages

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: EstimateThread -> CrawlerThread

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: remove rabin-specific options

We don't want to fiddle with these.

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_cdc: adjust tests a bit

Seed random buffers!

Include average in histogram, and do the binning slightly differently.

Signed-off-by: Sage Weil <sage@newdream.net>

common/FixedCDC: add 'fixed' chunker

Signed-off-by: Sage Weil <sage@newdream.net>

common/FastCDC: initial implementation

Loosely based on the paper, but with a few changes.

Signed-off-by: Sage Weil <sage@newdream.net>

common/CDC: add abstract interface

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: warn on hash collision

Signed-off-by: Sage Weil <sage@newdream.net>

unittest_rabin_chunk: add unit test

This includes a "shift" test that shows that the rabin chunking misbehaves
when 5+ bytes are inserted at the front (many chunk boundaries shift!)

Signed-off-by: Sage Weil <sage@newdream.net>

common/rabin: add helper set_target_bits()

Sets key chunking parameters in one easy call.

Signed-off-by: Sage Weil <sage@newdream.net>

common/rabin: some cleanups

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: rename var

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-dedup-tool: larger default op size

If we don't read the entire object the rabin calculation won't work.  Avoid
this by default by just using a sufficiently large chunk size.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: fix host matching when multiple hosts match for in osd
previews

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa/cephadm: Docker doesn't ship a registries.conf

I guess not using the mirror for docker based tests is ok for now.

For adding docker support, we need to:

1. change the docker config
2. restart the docker daemon

Note that Docker's config only supports to mirror the docker.io registry.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: Alow users to provide ssh config during bootstrap

Custom ssh config can be provided by using the '--ssh-config' option

Signed-off-by: Ricardo Marques <rimarques@suse.com>

cephadm: Allow users to provide ssh keys during bootstrap

Fixes: https://tracker.ceph.com/issues/45629
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34902 from p-se/wip-pse-doc-prom-security

mgr/dashboard: document Prometheus' security model

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

vstart_runner: set mounted to True at the end of mount()

Not doing so prevents cleanup methods from running.

Fixes: https://tracker.ceph.com/issues/45723
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34863 from rhcs-dashboard/45310-add-grafana-rgw-sync-info

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34321 from jan--f/c-v-manage-lv_tags-bulk

ceph-volume: add and delete lvm tags in a single lvchange call.

fmt: pickup change for adding MinGW support

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/Transaction: do not use anonymous aggregate with member having ctor

use a single `hint` to replace `hint_type` and `alloc_hint_flags` in
the anonymous union.

this helps to address the compile error reported by GCC-10

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: no need to check if an unsigned greater or equal to 0

this silences the warning of

../src/include/interval_set.h:512:23: warning: comparison of unsigned expression in ‘>= 0’ is always true [-Wtype-limits]
  512 |     ceph_assert(_size >= 0);

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35235 from sebastian-philipp/qa-cephadm-mirror

qa/cephadm: Add local dockerhub mirror

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #34931 from trociny/wip-45072-2

rbd-mirror: wait for in-flight start/stop/restart

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35245 from ivancich/wip-remove-unused-func

rgw: remove unused function obj_force_ns

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/cephadm: Add local registry mirror

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rgw: remove unused function obj_force_ns

Removing the function eliminates a compiler warning.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>