test: put net BlueStore tests un ifdef WITH_BLUESTORE

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #20497 from tchaikov/wip-ceph-disk-cmd-failure

ceph-disk: include output of failed command in exception

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20534 from tchaikov/wip-22413

librados: make OPERATION_FULL_FORCE the default for rados_remove()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20456 from b-ranto/wip-mgr-prom

mgr/prometheus: Skip bogus entries

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #19843 from ifed01/wip-ifed-bluestore-repair

os/bluestore: implement BlueStore repair

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #14727 from voidbag/wip-voidbag-bdev-discard

os/bluestore: add discard method for ssd's performance

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20488 from theanalyst/rgw/frontend-init

rgw:  make init env methods return an error

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20504 from dzafman/wip-pg-states-doc

doc: Add missing pg states from doc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20543 from ceph/wip-rm23080

ceph-volume correctly normalize mount flags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.util update tests for mount flags

Include parametrized flags for ensuring a combination of values will
still be normalized regardless on how they are on ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.prepare normalize mount flags for CLI

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20510 from ceph/wip-cdp-docs-update

docs update ceph-deploy reference to reflect ceph-volume API

Reviewed-by: Vasu Kulkarni <vakulkar@redhat.com>

Merge pull request #20537 from ceph/wip-rm22878

ceph-volume: adds support to zap encrypted devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20425 from yehudasa/wip-22982

rgw: parse old rgw_obj with namespace correctly

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13644 from yehudasa/wip-rgw-ragweed

rgw: changes to support ragweed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: use lv.encrypted property with lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds an encrypted property to api.lvm.Volume

This property will return a boolean indicating if the lv
is encrypted or not.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add zapping to lvm dmcrypt tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: test zapping for both lvm create and lvm dmcrypt

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds support to zap encrypted devices, lvs, and partitions

Fixes: http://tracker.ceph.com/issues/22878
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20533 from wido/issue-23083

systemd: Wait 10 seconds before restarting ceph-mgr

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rados/troubleshooting update ceph-deploy references with new ceph-voume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/deployment update ceph-deploy references with new ceph-volume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man update ceph-deploy for the new ceph-volume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

librados: make OPERATION_FULL_FORCE the default for rados_remove()

Fixes: http://tracker.ceph.com/issues/22413
Signed-off-by: Kefu Chai <kchai@redhat.com>

systemd: Wait 10 seconds before restarting ceph-mgr

We do this for the MON and OSD as well, wait for a few
seconds before we try to attempt a restart.

On boot in IPv6 networks it might take a few seconds longer
before a IP-address is usable and this does not allow the mgr
to start right away.

Fixes: http://tracker.ceph.com/issues/23083
Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #20400 from tchaikov/wip-22354

ceph-disk: erase 110MB for nuking existing bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #20514 from ifed01/wip-ifed-fio-build-fix

test/fio: fix fio objectstore plugin building broken by

ceph-disk: erase 110MB for nuking existing bluestore

10M is not enough for erasing the data partition *and* db partition,
let's make it 110M.

Fixes: http://tracker.ceph.com/issues/22354
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20513 from ceph/wip-rm23003

ceph-volume warn on mix of filestore and bluestore flags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20500 from theanalyst/doc/releases/12.2.3

doc: release notes for 12.2.3

Reviewed-By: Sage Weil <sage@redhat.com>

test/fio: fix fio objectstore plugin building broken by
ObjectStore::queue_transaction interface changes.

Signed-off-by: Igor Fedotov "ifedotov@suse.com"

doc quick-ceph-deploy update for newer ceph-volume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: release notes for 12.2.3

Also dropping the relevant changes from PendingReleaseNotes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #20503 from theanalyst/doc/changelog/v12.2.3

doc: changelog for v12.2.3

ceph-volume tests verify exclude_group_options works with groups of flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add exclusion checks in prepare

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add exclusion checks in create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators create an arg group exclusion check

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common remove --journal-size as it is not used/supported

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common define arg groups for objectstores

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #19342 into master

* refs/pull/19342/head:
osd: add 'ceph [tell|daemon] osd.id smart'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #20465 into master

* refs/pull/20465/head:
os/bluestore: avoid frequent allocator dump on bluefs rebalance failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #20467 into master

* refs/pull/20467/head:
mon: mark manager beacons as no_reply

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #20484 into master

* refs/pull/20484/head:
osd/PrimaryLogPG: cancel all objecter ops atomically
osdc/Objecter: allow multiple ops to be canceled atomically

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #20487 into master

* refs/pull/20487/head:
os/bluestore: do not account DB volume space in total one reported by statfs method
os/bluestore: prevent statfs available from going negative

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20374 from liewegas/wip-18351

msg/DispatchQueue: clear queue after wait()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: add an ability to reproduce issue #21040 and verify if
fsck repair fixes it.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20475 from tchaikov/wip-std-string-view

common/str_list: s/boost::string_view/std::string_view

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: implement BlueStore repair

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: simplify SharedBlob::put_ref interface

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20443 from rzarzynski/wip-common-inlineable-ceph_clock_now

common: make ceph_clock_now() inlineable.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20439 from jcsp/wip-percent_used

mon: dump percent_used PGMap field as float

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Add missing pg states from doc

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: changelog for v12.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #20177 from liewegas/wip-kill-onreadable

os,osd: initial work to drop onreadable/onapplied callbacks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20499 from idryomov/wip-krbd-discard-tests

qa: additional krbd discard test cases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20490 from dillaman/wip-rbd-journal-workunit

qa/workunits/rbd: remove sanity check in journal.sh test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20093 from supriti/wip_rgw_ganesha_df

RGW-NFS: Use rados cluster_stat to report filesystem usage

Merge pull request #20395 from Songweibin/wip-misc-cleanup

librbd: validate if dst group snap name is the same with src

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20420 from trociny/wip-unwatch-race

librbd: potential race in RewatchRequest when resetting watch_handle

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20478 from trociny/wip-api-mirror

librbd: fix structure size check in rbd_mirror_image_get_info/status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw:  make init env methods return an error

Since web frontends may signal an error when requests are malformed or so, let
us double check this and raise errors early. The current user of this is
civetweb frontend; which can potentially return null from `parse_http_headers`
when a HTTP header without a ":" is supplied at which point headers.value is
null which can lead to undefined behaviour later in RGW.

Fixes: http://tracker.ceph.com/issues/23039
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

objectstore/bluestore.yaml: enable bdev discard

set bdev_enable_discard and bdev_async_discard true.

Signed-off-by: Taeksang Kim <voidbag@gmail.com>

os/bluestore/KernelDevice: async discard has been implemented.

Signed-off-by: Taeksang Kim <voidbag@gmail.com>

os/bluestore: add discard in BlueFS and BlueStore

discard is added to BlueFS.cc and BlueStore.cc

Signed-off-by: Taeksang Kim <voidbag@gmail.com>

os/bluestore/KernelDevice: add discard command

Discard method is added for ssd's performance.

Signed-off-by: Taeksang Kim <voidbag@gmail.com>

qa: krbd_blkroset.t: attempt discards in addition to writes

The block layer used to not enforce ioctl(BLKROSET) and set_disk_ro().
This has been fixed in 4.16-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-disk: include output of failed command in exception

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20491 from dillaman/wip-23043

qa/workunits/rbd: fix issues within permissions test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20481 from dillaman/wip-rbd-remove-error

rbd: don't overwrite the error code from the remove action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20486 from dillaman/wip-11502

test/librbd: utilize unique pool for cache tier testing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20450 from dzafman/wip-22996

Snapset inconsistency is detected with its own error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20492 from tchaikov/wip-boost-cmake-3.0

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/ragweed: don't reference fastcgi

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/rgw: ragweed: change base branch for upgrade testing

Change directory structure to reflect base version of upgrade test.
Also set base version to kraken.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/rgw/upgrade: don't restart the entire cluster at once

and also whitelist 'wrongly marked me down'

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks: add ragweed tasks

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph/qa/suites: add rgw/upgrade/ragweed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: only init config when doing prepare

and only remove users when doing check.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: configurable stages

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: point at ceph.conf in config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: set system flag on system user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks: add ragweed task

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

see https://cmake.org/cmake/help/v3.1/module/ExternalProject.html and
https://cmake.org/cmake/help/v3.0/module/ExternalProject.html. the
former has DOWNLOAD_NO_PROGRESS, while the latter does not.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: remove sanity check in journal.sh test

When the OSDs are being concurrently thrashed, this can result in
sporadic failures due to the admin socket disappearing.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: correct permission test handling for clone v2 images

Fixes: http://tracker.ceph.com/issues/23043
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: utilize RBD profiles in permissions.sh test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: do not account DB volume space in total one reported by statfs method

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: prevent statfs available from going negative

Fixes: https://tracker.ceph.com/issues/23040
Signed-off-by: Sage Weil <sage@redhat.com>

qa: krbd_fallocate.sh: add -ENOENT handling test case

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon: mark manager beacons as no_reply

We never reply to manager beacons, and we have to mark them
that way or else forwarded messages pile up pending replies
and things eventually block.

Fixes: http://tracker.ceph.com/issues/22114
Reported-by: Hongpeng Lu <ludehp@163.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mgr/prometheus: Expose OSD Flags

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/prometheus: Fix pg_* counts

Currently, the pg_* counts are not computed properly. We split the
current state by '+' sign but do not add the pg count to the already
found pg count. Instead, we overwrite any existing pg count with the new
count. This patch fixes it by adding all the pg counts together for all
the states.

It also introduces a new pg_total metric for pg_total that shows the
total count of PGs.

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/prometheus: Skip bogus entries

The osd data can contain bogus '-' entries, skip these when populating
osd metadata and disk occupation.

Signed-off-by: Boris Ranto <branto@redhat.com>

test/librbd: utilize unique pool for cache tier testing

Fixes: http://tracker.ceph.com/issues/11502
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/PrimaryLogPG: cancel all objecter ops atomically

We want to avoid a situation like this:

- start proxy op A (epoch E)
- start proxy op B (epoch E)
- start proxy op C (epoch E)
- objecter sends none of these because target is down in epoch E
- osdmap update to E+1
- pg cancels requeues A, B
- objecter updates to E+1
- objecter sends C
- pg cancels/requeues C

Note that the key thing is that operations on each object are canceled
atomically.  On the interval change we do it all at once.  In the other
cases, we cancel everything on the given object together.

Fixes: http://tracker.ceph.com/issues/22123
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: allow multiple ops to be canceled atomically

This allows multiple cancellations without worrying about a race with an
(objecter) osdmap update.

Signed-off-by: Sage Weil <sage@redhat.com>