mds: add uptime to MDS status

Fixes: https://tracker.ceph.com/issues/23094
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20510 from ceph/wip-cdp-docs-update

docs update ceph-deploy reference to reflect ceph-volume API

Reviewed-by: Vasu Kulkarni <vakulkar@redhat.com>

Merge pull request #20537 from ceph/wip-rm22878

ceph-volume: adds support to zap encrypted devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20425 from yehudasa/wip-22982

rgw: parse old rgw_obj with namespace correctly

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13644 from yehudasa/wip-rgw-ragweed

rgw: changes to support ragweed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: use lv.encrypted property with lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds an encrypted property to api.lvm.Volume

This property will return a boolean indicating if the lv
is encrypted or not.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: add zapping to lvm dmcrypt tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: test zapping for both lvm create and lvm dmcrypt

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds support to zap encrypted devices, lvs, and partitions

Fixes: http://tracker.ceph.com/issues/22878
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20533 from wido/issue-23083

systemd: Wait 10 seconds before restarting ceph-mgr

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rados/troubleshooting update ceph-deploy references with new ceph-voume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/rados/deployment update ceph-deploy references with new ceph-volume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man update ceph-deploy for the new ceph-volume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

systemd: Wait 10 seconds before restarting ceph-mgr

We do this for the MON and OSD as well, wait for a few
seconds before we try to attempt a restart.

On boot in IPv6 networks it might take a few seconds longer
before a IP-address is usable and this does not allow the mgr
to start right away.

Fixes: http://tracker.ceph.com/issues/23083
Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #20400 from tchaikov/wip-22354

ceph-disk: erase 110MB for nuking existing bluestore

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Wido den Hollander <wido@42on.com>

Merge pull request #20514 from ifed01/wip-ifed-fio-build-fix

test/fio: fix fio objectstore plugin building broken by

ceph-disk: erase 110MB for nuking existing bluestore

10M is not enough for erasing the data partition *and* db partition,
let's make it 110M.

Fixes: http://tracker.ceph.com/issues/22354
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20513 from ceph/wip-rm23003

ceph-volume warn on mix of filestore and bluestore flags

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20500 from theanalyst/doc/releases/12.2.3

doc: release notes for 12.2.3

Reviewed-By: Sage Weil <sage@redhat.com>

test/fio: fix fio objectstore plugin building broken by
ObjectStore::queue_transaction interface changes.

Signed-off-by: Igor Fedotov "ifedotov@suse.com"

doc quick-ceph-deploy update for newer ceph-volume API

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: release notes for 12.2.3

Also dropping the relevant changes from PendingReleaseNotes

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #20503 from theanalyst/doc/changelog/v12.2.3

doc: changelog for v12.2.3

ceph-volume tests verify exclude_group_options works with groups of flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add exclusion checks in prepare

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add exclusion checks in create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators create an arg group exclusion check

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common remove --journal-size as it is not used/supported

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common define arg groups for objectstores

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #19342 into master

* refs/pull/19342/head:
osd: add 'ceph [tell|daemon] osd.id smart'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #20465 into master

* refs/pull/20465/head:
os/bluestore: avoid frequent allocator dump on bluefs rebalance failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #20467 into master

* refs/pull/20467/head:
mon: mark manager beacons as no_reply

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #20484 into master

* refs/pull/20484/head:
osd/PrimaryLogPG: cancel all objecter ops atomically
osdc/Objecter: allow multiple ops to be canceled atomically

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #20487 into master

* refs/pull/20487/head:
os/bluestore: do not account DB volume space in total one reported by statfs method
os/bluestore: prevent statfs available from going negative

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20374 from liewegas/wip-18351

msg/DispatchQueue: clear queue after wait()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20475 from tchaikov/wip-std-string-view

common/str_list: s/boost::string_view/std::string_view

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20443 from rzarzynski/wip-common-inlineable-ceph_clock_now

common: make ceph_clock_now() inlineable.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20439 from jcsp/wip-percent_used

mon: dump percent_used PGMap field as float

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: changelog for v12.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #20177 from liewegas/wip-kill-onreadable

os,osd: initial work to drop onreadable/onapplied callbacks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20499 from idryomov/wip-krbd-discard-tests

qa: additional krbd discard test cases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20490 from dillaman/wip-rbd-journal-workunit

qa/workunits/rbd: remove sanity check in journal.sh test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20093 from supriti/wip_rgw_ganesha_df

RGW-NFS: Use rados cluster_stat to report filesystem usage

Merge pull request #20395 from Songweibin/wip-misc-cleanup

librbd: validate if dst group snap name is the same with src

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20420 from trociny/wip-unwatch-race

librbd: potential race in RewatchRequest when resetting watch_handle

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20478 from trociny/wip-api-mirror

librbd: fix structure size check in rbd_mirror_image_get_info/status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: krbd_blkroset.t: attempt discards in addition to writes

The block layer used to not enforce ioctl(BLKROSET) and set_disk_ro().
This has been fixed in 4.16-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20491 from dillaman/wip-23043

qa/workunits/rbd: fix issues within permissions test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20481 from dillaman/wip-rbd-remove-error

rbd: don't overwrite the error code from the remove action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20486 from dillaman/wip-11502

test/librbd: utilize unique pool for cache tier testing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20450 from dzafman/wip-22996

Snapset inconsistency is detected with its own error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20492 from tchaikov/wip-boost-cmake-3.0

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/ragweed: don't reference fastcgi

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/rgw: ragweed: change base branch for upgrade testing

Change directory structure to reflect base version of upgrade test.
Also set base version to kraken.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/rgw/upgrade: don't restart the entire cluster at once

and also whitelist 'wrongly marked me down'

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks: add ragweed tasks

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph/qa/suites: add rgw/upgrade/ragweed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: only init config when doing prepare

and only remove users when doing check.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: configurable stages

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: point at ceph.conf in config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/ragweed: set system flag on system user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks: add ragweed task

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

see https://cmake.org/cmake/help/v3.1/module/ExternalProject.html and
https://cmake.org/cmake/help/v3.0/module/ExternalProject.html. the
former has DOWNLOAD_NO_PROGRESS, while the latter does not.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: remove sanity check in journal.sh test

When the OSDs are being concurrently thrashed, this can result in
sporadic failures due to the admin socket disappearing.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: correct permission test handling for clone v2 images

Fixes: http://tracker.ceph.com/issues/23043
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: utilize RBD profiles in permissions.sh test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: do not account DB volume space in total one reported by statfs method

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: prevent statfs available from going negative

Fixes: https://tracker.ceph.com/issues/23040
Signed-off-by: Sage Weil <sage@redhat.com>

qa: krbd_fallocate.sh: add -ENOENT handling test case

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon: mark manager beacons as no_reply

We never reply to manager beacons, and we have to mark them
that way or else forwarded messages pile up pending replies
and things eventually block.

Fixes: http://tracker.ceph.com/issues/22114
Reported-by: Hongpeng Lu <ludehp@163.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test/librbd: utilize unique pool for cache tier testing

Fixes: http://tracker.ceph.com/issues/11502
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/PrimaryLogPG: cancel all objecter ops atomically

We want to avoid a situation like this:

- start proxy op A (epoch E)
- start proxy op B (epoch E)
- start proxy op C (epoch E)
- objecter sends none of these because target is down in epoch E
- osdmap update to E+1
- pg cancels requeues A, B
- objecter updates to E+1
- objecter sends C
- pg cancels/requeues C

Note that the key thing is that operations on each object are canceled
atomically.  On the interval change we do it all at once.  In the other
cases, we cancel everything on the given object together.

Fixes: http://tracker.ceph.com/issues/22123
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: allow multiple ops to be canceled atomically

This allows multiple cancellations without worrying about a race with an
(objecter) osdmap update.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: don't overwrite the error code from the remove action

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix structure size check in rbd_mirror_image_get_info/status

Signed-off-by: Mykola Golub <mgolub@suse.com>

os/bluestore: avoid frequent allocator dump on bluefs rebalance failure

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20031 from tchaikov/wip-doc-macos

doc/dev/macos: add doc for building on MacOS

common/str_list: s/boost::string_view/std::string_view

since string_view is include by standard library, we can now switch from
boost::string_view to std::string_view.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20474 from wjwithagen/wjw-wip-pthread_set_name_2

include: compat.h, fix the return result of pthread_set_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20168 into master

* refs/pull/20168/head:
Handle dir->dentries.empty()
client: lookup_ino when i1 becomes NULL

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20424 into master

* refs/pull/20424/head:
qa: ignore io pause warnings in mds-full test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20428 into master

* refs/pull/20428/head:
qa: ignore MON_DOWN while thrashing mons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20470 into master

* refs/pull/20470/head:
qa: ignore full cluster warnings for mds-full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

include: compat.h, fix the return result of pthread_set_name()

On FreeBSD the function type is void(). Make it compatible with return
value on other platforms.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa: ignore full cluster warnings for mds-full

Fixes: http://tracker.ceph.com/issues/23033
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados/objectstore: increase open file limit

Signed-off-by: Sage Weil <sage@redhat.com>

os: do not inject read EIO on meta pool objects

Avoid EIO on, say, osdmaps until we fix
http://tracker.ceph.com/issues/23029

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20274 from adamemerson/wip-socket-to-me!

common/admin_socket: various cleanups

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test_filestore_idempotent_sequence: put txn object in each collection

The transactions are idependent in each collection/sequencer, so we
can't record to a single txn object with racing transactions.  Fix
it by doing one in each collection, and when reading the latest op,
use the highest txn value we see.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: Use unique_ptr instead of new/delete

Mostly for the default hooks, plus one Formatter.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Cleanup path cleanup

Don't use strdup/free explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config/admin_socket: Remove PFL_* stuff since nobody's using it

If we want to, we have other ways of getting data out of a thread.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use one map instead of three

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph_test_objectstore: fix split ordering

We need to flush between split.  This requirement unfortunately
doesn't quite go away with the FileStore tracking.

Also, flush for each batch.  This is just because the test environment
may have a low open file ulimit.  (The old code did apply_transaction,
so it's functionally equivalent to this.)

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: Use std::mutex/condition_variable/thread

And fix a bug where we exit in one code-path without releasing the
lock.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_named_thread

To allow us to make a std::thread with a name.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Copy fewer strings

Pass std::string_view to call, rather than passing strings by value.

Change cmdmap_t and the maps in admin_socket to use the std::less<>
transparent comparator, so we can compare string_views to strings. (Or
anything else with an equality operator with strings.)

Pass the cmdmap_t in to call by const reference, not reference.

The change to cmdmap_t requires changing some of the users of
cmdmap_t. (That is, make them actually use cmdmap_t explicitly, not
the open-coed map type.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>