ceph-volume tests add exclusion checks in prepare

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add exclusion checks in create

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators create an arg group exclusion check

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.create use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare use the new group option exclution

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common remove --journal-size as it is not used/supported

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common define arg groups for objectstores

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #19342 into master

* refs/pull/19342/head:
osd: add 'ceph [tell|daemon] osd.id smart'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #20465 into master

* refs/pull/20465/head:
os/bluestore: avoid frequent allocator dump on bluefs rebalance failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #20467 into master

* refs/pull/20467/head:
mon: mark manager beacons as no_reply

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #20484 into master

* refs/pull/20484/head:
osd/PrimaryLogPG: cancel all objecter ops atomically
osdc/Objecter: allow multiple ops to be canceled atomically

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #20487 into master

* refs/pull/20487/head:
os/bluestore: do not account DB volume space in total one reported by statfs method
os/bluestore: prevent statfs available from going negative

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20374 from liewegas/wip-18351

msg/DispatchQueue: clear queue after wait()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20475 from tchaikov/wip-std-string-view

common/str_list: s/boost::string_view/std::string_view

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20443 from rzarzynski/wip-common-inlineable-ceph_clock_now

common: make ceph_clock_now() inlineable.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20439 from jcsp/wip-percent_used

mon: dump percent_used PGMap field as float

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20177 from liewegas/wip-kill-onreadable

os,osd: initial work to drop onreadable/onapplied callbacks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20499 from idryomov/wip-krbd-discard-tests

qa: additional krbd discard test cases

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20490 from dillaman/wip-rbd-journal-workunit

qa/workunits/rbd: remove sanity check in journal.sh test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20093 from supriti/wip_rgw_ganesha_df

RGW-NFS: Use rados cluster_stat to report filesystem usage

Merge pull request #20395 from Songweibin/wip-misc-cleanup

librbd: validate if dst group snap name is the same with src

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20420 from trociny/wip-unwatch-race

librbd: potential race in RewatchRequest when resetting watch_handle

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20478 from trociny/wip-api-mirror

librbd: fix structure size check in rbd_mirror_image_get_info/status

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: krbd_blkroset.t: attempt discards in addition to writes

The block layer used to not enforce ioctl(BLKROSET) and set_disk_ro().
This has been fixed in 4.16-rc1.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #20491 from dillaman/wip-23043

qa/workunits/rbd: fix issues within permissions test

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20481 from dillaman/wip-rbd-remove-error

rbd: don't overwrite the error code from the remove action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20486 from dillaman/wip-11502

test/librbd: utilize unique pool for cache tier testing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20450 from dzafman/wip-22996

Snapset inconsistency is detected with its own error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20492 from tchaikov/wip-boost-cmake-3.0

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: disable DOWNLOAD_NO_PROGRESS if cmake ver is lower than 3.1

see https://cmake.org/cmake/help/v3.1/module/ExternalProject.html and
https://cmake.org/cmake/help/v3.0/module/ExternalProject.html. the
former has DOWNLOAD_NO_PROGRESS, while the latter does not.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: remove sanity check in journal.sh test

When the OSDs are being concurrently thrashed, this can result in
sporadic failures due to the admin socket disappearing.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: correct permission test handling for clone v2 images

Fixes: http://tracker.ceph.com/issues/23043
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: utilize RBD profiles in permissions.sh test case

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: do not account DB volume space in total one reported by statfs method

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: prevent statfs available from going negative

Fixes: https://tracker.ceph.com/issues/23040
Signed-off-by: Sage Weil <sage@redhat.com>

qa: krbd_fallocate.sh: add -ENOENT handling test case

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon: mark manager beacons as no_reply

We never reply to manager beacons, and we have to mark them
that way or else forwarded messages pile up pending replies
and things eventually block.

Fixes: http://tracker.ceph.com/issues/22114
Reported-by: Hongpeng Lu <ludehp@163.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test/librbd: utilize unique pool for cache tier testing

Fixes: http://tracker.ceph.com/issues/11502
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/PrimaryLogPG: cancel all objecter ops atomically

We want to avoid a situation like this:

- start proxy op A (epoch E)
- start proxy op B (epoch E)
- start proxy op C (epoch E)
- objecter sends none of these because target is down in epoch E
- osdmap update to E+1
- pg cancels requeues A, B
- objecter updates to E+1
- objecter sends C
- pg cancels/requeues C

Note that the key thing is that operations on each object are canceled
atomically.  On the interval change we do it all at once.  In the other
cases, we cancel everything on the given object together.

Fixes: http://tracker.ceph.com/issues/22123
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: allow multiple ops to be canceled atomically

This allows multiple cancellations without worrying about a race with an
(objecter) osdmap update.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: don't overwrite the error code from the remove action

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix structure size check in rbd_mirror_image_get_info/status

Signed-off-by: Mykola Golub <mgolub@suse.com>

os/bluestore: avoid frequent allocator dump on bluefs rebalance failure

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20031 from tchaikov/wip-doc-macos

doc/dev/macos: add doc for building on MacOS

common/str_list: s/boost::string_view/std::string_view

since string_view is include by standard library, we can now switch from
boost::string_view to std::string_view.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20474 from wjwithagen/wjw-wip-pthread_set_name_2

include: compat.h, fix the return result of pthread_set_name()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #20168 into master

* refs/pull/20168/head:
Handle dir->dentries.empty()
client: lookup_ino when i1 becomes NULL

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20424 into master

* refs/pull/20424/head:
qa: ignore io pause warnings in mds-full test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20428 into master

* refs/pull/20428/head:
qa: ignore MON_DOWN while thrashing mons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20470 into master

* refs/pull/20470/head:
qa: ignore full cluster warnings for mds-full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

include: compat.h, fix the return result of pthread_set_name()

On FreeBSD the function type is void(). Make it compatible with return
value on other platforms.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa: ignore full cluster warnings for mds-full

Fixes: http://tracker.ceph.com/issues/23033
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados/objectstore: increase open file limit

Signed-off-by: Sage Weil <sage@redhat.com>

os: do not inject read EIO on meta pool objects

Avoid EIO on, say, osdmaps until we fix
http://tracker.ceph.com/issues/23029

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20274 from adamemerson/wip-socket-to-me!

common/admin_socket: various cleanups

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test_filestore_idempotent_sequence: put txn object in each collection

The transactions are idependent in each collection/sequencer, so we
can't record to a single txn object with racing transactions.  Fix
it by doing one in each collection, and when reading the latest op,
use the highest txn value we see.

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: Use unique_ptr instead of new/delete

Mostly for the default hooks, plus one Formatter.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Cleanup path cleanup

Don't use strdup/free explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config/admin_socket: Remove PFL_* stuff since nobody's using it

If we want to, we have other ways of getting data out of a thread.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use one map instead of three

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

ceph_test_objectstore: fix split ordering

We need to flush between split.  This requirement unfortunately
doesn't quite go away with the FileStore tracking.

Also, flush for each batch.  This is just because the test environment
may have a low open file ulimit.  (The old code did apply_transaction,
so it's functionally equivalent to this.)

Signed-off-by: Sage Weil <sage@redhat.com>

common/admin_socket: Use std::mutex/condition_variable/thread

And fix a bug where we exit in one code-path without releasing the
lock.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_named_thread

To allow us to make a std::thread with a name.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Copy fewer strings

Pass std::string_view to call, rather than passing strings by value.

Change cmdmap_t and the maps in admin_socket to use the std::less<>
transparent comparator, so we can compare string_views to strings. (Or
anything else with an equality operator with strings.)

Pass the cmdmap_t in to call by const reference, not reference.

The change to cmdmap_t requires changing some of the users of
cmdmap_t. (That is, make them actually use cmdmap_t explicitly, not
the open-coed map type.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17020 from theanalyst/lc-time-disable

qa: re enable LC tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16757 from rzarzynski/wip-rgw-20883

rgw: dump Last-Modified in Swift's responses for GET/HEAD on container.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15054 from rzarzynski/wip-rgw-fix-browser-upload-error-handling

rgw: fix error handling in Browser Uploads.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12427 from linuxbox2/wip-rgw-ldap-shutdown

rgw: don't leak S3 LDAPHelper

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19065 from tianshan/fix_nfs_init

rgw: fix RGWLibIO did not init RGWEnv

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17094 from theanalyst/rgw-policy-tagging-2

rgw: add support for tagging and other conditionals in policy

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20226 from fangyuxiangGL/offset

rgw: can't download object with range when compression enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #20332 from pritha-srivastava/wip-rgw-role-admin-only

rgw: Correct permission evaluation to allow only admin users to work with Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20396 from tianshan/cancel_op_miss_update_header

rgw: fix index cancel op miss update header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20459 from ceph/wip-cv-lint-fix

ceph-volume tests remove unused import

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume tests remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

buffer: Append from std::string_view

Now that we have std::string_view.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20426 from batrick/i22991

mon/MgrStatMonitor: fix formatting of pending_digest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20409 from badone/wip-better-error-reporting-ceph_test_admin_socket_output

test/admin_socket_output: better error reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20370 from ovh/bp-no-normalizing-of-normalized

common/ConfUtils: check key before actually normalizing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20437 from tchaikov/wip-admin-socket

common/admin_socket: validate command json before feeding it to hook

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20408 from ceph/wip-rm22785

ceph-volume consume mount/format options from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/admin_socket: consolidate dump_cmd_to_json() and validate_cmd()

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: validate command json before feeding it to hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20158 from liaoxin01/fix-list-multi-upload

rgw: fix the max-uploads parameter not work
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20362 from tchaikov/wip-mgr-py3

mgr: fix py3 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20430 from tchaikov/wip-bluefs-demotion

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20410 from dzafman/wip-21218

osd: Fix assert when checking missing version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19322 from theanalyst/usage-trim-2

rgw: add an option to clear all usage entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19573 from Leeshine/wip-rgw-sync-status

rgw: improve sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19606 from cfanz/wip-rgw-fix-collect-return

rgw: fix collect()'s return in coroutine

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19815 from qrGitHub/wip-rgw-memory-optimize2

rgw: drop unused vector elements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19921 from fangyuxiangGL/lc-ret

rgw: lifecycle omap entry was removed in abnormal situation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20006 from jcsp/wip-rgw-perfcounters

rgw: set priority on perf counters

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20021 from tianshan/fix_bilog_list

cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20023 from badone/wip-rgw-better-ERANGE-error

rgw: Better ERANGE error message

Reviewed-by: <mkogan@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20024 from luomuyao/dev

rgw: fix a typo in rgw_perms[]

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd: Add new snapset_inconsistency error check

Includes new test case

Caused by: 5f58301a1364e948834dabe503200dda07fc2790
This changed attr consistency checking to exclude system keys,
which required snapset to be handled just like object info.

Fixes: http://tracker.ceph.com/issues/22996
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #20191 from yehudasa/wip-22833

rgw: sync tracing fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: <mkogan@redhat.com>

Merge pull request #20063 from cethikdata/rgw_quota

rgw: upldate the max-buckets when the quota is uploaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20095 from tianshan/fix_reload

rgw: fix RGWCompletionManager get_next stuck after going down

Reviewed-by: Casey Bodley <cbodley@redhat.com>