Merge pull request #20274 from adamemerson/wip-socket-to-me!

common/admin_socket: various cleanups

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: Use unique_ptr instead of new/delete

Mostly for the default hooks, plus one Formatter.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Cleanup path cleanup

Don't use strdup/free explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config/admin_socket: Remove PFL_* stuff since nobody's using it

If we want to, we have other ways of getting data out of a thread.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use one map instead of three

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use std::mutex/condition_variable/thread

And fix a bug where we exit in one code-path without releasing the
lock.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_named_thread

To allow us to make a std::thread with a name.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Copy fewer strings

Pass std::string_view to call, rather than passing strings by value.

Change cmdmap_t and the maps in admin_socket to use the std::less<>
transparent comparator, so we can compare string_views to strings. (Or
anything else with an equality operator with strings.)

Pass the cmdmap_t in to call by const reference, not reference.

The change to cmdmap_t requires changing some of the users of
cmdmap_t. (That is, make them actually use cmdmap_t explicitly, not
the open-coed map type.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17020 from theanalyst/lc-time-disable

qa: re enable LC tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16757 from rzarzynski/wip-rgw-20883

rgw: dump Last-Modified in Swift's responses for GET/HEAD on container.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15054 from rzarzynski/wip-rgw-fix-browser-upload-error-handling

rgw: fix error handling in Browser Uploads.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12427 from linuxbox2/wip-rgw-ldap-shutdown

rgw: don't leak S3 LDAPHelper

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19065 from tianshan/fix_nfs_init

rgw: fix RGWLibIO did not init RGWEnv

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17094 from theanalyst/rgw-policy-tagging-2

rgw: add support for tagging and other conditionals in policy

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20226 from fangyuxiangGL/offset

rgw: can't download object with range when compression enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #20332 from pritha-srivastava/wip-rgw-role-admin-only

rgw: Correct permission evaluation to allow only admin users to work with Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20396 from tianshan/cancel_op_miss_update_header

rgw: fix index cancel op miss update header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20459 from ceph/wip-cv-lint-fix

ceph-volume tests remove unused import

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume tests remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

buffer: Append from std::string_view

Now that we have std::string_view.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20426 from batrick/i22991

mon/MgrStatMonitor: fix formatting of pending_digest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20409 from badone/wip-better-error-reporting-ceph_test_admin_socket_output

test/admin_socket_output: better error reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20370 from ovh/bp-no-normalizing-of-normalized

common/ConfUtils: check key before actually normalizing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20437 from tchaikov/wip-admin-socket

common/admin_socket: validate command json before feeding it to hook

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20408 from ceph/wip-rm22785

ceph-volume consume mount/format options from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/admin_socket: consolidate dump_cmd_to_json() and validate_cmd()

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: validate command json before feeding it to hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20158 from liaoxin01/fix-list-multi-upload

rgw: fix the max-uploads parameter not work
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20362 from tchaikov/wip-mgr-py3

mgr: fix py3 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20430 from tchaikov/wip-bluefs-demotion

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20410 from dzafman/wip-21218

osd: Fix assert when checking missing version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19322 from theanalyst/usage-trim-2

rgw: add an option to clear all usage entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19573 from Leeshine/wip-rgw-sync-status

rgw: improve sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19606 from cfanz/wip-rgw-fix-collect-return

rgw: fix collect()'s return in coroutine

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19815 from qrGitHub/wip-rgw-memory-optimize2

rgw: drop unused vector elements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19921 from fangyuxiangGL/lc-ret

rgw: lifecycle omap entry was removed in abnormal situation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20006 from jcsp/wip-rgw-perfcounters

rgw: set priority on perf counters

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20021 from tianshan/fix_bilog_list

cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20023 from badone/wip-rgw-better-ERANGE-error

rgw: Better ERANGE error message

Reviewed-by: <mkogan@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20024 from luomuyao/dev

rgw: fix a typo in rgw_perms[]

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20191 from yehudasa/wip-22833

rgw: sync tracing fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: <mkogan@redhat.com>

Merge pull request #20063 from cethikdata/rgw_quota

rgw: upldate the max-buckets when the quota is uploaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20095 from tianshan/fix_reload

rgw: fix RGWCompletionManager get_next stuck after going down

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume util.prepare extend flags when found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.constants update mount flags to include rw, normalize on list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration normalize underscore vs. whitespace named options

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check for mount/format usage from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add ceph.conf stub for custom conf files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20423 from dillaman/wip-22945

librbd: new tag should use on-disk committed position

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20442 from JordanHus/master

doc: Fixes a spelling error and a broken hyperlink.

Reviewed-by: John Wilkins

doc: Fixes a spelling error and a broken hyperlink.

Signed-off-by: Jordan Hus <jordan.hus@dicecorp.com>

journal: flush commit positions should wait for refresh

Fixes: http://tracker.ceph.com/issues/22945
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support concurrent refresh requests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19859 from yaozongyou/rgw-remove-register-warning

rgw: silence not allow register storage class specifier warning

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20187 from wido/influxdb-connection-error

mgr/influx: Various fixes and improvements

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20265 from ceph/wip-rm22876

ceph-volume: lvm zap will unmount osd paths used by zapped devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: fix index cancel op miss update header

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #20406 from dillaman/wip-22979

librbd: fix issues discovered in clone v2 during upgrade tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

* we should update the _size field when demotion kicks in, otherwise the
  interval_set::m and interval_set::_size won't be consistent.
* also rename post_process to claim, as it "steals" the leftover(s) of
chopped segment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: use terminal.MultiLogger in `lvm zap`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use api helper methods for pvs and lvs in lvm zap

This is just a cleaner way of doing this instead of using PVolumes
and Volumes directly.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix documentation typos for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when zapping unmount osd directories

If you zap an lv, device or partition and it's currently mounted as a
ceph osd directory then it will be unmounted so the zap can complete.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a util.system.unmount function

This function will unmount the given path

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20427 from ceph/wip-rm22988

ceph-volume use realpath when checking mounts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #19263 into master

* refs/pull/19263/head:
qa: ignore bad backtrace cluster wrn
qa/cephfs: Add tests to validate scrub functionality
cephfs: Add option to load invalid metadata from disk
cephfs: Reset scrub data when inodes move

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore bad backtrace cluster wrn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests verify realpath usage in mounts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system optionally use realpath on devices

To check a mounted device it is needed to verify on a combination of
realpath and plain devices against realpath and plain paths. In LVM, two
different paths might refer to the same devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #19957 into master

* refs/pull/19957/head:
client: fixup parallel calls to ceph_ll_lookup_inode() in NFS FASL

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #20336 into master

* refs/pull/20336/head:
client: re-use free snap tag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20373 into master

* refs/pull/20373/head:
client: clamp I/O sizes to INT_MAX when we can't return larger values
test: new testcase for ceph_ll_readv and ceph_ll_writev
client: hook up ceph_ll_readv and ceph_ll_writev
client: type safety cleanup for _read and _write codepaths

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20386 into master

* refs/pull/20386/head:
mds: optimize getattr file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: Add tests to validate scrub functionality

Add tests to ensure the scrub operation is not adversly affected
by certain metadata pathologies.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

MgrStatMonitor: fix formatting of pending_digest

Fixes: http://tracker.ceph.com/issues/22991
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20147 from smithfarm/wip-15904

 tools: rados: make -f be --format for consistency with ceph tool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20422 from cbodley/wip-check-cls-rgw

cmake: 'make check' builds radosgw and its cls dependencies

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: 'make check' builds radosgw and its cls dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: flush journal commit position after stopping replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: disable clone v2 for negative test cases

Fixes: http://tracker.ceph.com/issues/22979
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: don't clear CLONE_CHILD op feature if still a child

After removing the last snapshot linked to a parent image,
don't clear the CLONE_CHILD op feature bit if the image HEAD
is still linked to the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: Fix assert when checking missing version

Fixes: http://tracker.ceph.com/issues/21218
Signed-off-by: David Zafman <dzafman@redhat.com>

test/admin_socket_output: better error reporting

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19939 from liewegas/wip-osd-new-class

mon,osd: do not use crush_device_class file to initalize class for new osds

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20363 from dongbula/e-rados-get-put-command

make rados get/put/append command help txt clear

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

client: clamp I/O sizes to INT_MAX when we can't return larger values

We have several API functions that allow the caller to request I/Os
larger than INT_MAX bytes, but that return an int. Ensure that we don't
try to do more I/O than we can represent in the return value.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: new testcase for ceph_ll_readv and ceph_ll_writev

Just copy the PreadvPwritev test and alter accordingly.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: hook up ceph_ll_readv and ceph_ll_writev

Break the core of _preadv_pwritev out into a function that takes a Fh.
Make _preadv_pwritev into a wrapper around that.

Then add in plumbing for ceph_ll_readv and ceph_ll_writev.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: type safety cleanup for _read and _write codepaths

In the read codepath, bl->length() returns an unsigned value, and that
could end up looking negative when cast to int. On the write side,
totalwritten is a uint64_t, which could look negative when cast to int.

Have the underlying layers pass back an int64_t and convert them to
int at a higher level. This prepares the underlying infrastructure for
ceph_ll_readv and ceph_ll_writev support.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

tools/rados: make rados get/put/append command help txt clearer

Fixes: http://tracker.ceph.com/issues/22958
Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

Merge pull request #20392 from adamemerson/wip-clangtastic-voyage

cmake, test/fs, client: fix build with clang

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20379 from tchaikov/wip-test-fix-freebsd

test/librados: s/invoke_result_t/result_of_t/

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #20360 from ifed01/wip-ifed-extents-cleanup

os/bluestore: cleanup around ExtentList, AllocExtent and bluestore_extent_t classes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20320 from rzarzynski/wip-common-md_config_cacher_t

common: introduce md_config_cacher_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20198 from wido/zabbix-health-checks

mgr/zabbix: Implement health checks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: correct the log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: various py3 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add comma in DASH_URLS and RESTFUL_URLS

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: import absolute_import to enforce absolute_import

to make sure new "import" statements in future in these files do not
assume relative import.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: move global variable into context.py

otherwise we need to tackle with the circular import. it's not fatal, if
we can "import foo" in py2 and "from . import foo" in py3, respectively.
but we cannot unify them using "from . import foo", because the circular
reference issue. and conditionalize the import statement is ugly. so, i
think we'd better break the loop. so i am moving the "instance" variable
into its own file.

Signed-off-by: Kefu Chai <kchai@redhat.com>