mds: fix check of underwater dentries

Underwater dentry is dentry that is dirty in our cache from journal
replay, but had already been flushed to disk before the mds failed.
To decide if an dentry is underwater, original code compares dirty
dentry's version to on-disk dirfrag's version. This method is racy
because CDir::log_mark_dirty() can increase dirfrag's version without
adding log event. After mds failover, version of dirfrag from journal
replay can be less than on-disk dirfrag's version. So newly dirtied
dentry can be equal to or less than the on-disk dirfrag's version.

The race can cause incorrect fragstat/rstat

Fixes: http://tracker.ceph.com/issues/23032
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #20274 from adamemerson/wip-socket-to-me!

common/admin_socket: various cleanups

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: Use unique_ptr instead of new/delete

Mostly for the default hooks, plus one Formatter.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Cleanup path cleanup

Don't use strdup/free explicitly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

config/admin_socket: Remove PFL_* stuff since nobody's using it

If we want to, we have other ways of getting data out of a thread.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use one map instead of three

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Use std::mutex/condition_variable/thread

And fix a bug where we exit in one code-path without releasing the
lock.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add make_named_thread

To allow us to make a std::thread with a name.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common/admin_socket: Copy fewer strings

Pass std::string_view to call, rather than passing strings by value.

Change cmdmap_t and the maps in admin_socket to use the std::less<>
transparent comparator, so we can compare string_views to strings. (Or
anything else with an equality operator with strings.)

Pass the cmdmap_t in to call by const reference, not reference.

The change to cmdmap_t requires changing some of the users of
cmdmap_t. (That is, make them actually use cmdmap_t explicitly, not
the open-coed map type.)

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #17020 from theanalyst/lc-time-disable

qa: re enable LC tests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #16757 from rzarzynski/wip-rgw-20883

rgw: dump Last-Modified in Swift's responses for GET/HEAD on container.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #15054 from rzarzynski/wip-rgw-fix-browser-upload-error-handling

rgw: fix error handling in Browser Uploads.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12427 from linuxbox2/wip-rgw-ldap-shutdown

rgw: don't leak S3 LDAPHelper

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19065 from tianshan/fix_nfs_init

rgw: fix RGWLibIO did not init RGWEnv

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #17094 from theanalyst/rgw-policy-tagging-2

rgw: add support for tagging and other conditionals in policy

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20226 from fangyuxiangGL/offset

rgw: can't download object with range when compression enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #20332 from pritha-srivastava/wip-rgw-role-admin-only

rgw: Correct permission evaluation to allow only admin users to work with Roles.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20396 from tianshan/cancel_op_miss_update_header

rgw: fix index cancel op miss update header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20459 from ceph/wip-cv-lint-fix

ceph-volume tests remove unused import

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume tests remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

buffer: Append from std::string_view

Now that we have std::string_view.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20426 from batrick/i22991

mon/MgrStatMonitor: fix formatting of pending_digest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20409 from badone/wip-better-error-reporting-ceph_test_admin_socket_output

test/admin_socket_output: better error reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20370 from ovh/bp-no-normalizing-of-normalized

common/ConfUtils: check key before actually normalizing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20437 from tchaikov/wip-admin-socket

common/admin_socket: validate command json before feeding it to hook

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20408 from ceph/wip-rm22785

ceph-volume consume mount/format options from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/admin_socket: consolidate dump_cmd_to_json() and validate_cmd()

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: validate command json before feeding it to hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20158 from liaoxin01/fix-list-multi-upload

rgw: fix the max-uploads parameter not work
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20362 from tchaikov/wip-mgr-py3

mgr: fix py3 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20430 from tchaikov/wip-bluefs-demotion

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20410 from dzafman/wip-21218

osd: Fix assert when checking missing version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19322 from theanalyst/usage-trim-2

rgw: add an option to clear all usage entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19573 from Leeshine/wip-rgw-sync-status

rgw: improve sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19606 from cfanz/wip-rgw-fix-collect-return

rgw: fix collect()'s return in coroutine

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19815 from qrGitHub/wip-rgw-memory-optimize2

rgw: drop unused vector elements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19921 from fangyuxiangGL/lc-ret

rgw: lifecycle omap entry was removed in abnormal situation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20006 from jcsp/wip-rgw-perfcounters

rgw: set priority on perf counters

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20021 from tianshan/fix_bilog_list

cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20023 from badone/wip-rgw-better-ERANGE-error

rgw: Better ERANGE error message

Reviewed-by: <mkogan@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20024 from luomuyao/dev

rgw: fix a typo in rgw_perms[]

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20191 from yehudasa/wip-22833

rgw: sync tracing fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: <mkogan@redhat.com>

Merge pull request #20063 from cethikdata/rgw_quota

rgw: upldate the max-buckets when the quota is uploaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20095 from tianshan/fix_reload

rgw: fix RGWCompletionManager get_next stuck after going down

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume util.prepare extend flags when found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.constants update mount flags to include rw, normalize on list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration normalize underscore vs. whitespace named options

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check for mount/format usage from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add ceph.conf stub for custom conf files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20423 from dillaman/wip-22945

librbd: new tag should use on-disk committed position

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20442 from JordanHus/master

doc: Fixes a spelling error and a broken hyperlink.

Reviewed-by: John Wilkins

doc: Fixes a spelling error and a broken hyperlink.

Signed-off-by: Jordan Hus <jordan.hus@dicecorp.com>

journal: flush commit positions should wait for refresh

Fixes: http://tracker.ceph.com/issues/22945
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support concurrent refresh requests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19859 from yaozongyou/rgw-remove-register-warning

rgw: silence not allow register storage class specifier warning

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20187 from wido/influxdb-connection-error

mgr/influx: Various fixes and improvements

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20265 from ceph/wip-rm22876

ceph-volume: lvm zap will unmount osd paths used by zapped devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: fix index cancel op miss update header

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #20406 from dillaman/wip-22979

librbd: fix issues discovered in clone v2 during upgrade tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

* we should update the _size field when demotion kicks in, otherwise the
  interval_set::m and interval_set::_size won't be consistent.
* also rename post_process to claim, as it "steals" the leftover(s) of
chopped segment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: use terminal.MultiLogger in `lvm zap`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use api helper methods for pvs and lvs in lvm zap

This is just a cleaner way of doing this instead of using PVolumes
and Volumes directly.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix documentation typos for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when zapping unmount osd directories

If you zap an lv, device or partition and it's currently mounted as a
ceph osd directory then it will be unmounted so the zap can complete.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a util.system.unmount function

This function will unmount the given path

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20427 from ceph/wip-rm22988

ceph-volume use realpath when checking mounts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #19263 into master

* refs/pull/19263/head:
qa: ignore bad backtrace cluster wrn
qa/cephfs: Add tests to validate scrub functionality
cephfs: Add option to load invalid metadata from disk
cephfs: Reset scrub data when inodes move

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore bad backtrace cluster wrn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests verify realpath usage in mounts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system optionally use realpath on devices

To check a mounted device it is needed to verify on a combination of
realpath and plain devices against realpath and plain paths. In LVM, two
different paths might refer to the same devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #19957 into master

* refs/pull/19957/head:
client: fixup parallel calls to ceph_ll_lookup_inode() in NFS FASL

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #20336 into master

* refs/pull/20336/head:
client: re-use free snap tag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20373 into master

* refs/pull/20373/head:
client: clamp I/O sizes to INT_MAX when we can't return larger values
test: new testcase for ceph_ll_readv and ceph_ll_writev
client: hook up ceph_ll_readv and ceph_ll_writev
client: type safety cleanup for _read and _write codepaths

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20386 into master

* refs/pull/20386/head:
mds: optimize getattr file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: Add tests to validate scrub functionality

Add tests to ensure the scrub operation is not adversly affected
by certain metadata pathologies.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

MgrStatMonitor: fix formatting of pending_digest

Fixes: http://tracker.ceph.com/issues/22991
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20147 from smithfarm/wip-15904

 tools: rados: make -f be --format for consistency with ceph tool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20422 from cbodley/wip-check-cls-rgw

cmake: 'make check' builds radosgw and its cls dependencies

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: 'make check' builds radosgw and its cls dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: flush journal commit position after stopping replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: disable clone v2 for negative test cases

Fixes: http://tracker.ceph.com/issues/22979
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: don't clear CLONE_CHILD op feature if still a child

After removing the last snapshot linked to a parent image,
don't clear the CLONE_CHILD op feature bit if the image HEAD
is still linked to the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: Fix assert when checking missing version

Fixes: http://tracker.ceph.com/issues/21218
Signed-off-by: David Zafman <dzafman@redhat.com>

test/admin_socket_output: better error reporting

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19939 from liewegas/wip-osd-new-class

mon,osd: do not use crush_device_class file to initalize class for new osds

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20363 from dongbula/e-rados-get-put-command

make rados get/put/append command help txt clear

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

client: clamp I/O sizes to INT_MAX when we can't return larger values

We have several API functions that allow the caller to request I/Os
larger than INT_MAX bytes, but that return an int. Ensure that we don't
try to do more I/O than we can represent in the return value.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: new testcase for ceph_ll_readv and ceph_ll_writev

Just copy the PreadvPwritev test and alter accordingly.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: hook up ceph_ll_readv and ceph_ll_writev

Break the core of _preadv_pwritev out into a function that takes a Fh.
Make _preadv_pwritev into a wrapper around that.

Then add in plumbing for ceph_ll_readv and ceph_ll_writev.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: type safety cleanup for _read and _write codepaths

In the read codepath, bl->length() returns an unsigned value, and that
could end up looking negative when cast to int. On the write side,
totalwritten is a uint64_t, which could look negative when cast to int.

Have the underlying layers pass back an int64_t and convert them to
int at a higher level. This prepares the underlying infrastructure for
ceph_ll_readv and ceph_ll_writev support.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

tools/rados: make rados get/put/append command help txt clearer

Fixes: http://tracker.ceph.com/issues/22958
Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

Merge pull request #20392 from adamemerson/wip-clangtastic-voyage

cmake, test/fs, client: fix build with clang

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20379 from tchaikov/wip-test-fix-freebsd

test/librados: s/invoke_result_t/result_of_t/

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #20360 from ifed01/wip-ifed-extents-cleanup

os/bluestore: cleanup around ExtentList, AllocExtent and bluestore_extent_t classes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20320 from rzarzynski/wip-common-md_config_cacher_t

common: introduce md_config_cacher_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20198 from wido/zabbix-health-checks

mgr/zabbix: Implement health checks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: correct the log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: various py3 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add comma in DASH_URLS and RESTFUL_URLS

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: import absolute_import to enforce absolute_import

to make sure new "import" statements in future in these files do not
assume relative import.

Signed-off-by: Kefu Chai <kchai@redhat.com>