Merge pull request #20426 from batrick/i22991

mon/MgrStatMonitor: fix formatting of pending_digest

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20409 from badone/wip-better-error-reporting-ceph_test_admin_socket_output

test/admin_socket_output: better error reporting

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20370 from ovh/bp-no-normalizing-of-normalized

common/ConfUtils: check key before actually normalizing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20437 from tchaikov/wip-admin-socket

common/admin_socket: validate command json before feeding it to hook

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #20408 from ceph/wip-rm22785

ceph-volume consume mount/format options from ceph.conf

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/admin_socket: consolidate dump_cmd_to_json() and validate_cmd()

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/admin_socket: validate command json before feeding it to hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20158 from liaoxin01/fix-list-multi-upload

rgw: fix the max-uploads parameter not work
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20362 from tchaikov/wip-mgr-py3

mgr: fix py3 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20430 from tchaikov/wip-bluefs-demotion

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20410 from dzafman/wip-21218

osd: Fix assert when checking missing version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19322 from theanalyst/usage-trim-2

rgw: add an option to clear all usage entries

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19573 from Leeshine/wip-rgw-sync-status

rgw: improve sync status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19606 from cfanz/wip-rgw-fix-collect-return

rgw: fix collect()'s return in coroutine

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #19815 from qrGitHub/wip-rgw-memory-optimize2

rgw: drop unused vector elements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19921 from fangyuxiangGL/lc-ret

rgw: lifecycle omap entry was removed in abnormal situation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #20006 from jcsp/wip-rgw-perfcounters

rgw: set priority on perf counters

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20021 from tianshan/fix_bilog_list

cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20023 from badone/wip-rgw-better-ERANGE-error

rgw: Better ERANGE error message

Reviewed-by: <mkogan@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20024 from luomuyao/dev

rgw: fix a typo in rgw_perms[]

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20191 from yehudasa/wip-22833

rgw: sync tracing fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: <mkogan@redhat.com>

Merge pull request #20063 from cethikdata/rgw_quota

rgw: upldate the max-buckets when the quota is uploaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20095 from tianshan/fix_reload

rgw: fix RGWCompletionManager get_next stuck after going down

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume util.prepare extend flags when found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.constants update mount flags to include rw, normalize on list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume configuration normalize underscore vs. whitespace named options

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests check for mount/format usage from ceph.conf

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add ceph.conf stub for custom conf files

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20423 from dillaman/wip-22945

librbd: new tag should use on-disk committed position

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20442 from JordanHus/master

doc: Fixes a spelling error and a broken hyperlink.

Reviewed-by: John Wilkins

doc: Fixes a spelling error and a broken hyperlink.

Signed-off-by: Jordan Hus <jordan.hus@dicecorp.com>

journal: flush commit positions should wait for refresh

Fixes: http://tracker.ceph.com/issues/22945
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support concurrent refresh requests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19859 from yaozongyou/rgw-remove-register-warning

rgw: silence not allow register storage class specifier warning

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20187 from wido/influxdb-connection-error

mgr/influx: Various fixes and improvements

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20265 from ceph/wip-rm22876

ceph-volume: lvm zap will unmount osd paths used by zapped devices

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20406 from dillaman/wip-22979

librbd: fix issues discovered in clone v2 during upgrade tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

os/bluestore: fix the demotion in StupidAllocator::init_rm_free

* we should update the _size field when demotion kicks in, otherwise the
  interval_set::m and interval_set::_size won't be consistent.
* also rename post_process to claim, as it "steals" the leftover(s) of
chopped segment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: use terminal.MultiLogger in `lvm zap`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: use api helper methods for pvs and lvs in lvm zap

This is just a cleaner way of doing this instead of using PVolumes
and Volumes directly.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: fix documentation typos for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: when zapping unmount osd directories

If you zap an lv, device or partition and it's currently mounted as a
ceph osd directory then it will be unmounted so the zap can complete.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: adds a util.system.unmount function

This function will unmount the given path

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20427 from ceph/wip-rm22988

ceph-volume use realpath when checking mounts

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #19263 into master

* refs/pull/19263/head:
qa: ignore bad backtrace cluster wrn
qa/cephfs: Add tests to validate scrub functionality
cephfs: Add option to load invalid metadata from disk
cephfs: Reset scrub data when inodes move

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore bad backtrace cluster wrn

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume tests verify realpath usage in mounts

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.system optionally use realpath on devices

To check a mounted device it is needed to verify on a combination of
realpath and plain devices against realpath and plain paths. In LVM, two
different paths might refer to the same devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #19957 into master

* refs/pull/19957/head:
client: fixup parallel calls to ceph_ll_lookup_inode() in NFS FASL

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #20336 into master

* refs/pull/20336/head:
client: re-use free snap tag

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20373 into master

* refs/pull/20373/head:
client: clamp I/O sizes to INT_MAX when we can't return larger values
test: new testcase for ceph_ll_readv and ceph_ll_writev
client: hook up ceph_ll_readv and ceph_ll_writev
client: type safety cleanup for _read and _write codepaths

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20386 into master

* refs/pull/20386/head:
mds: optimize getattr file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: Add tests to validate scrub functionality

Add tests to ensure the scrub operation is not adversly affected
by certain metadata pathologies.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

MgrStatMonitor: fix formatting of pending_digest

Fixes: http://tracker.ceph.com/issues/22991
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20147 from smithfarm/wip-15904

 tools: rados: make -f be --format for consistency with ceph tool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #20422 from cbodley/wip-check-cls-rgw

cmake: 'make check' builds radosgw and its cls dependencies

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: 'make check' builds radosgw and its cls dependencies

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: flush journal commit position after stopping replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: disable clone v2 for negative test cases

Fixes: http://tracker.ceph.com/issues/22979
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: don't clear CLONE_CHILD op feature if still a child

After removing the last snapshot linked to a parent image,
don't clear the CLONE_CHILD op feature bit if the image HEAD
is still linked to the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: Fix assert when checking missing version

Fixes: http://tracker.ceph.com/issues/21218
Signed-off-by: David Zafman <dzafman@redhat.com>

test/admin_socket_output: better error reporting

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19939 from liewegas/wip-osd-new-class

mon,osd: do not use crush_device_class file to initalize class for new osds

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20363 from dongbula/e-rados-get-put-command

make rados get/put/append command help txt clear

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

client: clamp I/O sizes to INT_MAX when we can't return larger values

We have several API functions that allow the caller to request I/Os
larger than INT_MAX bytes, but that return an int. Ensure that we don't
try to do more I/O than we can represent in the return value.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: new testcase for ceph_ll_readv and ceph_ll_writev

Just copy the PreadvPwritev test and alter accordingly.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: hook up ceph_ll_readv and ceph_ll_writev

Break the core of _preadv_pwritev out into a function that takes a Fh.
Make _preadv_pwritev into a wrapper around that.

Then add in plumbing for ceph_ll_readv and ceph_ll_writev.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: type safety cleanup for _read and _write codepaths

In the read codepath, bl->length() returns an unsigned value, and that
could end up looking negative when cast to int. On the write side,
totalwritten is a uint64_t, which could look negative when cast to int.

Have the underlying layers pass back an int64_t and convert them to
int at a higher level. This prepares the underlying infrastructure for
ceph_ll_readv and ceph_ll_writev support.

Tracker: http://tracker.ceph.com/issues/22948
Signed-off-by: Jeff Layton <jlayton@redhat.com>

tools/rados: make rados get/put/append command help txt clearer

Fixes: http://tracker.ceph.com/issues/22958
Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

Merge pull request #20392 from adamemerson/wip-clangtastic-voyage

cmake, test/fs, client: fix build with clang

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20379 from tchaikov/wip-test-fix-freebsd

test/librados: s/invoke_result_t/result_of_t/

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #20360 from ifed01/wip-ifed-extents-cleanup

os/bluestore: cleanup around ExtentList, AllocExtent and bluestore_extent_t classes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20320 from rzarzynski/wip-common-md_config_cacher_t

common: introduce md_config_cacher_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20198 from wido/zabbix-health-checks

mgr/zabbix: Implement health checks

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: correct the log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: various py3 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add comma in DASH_URLS and RESTFUL_URLS

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: import absolute_import to enforce absolute_import

to make sure new "import" statements in future in these files do not
assume relative import.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: move global variable into context.py

otherwise we need to tackle with the circular import. it's not fatal, if
we can "import foo" in py2 and "from . import foo" in py3, respectively.
but we cannot unify them using "from . import foo", because the circular
reference issue. and conditionalize the import statement is ugly. so, i
think we'd better break the loop. so i am moving the "instance" variable
into its own file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

restful: various py3 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: depend on the right version of python bindings

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: set PYTHONPATH correctly

it varies depending on the major version of the python we are using.

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: introduce get_cmake_variable()

Signed-off-by: Kefu Chai <kchai@redhat.com>

init-ceph: s/lib.2/lib.$MGR_PYTHON_VERSION_MAJOR/

we should not hardwire cython_modules's path to lib.2, it should be
cython_modules/lib.3 if ceph-mgr requires python3.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: load ceph_{logger,module} globally

no need to add them for each sub-interpreter. extensions are shared
among them.

Fixes: http://tracker.ceph.com/issues/22880
Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: use relative import

* in python3, we cannot assume default relative import. instead, we should
  be explicit. also, only import the `Module` and `StandbyModule` classes
  from plugins, as they are what we are interested.
* and import urllib.parse in python3.

Fixes: http://tracker.ceph.com/issues/22880
Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: fix the TabError error

018-02-08 18:50:10.790 7fcfbb679700 -1 mgr[py] Module not found:
'dashboard'
2018-02-08 18:50:10.790 7fcfbb679700 -1 mgr[py] Traceback (most recent
call last):
  File "/var/ceph/ceph/src/pybind/mgr/dashboard/__init__.py", line 2, in
<module>
    from . module import Module  # NOQA
  File "/var/ceph/ceph/src/pybind/mgr/dashboard/module.py", line 1189
    template = env.get_template("rgw_detail.html")
                                                 ^
TabError: inconsistent use of tabs and spaces in indentation

Fixes: http://tracker.ceph.com/issues/22880
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20378 from tchaikov/wip-os-perf-stat-in-ns

osd: check feature bits when encoding objectstore_perf_stat_t

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20275 from badone/wip-osdmonitor-unused-function

mon/OSDMonitor: Comment out unused function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20371 from tchaikov/wip-admin_socket_output

test/admin_socket_output: add --vstart=path/to/asok option

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20227 from rzarzynski/wip-bs-final-aiocontexts

os/bluestore: mark derivatives of AioContext as final.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20376 from dillaman/wip-22873

librbd: auto-remove trash snapshots when image is deleted

Reviewed-by: Mykola Golub <mgolub@suse.com>

build: Use the right experimental library

Test whether we're building with libstdc++ or libc++.

Use stdc++filesystem if the former and c++experimental if the later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{client,test}: Explicitly initialize empty inode_t

Using an explicit {} initializer. This unbreaks Clang.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

librbd: auto-remove trash snapshots when image is deleted

Fixes: http://tracker.ceph.com/issues/22873
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados: s/invoke_result_t/result_of_t/

to workaround buggy libc++ implementation. see
https://cplusplus.github.io/LWG/issue2807 . we should switch to
invoke_result_t once issue2807 is fixed in the latest release of
libc++.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: cleanup around ExtentList class.

Also move it out of bluestore_types.h.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: kill AllocExtent[Vector] to simplify code a bit

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20387 from ifed01/wip-ifed-fio-build-fix

test/fio: fix build failure caused by sequencer replacement.

Merge pull request #20043 from Rubab-Syed/configuration_setting_browser

mgr/dashboard: add configuration setting browser

Reviewed-by: John Spray <john.spray@redhat.com>