qa/tasks/mgr: correct the log message

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: various py3 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add comma in DASH_URLS and RESTFUL_URLS

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: import absolute_import to enforce absolute_import

to make sure new "import" statements in future in these files do not
assume relative import.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/restful: move global variable into context.py

otherwise we need to tackle with the circular import. it's not fatal, if
we can "import foo" in py2 and "from . import foo" in py3, respectively.
but we cannot unify them using "from . import foo", because the circular
reference issue. and conditionalize the import statement is ugly. so, i
think we'd better break the loop. so i am moving the "instance" variable
into its own file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

restful: various py3 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: depend on the right version of python bindings

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: set PYTHONPATH correctly

it varies depending on the major version of the python we are using.

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: introduce get_cmake_variable()

Signed-off-by: Kefu Chai <kchai@redhat.com>

init-ceph: s/lib.2/lib.$MGR_PYTHON_VERSION_MAJOR/

we should not hardwire cython_modules's path to lib.2, it should be
cython_modules/lib.3 if ceph-mgr requires python3.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: load ceph_{logger,module} globally

no need to add them for each sub-interpreter. extensions are shared
among them.

Fixes: http://tracker.ceph.com/issues/22880
Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: use relative import

* in python3, we cannot assume default relative import. instead, we should
  be explicit. also, only import the `Module` and `StandbyModule` classes
  from plugins, as they are what we are interested.
* and import urllib.parse in python3.

Fixes: http://tracker.ceph.com/issues/22880
Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: fix the TabError error

018-02-08 18:50:10.790 7fcfbb679700 -1 mgr[py] Module not found:
'dashboard'
2018-02-08 18:50:10.790 7fcfbb679700 -1 mgr[py] Traceback (most recent
call last):
  File "/var/ceph/ceph/src/pybind/mgr/dashboard/__init__.py", line 2, in
<module>
    from . module import Module  # NOQA
  File "/var/ceph/ceph/src/pybind/mgr/dashboard/module.py", line 1189
    template = env.get_template("rgw_detail.html")
                                                 ^
TabError: inconsistent use of tabs and spaces in indentation

Fixes: http://tracker.ceph.com/issues/22880
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20378 from tchaikov/wip-os-perf-stat-in-ns

osd: check feature bits when encoding objectstore_perf_stat_t

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20275 from badone/wip-osdmonitor-unused-function

mon/OSDMonitor: Comment out unused function

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20371 from tchaikov/wip-admin_socket_output

test/admin_socket_output: add --vstart=path/to/asok option

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20227 from rzarzynski/wip-bs-final-aiocontexts

os/bluestore: mark derivatives of AioContext as final.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20376 from dillaman/wip-22873

librbd: auto-remove trash snapshots when image is deleted

Reviewed-by: Mykola Golub <mgolub@suse.com>

librbd: auto-remove trash snapshots when image is deleted

Fixes: http://tracker.ceph.com/issues/22873
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20387 from ifed01/wip-ifed-fio-build-fix

test/fio: fix build failure caused by sequencer replacement.

Merge pull request #20043 from Rubab-Syed/configuration_setting_browser

mgr/dashboard: add configuration setting browser

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20389 from trociny/wip-cli_generic

qa/workunits/rbd: fix cli_generic test_purge for rbd default format 1

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd/cli_generic: fix test_purge for rbd default format 1

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20364 from Songweibin/wip-fix-test-rbd

test/rbd: cli_generic fails if v1 image format or deep-flatten disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

tools/ceph-objectstore-tool: do not fail on lack of OI_ATTR for an
object.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/fio: create superblock on mkfs to provide proper tools functioning

E.g. ceph-objectstore-tool requires that

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/fio: fix build failure caused by sequencer replacement.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20372 from tchaikov/wip-mgr-doc

doc/mgr: remove non user-facing code from doc

Reviewed-by: John Spray <john.spray@redhat.com>

test/rbd: cli_generic fails if v1 image format or deep-flatten disabled

Fixes: http://tracker.ceph.com/issues/22950
Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #20382 from tchaikov/wip-fix-ftbfs-store-test

test/store_test: fix FTBFS as Sequencer is removed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

check feature bits when encoding objectstore_perf_stat_t

OS_PERF_STAT_NS feature bit is added. it's sharing the same bit with
QOS_DMC.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/store_test: fix FTBFS as Sequencer is removed

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20294 from rzarzynski/wip-bs-drop-std_function

os/bluestore: avoid overhead of std::function in blob_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19232 from socketpair/precision

mgr: increase time resolution of Commit/Apply OSD latencies.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18804 from majianpeng/bluestore-collection-prealloc

os/bluestore: Prealloc memory avoid realloc in list_collection.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18494 from ifed01/wip-stupidalloc-fix2

os/bluestore: do not assert if BlueFS rebalance is unable to allocate sufficient space

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18343 from shinobu-x/sk-remove-osdmap

mon/PGMap: Remove unnecessary header

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #19954 into master

* refs/pull/19954/head:
test/encoding: refactor to avoid escaping shell magic
mds: minor refactor of SimpleLock
mds: track Capability in mempool
mds: move CInode container members to mempool
mds: move CDentry container members to mempool
mds: move CDir container members to mempool
mds: put MDSCacheObject compact_map in mempool
common: use size_t for object size
mds: convert to allocator agnostic string_view
mds: simplify initialization
compact_*: support mempool allocated containers

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #20007 from mogeb/steady-clock-librbd

librbd: use steady clock to measure elapsed time in AioCompletion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20008 from mogeb/steady-clock-tools-rbd

tools/rbd: use steady clock in bencher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20218 from shun-s/wip-speedup-diskusage-resize

librbd: speed up object map disk usage and resize

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20311 from Songweibin/wip-group-snap-ls

rbd: do not show title if there is no group snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20349 from trociny/wip-22932

rbd-mirror: fix potential infinite loop when formatting status message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #20310 into master

* refs/pull/20310/head:
qa: adjust cephfs full test for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20148 into master

* refs/pull/20148/head:
mds: reset connection's priv when marking down connection
mds: fix session reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20155 into master

* refs/pull/20155/head:
osdc/Journaler: make sure flush() writes enough data

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20190 into master

* refs/pull/20190/head:
mon: allow removal of tier of ec overwritable pool

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #20200 into master

* refs/pull/20200/head:
client: add cap_dirtier_uid/gid to CapSnap

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20246 into master

* refs/pull/20246/head:
mds: remove extra 0x in ino prints
mds: print inode number not CInode ptr

doc/mgr: remove none user-facing code from doc

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/test_admin_socket_output: allow different asok directory

the asok files are not necessary located in ./out, actually, they are
put in $TMPDIR/ceph-asok.$random_stuff .

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/admin_socket_output: no need to create a copy of fs::path

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20367 from ceph/simple-custom-cluster

ceph-volume: adds custom cluster name support to simple

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: use a custom cluster name in simple functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20305 from xiexingguo/wip-more-balancer-fixes

pybind/mgr/balancer: more specific command outputs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20355 from luomuyao/typo

rgw: fix a typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

test/encoding: refactor to avoid escaping shell magic

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: minor refactor of SimpleLock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: track Capability in mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move CInode container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: avoid overhead of std::function in blob_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

mon/OSDMonitor: Remove unused functions

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20161 from tchaikov/wip-20086

test/librados: be more tolerant with timed lock tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20173 from liewegas/wip-os-ch

os: unify Sequencer and CollectionHandle

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: set conf.cluster name during simple activate

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20144 from adamemerson/wip-cache-register

Cache Register!

Provide the ability to examine and delete elements from the cache.

rgw: Add cache introspection and manipulation

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20176 from dillaman/wip-librbd-clone-v2-1

librbd: initial hooks for clone v2 support

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add cache introspection and manipulation

Provide the ability to examine and delete elements from the cache.

Fixes: http://tracker.ceph.com/issues/22603
Fixes: http://tracker.ceph.com/issues/22604
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd-mirror: fix potential infinite loop when formatting status message

The improvements include:

- tag_tid values should always be increasing, so loop only if
  master.tag_tid > mirror_tag_tid in calculate_behind_master_or_send_update;
- in send_update_tag_cache don't refetch a tag if it is already in the
  cache;
- make fake tags with tag_data.predecessor.tag_tid set to zero;
- make sure the new tag is inserted to the cache if an old entry
  with this id happens to exist.

Fixes: http://tracker.ceph.com/issues/22932
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20318 from ovh/bp-limit-normalize-key

common/config: limit calls to normalize_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19958 from renhwztetecs/renhw-wip-osd-sighup

osd: Sign in early SIGHUP signal

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: move CDentry container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move CDir container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: put MDSCacheObject compact_map in mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use size_t for object size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: convert to allocator agnostic string_view

This is necessary to allow many interfaces to take mempool allocated strings.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20264 from ceph/wip-rm22620

ceph-volume dmcrypt support for simple

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20307 from tchaikov/wip-std-filesystem

test/admin_socket_output: switch to std::experimental::filesystem

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20324 from tchaikov/wip-cmake-build-boost

cmake: fix the include dir for building boost::python

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

pybind/mgr/balancer: cancel plan if distribution is already perfect

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: allow customized optimization threshold

In practice it is hard to achieve a 100% (perfect) balanced distribution.
This patch introduces a user visiable threshold configuration here, which
allow user to define the minimal deviation to trigger an optimization.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: be more specific on creating an optimization plan

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20347 from Songweibin/wip-osd-cleanup

osd: drop redundant comment

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: drop redundant comment

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge PR #20288 into master

* refs/pull/20288/head:
ptl-tool: open githubmap with utf-8 code

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: remove extra 0x in ino prints

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/objectstore/TestObjectStoreState: make objects match collection

- start pool at 1
- make hash fall within pg (just match it for now)

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: remove void *osr from Transaction

No longer used!  (It was only needed for an ugly FileStore hack.)

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: do not rely on ObjectStore::Transaction::osr

This was used only to name the osr for the purposes of the tracepoint. We
can do that ourselves!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop explicit calls to osr->flush

These are no longer needed as the implementations do it themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop unnecessary flush() on shutdown

The ObjectStore implementation should handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os: make implementation flush before collection_{list,empty}

This takes the (awkward) burden off the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: make all read operations CollectionHandle-based

Drop the coll_t shims, and convert the last few that weren't already
handle-based.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #20313 from guzhongyan/fix-22350

osd/OSDMap: misleading message in print_oneline_summary()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20317 from liewegas/wip-pg-scrub-priority

osd/PG: pass scrub priority to replica

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20319 from tchaikov/wip-ops_blocked_by_scrub-cleanup

osd: pass ops_blocked_by_scrub() to requeue_scrub()

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: do not assert on insufficient allocation during bluefs
rebalance

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/store_test: add test case to for excessive fragmentation

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: fix lack of extent demotion in StupidAllocator::init_rm_free

Signed-off-by: Igor Fedotov <ifedotov@suse.com>