Merge pull request #20660 from Leeshine/wip-mon-op-tracker

mon,mgr: make  osd_metric more popular and report slow ops to mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21057 from tchaikov/wip-22114

mon: mark mgr reports as no_reply

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #20976 from sebastian-philipp/dashboard-restcontroller-default-get

mgr/dashboard: Fix objects named `default` are inaccessible

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #20870 from rjfd/wip-dashboard-tasks

mgr/dashboard: asynchronous task support

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20884 from zouaiguo/wip_for_mgr_rbd_info

mgr/dashboard: add image id to mgr rbd info instead of block_name_prefix

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr: summarize mon metrics in MMgrReport and sent it

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

mon: send health_metrics to mgr

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

mgr,osd: make osd_metric more popular

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

mon: add admin command to dump historic/slow ops

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

[mgr/dashboard] add image id to rbd info instead of block_name_prefix

when create images use data_pool parameter, such as:
rbd create -p pool1 -s 1G --data-pool pool2  image1

then get image info from mgr use request http://192.7.7.36:7000/rbd_pool_data/pool1
we got the response:
[{"parent_name": "", "parent_pool": 18446744073709551615, "name": "image1", "num_objs": 256, "block_name_prefix": "rbd_data.14.f561643c986", "obj_size": 4194304, "features": 189, "features_name": "fast-diff, layering, exclusive-lock, deep-flatten, object-map, data-pool", "order": 22, "size": 1073741824}]

notice:
since block_name_prefix is deprecated,
from block_name_prefix we can not get image id(the real id is ''f561643c9869')
we need add image id to rbd info instead block_name_prefix

Python 2.7.5 (default, Jun 17 2014, 18:11:42)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rados
>>> import rbd
>>> RADOS_NAME = 'client.admin'
>>> cluster_name="ceph"
>>> cluster_handle = rados.Rados(name=RADOS_NAME, clustername=cluster_name, conffile='')
>>> cluster_handle.connect()
>>> ioctx = cluster_handle.open_ioctx("pool1")
>>> image_inst = rbd.Image(ioctx, "image1")
>>> image_inst.stat()
{'parent_name': '', 'parent_pool': 18446744073709551615L, 'num_objs': 256L, 'block_name_prefix': u'rbd_data.14.f561643c986', 'obj_size': 4194304L, 'order': 22, 'size': 1073741824L}
>>> image_inst.id()
u'f561643c9869'

after fix:

[root@tfg36 site-packages]# curl http://192.7.7.36:7000/rbd_pool_data/pool1
[{"parent_name": "", "parent_pool": 18446744073709551615, "name": "image1", "num_objs": 256, "block_name_prefix": "rbd_data.14.f561643c986", "obj_size": 4194304, "features": 189, "id": "f561643c9869", "features_name": "fast-diff, layering, exclusive-lock, deep-flatten, object-map, data-pool", "order": 22, "size": 1073741824}]

Signed-off-by: zouaiguo <zou.aiguo@zte.com.cn>

Merge pull request #21064 from Songweibin/wip-spell-error

osd: correct several spell errors in comments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21051 from neha-ojha/wip-doc-async-recovery

doc: dev description of async recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21023 from Liuchang0812/delete-asok-directories

test: delete asok directories correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21052 from zhsj/fix-distro-detect-test

test: remove distro_version assert in distro detect test

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: dev description of async recovery

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #20798 into master

* refs/pull/20798/head:
ceph-fuse: trim ceph-fuse -V output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20828 into master

* refs/pull/20828/head:
client: cleanup ::_read return

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #20854 into master

* refs/pull/20854/head:
client: add ctime/btime to Inode dump

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge PR #20908 into master

* refs/pull/20908/head:
ceph-fuse: Fix Segmentation fault --localize-reads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20913 into master

* refs/pull/20913/head:
client: add ceph_ll_sync_inode

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #20997 into master

* refs/pull/20997/head:
client: fix Client::_read return

Merge PR #21012 into master

* refs/pull/21012/head:
Document the new '$' suite file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21070 from ceph/wip-cv-vagrant-reload

ceph-volume improve robustness when reloading vms in tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20494 from wido/influx-time-fixes

mgr/influx: Various time fixes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21063 from guzhongyan/bs-cleanup

os/bluestore: remove unused variable.

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #21043 from wido/zabbix-osd-numpgs

mgr/zabbix: Send max, min and avg PGs of OSDs to Zabbix

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21037 from guzhongyan/bf-format

os/bluestore: fix some code formatting

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20951 from tchaikov/wip-FTBFS-bluestore-armhf

bluestore: fix build on armhf

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21034 from rzarzynski/wip-bs-reorder-bluefs_extent_t

os/bluestore: reorder members of bluefs_extent_t for space efficiency

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20900 from ifed01/wip-ifed-fix-interals-in-repair

os/bluestore: bluestore repair should use interval_set::union_insert

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20774 from rishabh-d-dave/fix-tilde-expansion

config: expand tilde for ~/.ceph/$cluster.conf

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20558 from chardan/jfw-wip-show_compression_methods-22420

osd,compressor: Expose compression algorithms via MOSDBoot.

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20314 from ovh/bp-librados-nntk

librados: extend C API for so it accepts keys with NUL chars

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume tests create a retry script for vagrant reload

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests use a reload script in simple functional tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests use a reload script in lvm functional tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Fix objects named `default` are inaccessible

These two calls to `default()` are identical: `vpath` and
params` are both empty:

$ curl 'http://localhost/api/cp_path/'

and

$ curl 'http://localhost/api/cp_path/default'

But we need to dinstinguish them. To fix this, we need
to add the missing `default`

We now require that all exposed controllers have a @ApiController
decorator.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Use `@ApiController` for PerfCounters

The next commit will require that all controllers
have a `@ApiController` decorator.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: task manager unit tests implementation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: add task manager usage instructions to HACKING.rst

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: added tasks info to summary controller

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: async tasks controller

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Asynchronous tasks implementation

Signed-off-by: Ricardo Dias <rdias@suse.com>

mon: mark mgr reports and osd_failure as no_reply

see also: #20517

Fixes: http://tracker.ceph.com/issues/22114
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: implemented NotificationQueue listener removal

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: fix NotificationQueue waiting loop

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Support for handler priorities in NotificationQueue

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: privatize NotificationQueue methods

Signed-off-by: Ricardo Dias <rdias@suse.com>

osd: correct several spell errors in comments

Signed-off-by: songweibin <song.weibin@zte.com.cn>

os/bluestore: remove unused variable.

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

Merge pull request #21005 from rjfd/wip-mgr-fix-error-handling

mgr: fixes python error handling

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #21024 from s0nea/wip-dashboard-return-features-list

mgr/dashboard: Convert the RBD feature names to a list of strings

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #21004 from s0nea/wip-dashboard-skip-datapool-testcase

mgr/dashboard: skip data pool testcase for none-bluestore clusters

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20873 from sebastian-philipp/dashboard_v2-simple-html-debug

mgr/dashboard: Add minimalistic browsable API

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21047 from rjfd/wip-mgr-smart-fix-py3

mgr/smart: fix python3 module loading

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

test: remove distro_version assert in distro detect test

VERSION_ID is optional in /etc/os-release, we removed the error
log in #17787, and the test also needs to be fixed.

Signed-off-by: Shengjing Zhu <zhsj@umcloud.com>

Merge pull request #21027 from cbodley/wip-qa-swift-force-branch

qa/tasks/swift: add support for the "force-branch" configurable.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/dashboard: New API tests  for `perf_couters`

Added OSD and MDS tests.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add minimalistic browsable API

Also provides a simple HTML form to POST
data to a `RESTController`'s `create()` method.

Also added ENABLE_BROWSABLE_API setting to the dashboard

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Renamed `types` as it clashes with std lib

This will lead to a name clash, when importing `types`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/smart: fix python3 module loading

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr: fixes python error handling

The current `handle_pyerror` function implementation relies in the
`traceback.format_exception_only` python function to format the
exception object. The problem is that this python function might also
raise an exception. This commit fixes it by enclosing that python
function call in try...catch block.

Fixes: http://tracker.ceph.com/issues/23406
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #19439 from chardan/jfw-wip-14363-normalize-rm

rgw, mon: normalize delete/remove in admin console (cleanup 22444)

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21011 from tspmelo/wip-submit-btn

mgr/dashboard: Add submit button component

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #21032 from simransinghal/fixtypo

rgw: Fix typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #21020 from trociny/wip-vstart-no-mon-config

stop.sh: use --no-mon-config when trying to unmap rbd devices

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: Send max, min and avg PGs of OSDs to Zabbix

We already send the max, min and avg fill ratio of OSDs but
knowing the OSD with the highest amount of PGs is also useful.

This allows admins to create a trigger should it happen that there
is a OSD with too many PGs.

This could happen if a lot of OSDs fail and PGs start to move filling
up one or more OSDs with many PGs.

As PGs eat CPU and Memory people usually like to watch out for these
situations.

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #21036 from xiexingguo/wip-wrong-tip

os/bluestore: s/bluefs_total/bluefs_free/

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #21007 from rjfd/wip-dashboard-fix-py3

mgr/dashboard: fix dashboard python 3 support

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #21006 from rjfd/wip-dashboard-fix-auth-py3

mgr/dashboard: fixed password generation in Auth controller

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20865 from sebastian-philipp/dashboard_v2_pool-create

mgr/dashboard: Add Pool-create to the backend

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: convert RBD features to a list of strings

Until now the API returns the features provided by an RBD as one comma
separated string. It's more usable for the frontend to receive a list
of strings instead.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

os/bluestore: fix some code formatting

Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>

os/bluestore: s/bluefs_total/bluefs_free/

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: reorder members of bluefs_extent_t to make it smaller.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

src: Fix typo

Replace "dont" with "don't". "Dont" is not same as "Do not" or "Don't".

Signed-off-by: Simran Singhal <singhalsimran0@gmail.com>

Merge pull request #21002 from smithfarm/wip-23437

build/ops: setup-virtualenv.sh: do not hardcode python binary

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add submit button component

This component is to be used inside a form.

It will render a submit button with the given label.
The button will disabled itself and show a loading icon when the user clicks it,
usually initiating a request to the server, and it will stay in that state
until the request is finished.

To indicate that the request failed, returning the button to the enable state,
you need to insert an error in the form with the 'cdSubmitButton' key.

It will also check if the form is valid, when clicking the button, and will
focus on the first invalid input.

Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/tasks/swift: add support for the "force-branch" configurable.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #21013 from dzafman/wip-remove-bc

test: Replace bc command with printf command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Add Teuthology test for `/api/pool/_info`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Minor tweak to `DashboardTestCase._request`

Don't fall through silently.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Change `crush_rule` serialization of Pools from id to name

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add `pool/_info` api endpoint to fill the create-pool dialog

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add create() to Pool controller

Also:

* Added tests.
* Renamed `DashboardTest` to `PoolTest`.
* Added `delete()`.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Changed `@RESTController.args_from_json`

It now allows any keyword arguments. This is needed for ECP
and Pool creation.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Refactor `send_command` into CephService

* `CephService.send_command` is much easier to use.
* Refactored `CephFSClients.get` and `Dashboard.load_bufer`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #21008 from trociny/wip-23430

vstart: set osd_pool_default_erasure_code_profile in initial ceph.conf

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #20981 from cbodley/wip-qa-fix-ssl

qa: fix overrides for openssl_keys task

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #20841 from oritwas/wip-rgw-metadata

rgw: trim all spaces inside a metadata value

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #20842 from mkogan1/wip-rgw-23147

rgw: enable 'qlen' & 'qactive' performance counters

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #20846 from theanalyst/rgw/fix-empty-stats

rgw: radosgw-admin abort early for user stats for empty uids

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #21021 from sebastian-philipp/dashboard-auth-qa-fix

qa/tasks/mgr/dashboard: Fix login expires too soon

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #20972 from ricardoasmarques/wip-dimless-binary-directive

mgr/dashboard: Add DimlessBinaryDirective

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #20971 from ricardoasmarques/wip-helper-component

mgr/dashboard: Add helper component

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test: delete asok directories correctly

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

mgr/dashboard: Add helper component

Signed-off-by: Ricardo Marques <rimarques@suse.com>

tools: setup-virtualenv.sh: improve error message

Signed-off-by: Nathan Cutler <ncutler@suse.com>

tools: setup-virtualenv.sh: add usage function and --help option

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/mgr/dashboard: Fix login expires too soon

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>