common: always include priority in perf counter dump

JSON output with inconsistent sets of members is
annoying to use on the receiving side.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit e631f1a72735ec618e2f3012ad7b9c5830d6c0eb)

mgr: add get_all_perf_counters to MgrModule interface

This is for use by modules that dump counters
in bulk, e.g. to a TSDB.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 9a42d4255d9d968d6162b53b71db292d9d3de2e4)

pybind/mgr/prometheus: export cluster-wide pg stats, not per osd

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 13b1236b96d4563e0985cad40d3009b60cc475e7)

pybind/mgr/prometheus: add more osd metadata

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit e7704fa9cc35549dba526212c2830df589670416)

pybind/mgr/prometheus: don't get perf counters that are not in schema

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit d4ba07d04477ccae3a89dcdcafbb7e76149dfd1c)

pybind/mgr/prometheus: add mon and osd perf counters to export

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit fa25d31263a26074225e2a00cb82448066b54069)

pybind/mgr/prometheus: add index page, export metrics under metrics/

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit d99a506ed37c2d0991d68ecd34ac5fb213a3eea4)

pybind/mgr/prometheus: export selected perf_counters

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f6e2e36ba72caf6347f3bb6a985925d0e35077a2)

pybind/mgr/prometheus: export osd and pool metadata

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 2bea3814699c27baa8f633b56a8800d697685898)

pybind/mgr/prometheus: actually emit reported pg counts

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c288624eed862559b2c86c5dfc85c837716739ab)

pybind/mgr/prometheus: no need to wait for notify event

If stats or perf counters are not available they won't be emitted.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ead0973d7dd12fe985390891c80f1bc15f7b9aec)

pybind/mgr/prometheus: no need to convert perf_schema to ordered_dict

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 5e4b4b5ea2a217731691c1c391c252b08452798a)

pybind/mgr/prometheus: add device_class label to osd metrics

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 76d1918724320b7d6b1120b57b3002bb24099001)

pybind/mgr/prometheus: add cluster wide metrics; no perf counters for now

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 49b3ff83cd231066d2a8f1809fadbdeb2c0c1f88)

pybind/mgr/prometheus: prefix metrics with 'ceph'; replace :: with _

Both follow prometheus best practices. While : is a legal metric
character, "Exposed metrics should not contain colons, these are for
users to use when aggregating."

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 177afcc7886aa3898d092ebd1e101697bc6539fd)

doc/mgr: add influx plugin docs

Signed-off-by: My Do <mhdo@umich.edu>
(cherry picked from commit e345fe3c5780976a4e33488b3a75cd24bb2c96c5)

mgr/influx: added influx plugin

Signed-off-by: My Do <mhdo@umich.edu>
(cherry picked from commit 68ae26c014d0471cc3f2f979dc8d822b2e50740f)

mgr: store declared_types in MgrSession

Because we don't (yet) properly prevent multiple sessions
from daemons reporting the same name (e.g. rgws), storing
it in the DaemonPerfCounters meant that one daemon's report
was referring to another daemon's set of reported types.

This should always have been a property of the session.

The behaviour will still be ugly when multiple daemons
are using the same name (stomping on each other's stats/statsu)
but it shouldn't crash.

Fixes: http://tracker.ceph.com/issues/21197
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit dc415f1ae09a308bd448614934a4c168eb9cf07b)

mgr: make pgmap_ready atomic to avoid taking lock

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d20915741d985e080a723cd6563bc6f4a657276f)

mgr/DaemonServer: handle MMgrReports in parallel

The DaemonStateIndex locking is sufficient to make all
the report processing safe: holding DaemonServer::lock
through all ms_dispatch was unnecessarily serializing
dispatch.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 64af9d3da0fceff9ad0ff668f60d272c46912f34)

mgr: clean up DaemonStateIndex locking

Various things here were dangerously operating
outside locks.

Additionally switch to a RWLock because this lock
will be relatively read-hot when it's taken every time
a MMgrReport is handled, to look up the DaemonState
for the sender.

Fixes: http://tracker.ceph.com/issues/21158
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 806f10847cefe5c7a78fc319b1b130d372197dd3)

mgr: runtime adjustment of perf counter threshold

ceph-mgr has missed out on the `config set` command
that the other daemons got recently: add it here
and hook it all up to the stats period and threshold
settings.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 057b73d641decb9403aba50caae9d139f3a34dd4)

mgr: apply a threshold to perf counter prios

...so that we can control the level of load
we're putting on ceph-mgr with perf counters.  Don't collect
anything below PRIO_USEFUL by default.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit bdc775fdd8acdad5c58ff3065a21396f80ce5db4)

pybind/mgr/balancer: make auto mode work

(with upmap at least)

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ef1a3be05671ad31907cf8c4beb64a766359bc66)

src/pybind/mgr/balancer/module.py: improve scoring method

* score lies in [0, 1), 0 being perfect distribution
* use shifted and scaled cdf of normal distribution
  to prioritize highly over-weighted device.
* consider only over-weighted devices to calculate score

Signed-off-by: Spandan Kumar Sahu <spandankumarsahu@gmail.com>
(cherry picked from commit c09308c49ca087fb8c5e7d4261b0234190f863d9)

pybind/mgr/balancer: make 'crush-compat' sort of work

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7a00e02acd1b2ff21dac829de30f80fd69eae602)

pybind/mgr/balancer: rough framework

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit d5e5c68c374e7d5514f89aac2d3df6008d103a76)

mgr/PyOSDMap: OSDMap.map_pool_pgs_up, CRUSHMap.get_item_name

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a928bf62316c32f37dd1791192fd9a2ddaef0d33)

mgr/PyOSDMap: get_crush, find_takes, get_take_weight_osd_map

These let us identify distinct CRUSH hierarchies that rules distribute
data over, and create relative weight maps for the OSDs they map to.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3b8a276c437cfd599c55a935d141375afda676ff)

crush/CrushWrapper: rule_has_take

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ef140de639078b40c05971fb219f7b8c12d83228)

crush/CrushWrapper: refactor get_rule_weight_osd_map to work with roots too

Allow us to specify a root node in the hierarchy instead of a rule.
This way we can use it in conjunction with find_takes().

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 69454e0570274ff7f252e7f081965dcc9bb04459)

pybind/mgr/balancer: do upmap by pool, in random order

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 028a66d43244c15a77e71f3d3e4f41773837ab02)

pybind/mgr/balancer: add balancer module

- wake up every minute
- back off when unknown, inactive, degraded
- throttle against misplaced ratio
- apply some optimization step
  - initially implement 'upmap' only

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 0d9685c50f79fbb53dbc8bd98c95900ef6e902b8)

pybind/mgr/mgr_module: add default arg to get_config

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 39c42ddb9339c1950a3a474e8083db8b24e775a6)

mgr: add trivial OSDMap wrapper class

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2ef005196ba2eb49c34c32def624938c7a8beb03)

mgr/PyModules: add 'pg_dump' get

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit bfb9286f4212947183c46543d609b664ea13b489)

mgr/PyModules: add 'pg_status' dump

This is summary info, same as what's in 'ceph status'.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 85b5b80906d00e098d4b1af1354c60a357022dd2)

Merge pull request #18674 from ceph/wip-rgw-s3-branch

qa/tests: use ceph-luminous branch for s3tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: use ceph-luminous branch for s3tests

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #18412 from kmroz/wip-21659-luminous

luminous: mgr: fix crashable DaemonStateIndex::get calls

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18113 from jcsp/wip-prometheus-port-backport

luminous: ceph-mgr: can not change prometheus port for mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18655 from ceph/wip-yuriw-bp-PR18634-luminous

qa: add "restful" to ceph_mgr_modules in ceph-ansible suite

qa: add "restful" to ceph_mgr_modules in ceph-ansible suite

backport of https://github.com/ceph/ceph/pull/18634

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit caf9ee5c60d999951979d0b67afda8d56e1cd91d)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #17889 from smithfarm/wip-21372-luminous

luminous: core: Improve OSD startup time by only scanning for omap corruption once

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18004 from linuxbox2/luminous-rgwf-wr

luminous: rgw_file: fix write error when the write offset overlaps.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18438 from theanalyst/wip-21696-luminous

luminous: fix a bug about inconsistent unit of comparison

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18410 from kmroz/wip-21732-luminous

luminous: qa/suites/rest/basic/tasks/rest_test: whitelisting

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18629 from ceph/wip-yuriw-21978-luminous

qa/suites/upgrade/jewel-x: Changed typo ('hammer' to 'jewel')

Changed typo ('hammer' to 'jewel')

Fixes http://tracker.ceph.com/issues/21978
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #18627 from ceph/backport-18513

ceph-volume lvm zap backport

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: set journal_uuid and journal_device when using a partition

This correctly sets the tags when a partition is used for a filestore
journal.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1a8561d38dd7363dc920ae82ec1343b2b75a1ca2)

docs for ceph-volume lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3e93a31deda96ed6fb23fa19ce3e273de05f7d88)

ceph-volume: add tests for ceph-volume lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3d5e391693fc64747a4774287c968f842294eaa6)

ceph-volume: print success message if zap succeeds

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 6d70762dee0cd9047c291cf9869f666a375e956b)

ceph-volume: terminal.write only takes one argument

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a5454eba241abd8cc1b1a660b12a1aec7c3c16e0)

ceph-volume: the metadata field for the lv path is lv_path not path

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit e3a9113e7025f1e3a3130ec1e2d565f37bf3d2dc)

ceph-volume: update help text for ceph-volume lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit cbc38c4e4a7dcc24b31bcfa6af73eb8cf04f56ad)

ceph-volume: remove lvm metadata when zapping an lv

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5f57fc87205c2d35da9a1f28c72e233ffb3fe4d9)

ceph-volume: adds Volume.clear_tags

Will remove all tags for the current lv

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3c9401f1618f7fcbea827fe7279e8dfde617d957)

ceph-volume: zap should leave lvs and partitions intact

This will remove filesystems and wipe data from any lv or partition
given to 'ceph-volume lvm zap' but still leave it intact for further
use.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit caba9fb80b358222192f736bd1b0ab31dca25cec)

ceph-volume: adds utilites to wipe the fs and clear data when zapping

These should eventually move to a disk api.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2f64d4a0abd41afbcb9eba6a237642502d9a9135)

ceph-volume: adds tests for api.lvm.remove_lv

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 937b57438147681c2c6e32c6db38d8bea68d4731)

ceph-volume: api.lvm.remove_lv should return True when successful

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7f055aa6deb3904cf0334e214c13a26098b08aa8)

ceph-volume: zap logical volumes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1d083a2191315ee94301c2f5f102a0906dd05fa8)

ceph-volume: adds a remove_lv command to the lvm api

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit cf98b6971ced0caa29ff4f7ac346df652177fa29)

ceph-volume: stubs out the ceph-volume lvm zap command

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 2e64b797ef6ae91623ffba8ae28d3f8ccc7d7b93)

Merge PR #18385 into luminous

* refs/pull/18385/head:
mds: fix race in PurgeQueue::wait_for_recovery()
mds: open purge queue when transitioning out of standby replay
mds: always re-probe mds log when standby replay done

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #18316 into luminous

* refs/pull/18316/head:
mds: prevent trim count from underflowing

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #18300 into luminous

* refs/pull/18300/head:
mds: keep CInode::STATE_QUEUEDEXPORTPIN state when exporting inode

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #17729 into luminous

* refs/pull/17729/head:
ceph.in: validate service glob

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #18299 into luminous

* refs/pull/18299/head:
mds: update client metadata for already open session

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge PR #18298 into luminous

* refs/pull/18298/head:
osdc/ObjectCacher: limit memory usage of BufferHead

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #18085 into luminous

* refs/pull/18085/head:
ceph_volume_client: fix setting caps for IDs

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #18030 into luminous

* refs/pull/18030/head:
qa: relax cap expected value check
mds: improve cap min/max ratio descriptions
mds: fix whitespace
mds: cap client recall to min caps per client
mds: fix conf types
mds: fix whitespace
doc/cephfs: add client min cache and max cache ratio describe
mds: adding tunable features for caps_per_client

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17921 into luminous

* refs/pull/17921/head:
ceph_volume_client: perform snapshot operations in

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #18593 from ceph/luminous-wip-bz1499840

luminous ceph-volume lvm bluestore support

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18334 from kmroz/wip-21816-luminous

luminous: rgw: fix bilog entries on multipart complete

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17861 from smithfarm/wip-21441-luminous

luminous: rbd: [cli] mirror getter commands will fail if mirroring has never been enabled

Merge pull request #18416 from kmroz/wip-21855-luminous

luminous: librbd: object map batch update might cause OSD suicide timeout

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #18337 from kmroz/wip-21640-luminous

luminous: rbd-mirror: forced promotion can result in incorrect status

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #18336 from kmroz/wip-21639-luminous

luminous: librbd: snapshots should be created/removed against data pool

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #17860 from smithfarm/wip-21299-luminous

luminous: rbd: [rbd-mirror] asok hook names not updated when image is renamed

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

ceph-volume lvm.activate remove links on activate before priming

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 634b5caaed7c9188e426727e83a0768bdbc51f0d)

ceph-volume lvm.activate ceph-bluestore-tool needs --cluster too

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fb36087bc3c7e7bb8358cbf8ad84409fa23fac68)

ceph-volume lvm.activate consume cluster_name from lvm metadata

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2b7ff497fe9e0e167ab01bf0d8ce5e26f0da836f)

ceph-volume lvm.prepare persist cluster_name on lvm metadata

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6d38ad987e08791be379f6c006c1eb5bf9464415)

ceph-volume util.prepare bluestore db and wal paths are to devices, not osd dir paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fd616acd6ae0f184c1757e7de38fc746d526908c)

ceph-volume lvm.prepare bluestore wal and db links get created with mkfs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 80698d2309a221d2318b11be9b8f8a95b51416ea)

ceph-volume lvm.prepare use wal and db for mkfs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4970ab5c804111ce4b2b57817f1e29d58b18a893)

ceph-volume lvm.prepare ensure tags are set for ceph.type correctly for block and data

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 58dbefd3be7bad51e6c56c50fbfb450e205b3036)

ceph-volume lvm.activate check if lv device is mounted at path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1ead135bef801280e7cec21283c95c7ba440d84b)

ceph-volume lvm.prepare default to using bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 39dd0d1d10aba08c3e8299eb1702260274a756a7)

ceph-volume lvm.create default to using bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d3145da6e643dedb2fbed72e7d4d0cddf397c5e3)

ceph-volume lvm.common remove boolean defaults for CLI flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a7d5965ab532a621f402bad55d45dfed61d90cc5)

ceph-volume lvm.prepare update to use create_osd_path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bafb72144666f891773bed95e9a0f757673af2e0)

Merge pull request #18138 from kmroz/wip-21684-luminous

luminous: rgw: stop/join TokenCache revoke thread only if started.

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17994 from ukernel/luminous-21337

luminous: mds: make sure snap inode's last matches its parent dentry's last

Reviewed-By: Patrick Donelly <pdonelly@redhat.com>

Merge pull request #18431 from theanalyst/wip-21857-luminous

luminous: rgw: We cant't get torrents if  objects are encrypted using SSE-C

Reviewed-By: Casey Bodley <cbodley@redhat.com>

ceph-volume lvm.activate only prime-osd-dir when directory is empty

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9bb6cfd867c2d8e972093d023622f8c2e5b440d4)

ceph-volume lvm.activate safeguard against auto detect objectstore flag not being set

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c9b5f352a8f9e2743e5dca3a0479c8099a59fd12)