mds: don't decay load counters when printing load

Changing load counters while printing load results confusing output.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 831c5d91cc3863c8ec6ef76ff71f4b7d237ee089)

mds: limit run time of load balancer

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit fc0d340876fbcd7d1d262c00b987849c052a296d)

mds: add list to track recently used sub-directories

MDBalancer::find_exports() can be more efficient with LRU list.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 65a35e2a76bbf1ce21a8848a4a92f70a0c6b3728)

mds: calculate other mds' last_epoch_under locally

No need to get this information from MHeartbeat

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 376145d9199ef8bdb17b18ca873dec0b588509b2)

 Conflicts:
src/messages/MHeartbeat.h

mds: cleanup mds_load map access/update

using map::insert to access map items looks strang

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 40759407bf654eb66685b877a8f482a8ef4c8965)

mds: check export pin when choosing dirfrags for exporting

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit eb095453b9aab0a7520d5540237e61a3ebd0d558)

mds: optimize MDBalancer::find_exports()

stop at subtree bounds; make freezing/frozen check more efficient.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 04831d463c63c90124a28307908afe3d478466dd)

mds: avoid creating unnecessary subtrees during load balance

When choosing dirfrags for exporting, consider subtrees imported
from other MDS first. It's because that exporting dirfrags chosen
by MDBalancer::find_export() will create new subtrees.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit b9fb3a4cffdad1ac2a01a415edeaddbf7236edbc)

mds: mds: optimize MDBalancer::try_rebalance()

1. change import_pop_map to multimap because subtrees may have the same
   popularity.
2. avoid calculating subtrees' popularity multiple times

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit cd6f77c6a08cf0d7f0a995664e7a8981675b1b9c)

mds: don't try exporting subdir if dirfrag is already being exported

It doesn't make sense to call MDBalancer::find_exports() for a dirfrag
that is already being exported.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 15810b436051c220e983b9b6cc268f2bc14acc2d)

mds: don't try exporting dirfrags under mds's own mdsdir

Migrator will refuse to export them. Besides, stray directory can
never be subtree root.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit bb6d381f30bd8b51651db813d5b19ac7137ec773)

mds: cleanup MDBalancer::try_rebalance()

no logic changes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 08116b71f6cf11d16be573ce4fd0be96ee9bbdce)

mds: always pass current time to MDBalancer::{hit_inode,hit_dir}

There were codes that pass mdr->get_mds_stamp() to {hit_inode,hit_dir}.
The stamp can be older than 'last_decay' in decay counters,  which can
cause DecayCounter::decay() to malfunction.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 896491268f90d53775e4a76fb808fa8f65bdb513)

 Conflicts:
src/mds/Server.cc

mds: remove unused MDBalancer::export_empties()

MDCache::trim() already does the same job

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 29e19c1d7d032dbb4fc3a06cb46ee88c5b22587f)

mds: adjust subtree popularity after rename

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit bedd98f878352651e97c173306b315e19e203a3d)

mds: fix request rate calculation

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: http://tracker.ceph.com/issues/21745
(cherry picked from commit e9689c1ff7e75394298c0e86aa9ed4e703391c3e)

 Conflicts:
src/mds/MDBalancer.cc

simplify mds overload judgement logic

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit e5ca899260ca3e889448c9cb1c583f575145e390)

Make MDS evaluates the overload situation with the same criterion

Currently, the prep_rebalancer requires the mds_load of itself
being greater than average level mds_bal_min_rebalance percent
to be counted as exporter, however, for the other mds, as long
as their mdsload is greater than average level, they could be
consider as exporter. This inconsistent standard would cause
gray area in the rebalance activity: one may expect the other
do some work, but it didn't. With the increasing amount of mds,
this inconsistent decision would get worse and affects the
optimization effect of rebalancer.

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 9a276c2733415f41d54524db20a9dd8eb62359bd)

 Conflicts:
src/messages/MHeartbeat.h

mds: add asok command that dumps metadata popularity

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit c078591ad1a33447f4de731c846844f4a0433513)

Merge pull request #21233 from smithfarm/wip-jinja-mgr-luminous

luminous: build/ops: deb: move python-jinja2 dependency to mgr

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: James Page <james.page@ubuntu.com>

Merge pull request #21283 from liewegas/wip-pgpool-enc-luminous

luminous: osd/osd_types: fix pg_pool_t encoding for hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21307 from smithfarm/wip-23606-luminous

luminous: tests: ENGINE Error in 'start' listener <bound  in rados

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #21330 from tchaikov/wip-23610-luminous

luminous: osd: resend osd_pgtemp if it's not acked

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21352 from pdvian/wip-23634-luminous

luminous: doc: outline upgrade procedure for mds cluster

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19883 from pdvian/wip-22635-luminous

luminous: rgw: add xml output header in RGWCopyObj_ObjStore_S3 response msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20829 from xiexingguo/backport-pr-20653

luminous: mon, osd: fix potential collided *Up Set* after PG remapping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20840 from xiexingguo/backport-pr-20655

luminous: osd/OSDMap: skip out/crush-out osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20674 from rzarzynski/wip-bs-drop-std_function-luminous

luminous: os/bluestore: avoid overhead of std::function in blob_t.

Merge pull request #20907 from pdvian/wip-23315-luminous

luminous: mon/OSDMonitor.cc: fix expected_num_objects interpret error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21231 from pdvian/wip-23544-luminous

luminous: os/bluestore: change the type of aio_t:res to long

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20973 from pdvian/wip-23409-luminous

luminous: mgr: fix MSG_MGR_MAP handling

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21229 from liewegas/wip-release-reserved-luminous

luminous: osd: do not release_reserved_pushes when requeuing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21103 from dzafman/wip-23267-luminous

luminous: scrub errors not cleared on replicas can cause inconsistent pg state when replica takes over primary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21335 from ceph/backport-rm23644

luminous: ceph-volume lvm.activate error if no bluestore OSDs are found

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

doc: outline upgrade procedure for mds cluster

Fixes: http://tracker.ceph.com/issues/23568
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit baa59f3821febaac869e3a8d1f16e775b4b855c3)

osd: Fix stale scrub stats when a primary takes over

Fixes: http://tracker.ceph.com/issues/23267
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5cfb8241f482ed53c63bb97262425b2acb733d7d)

test: Eliminate use of bc (use awk) in get_timeout_delays()

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ce9c029858d8deb249afe8effd5d80ae623f69a2)

osd: Improve object_stat_sum_t handling

Keep post split PGs as inconsistent from original
Keep num_scrub_errors as total of other 2

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 39adde2655ebb40ad5c010fa77509acc35b25b91)

Conflicts:
src/osd/osd_types.h (trivial)

Merge pull request #21124 from smithfarm/wip-23507-luminous

luminous: test_admin_socket.sh may fail on wait_for_clean

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #21192 from pdvian/wip-23524-luminous

luminous: is_qemu_running in qemu_rebuild_object_map.sh and qemu_dynamic_features.sh may return false positive

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21230 from pdvian/wip-23542-luminous

luminous: rbd-nbd: fix ebusy when do map

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume lvm.activate error if no bluestore OSDs are found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b1920d6e731f194c4829810d849dba6b087ce3a4)

osd: resend osd_pgtemp if it's not acked

if the osd_pgtemp message is dropped before monitor receives it, we need
to resend it. otherwise a pg could be stuck in activating state if the
pg creation was withheld by the max-pg-per-osd on the replica, and then
the replica osd removes some existing pg.

Fixes: http://tracker.ceph.com/issues/23610
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 9c5cb4ea620da4ef14345ebc8018f9ed008c4345)

Conflicts:
src/osd/OSD.cc: structured binding is a feature introduced in
C++17. while we are using C++11 in luminous. so implement it with C++11
syntax.
src/osd/OSD.h: trivial resolution

Merge pull request #20431 from pdvian/wip-22967-luminous

luminous: osdc/Journaler: make sure flush() writes enough data

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21173 from smithfarm/wip-22383-luminous

luminous: qa: src/test/libcephfs/test.cc:376: Expected: (len) > (0), actual: -34 vs 0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21176 from smithfarm/wip-23154-luminous

luminous: mds: FAILED assert (p != active_requests.end()) in MDRequestRef MDCache::request_get(metareqid_t)

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21267 into luminous

* refs/pull/21267/head:
discard the mdsload clear after prep_rebalance in case we want to export it for debugging
make sure that MDBalancer uses heartbeat info from the same epoch

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21266 into luminous

* refs/pull/21266/head:
make popular counter decay at proper rate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21256 into luminous

* refs/pull/21256/head:
mds: bump mds_log_max_segments for trim buffer

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: assign prometheus ports during selftest

This was throwing IOError("Port 9283 not free on '::'",)
when trying to serve, since merging https://github.com/ceph/ceph/pull/19744

It's because the standbys (on the same node as the active) are
now trying to listen too.

Fixes: https://tracker.ceph.com/issues/22755
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit e2c68d5e2582fe610098bc6ea4f0b44ad1df28fd)

Merge pull request #20885 from mslovy/fix-bug-21811

luminous: rgw: Admin API Support for bucket quota change

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21048 from cbodley/wip-qa-swift-force-branch-luminous

luminous: tests: rgw: swift tests target ceph-luminous branch

Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/osd_types: fix pg_pool_t encoding for hammer

If we are missing the hammer feature, we will also be missing the
luminous feature, but we should still encode like hammer.

Analogous fix to e28e0c617af8825ae92cced5d87cc4f403709e48

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 37c22b273959c6ca54808f3cb8d22a2d250d0c1b)

Merge pull request #20890 from pdvian/wip-23312-luminous

luminous: invalid JSON returned when querying pool parameters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20998 from pdvian/wip-23412-luminous

luminous: log: Fix AddressSanitizer: new-delete-type-mismatch

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #21053 from pdvian/wip-23224-luminous

luminous: mgr: quieten logging on missing OSD stats

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21059 from pdvian/wip-23226-luminous

luminous: os/bluestore: trim cache every 50ms (instead of 200ms)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21090 from tchaikov/wip-23477-luminous

luminous: common: should not check for VERSION_ID

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20675 from rzarzynski/wip-bs-no-blobrefing-in-do_read-luminous

luminous: os/bluestore: avoid unneeded BlobRefing in _do_read().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20676 from rzarzynski/wip-bs-dump-onode-by-ref-luminous

luminous: os/bluestore: _dump_onode() don't prolongate Onode anymore.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20712 from pdvian/wip-23175-luminous

luminous: mgr: die on bind() failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20867 from pdvian/wip-23306-luminous

luminous: msg/async: size of EventCenter::file_events should be greater than fd.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #21198 into luminous

* refs/pull/21198/head:
ceph_authtool: add mode option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20642 from b-ranto/wip-mgr-prom-luminous

[luminous] Backport recent prometheus exporter changes

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20668 from pdvian/wip-23160-luminous

luminous: osd: Remove partially created pg known as DNE

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20710 from pdvian/wip-23174-luminous

luminous: common/dns_resolv.cc: Query for AAAA-record if ms_bind_ipv6 is True

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #20766 from t19903/wip-cache-tier-forward-hang

luminous: Objecter: add ignore overlay flag if got redirect reply

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21079 from tchaikov/wip-23471-luminous

luminous: add --add-bucket and --move options to crushtool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton/diverget_priors*: broaden whitelist

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 39e5efbad2c2834acf8e6463258b148321ea85a4)

discard the mdsload clear after prep_rebalance in case we want to export it for debugging

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 513f2e57666bd08242fc639ebc5c65db12742a01)

make sure that MDBalancer uses heartbeat info from the same epoch

Currently mds saves the heartbeat info from others in mds_load, once the mds_load.size( ) equals mds number, it considers that have received all heartbeats info and start the rebalance work. However, after prep_rebalance returns, it doesn't clear the mds_load immediately, but wait until receives the next round hearbeat from mds0. If there are mutiple mds(e.g. greater than 2), there is a chance for one mds receiving the first next round heartbeat other than mds0 due to the network delay.

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 591fd4970863ae51dad5e16be3cef24c1c6c92d6)

make popular counter decay at proper rate

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 6d4709e649beacb5f2d6197d1bea57c8590c5f71)

Merge pull request #20762 from pdvian/wip-23180-luminous

luminous: radosgw-admin data sync run crashes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20953 from pdvian/wip-23347-luminous

luminous: rgw: fix memory fragmentation problem reading data from client.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21054 from pdvian/wip-23225-luminous

luminous: cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20722 from theanalyst/wip-luminous-23221

luminous: fixes for openssl & libcurl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21214 from smithfarm/wip-23025-luminous

luminous: rgw: data sync of versioned objects, note updating bi marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20896 from smithfarm/wip-23349-luminous

luminous: Couldn't init storage provider (RADOS)

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20250 from smithfarm/wip-22856-luminous

luminous: build Debian installation packages failure

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20737 from pdvian/wip-23178-luminous

luminous: build/ops: run-make-check.sh: fix SUSE support

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20748 from smithfarm/wip-jinja2-luminous

build/ops: deb: move python-jinja2 dependency to mgr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21016 from pdvian/wip-23077-luminous

luminous: mon: ops get stuck in "resend forwarded message to leader"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21111 from wido/luminous-21813

luminous: common/ipaddr: Do not select link-local IPv6 addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21118 from smithfarm/wip-23500-luminous

luminous: snapmapper inconsistency, crash on luminous

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21134 from smithfarm/wip-23522-luminous

luminous: tests: unittest_pglog timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: bump mds_log_max_segments for trim buffer

Under create heavy workloads, the MDS will sometimes get behind trimming but
catch up. This avoids unnecessary warnings.

Fixes: http://tracker.ceph.com/issues/23560
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 86e76d15f3b3c06e96d7b27bb01d5840d311cf5d)

Conflicts:
    src/common/options.cc

tests: swift.py: clone the ceph-luminous branch

The master branch of ceph/swift.git contains tests that are incompatible with
Jewel and Luminous. The ceph-luminous branch omits these tests.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit a86ce728954a765797ce634025d43650d990e480)

Conflicts: qa/tasks/swift.py (cherry-picked jewel version to luminous)

doc: Add usage doc for individual bucket quota Admin API

Add usage doc for a new Admin API for individual bucket quota management

Fixes: http://tracker.ceph.com/issues/21811
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>
(cherry picked from commit 0cbd26e88de3c9c71c1ca4532308502f3a8c7ff8)

rgw: Add URL format support for bucket quota modification

Allow admin to change the quota of a individual bucket through Admin API
with params in URL

Fixes: http://tracker.ceph.com/issues/21811
Signed-off-by: Jeegn Chen <jeegnchen@gmail.com>
(cherry picked from commit 46b1186f144c0928394835301fc993d89598398b)

build/ops: deb: move python-jinja2 dependency to mgr

87399bea8321a7a9ab554085be3a3d5e0563f27f introduced an explicit dependency on
python-jinja2, but mistakenly as an overall build dependency instead of as a
runtime dependency of ceph-mgr as intended.

Fixes: http://tracker.ceph.com/issues/22457
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit b7e638f7e42ac232958f2ca2f16f8202cb5ae5bf)

os/bluestore: change the type of aio_t:res to long

io_event::res was assigned to io_t::rval, but io_event::res
is long, if rval is int, this will cause overflow.

Fixes: http://tracker.ceph.com/issues/23527
Signed-off-by: kungf <yang.wang@easystack.cn>
(cherry picked from commit a9b9477a44d4556f6827e6dc4b94a5f1f61ce8e1)

osd: do not release_reserved_pushes when requeuing

Back in 3cc48278bf0ee5c9535d04b60a661f988c50063b we refactoring the sharded
wq and incorrectly included code that would release_reserved_pushes for
items that were queued and deferred and then woken and put back in the
queue. The reserved_pushes are for recovery ops that are in flight in the
queue, which includes the priority queue *and* the waiting_for_pg; the code
we replaced would release these only when dequeueing an item (or items) for
processing (or discard).

In master, this code is fixed as part of the peering fast dispatch and
OSDShard refactor.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-nbd: fix ebusy when do map

When doing rbd-nbd map, if the Ceph service is not available,
the codes will wait on rados.connect(), unless killing the process.
In that case, the close_nbd logic is skipped with NBD_CLEAR_SOCK ioctl
not called. On the CentOS 7 kernel, it leaves nbd->file not cleared, which
causes the subsequent map requests return EBUSY, this patch fixes it
by connecting Ceph first prior to calling NBD_SET_SOCK ioctl

Fixes: http://tracker.ceph.com/issues/23528
Signed-off-by: Li Wang <laurence.liwang@gmail.com>
(cherry picked from commit ab77dcc0170c0d63795fe0d50427cda630bfd593)

pybind/mgr/prometheus: extend daemon metadata with hostname and ceph version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit d03c79676b1761b016ceaed82f7e3f03f99cda73)

pybind/mgr/prometheus: add fs and mds metadata metrics

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 0095e6a1ba0fb9473cfb60b6977498f2c1159079)

pybind/mgr/prometheus: new mon metrics

Remove mon_quorum_count and replace it with per-MON quorum status
(mon_quorum_status). Also add mon_metadata metrics.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 71a6408b1adfedab64208fe1a23231df43960924)

pybind/mgr/prometheus: add Metrics class to manage Metric instances

The central change of this commit is that per-daemon metrics are now
managed by first appending the metric (using Metrics.append) to a
staging area. Then the metrics for specific paths (metric names) are
overwritten by the staged metrics (by calling Metrics.reset). This gets
rid of metrics from daemon that are no longer in the cluster. I.e. when
ceph no longer reports metrics for one OSD daemon (because it was
removed from the cluster) the prometheus module will no longer export
metrics for that daemon.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f88877ab0bd895d8b67222225c22a12d479b7548)

pybind/mgr/prometheus: don't crash on OSDs without metadata

Fix issue where the ceph_exporter crashes after a Ceph upgrade with a broken OSD - that OSD was never online with Luminous and thus we have no metadata for it

Signed-off-by: Christopher Blum zeichenanonym@web.de
(cherry picked from commit 3beee663f6fd6a16cef5a65ea5f9937cbea7a537)

pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ff471d485cd2068b6425b74a17875d8d6a823595)

pybind/mgr/prometheus: return 503 if MON cluster is down

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit eda9f15dd939de3e7205ceb0dfcc9fcfce7d89bb)