pybind/mgr/prometheus: add Metrics class to manage Metric instances

The central change of this commit is that per-daemon metrics are now
managed by first appending the metric (using Metrics.append) to a
staging area. Then the metrics for specific paths (metric names) are
overwritten by the staged metrics (by calling Metrics.reset). This gets
rid of metrics from daemon that are no longer in the cluster. I.e. when
ceph no longer reports metrics for one OSD daemon (because it was
removed from the cluster) the prometheus module will no longer export
metrics for that daemon.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20630 from joscollin/wip-doc-monitoring

doc: Update monitoring.rst

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>

doc: Update monitoring.rst

Updated monitoring.rst with
* Latest command output
* misc doc fixes

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #20646 from tchaikov/wip-qa/run-standalone.sh/python-path

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20634 from tchaikov/wip-delete-pool-in-standalone-test

qa/standalone: extract delete_pool()

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20619 from theanalyst/doc/release-notes/12.2.4

doc: 12.2.4 release notes

Reviewed-By: Alfredo Deza <adeza@redhat.com>
Reviewed-By: Lenz Grimmer <lgrimmer@suse.com>

qa/run-standalone.sh: set PYTHONPATH for FreeBSD also

do not set PYTHONPATH=/usr/lib/python2.7/dist-packages anymore, it will
be overridden later on anyway.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19015 from hansbogert/master

mon: Add `ceph osd get-require-min-compat-client` command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #20607 from shangfufei/test_stack

msg/async:fix the incoming parameter type of EventCenter::process_events()

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20509 from liewegas/wip-less-filestore-idempotent

qa/tasks/filestore_idempotent: shorter test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20498 from liewegas/wip-22616

os/bluestore: trim cache every 50ms (instead of 200ms)

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #20616 from tchaikov/wip-cmake

cmake/BuildBoost: fixes to ready seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20636 from tchaikov/wip-23169

Revert "core: hint the dout()'s message crafting as a cold code."

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #20631 from tchaikov/wip-cmake-systemd

cmake: s/sysconf/sysconfig/

Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Lenz Grimmer <lenz@grimmer.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #15849 from theanalyst/wip-doc-es

doc: rgw add some basic documentation for sync plugins & ES

Reviewed-By: Yehuda Sadeh <yehuda@redhat.com>

doc: 12.2.4 release notes

And updating the release table

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Revert "core: hint the dout()'s message crafting as a cold code."

This reverts commit b265ed2955570c17b954270e6a1449637790e9b8.

with b265ed29, we cannot use `dout(10) << __func__ << ...` anymore for
logging the name of the inner most function. what we will have is
always `operator()`, because the lambda closure will be the inner most
function with the the hinting change.

Fixes: http://tracker.ceph.com/issues/23169
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/standalone: extract delete_pool()

some tests, like osd-backfill-stats.sh are using delete_pool(), but
they don't have this function defined. and this function is defined
in standalone tests separately, so would be simpler if we can
consolidate them in ceph-helper.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20411 from majianpeng/sync-nvml-submodule

cmake: sync nvml submodule to latest code.

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: make PMEMDevice work based on latest pmdk.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: sync nvml submodule

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

cmake: s/sysconf/sysconfig/

it's a regression caused by 638aadf

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20618 from tchaikov/wip-systemd-deb-env

cmake,deb: install system units using cmake

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20602 from gregsfortytwo/wip-23141-rotational

os/bluestore: correctly check all block devices to decide if journal is_rotational

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20595 from jcsp/wip-23037

mgr: die on bind() failure

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20593 from tchaikov/wip-23121

ceph-objectstore-tool: don't destroy SnapMapper until the txn is completed

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20540 from ovh/bp-optracker-cleanup

common/OpHistory: move insert/cleanup into separate thread

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20539 from zeichenanonym/mgr-prometheus-fix-metadata-does-not-exist

mgr: prometheus: Don't crash on OSDs without metadata

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20530 from wido/issue-23078

common/dns_resolv.cc: Query for AAAA-record if ms_bind_ipv6 is True

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20394 from liewegas/wip-pg-log-size

osd: min_pg_log_entries == max == pg_log_dups_tracked

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20290 from rzarzynski/wip-common-subsystemmap

core: make the main dout() paths faster and more maintanable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20273 from adamemerson/wip-any-singleton-will-do

common: Switch singletons to use immobile_any and cleanups

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20528 from tchaikov/wip-22462

mon: add MMonHealth back

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20415 from wumingqiao/wip-fallocate

filestore: include <linux/falloc.h>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20620 from ceph/wip-rm23140

ceph-volume: adds raw device support to 'lvm list'

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20599 from dillaman/wip-librbd-devstack-tempest

qa/workunits/rbd: switch devstack tempest to 17.2.0 tag

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20612 from fangyuxiangGL/com

rgw: download object might fail for local invariable uninitialized

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19845 from fangyuxiangGL/escape

rgw: rework json/xml escape usage follow #19806

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20352 from yehudasa/wip-22928

rgw: use explicit index pool placement

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #20440 from cbodley/wip-qa-rgw-beast

qa/rgw: remove some civetweb overrides for beast testing

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #20471 from mdw-at-linuxbox/wip-master-swacldel

rgw: Allow swift acls to be deleted.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: download object might fail for local invariable uninitialized

This is an omission in #20226

Fixes: http://tracker.ceph.com/issues/23146
Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #20234 from smithfarm/wip-run-make-check

build/ops: run-make-check.sh: fix SUSE support

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake,deb: set EnvironmentFile using cmake

this change also fix the EnvironmentFile specified in rbdmap.service.
without this change EnvironmentFile in rbdmap.service is always
/etc/sysconfig/ceph even on debian derived distros. after this change,
this variable is /etc/default/ceph in rbdmap.service shipped by the deb
packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: install system units using cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: if using lvm list on a device with many lvs, show them all

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20436 from smithfarm/wip-22999

install-deps.sh: set python2 %bcond by environment

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20611 from Leeshine/wip-rgw-22080-1

rgw: create sync-module instance when radosgw-admin sync run

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20397 from tdb/fix/perms-ocf-rbd

rbd: resource agent needs to be executable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20577 from Songweibin/wip-group-rename

rbd: add group rename methods

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: add implicit dependencies between boost libs

so the dependencies can be included when linking against the one which
depends on them. for example, libboost_filesystem depends on
libboost_system, if we don't link against the latter, linker will bail
out when linking an executable using Boost::filesystem.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add an alias for unit_test_framework

should pass --with-test for building Boost::unit_test_framework

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: skip boost dependency on ALIAS targets

the boost override for add_library() does not work for ALIAS targets:

CMake Error at cmake/modules/BuildBoost.cmake:227 (add_dependencies):
  Cannot add target-level dependencies to alias target "Seastar::seastar".

Signed-off-by: Casey Bodley <cbodley@redhat.com>

install-deps.sh: set python2 %bcond by environment

If there is no /usr/bin/python2 we need to set %bcond_with python2.

Fixes: http://tracker.ceph.com/issues/22999
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20434 from smithfarm/wip-22998

build/ops: install-deps.sh: pass --no-recommends to zypper

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #20404 from smithfarm/wip-cmake-with-systemd

build/ops: change WITH_SYSTEMD default to ON

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

build/ops: run-make-check.sh: fix SUSE support

Get /usr/bin/which from the right RPM.

Fixes: http://tracker.ceph.com/issues/22875
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20245 from smithfarm/wip-p2p-ripout

tests: cleanup: drop upgrade/jewel-x/point-to-point-x

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20138 from smithfarm/wip-recommend-chrony

build/ops: rpm: recommend chrony instead of ntp-daemon

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #20551 from shun-s/wip-add-notify-name-print

rbd: output notifyOp request name when watching

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

log: prohibit negative values for debug_* configurables.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

log: remove SubsystemMap::m_max_name_len.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph: switch to SubsystemMap::should_gather<SubV, LvlV>().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: resurrect the old check on dout's levels where possible.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: hint the dout()'s message crafting as a cold code.

The idea is to:
  1. Do not put the dout()'s crafting stuff on the hot, fall-through
     path. Cheapest branches are those that are forward and never taken.
  2. Move it to separated sections placed far away from the main path
     to be more friendly to ICache and ITLB. That is, dout_impl constructs
     a function now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: make SubsystemMap more statical and optimize should_gather().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: remove the PGLog::prefix_provider as it's always nullptr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: move subsys-dependant types to a dedicated header.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

core: dout checks ceph_subsys_* in compile time if possible.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #19471 into master

* refs/pull/19471/head:
cephfs-journal-tool: support purge queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20412 into master

* refs/pull/20412/head:
mds/CDir: cleanup in the lookup_exact_snap()
mds/cdir: add CDir missing state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20468 into master

* refs/pull/20468/head:
mds: fix check of underwater dentries

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20489 into master

* refs/pull/20489/head:
common: omit short option for id in help for clients

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #20547 into master

* refs/pull/20547/head:
mds: add uptime to MDS status

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

Merge PR #20553 into master

* refs/pull/20553/head:
mds: always handle SESSION_REQUEST_RENEWCAPS messages
mds: don't cleanup request that has pending remote authpin/wrlock/xlock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: create sync-module instance when radosgw-admin sync run

Signed-off-by: lvshanchun <lvshanchun@gmail.com>

msg/async:change EventCenter::process_events(int timeout_microseconds) to EventCenter::process_events(unsigned timeout_microseconds)

Signed-off-by: shangfufei <shangfufei@inspur.com>

Merge pull request #20532 from Songweibin/wip-du-user

rbd: filter out UserSnapshotNamespace in do_disk_usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20529 from Songweibin/wip-fix-nbd-map

 rbd: set a default value for options in `nbd map`

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20507 from trociny/wip-remote-rename

 librbd: make rename request complete with filtered code

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20600 from liewegas/wip-msgr-level

msg/async/AsyncConnection: less noisy debug

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #20572 from batrick/erasure-code-cephfs-doc

doc: cleanup erasure coded pool doc on cephfs use

bluestore: correctly check all block devices to decide if journal is_rotational

Fixes: http://tracker.ceph.com/issues/23141
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

msg/async/AsyncConnection: less noisy debug

This spams logs of healthy OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: lvm list now works with raw devices

If you used a raw device with --data to create an OSD
you can now use 'lvm list' to get information about that OSD
using the raw device name like:

ceph-volume lvm list /dev/sda

fixes http://tracker.ceph.com/issues/23140

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20571 from dzafman/wip-21833-2

Fix asserts caused by DNE pgs left behind after lots of OSD restarts

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20594 from ceph/wip-rm23004

ceph-volume: log the current running command for easier debugging

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rbd: switch devstack tempest to 17.2.0 tag

Fixes: http://tracker.ceph.com/issues/22961
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20560 from dillaman/wip-22960-1

librbd: minor cleanup of the IO pathway

Reviewed-by: Mykola Golub <mgolub@suse.com>

librbd: moved write same buffer assembler to common location

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: decoupled object read requests from result assembler

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use move-semantics for object IO data payloads

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track the originating source of a flush request

This will be used in a later cleanup to remove the IO flush handling
from ImageCtx.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: separated queued image IO requests from state machine

This breaks the tight coupling between the IO work queue and
the actual dispatch of IO requests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr: die on bind() failure

Previously, the daemon would get wedged if it
competed for the same port as another daemon
on the same host and lost.

Fixes: https://tracker.ceph.com/issues/23037
Signed-off-by: John Spray <john.spray@redhat.com>

ceph-objectstore-tool: do not check action_on_object_t::call()'s retval

it never fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-objectstore-tool: do not check queue_transaction()'s retval

it never fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-objectstore-tool: don't destroy SnapMapper until the txn is completed

ch->flush() only ensure that the op is dequeued from osr/collection's
queue. but the ops could be still in the flight and they could still be
holding a *weak* reference of SnapMapper. we should make sure that the
mapper is still available when they reference it,

Fixes: http://tracker.ceph.com/issues/23121
Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: log the current running command for easier debugging

Fixes: http://tracker.ceph.com/issues/23004
Signed-off-by: Andrew Schoen <aschoen@redhat.com>

osd: Remove partially created pg known as DNE

Fixes: http://tracker.ceph.com/issues/21833
Signed-off-by: David Zafman <dzafman@redhat.com>