mgr/influx: enable self-test without dependencies

The idea of self-test commands is that they're self
contained and just exercise the module's calls
to the Ceph-side.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/influx: revise perf counter handling

- Use new get_all_perf_counters path
- Consequently get counters for all daemons, not just OSD
- Tag stats with ceph_daemon rather than osd_id, as some
  stats appear from more than one daemon type
- Remove summing of perf counters, external TSDB and/or queries
  can do this.
- Remove mgr_id tag: this would change depending on which
  mgr was active, which is certainly not desirable.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: omit module list in beacon logging

This is useful in itself, but awkward when dealing
with logs generally, because it means that when you
grep on the name of a module, you get mostly beacon
messages rather than the log messages from the
module.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: define perf counter constants in mgr_module

So that modules can consume perf counter data
intelligently without having to hunt around
in C land for these constants and redefine them.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph.in: use PRIO_INTERESTING as daemonperf threshold

Using PRIO_USEFUL as the threshold for what goes into
time series databases.  I'm claiming that we have
more "useful" counters than fit on the screen,
so daemonperf's "a screen's worth" threshold
should be at the "interesting" level.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: set some priorities on perf counters

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/prometheus: tag stats by daemon name

Using osd=0 or similar tags was problematic because
daemons of different types have some same-named
counters (e.g. MDS and OSD both have objecter
perf counters).

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/prometheus: use new get_all_perf_counters interface

Signed-off-by: John Spray <john.spray@redhat.com>

common: used fixed size int for perf counter prio

...to avoid any ambiguity in allowed range and
make clear how to encode it down the wire.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: transmit perf counter prio to the mgr

Signed-off-by: John Spray <john.spray@redhat.com>

common: always include priority in perf counter dump

JSON output with inconsistent sets of members is
annoying to use on the receiving side.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: add get_all_perf_counters to MgrModule interface

This is for use by modules that dump counters
in bulk, e.g. to a TSDB.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr/prometheus: export cluster-wide pg stats, not per osd

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add more osd metadata

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: don't get perf counters that are not in schema

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add mon and osd perf counters to export

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add index page, export metrics under metrics/

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: export selected perf_counters

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: export osd and pool metadata

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: actually emit reported pg counts

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: no need to wait for notify event

If stats or perf counters are not available they won't be emitted.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: no need to convert perf_schema to ordered_dict

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add device_class label to osd metrics

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: add cluster wide metrics; no perf counters for now

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr/prometheus: prefix metrics with 'ceph'; replace :: with _

Both follow prometheus best practices. While : is a legal metric
character, "Exposed metrics should not contain colons, these are for
users to use when aggregating."

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

doc/mgr: add influx plugin docs

Signed-off-by: My Do <mhdo@umich.edu>

mgr/influx: added influx plugin

Signed-off-by: My Do <mhdo@umich.edu>

Merge pull request #17978 from alram/rbdmap-fix

rbdmap: fix umount when multiple mounts use the same RBD

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17962 from joscollin/wip-test-signed-unsigned-warning

test: silence warnings from -Wsign-compare

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17988 from wumingqiao/wip-rados-update

include/rados: fix typo in librados.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

include/rados: fix typo in librados.h

opeprate -> operate

Signed-off-by: wumingqiao <wumingqiao@inspur.com>

rbdmap: fix umount when multiple mounts use the same RBD

When a Kubernetes Pod consumes a RBD it is mounted two
times on the same host. When the host shutdown umount will
fail leading to a hung system

Signed-off-by: Alexandre Marangone <amarango@redhat.com>

Merge pull request #17063 from bspark8/wip-dynamic-cli-info-f

osd: use dmclock library client_info_f function dynamically

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #17335 from Carudy/cxw

crush: safe check for 'ceph osd crush swap-bucket'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17410 from zmedico/PGPool-update-optimize-removed_snaps-comparison-when-possible

PGPool::update: optimize removed_snaps comparison when possible

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #17980 from liewegas/wip-boost-url

make-dist,cmake: move boost tarball location to download.ceph.com

Reviewed-by: Jeff Layton <jlayton@redhat.com>

make-dist,cmake: move boost tarball location to download.ceph.com

Sourceforge is down.  Also, we can hammer our servers instead of
theirs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17589 from myENA/feature/20883-adminapi_sync_stats_master

rgw: admin api - add ability to sync user stats from admin api

Merge pull request #17433 from C2python/check_action

rgw: Return Error if Bucket Policy Contians Undefined Action

Merge pull request #17969 from ceph/rbd-mirror-reboot

systemd: rbd-mirror does not start on reboot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17929 from renhwztetecs/renhw-wip-mgrmon-command_descs_prefix

mon/mgr: sync "mgr_command_descs","osd_metadata" and "mgr_metadata" prefixes to new mons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17932 from jcsp/wip-21197

mgr: store declared_types in MgrSession

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17933 from jcsp/wip-17737

mgr: fix crashable DaemonStateIndex::get calls

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17944 from thmour/tmour_upmap

osd/OSDMap: upmap should respect the osd reweights

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17880 from amitkumar50/cov-rgw-9

rgw: Error check on return of read_line()

rbd-mirorr: does not start on reboot

The current systemd unit file misses 'PartOf=ceph-rbd-mirror.target',
which results in the unit not starting after reboot.
If you have ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0, it won't start
after reboot even if enabled.
Adding 'PartOf=ceph-rbd-mirror.target' will enable
ceph-rbd-mirror.target when ceph-rbd-mirror@rbd-mirror.ceph-rbd-mirror0
gets enabled.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #17961 from tchaikov/wip-doc-local-pool-in-toc

doc/mgr: add "local pool" plugin to toc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17949 from asomers/nproc_freebsd

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17940 from jcsp/wip-21534

mon: update get_store_prefixes implementations

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: silence warnings from -Wsign-compare

Fixed the warnings:

ceph/src/test/bufferlist.cc:2150:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

ceph/src/test/bufferlist.cc:2151:3:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1392:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (lhs == rhs) {

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/mgr: add "local pool" plugin to toc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17842 from MahatiC/fix_example

examples: fix link order in librados example Makefile

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17633 from yonghengdexin735/wip-zzz-doc

doc/rbd: add info for rbd group

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17803 from amitkumar50/cov-librbd-5

test: address coverity false positives

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17800 from amitkumar50/cov-misc-8

test/librbd: initialize on_finish,locker,force,snap_id

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

do_{cmake,freebsd}: Don't invoke nproc(1) on FreeBSD

Use sysctl(8) instead.  Also, there's no longer any need for /bin/bash
on FreeBSD.

Signed-off-by: Alan Somers <asomers@gmail.com>

Merge pull request #17813 from liewegas/wip-21410-note

PendingReleaseNotes: note about upmap mapping change in luminous release notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

PendingReleaseNotes: note about upmap mapping change in luminous release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17809 from yaozongyou/fix-rgw-file-write-error

rgw_file: fix write error when the write offset overlaps

Merge pull request #17947 from batrick/ceph-osd-core-dbg

ceph-debug-docker.sh: add ceph-osd-dbg package

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-debug-docker.sh: add ceph-osd-dbg package

So we can debug ceph-osd cores.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: upmap should respect osd reweights

Modify OSDMap::calc_pg_upmaps to take the osd reweight into account when
computing the size of each OSD.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>
Fixes: http://tracker.ceph.com/issues/21538

test/cli/osdmaptool: test upmap with an out osd

Test upmap with one osd marked out.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #17810 from hjwsm1989/wip-21294

qa/ceph_manager: check pg state again before timedout

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17661 from yunfeiguan/master

librados: Fix a potential risk of buffer::list::claim_prepend(list& b…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17790 from joscollin/wip-test-warning-signed-unsigned

test: silence warning from -Wsign-compare

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #17928 from xiexingguo/wip-more-interval-set-size

osd/PrimaryLogPG: calc clone_overlap size in a more efficient and concise way

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17528 from liewegas/wip-mgr-localpool

pybind/mgr/localpool: module to automagically create localized pools

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/mgr/test_localpool: simple test for localpool mode

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: add --mark-out option

Add a --mark-down <id> option which sets the reweight for a given
osd id to 0.0.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

examples: fix link order in librados example Makefile

The library link order in librados example Makefile is incorrect
and as a result 'make' throws an error. This change fixes the order.

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

mon/mgr: sync mgr_command_descs table and mgr_metadata table

sync mgr_command_descs table and mgr_metadata table to new mons

Fixes: http://tracker.ceph.com/issues/21527
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon/osd_metadata: sync osd_metadata table

sync osd_metadata table to new mons when add new mons

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mgr: fix crashable DaemonStateIndex::get calls

This function was recently fixed to return null
on missing entries: handle that properly.

Fixes: http://tracker.ceph.com/issues/17737
Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement MDSMonitor::get_store_prefixes

Fixes: http://tracker.ceph.com/issues/21534
Signed-off-by: John Spray <john.spray@redhat.com>

mon/OSDMonitor: tidy prefix definitions

We should define them in one place to make it easy
when updating get_store_prefixes.

Fixes: http://tracker.ceph.com/issues/21534
Signed-off-by: John Spray <john.spray@redhat.com>

mon/osd_metadata: sync osd_metadata table

sync osd_metadata table to new mons when add new mons

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #17831 from badone/wip-require-osd-release-with-no-up-osds

mon: osd feature checks with 0 up osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17848 from liewegas/wip-ruleset-errors

mon/OSDMonitor: error out if setting ruleset-* ec profile property

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #17902 from tchaikov/wip-21474

qa/tasks: prolong revive_osd() timeout to 6 min

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17903 from lixiaoy1/add_func_log

os/bluestore: Add missing __func__ in dout

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #17905 from tchaikov/wip-osd_types-cleanup

osd/osd_types: object_info_t: remove unused function

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17927 from liewegas/wip-21511

qa/standalone/scrub/osd-scrub-snaps: adjust test for lack of snapdir objects

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #17919 into master

* refs/remotes/upstream/pull/17919/head:
qa: whitelist cache too large in client-limits

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17918 into master

* refs/remotes/upstream/pull/17918/head:
qa: whitelist failover log warnings

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #17917 into master

* refs/remotes/upstream/pull/17917/head:
qa: ignore evicted client in client-recovery

Reviewed-by: Zheng Yan <zyan@redhat.com>

mon/OSDMonitor: error out if setting ruleset-* ec profile property

We change ruleset -> crush back in dc7a2aaf7a34b1e6af0c7b79dc44a69974c1da23.
If someone tries to use the old property, error out early, instead of
silently not doing the thing they thought they told us to do.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw_file: fix write error when the write offset overlaps.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

mgr: store declared_types in MgrSession

Because we don't (yet) properly prevent multiple sessions
from daemons reporting the same name (e.g. rgws), storing
it in the DaemonPerfCounters meant that one daemon's report
was referring to another daemon's set of reported types.

This should always have been a property of the session.

The behaviour will still be ugly when multiple daemons
are using the same name (stomping on each other's stats/statsu)
but it shouldn't crash.

Fixes: http://tracker.ceph.com/issues/21197
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #17850 from batrick/vstart-confg-path

qa: revert "qa: use config_path property instead of literal"

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #17881 from joscollin/wip-mgr-signed-unsigned-warning

mgr: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: calc clone_overlap size in a more efficient and concise way

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/standalone/scrub/osd-scrub-snaps: adjust test for lack of snapdir objects

The head_exists stuff is totally gone; those test failures go away.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17756 from jcsp/wip-19291

mgr: set explicit thread name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17808 from xiexingguo/wip-eqduot-vs-enospc

osd/PrimaryLogPG: prepare_transaction - fix EDQUOT vs ENOSPC

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17819 from neha-ojha/wip-cbt-teuthology-fix-pg-num

qa/suites/rados/perf: create pool with lower pg_num

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #17827 from tchaikov/wip-mono-heartbeat

common: use mono clock for HeartbeatMap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17844 from liewegas/wip-21259

os/bluestore: debug bluestore cache shutdown

Merge pull request #17845 from liewegas/wip-21408

os/bluestore: ignore 0x2000~2000 extent oddity from luminous upgrade

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge PR #17853 into master

* refs/remotes/upstream/pull/17853/head:
mds: check ongoing catter-gather process before capping log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>