pybind/mgr: add MgrStandbyModule

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: standby modules come up and run now

...they still don't have access to any config though.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: enable running modules in standby mode

Modules can implement a second, separate class
that has access to very little state about the
system and can't implement commands.

They have just enough information to redirect
or forward incoming requests/traffic to the
active instance of the module on the active mgr.

This enables module authors to create modules
that end users can access via any (running) mgr node
at any time, rather than having to first work out
which mgr node is active.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: clean up python source file naming

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor python module management

Separate out the *loading* of modules from
the *running* of modules.

This is a precursor to enabling modules to run
in standby mode.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind/mgr: use set_uri hook from dashboard+restful modules

No more guessing the URL!

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: enable python modules to advertise their service URI

Fixes: http://tracker.ceph.com/issues/17460
Signed-off-by: John Spray <john.spray@redhat.com>

mon/MgrMonitor: store services in map and expose with command

Signed-off-by: John Spray <john.spray@redhat.com>

messages: `services` in MMgrBeacon

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MgrMap: store list of services

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: carry PyModules ref in MonCommandCompletion

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: update MgrModule for ceph_state->ceph_module

& tidy up the places where ceph_state was getting
used outside of MgrModule.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: refactor python interface

Expose a python class instead of a module,
so that we have a place to carry our reference
to our MgrPyModule* and to PyModules*, rather than
passing a handle for the former and using
a global pointer for the latter.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: remove blue highlight on scrubbing pg states

This was kind of unnecessary, highlighting a completely normal
and healthy situation in a different colour.  The blue was
also really hard to read against a grey background.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: clean up fs standby list when empty

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #18243 from jan--f/mgr-prom-osd-status

pybind/mgr/prometheus: add osd_in/out metric; make osd_weight a metric

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18651 from aermolaev/patch-1

doc: fix typo in radosgw/dynamicresharding.rst

Reviewed-By: Kefu Chai <kchai@redhat.com>

doc: fix typo in dynamicresharding.rst: admon => admin

Signed-off-by: Alexander Ermolaev <ave@integros.com>

Merge pull request #18660 from ownedu/wip-fix-async-rdma-typo

msg/async/rdma: a tiny typo fix.

Reviewed-by: Kefu Chai <kefu@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18268 from jcsp/wip-20629

mon: handle monitor lag when killing mgrs

Reviewed-By: Kefu Chai <kchai@redhat.com>

msg/async/rdma: a tiny typo fix.

Signed-off-by: Yan Lei <yongyou.yl@alibaba-inc.com>

Merge pull request #18329 from ifed01/wip-fix-mempool-dump

mempool: fix lack of pool names in mempool:dump output for JSON format

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18575 from majianpeng/msg-cleanup-2

msg/async:  misc cleanup.

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #18581 from wzliao/wip-drop-unused-parameter

osd/PrimaryLogPG: drop unused parameters

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18588 from mogeb/rados-bench-clock

tools/rados: use the monotonic clock in rados bench

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18595 from batrick/mon-clock

mon: a few conversions to monotonic clock

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18636 from tchaikov/wip-osdmap-cleanup

osd/OSDMap: remove the unnecessary checks for null

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18647 from tchaikov/wip-pgmap-cleanup

mon/PGMap: use new-style options and cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #18648 from tchaikov/wip-vstart-drop-num-opts

vstart.sh: drop --{mon,osd,mds,rgw,mgr}_num options

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #18618 into master

* refs/pull/18618/head:
mds: pass a reference of function to sanitize() not a copy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #18572 from linuxbox2/wip-rgwfile-env

rgw_log, rgw_file: account for new required envvars

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #18327 from PCzhangPC/clonelostmeta

librbd: cannot clone all image-metas if we have more than 64 key/value pairs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18546 from liewegas/wip-deprecate-features

include/ceph_features: deprecate a bunch of features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #18616 from xiexingguo/wip-hunt-scrub-mismatch

osd/PrimaryLogPG: more oi.extents fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18634 from tchaikov/wip-qa-add-restful

qa: add "restful" to ceph_mgr_modules in ceph-ansible suite

Merge pull request #18520 from alram/kube-helm-doc

doc: Kube + Helm installation

Merge pull request #18639 from yaozongyou/make-readme-properly-align

doc: make the commands in README.md properly aligned

Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #18499 from batrick/ptl-tool-testing-time

ptl-tool: make branch name configurable

Reviewed-by: Jos Collin <jcollin@redhat.com>

ptl-tool: make branch name configurable

Also, include hours/minutes in branch so multiple branches in a day have
different names.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18642 from zhangsw/cleanup-rgwputobj-getdata

rgw: add logs if get_data returns error in RGWPutObj::execute.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #18645 from tchaikov/wip-cmake-Findgperftools

cmake: should use the value of GPERFTOOLS_LIBRARIES as REQUIRED_VARS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18283 from tchaikov/wip-laggy-mgr

 qa: reduce mon-client-hunt-interval-max-multiple to 2

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

vstart.sh: drop --{mon,osd,mds,rgw,mgr}_num options

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: drop unused PGMap::update_global_delta()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/PGMap: use new-style options

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should check the availability of aligned_alloc() by running the test

this mimics the failure of rocksdb without a working
aligned_alloc()/free().

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: should use the value of GPERFTOOLS_LIBRARIES as REQUIRED_VARS

otherwise it's always true, even if the libaries are not found. also
the cmake output will be wrong without this change:

without this change, it is:

-- Found gperftools: GPERFTOOLS_TCMALLOC_LIBRARY;GPERFTOOLS_TCMALLOC_MINIMAL_LIBRARY;GPERFTOOLS_PROFILER_LIBRARY (found version "2.5.93")

with this change, it would be:

-- Found gperftools: /tmp/gperf/lib/libtcmalloc.so (found version "2.5.93")

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add logs if get_data returns error in RGWPutObj::execute.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

doc: make the commands in README.md properly aligned

The commands in README.md are aligned using one tab or eight spaces,
the github markdown engine not handle this properly, so using one tab
universely.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

osd/PrimaryLogPG: drop unused parameters

Signed-off-by: Liao Weizhong <liaoweizhong@unionpay.com>

osd/OSDMap: remove the unnecessary checks for null

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18547 from batrick/ptl-tool-trackers

ptl-tool: print bzs/tickets cited in commit msgs/comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #18495 from ceph/wip-da-SCA-20171013

osd,rgw,librbd: SCA fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18535 from yanghonggang/master

os/bluestore/bluestore_tool: add log-dump command to dump bluefs's log

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #18574 from shashalu/fix-strict-strtol

common: strict_strtol INT_MAX and INT_MIN is valid

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18577 from tchaikov/wip-kill-clang-warning

ceph-syn: silence clang analyzer warning

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa: add "restful" to ceph_mgr_modules in ceph-ansible suite

Signed-off-by: Kefu Chai <kchai@redhat.com>

ptl-tool: print bzs/tickets cited in commit msgs

This makes it easier to update issues after a merge.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18631 from gohighsec/cleanup

os/bluestore: delete redundant header file in KernelDevice.cc

Reviewed-by: Sage Weil <sage@redhat.com>

bluestore: delete redundant header file in KernelDevice.cc

Signed-off-by: Jing Li <lijing@gohighsec.com>

Merge pull request #18603 from batrick/i21945

mon/MDSMonitor: wait for readable OSDMap before sanitizing

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #18039 into master

* refs/pull/18039/head:
mgr/dashboard: redirect away if viewed filesystem is removed
mgr/dashboard: 404 instead of 500 on missing filesystem
mds: fix FSMap copy constructor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #18477 from ZVampirEM77/wip-em-cleanup

rgw: drop unused variable in copy_obj_data()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #18579 from idryomov/wip-doc-kclient-snapshots

doc/cephfs/experimental-features: kernel client snapshots limit

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #18513 from ceph/ceph-volume-zap

ceph-volume: adds the ceph-volume lvm zap subcommand

Reviewed-by: Alfredo Deza <adeza@redhat.com>

osd/PrimaryLogPG: add_object_context_to_pg_stat - fix stat

There might be holes in the extents map, thus it is not always
consistent to calc the clone bytes this way.
Instead we could switch to its own (simpler and consistent) way to
count used bytes for head-object and clone respectively.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/osd_type.cc: use "<<" directly for oi.extents

there's already a builtin operator<< for interval_set,
which is simpler.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/PrimaryLogPG: make oi.extents handle 0-length write/write_full properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #18615 from rgdoliveira/alpine

rbd-fuse: make sure PATH_MAX is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17363 from tchaikov/wip-ceph-disk

ceph-disk: unlock all partitions when activate

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #18585 from yaozongyou/fix-bluestore-tool-arg-bug

ceph-bluestore-tool: link target shouldn't ending with "\n"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17995 from scienceluo/wip-luo-fix-bluestore-metadata-branch

os/bluestore/BlueFS: Reduce unnecessary operations in collect_metadata

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rbd-fuse: make sure PATH_MAX is defined

On systems without glibc, as Alpine Linux, you might lack definition of
PATH_MAX. This patch adds the limits.h header to solve this issue.

Signed-off-by: Roberto Oliveira <robertoguimaraes8@gmail.com>

Merge pull request #18611 from liewegas/wip-interval-set

include/interval_set: parameterize by map type and kill btree_interval_set.h

Reviewed-by: Adam Emerson <aemerson@redhat.com>

tools/rados: use the monotonic clock in rados bench

Use the steady (CLOCK_MONOTONIC) clock in the bencher. The clock is used
to measure the passage of time between events, making the monotonic clock
more suited. The system clock is affected by all changes to the system's wall
clock time, which makes it unfit to compare events times relatively to each
other.

Fixes http://tracker.ceph.com/issues/21375

Signed-off-by: Mohamad Gebai <mgebai@suse.com>

Merge pull request #18507 from hjwsm1989/skip-generation-object

ceph-objectstore-tool: skip object with generated version

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18493 from liewegas/wip-unclean-degraded

mon/PGMap: 'unclean' does not imply damaged

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mds: pass a reference of function to sanitize() not a copy

to save the overhead of creating and destoying a copy of the function
closure.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/bluestore_tool: add bluefs-log-dump command to dump bluefs's log

./bin/ceph-bluestore-tool --command bluefs-log-dump --path dev/osd0/
...
0x1000: txn(seq 2 len 0xd7 crc 0x306e389b)
0x1000:  op_dir_create db
0x1000:  op_dir_create db.wal
0x1000:  op_dir_create db.slow
0x1000:  op_file_update  file

Signed-off-by: Yang Honggang <joseph.yang@xtaotech.com>

Merge pull request #18553 from liewegas/wip-preempt-feature

osd/PG: remote recovery preemption, and new feature bit to condition it on

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #18594 from zmc/wip-cephmetrics

suites/cephmetrics: Add Centos 7

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #18601 from dillaman/wip-21961

rbd-mirror: clean up spurious error log messages

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

osd/PG: handle spurious remote revoke or toofull

If we restart the recovery reservation process for some reason (e.g.,
local preemption), we may still get a racing REVOKE from the replicas in
the previous round.  Ignore.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: journal debug message should not be logged as error

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: avoid attempting to decode an empty client data buffer

Fixes: http://tracker.ceph.com/issues/21961
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #18604 from dillaman/wip-21960

cls/journal: ensure tags are properly expired

Reviewed-by: Mykola Golub <to.my.trociny@gmail.com>

unittest_interval_set: test with flat_map<> too

Signed-off-by: Sage Weil <sage@redhat.com>

include/btree_interval_set: remove it!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use interval_set<...,btree_map<...>>

This avoid the code duplication, yay!

Signed-off-by: Sage Weil <sage@redhat.com>

include/btree_map: clean up wrapper header around btree_map

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitMapAllocator: drop useless include

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: cbegin/cend -> begin/end

c{begin,end} don't exist for btree_map<>.

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: tolerate maps that invalidate iterator on change

These changes picked out of the diff between the original
btree_interval_set.h and interval_set.h (sadly I had it rolled into the
initial commit so it was tedious to identify these).

Signed-off-by: Sage Weil <sage@redhat.com>

include/interval_set: templating map used

Default to std::map for backward compat.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: handle RELEASE race with Remote*Reserved on replica

If we get a REQUEST and RELEASE in quick succession, we may process a
local Remote*Reserved back in the RepNotRecovering state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18602 from adamemerson/wip-optional-failure

rgw: Fix dereference of empty optional

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

osd/PG: fix Remote*Preempted handling in ReplicaActive

This was only half-implemented by c1e652a4547d89b889954ca4bdb82506ca0f1d83

Signed-off-by: Sage Weil <sage@redhat.com>

cls/journal: ensure tags are properly expired

Previously, if only the local image was using the journal or if
a disconnected peer was attached, the tag entries could not be
expired even if unreferenced.

Fixes: http://tracker.ceph.com/issues/21960
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph-bluestore-tool: the link target should not ending with new line

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

ceph-bluestore-tool: the right action is prime-osd-dir not prime-osd-dev

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>