mon/PGMonitor: link PGStatService to the actual PGMap

Signed-off-by: Sage Weil <sage@redhat.com>

mon: adjust pgservice based on OSDMap require_luminous flag

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove last trace of PGMap from MgrMonitor

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: send PGMapDigest instead of PGMap to mon

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: avoid pulling pgmap ratio on mkfs

If this is the first osdmap epoch there is nothing to be learned from
the PGMap.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGStatService: implement pre-luminous methods in parent

Many of these are only needed to support PGMonitor.  Implement those
in the parent and make them ceph_abort() so that the new child can avoid
the clutter.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: move summary information into parent PGMapDigest object

Everything summary-ish that we need to send to the mon is moved into a
parent class.  The child PGMap retains the detail.

The parent gets its own encode(), and PGMap::encode_digest() will call it
to encode just the summary info.

Squashed in here is a new num_pg_by_osd that could have been done
in a preceding patch but I did things in the wrong order.  :(

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: condition PGMap->creating priming on !luminous

Once we set the luminous flag we shouldn't need this anymore.  In fact,
we should stop needing this immediately after we set the monitor
luminous flag.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: use num_pg where available

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: remove version from summary

Signed-off-by: Sage Weil <sage@redhat.com>

mon: give PGMonitor and MgrMonitor their own PGStatService implementations

Right now they pretty much just copy the PGMapStatService, but it'll
let us slim down the MgrMonitor's one more easily.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: move the PGStatService lifecycle into the MgrMon/PGMon

I remain lazy enough that we just use the MgrMon right now for easier
testing.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: pgstats: move some creating_pgs maintenance into the PGStatService

And remove get_pg_map() now that there are no users!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: pgstat: remove unneeded StatIter functions

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: pgstat: expose the full pg_stat map to remove a get_pg_map() call

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: pgstat: expose the full osd_stat map to remove a get_pg_map() call

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: subclass PGStatService into PGMapStatService

Here we begin moving chunks into more appropriate locations.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: give PGStatService a reweight_by_utilization indirection

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: give PGStatService a process_pg_command indirection

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: use a pointer to PGStatService instead of putting it in Monitor

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: kill direct references to pgmon from OSDMonitor

The only one left now is where the OSDMonitor triggers
PGMonitor::check_osd_map() and that's going to die post-Luminous,
so no need to do anything fancy with it.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: get the OSDMonitor's pg info dump functions away from using PGMap

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: stop dereferencing pg_map for object stats in OSDMonitor

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: yet more OSDMonitor dep-pgmap-ification

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: more OSDMonitor de-pgmap-ification

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: OSDMonitor starting to abstract PGMap via PGService

We added an unfortunate get_pg_map() function but it's still easy
to find and puts off some of the logic work. We'll get to it soon.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: FSCommands gets pool stats via PGService

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: FSCommands gets pool stats via PGService

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: outline a PGStatService to replace raw PGMap calls

Right now, this is a quick hack that derives from PGMap. Soon, it will
be more interesting.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: mgr: send a pgmap in reports

This definitely isn't appropriate for long term but is a quick route to testing

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon: route manager reports down to the MgrMon

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mgr: introduce tick() calls through the system

Still empty for the moment

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mgr: create a skeleton MMonMgrReport message type

We'll use this for sending data to the monitor at tick intervals.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mgr: switch from send_beacon() to tick()

We are about to want similar time-based stuff in our other
classes and it's simplest to coalesce them into a single wakeup.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

MonClient: support "passthrough" of MonMap messages so other Dispatchers can see them

Use this in the Manager so it can notify of new monitor maps.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mon/MgrMap: prefix summary with epoch; add comma

This matches the others now:

    cluster 0d553373-5de6-4d90-bc60-cfd6f6554631
     health HEALTH_OK
     monmap e2: 1 mons, quorum a
      fsmap e2: 0/0/1 up
        mgr e3: active: x(starting), standbys: y
     osdmap e5: 1 osds: 0 up, 0 in
      pgmap v6: 24 pgs, 3 pools, 0 bytes data, 0 objects
            0 kB used, 0 kB / 0 kB avail
            100.000% pgs inactive
                  24 creating

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15427 from tchaikov/wip-vstart-osd-id

vstart: print "start osd.$id" instead of "start osd$id"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15217 from badone/wip-async-sleep-timer-fix

osd: Move scrub sleep timer to osdservice

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15396 from jcsp/wip-ceph-status

mon: revise "ceph status" output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15011 from xiexingguo/wip-crush-tool

tools/crushtool: replicated-rule API support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15176 from SUSE/wip-mgr-fix-help

mgr,osd:  ceph-mgr --help, unify usage text of other daemons

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15395 from liewegas/wip-log-entity

common/LogEntry: include EntityName in log entries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15398 from liewegas/wip-bluestore-leak

os/bluestore: fix buffers pinned by indefinitely deferred writes

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15402 from liewegas/wip-bluestore-mempools

os/bluestore: improve mempool usage

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15403 from liewegas/wip-mempool-dump

mempool: improve dump; fix buffer accounting bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15431 from rzarzynski/wip-rgw-vim-modeline-for-orhpantool

rgw: add the Vim's modeline into rgw_orphan.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15425 from yaozongyou/fix-build-error

cmake: link against fcgi only if enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add the Vim's modeline into rgw_orphan.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

vstart: print "start osd.$id" instead of "start osd$id"

"osd.9" is more consistent with other places where an osd is referenced.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15408 from yonghengdexin735/wip-zzz-add_const

include/lru.h: add const to member functions

Reviewed-by: Jos Collin <jcollin@redhat.com>

Fix cmake build error.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #15420 from yuriw/wip_remove_default_idle_timeout

qa: Removed all 'default_idle_timeout' due to chnage in rwg task

Merge pull request #15397 from dzafman/wip-20089

osd: Object level shard errors are tracked and used if no auth available

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/cephtool/test.sh: adjust watch test

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15417 from smithfarm/wip-cond-epoch-2

build/ops: rpm: move _epoch_prefix below Epoch definition

include/lru: add const to member function

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Removed all 'default_idle_timeout' due to chnage in rwg task
https://github.com/ceph/ceph/commit/8c74c8a639cd3dead52e1942b36f6eb3f1ceab2c#diff-995b04809fcabacc3e3ecfaea903a41aL539

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

build/ops: rpm: move _epoch_prefix below Epoch definition

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #15400 from yehudasa/wip-20128

qa/rgw_snaps: move default_idle_timeout config under the client

osd: Object level shard errors are tracked and used if no auth available

Shards with object mismatch are tracked to mark them inconsistent
Fix test because storing omap_digest in object_info not behaving as before

Fixes: http://tracker.ceph.com/issues/20089
Signed-off-by: David Zafman <dzafman@redhat.com>

osd-scrub-repair: Fix broken test and correctly handle ec overwrite testing

Caused by: 3ca750d41dfe33c6efea4abc96d2bd426a9742b9

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #15212 from neha-ojha/wip-fix-recovery-sleep-configurable

osd: Implement asynchronous recovery sleep

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15262 from Adirl/revert_cm

msg/async: Revert RDMA-CM

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15307 from ovh/bp-64bit-mem-is-zero

inline_memory: optimized mem_is_zero for non-x64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15340 from wjwithagen/wip-wjw-libcommon

cmake: do not link libcommon against some libs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15342 from chardan/jfw-wip-add_VLA_warning

build: Adds C++ warning flag for C Variable-Length Arrays.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15364 from joscollin/wip-cleanup-putto-operator-const

src: put-to operator function - const input cleanup

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15372 from ganeshmaharaj/reuse-isal

erasure-code: Remove duplicate of isa-l files

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: release aios and pinned buffers on io complete

Once we're done with our IO, clear the aio list so that the pinned buffers
are unpinned.  This ensures we release memory quickly, even if the
TransContext sticks around for a while (e.g., in the osr q).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: submit deferred if txc cleanup is blocked

If we have a single deferred write, and then a uniform workload with *no*
deferred writes, we will never actually submit it.  Meanwhile, the txc is
stuck on the osr q and nothing ever gets retired.

Simple fix is to submit any deferred ops if the osr queue is blocked by
a queued deferred write and the osr queue length is above some
threshold.  This prevents memory from being pinned indefinitely.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15209 from liewegas/wip-sortbitwise-notes

doc/release-notes: update which jewel version does sortbitwise warning

Merge pull request #15274 from liewegas/wip-auto-ccache

do_cmake.sh: enable ccache if installed

Reviewed-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: enable ccache if installed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15391 from liewegas/wip-authors

AUTHORS: update with release manager, backport team

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15410 from dachary/wip-19941-ceph-disk

ceph-disk: do not setup_statedir on trigger

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15412 from dachary/wip-choose-args

crush: add missing tunable in tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

crush: add missing tunable in tests

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15286 from smithfarm/wip-cond-epoch

build/ops: rpm: apply epoch only if %epoch macro is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15240 from ukernel/wip-20055

osdc/Journaler: avoid executing on_safe contexts prematurely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15282 from david-z/wip-fix-mantle

mds: Pass empty string to clear mantle balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15314 from renhwztetecs/renhw-wip-client-cleanup

client: fix display ino in the ldout

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15300 from ukernel/wip-20083

mds: properly create aux subtrees for pinned directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15278 from viCEnTECheNg/wip-data-pools-type

mds: change the type of data_pools

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15205 from batrick/i20039

mds: check export pin during replay

Reviewed-by: Yan, Zheng <zyan@redhat.com>

ceph-disk: do not setup_statedir on trigger

trigger may run when statedir is unavailable and does not use it.

Fixes: http://tracker.ceph.com/issues/19941
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15406 from tchaikov/wip-setuptools-vendors-six-or-not

install-deps.sh: workaround setuptools' dependency on six

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps.sh: workaround setuptools' dependency on six

see also https://github.com/pypa/setuptools/issues/1042

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: mempools for writing and writing_deferred

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_txc mempool

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rename cache mempools to bluestore_cache_*

These map directly onto the bluestore_cache_size setting.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: maybe_rebuild before adding to cache too

Signed-off-by: Sage Weil <sage@redhat.com>

AUTHORS: update with release manager, backport team

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15370 from liewegas/wip-jewel-tunables

crush, mon: make jewel the lower bound for client/crush compat for new clusters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/cli/osdmaptool/: fix tests

CRUSH tunables now default to jewel; outputs vary and
upmap results are a bit different.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix some append_buffer vs mempool assignment; add tests

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: make dump more concise

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool: adjust tests

Specify chooseleaf_stable for --build tests to match prior output.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: set mon_osd_initial_require_min_compat_client = jewel

IMO a jewel lower bound is reasonable for new clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: set default tunables to jewel

For new clusters.

This is the last "disruptive" tunable change we've made (where adjusting/
fixing it requires lots of data movement).  For new clusters a lower bound
of jewel seems pretty reasonable, and this has been upstream in Linux
for a while now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15267 from joscollin/wip-cleanup-redundant-headers-5

common: Remove redundant includes - 5

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>