buffer: remove buffer_data mempool

We are explicitly accounting for buffers via arbitrary pools, defaulting
to buffer_anon, so we don't need buffer_data anymore.  It leads to
double-counting.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: allow buffers to map into arbitrary mempools

- default is buffer_anon
- buffer_data includes *all* buffers across all mempools, and is the
*actual* allocation (includes raw_combined).
- buffer_meta is other buffer-related overhead.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: allow pool counts to be adjusted explicitly

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix mempool accounting for claimed chars

These are allocated by the caller, outside of the mempool; do not
deallocate via the mempool.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: kill unittest_? mempools

These were just annoying.  Use some real ones that will be empty
for the unit tests.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15291 from liewegas/wip-osdmap-pgtemp

osd/OSDMap: make pg_temp more efficient

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15321 from liewegas/wip-failed-msg

mon/OSDMonitor: change info in 'osd failed' messages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15388 from idryomov/wip-krbd-fallocate-tests

qa: krbd discard/zeroout tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15135 from cbodley/wip-denc-sstring

denc: add encode/decode for basic_sstring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13802 from LiumxNL/wip-170306

mon/OSDMonitor: use up set instead of acting set in reweight_by_utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15295 from liewegas/wip-bluestore-fsck

os/bluestore: fix fsck deferred_replay

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

qa: krbd discard/zeroout tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #15374 from yonghengdexin735/wip-zzz-mds

mon: Removed unnecessary function declaration in MDSMonitor.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15373 from kestrels/wip-asm-capital-S

build: Use .S suffix for ppc64le assembly files

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

mon:remove useless function for MDSMonitor::generate_mds_map()

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #15351 from liewegas/wip-osdmap-require-min-client

osd/OSDMap: replace string-based min_compat_client with a CEPH_RELEASE_* uint8_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osdmaptool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

common/version: include release name and type in version string

 ceph version 12.0.2-1687-ge042051 (e0420518d712d87a4882b3a327eaf1d3fcea12df) luminous (dev)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: switch [require_]min_compat_client to CEPH_RELEASE_* values

Instead of using a string, use the defined CEPH_RELEASE_* values.  This
is simpler.

We allow decoding of post-kraken pre-luminous maps that use the string
values.

Signed-off-by: Sage Weil <sage@redhat.com>

common: expand CEPH_RELEASE_ names; rename helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15331 from yonghengdexin735/wip-zzz-override

mon: Add override for FsNewHandler::handle()

Reviewed-by: Jos Collin <jcollin@redhat.com>

build: Use .S suffix for ppc64le assembly files

The assembly file name suffix was changed from .S to .s,
which caused a build break on ppc64le architecture.

To fix this issue, the suffix will be changed back to .S
for ppc64le assembly files only.  The x86 assembly files
can retain the .s suffix.

Fixes: http://tracker.ceph.com/issues/20106
Signed-off-by: Andrew Solomon <asolomon@us.ibm.com>

mon: Add override for FsNewHandler::handle()

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #15184 from cbodley/wip-qa-rgw-cleanup

qa/rgw: remove apache/fastcgi and radosgw-agent tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13851 from zhangsw/optimize-datasync

rgw: optimize data sync. Add zones_trace in log to avoid needless sync.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15368 from dzafman/wip-20089

osd: When scrub finds an attr error mark shard inconsistent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15350 from mrbojangles3/master

libradosstriper: Add example code

Merge pull request #15356 from joscollin/wip-test-msgr-warning

test/msgr: silence warnings from -Wsign-compare

Merge pull request #15362 from andihit/doc-rados-xattr-commands

doc: add rados xattr commands to manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15366 from joscollin/wip-cleanup-test-remove-commented-code

test/old: Removed commented code

Reviewed-by: Sage Weil <sage@redhat.com>

osd: When scrub finds an attr error mark shard inconsistent

Fixes: http://tracker.ceph.com/issues/20089
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #14971 from SUSE/wip-mgr-py-sub-interpreter

mgr: load modules in separate python sub-interpreters

Reviewed-by: John Spray <john.spray@redhat.com>

os/bluestore: bluestore_debug_fsck_abort

Abort fsck early to get a massif result.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15259 from zealoussnow/wip-0524

build: remove ceph-disk-udev entirely

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15285 from smithfarm/wip-kill-fcgi

build/ops: drop libfcgi build dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/old: Removed commented code

There were lot of commented code inside src/test/old/testcrush.cc. This was found while doing
the PR#15364. Removed them.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15359 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: add rados xattr commands to manpage

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@catalysts.cc>

Merge pull request #15306 from mikulely/fix-test_multi

rgw: fix test_multi.py default config file path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15294 from liewegas/wip-osd-tree

osd: 'osd tree in|out|up|down' to filter tree results

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15225 from smithfarm/wip-s390-rgw

build/ops: conditionalize rgw Beast frontend so it isn't built on s390x architecture

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: add up|down|in|out filters to 'osd tree'

For example, 'ceph osd tree down' will show *just* down OSDs and their
ancestors.  \o/

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: link radosgw with boost libraries

Signed-off-by: Nathan Cutler <ncutler@suse.com>

test/msgr: silence warnings from -Wsign-compare

The following warning appears during make:

ceph/src/test/msgr/test_msgr.cc: In member function ‘virtual void MessengerTest_MarkdownTest_Test::TestBody()’:
ceph/src/test/msgr/test_msgr.cc:1474:46: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     CHECK_AND_WAIT_TRUE(srv_dispatcher.count > last + 1);
                         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
ceph/src/test/msgr/test_msgr.cc:54:9: note: in definition of macro ‘CHECK_AND_WAIT_TRUE’
     if (expr)                           \
         ^~~~
ceph/src/test/msgr/test_msgr.cc:1475:30: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (srv_dispatcher.count == last) {
         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15355 from joscollin/wip-cleanup-testdenc-warning

test: test_denc.cc: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: test_denc.cc: silence warning from -Wsign-compare

The following warning appears during make:
ceph/src/test/test_denc.cc:669:5:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1530:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
ceph/src/googletest/googletest/include/gtest/gtest.h:1510:7:
   if (val1 op val2) {\

Signed-off-by: Jos Collin <jcollin@redhat.com>

os/bluestore: deep decode onode value

In particular, we want the attrs (map<string,bufferptr>) to be a deep
decode so that we do not pin this buffer, and so that any changed attr
will free the previous memory.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: use btree_set to replace set<uint64_t>

Signed-off-by: Sage Weil <sage@redhat.com>

include/cpp-btree/btree_set: add btree_set

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: init deep = false

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rebuild Buffer buffers with too much waste

Avoid pinning extra memory by rebuilding Buffer buffers when we waste too
much.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: make wasted() const

Remove useless assert (we'll segv on the next line anyway).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: wait for kv thread to start before stopping it

Otherwise we can assert out when we try to join a thread that
hasn't started.

- move everything into _kv_start() and _kv_stop()
- separate stop bools for each thread
- wait until thread starts before signalling stop (and potentially calling
join()).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fsck deferred_replay

_deferred_replay needs the kv_sync_thread to complete IOs; start them
just for that, but then shut them down again.  (We might revisit that
later if/when fsck does any sort of repair.)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15058 from yuyuyu101/wip-rocksdb-opt

kv/rocksdb: supports SliceParts interface

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15068 from liewegas/wip-osdmap-require

osd/OSDMap: replace require_*_osds flags with a single require_osd_release field

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15313 from Linbing/wip-pg-scrub-code-optimize

mon: it's no need to get pg action_primary osd twice in pg scrub

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: fix osd metadata update ordering

Do not do both.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: replace all require_*_osds flag checks with a require_osd_release comparison

Signed-off-by: Sage Weil <sage@redhat.com>

osd: replace require_*_osds flags with require_osd_release field

- OSDMap encode and decode translate between the flags and int
representations.
- OSDMap::Incremental only does decode; we do not expect to ever encode
an incremental osdmap for an old osd that sets any of these flags.
- the 'osd set' command still lets you set the jewel and kraken flags,
but not luminous.
- OSDMap::apply_incremental handles the conversion of legacy require flags
to the new field if the jewel or kraken flags have to be set before
starting the osd upgrade.
- clear out the legacy flags when we make the luminous transition only;
until then we keep using the old flag in the encoded and decoded version
(although the require_osd_release field will be accurate in memory in all
cases).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: new 'osd require-osd-release' command

Note that we only allow setting this to 'luminous'.  If the 'jewel' or 'kraken'
flags need to be set that can be done through the legacy interface.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add require_osd_release numeric field, CEPH_RELEASE_*

Instead of eating up a flag for each release, which is super-awkward and
annoying, just keep a numeric release version min.

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: delete excidentail reintroduced line.

- This target got reintroduced by accident after the creation
  of ceph-common in commit:
    https://github.com/ceph/ceph/commit/046b2bde6d93992c36d4943c4f7859993f034ca

  Detection was trigger by refering to libresolv, and results in a
  linking error.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

build/ops: deb: no rgw Beast frontend on s390x

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: no rgw Beast frontend on s390x

Since the Beast frontend uses boost::context which is not supported on
s390x.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

build/ops: rpm: reverse s390 bcond conditional block

I'm not a fan of "if NOT x - then - else" blocks.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: build boost::context and coroutine only with rgw Beast frontend

boost::context is currently (1.63) unsupported for s390x and anyway
it makes sense to conditionalize Boost components so they are only
built with the Ceph components that need them (like is already being
done for mgr).

Fixes: http://tracker.ceph.com/issues/20048
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15324 from yuyuyu101/wip-20093

msg/async: avoid requeue racing with handle_write

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15325 from liewegas/wip-redirect

osd,librados: add manifest, redirect

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15326 from xiexingguo/fix-kvtool-tip

tools/ceph_kvstore_tool: add "bluestore-kv" to usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15337 from wjwithagen/wip-wjw-freebsd-fix-includes

common: include: Redo some includes for FreeBSD

Added libradosstriper example Signed-off-by: Logan Blyth <mrbojangles3@gmail.com>

Merge pull request #15338 from trociny/wip-get_process_name

common: get_process_name: use getprogname on bsd systems

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15345 from badone/wip-tracker-19900-wait_auth_rotating-crash-2

osd: Return early on shutdown

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: Return early on shutdown

eb5c02d was missing a return statement, rectify that.

Fixes: http://tracker.ceph.com/issues/19900
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #15336 from wjwithagen/wjw-wip-iso_8601

common/iso_8601.cc: Make return expression Clang compatible

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

erasure-code/isa/ErasureCodePluginIsa.cc: fixup includes

 - Buffer.h is needed to prevent Clang seriously complaining
   about missing and wrong forward declarations of
ceph::buffer:ptr
   including buffer_fwd.h does not help.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

build/include: Redo som includes for FreeBSD

- During include cleanup just a too bit much was removed for FreeBSD
  to get things compiled.
  This redoes some of the includes.

Tracker: http://tracker.ceph.com/issues/19883
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15055 from mogeb/wip-with-instrument-functions

cmake: Add -finstrument-functions flag to OSD code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15319 from liewegas/wip-upmap

mon,osd/OSDMap: a couple pg-upmap fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15292 from liewegas/wip-map-cache

osd: reduce map cache size

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15296 from liewegas/wip-fix-at-end

qa/tasks/repair_test: unset flags we set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15297 from liewegas/wip-19595

mgr/MgrStandby: prevent use-after-free on just-shut-down Mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14284 from tchaikov/wip-19439

qa/tasks/rebuild_mondb: grant "mgr:allow *" to client.admin

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15284 from majianpeng/bluestore-batch-throotle

os/bluestore: batch throttle

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15302 from xiexingguo/wip-fix-reg-leak

common/ceph_context: fix leak of registered commands on exit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15320 from liewegas/wip-peon-mapping

mon/OSDMonitor: cancel mapping job from update_from_paxos

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/tasks/rebuild_mondb: update to address ceph-mgr changes

- revive ceph-mgr after updating the keyring cap
- grant "mgr:allow *" to client.admin
- minor refactors

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: get_process_name: use getprogname on bsd systems

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

common/iso_8601.cc: Make return expression Clang compatible

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

tools/ceph_kvstore_tool: add "bluestore-kv" to usage

We are soon going to use BlueStore as the default OS, and
hence "bluestore-kv" shall be the preferred option.

By applying this patch, the help message shall be slightly
nicer to newcomers to this tool now.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph_test_rados_api_tier: skip redirect tests if not luminous

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15111 from Liuchang0812/wip-19885-mgr-help

ceph: introduce "tell x help" subcommand

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-luminous: redirects require luminous

...so skip the at-end variant of d-luminous.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-extensible_tier-redirect' of git://github.com/myoungwon/ceph into wip-redirect

rgw: fix test_multi.py default config file path

$ python test_multi.py --num-zonegroups=2 --num-zones 2 --gateways-per-zone 1
Traceback (most recent call last):
  File "test_multi.py", line 315, in <module>
      init(True)
        File "test_multi.py", line 162, in init
    path = tpath('test_multi.conf')
    NameError: global name 'tpath' is not defined

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

msg/async: keep _has_next_outgoing calling under write_lock

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: avoid requeue racing with handle_write

when the thread is calling AsyncConnection::handle_write, another thread may
replace it and requeue all messages. Because we remove the write_lock
protection for handle_write caller, it may happen sent racing with out_q

Fix: http://tracker.ceph.com/issues/20093

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15289 from ifed01/wip-bluestore-fix-bmap-assert

os/bluestore: fix BitMapAllocator assert on out-of-bound hint value

Reviewed-by: Varada Kari <varada.kari@gmail.com>