Merge pull request #15274 from liewegas/wip-auto-ccache

do_cmake.sh: enable ccache if installed

Reviewed-by: Kefu Chai <kchai@redhat.com>

do_cmake.sh: enable ccache if installed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15391 from liewegas/wip-authors

AUTHORS: update with release manager, backport team

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #15410 from dachary/wip-19941-ceph-disk

ceph-disk: do not setup_statedir on trigger

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15412 from dachary/wip-choose-args

crush: add missing tunable in tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

crush: add missing tunable in tests

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15286 from smithfarm/wip-cond-epoch

build/ops: rpm: apply epoch only if %epoch macro is defined

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15240 from ukernel/wip-20055

osdc/Journaler: avoid executing on_safe contexts prematurely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #15282 from david-z/wip-fix-mantle

mds: Pass empty string to clear mantle balancer

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15314 from renhwztetecs/renhw-wip-client-cleanup

client: fix display ino in the ldout

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #15300 from ukernel/wip-20083

mds: properly create aux subtrees for pinned directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #15278 from viCEnTECheNg/wip-data-pools-type

mds: change the type of data_pools

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15205 from batrick/i20039

mds: check export pin during replay

Reviewed-by: Yan, Zheng <zyan@redhat.com>

ceph-disk: do not setup_statedir on trigger

trigger may run when statedir is unavailable and does not use it.

Fixes: http://tracker.ceph.com/issues/19941
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15406 from tchaikov/wip-setuptools-vendors-six-or-not

install-deps.sh: workaround setuptools' dependency on six

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

install-deps.sh: workaround setuptools' dependency on six

see also https://github.com/pypa/setuptools/issues/1042

Signed-off-by: Kefu Chai <kchai@redhat.com>

AUTHORS: update with release manager, backport team

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15370 from liewegas/wip-jewel-tunables

crush, mon: make jewel the lower bound for client/crush compat for new clusters

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test/cli/osdmaptool/: fix tests

CRUSH tunables now default to jewel; outputs vary and
upmap results are a bit different.

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/crushtool: adjust tests

Specify chooseleaf_stable for --build tests to match prior output.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: set mon_osd_initial_require_min_compat_client = jewel

IMO a jewel lower bound is reasonable for new clusters.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: set default tunables to jewel

For new clusters.

This is the last "disruptive" tunable change we've made (where adjusting/
fixing it requires lots of data movement).  For new clusters a lower bound
of jewel seems pretty reasonable, and this has been upstream in Linux
for a while now.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15267 from joscollin/wip-cleanup-redundant-headers-5

common: Remove redundant includes - 5

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15311 from dachary/wip-crush-add-remove

crush: update choose_args when items are added/removed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15318 from tchaikov/wip-20067

mon: trim the creating_pgs after updating it with pgmap

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15352 from liewegas/wip-buffer-mempools

buffer: allow buffers to be accounted in arbitrary mempools

Merge pull request #15357 from ceph/typo_bluestore

os/bluestore: fix a typo about bleustore

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #15385 from myoungwon/wip-20114

test/osd/TestRados.cc: run set-redirect test after finishing setup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14457 from ceph/wip-rest-flask

ceph-mgr: Implement new pecan-based rest api

Merge pull request #15222 from guihecheng/rgw_file-fix-unlink

rgw_file: fix flags set on unsuccessful unlink

Merge pull request #15329 from guihecheng/rgw_file-readdir

rgw_file: add lock protection for readdir against gc

Merge pull request #14871 from ceph/wip-selinux-optimize

selinux: Do parallel relabel on package install

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

mempool: dump total, too

Signed-off-by: Sage Weil <sage@redhat.com>

osd: (loosely) mark cached map bl buffers as osd_mapbl

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: mark buffer caches buffers as bluestore_data

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: remove buffer_data mempool

We are explicitly accounting for buffers via arbitrary pools, defaulting
to buffer_anon, so we don't need buffer_data anymore.  It leads to
double-counting.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: allow buffers to map into arbitrary mempools

- default is buffer_anon
- buffer_data includes *all* buffers across all mempools, and is the
*actual* allocation (includes raw_combined).
- buffer_meta is other buffer-related overhead.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: allow pool counts to be adjusted explicitly

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: fix mempool accounting for claimed chars

These are allocated by the caller, outside of the mempool; do not
deallocate via the mempool.

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: kill unittest_? mempools

These were just annoying.  Use some real ones that will be empty
for the unit tests.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15291 from liewegas/wip-osdmap-pgtemp

osd/OSDMap: make pg_temp more efficient

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15321 from liewegas/wip-failed-msg

mon/OSDMonitor: change info in 'osd failed' messages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15388 from idryomov/wip-krbd-fallocate-tests

qa: krbd discard/zeroout tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15135 from cbodley/wip-denc-sstring

denc: add encode/decode for basic_sstring

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13802 from LiumxNL/wip-170306

mon/OSDMonitor: use up set instead of acting set in reweight_by_utilization

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15295 from liewegas/wip-bluestore-fsck

os/bluestore: fix fsck deferred_replay

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

test/osd/TestRados.cc: run test after finishing redirect setup

set-redirect test creates more objects than default due to redirect mapping
(let's assume that we create 500 objects, oid 1 -> oid 501, oid 2 -> oid 502)
But the test should use objects within pre-defined range. For example,
If the number of object is 500, only 1~500 oid are allowed for test operations.

In the failure cases, test operation tried to use wrong oids (which means
higher than 500). This is because the bug in the setup phase.
If max_inflights > 1, set-redirect can be finished after setup is completed.
(so wrong oid is included in oid_in_use).
We need to completely remove redirect objects in oid_in_use before start the test.

Fixes: http://tracker.ceph.com/issues/20114
Signed-off-by: Myoungwon Oh <omwmw@sk.com>

qa: krbd discard/zeroout tests

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rpm: Move ceph-disk to ceph-base

The SELinux package now requires the ceph-disk binary but that one was
part of the ceph-osd package. The ceph-disk python library is already
packaged in ceph-base so moving ceph-disk to ceph-base seems like a
reasonable next step.

Signed-off-by: Boris Ranto <branto@redhat.com>

rgw_file: add lock protection for readdir against gc

The state.mtx protects state.events accessed both from readdir and gc.
Fixes: http://tracker.ceph.com/issues/20121
Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

Merge pull request #15374 from yonghengdexin735/wip-zzz-mds

mon: Removed unnecessary function declaration in MDSMonitor.h

Reviewed-by: Jos Collin <jcollin@redhat.com>

crush: update choose_args on adjust_item_weight

The osd_crush_update_weight_set (true by default) can be used to disable
the update of the weights.

Signed-off-by: Loic Dachary <loic@dachary.org>

crush: update choose_args when items are added/removed

When items are added:

- the value in the weight set is set to the target weight. It is assumed
  that weight set are updated on a regular basis and will eventually be
  set to a value that prevents excessive over/under filling.

- the value in the id list is set to the item id.

When items are removed, their weight / ids in all choose_args are
removed.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #15373 from kestrels/wip-asm-capital-S

build: Use .S suffix for ppc64le assembly files

Reviewed-by: Bassam Tabbara <bassam.tabbara@quantum.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

mon:remove useless function for MDSMonitor::generate_mds_map()

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

Merge pull request #15351 from liewegas/wip-osdmap-require-min-client

osd/OSDMap: replace string-based min_compat_client with a CEPH_RELEASE_* uint8_t

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osdmaptool: fix cli tests

Signed-off-by: Sage Weil <sage@redhat.com>

common/version: include release name and type in version string

 ceph version 12.0.2-1687-ge042051 (e0420518d712d87a4882b3a327eaf1d3fcea12df) luminous (dev)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: switch [require_]min_compat_client to CEPH_RELEASE_* values

Instead of using a string, use the defined CEPH_RELEASE_* values.  This
is simpler.

We allow decoding of post-kraken pre-luminous maps that use the string
values.

Signed-off-by: Sage Weil <sage@redhat.com>

common: expand CEPH_RELEASE_ names; rename helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15331 from yonghengdexin735/wip-zzz-override

mon: Add override for FsNewHandler::handle()

Reviewed-by: Jos Collin <jcollin@redhat.com>

build: Use .S suffix for ppc64le assembly files

The assembly file name suffix was changed from .S to .s,
which caused a build break on ppc64le architecture.

To fix this issue, the suffix will be changed back to .S
for ppc64le assembly files only.  The x86 assembly files
can retain the .s suffix.

Fixes: http://tracker.ceph.com/issues/20106
Signed-off-by: Andrew Solomon <asolomon@us.ibm.com>

mon: Add override for FsNewHandler::handle()

Signed-off-by: yonghengdexin735 <zhang.zezhu@zte.com.cn>

ceph-disk: Fix the file ownership, skip missing

This commit fixes the file ownership for the /usr/bin/ and /etc/ceph
files and skips missing files as some of the files that we do specify
now can be missing from the system (not installed, e.f. radosgw).

Fixes: http://tracker.ceph.com/issues/20077
Signed-off-by: Boris Ranto <branto@redhat.com>

selinux: Do parallel relabel on package install

We can take advantage of ceph-disk fix subcommand when doing a package
install. We will keep using the differential fixfiles command otherwise.

We also need to add relabel for /usr/bin/ daemons so that we could use
this.

Fixes: http://tracker.ceph.com/issues/20077
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #15184 from cbodley/wip-qa-rgw-cleanup

qa/rgw: remove apache/fastcgi and radosgw-agent tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13851 from zhangsw/optimize-datasync

rgw: optimize data sync. Add zones_trace in log to avoid needless sync.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15368 from dzafman/wip-20089

osd: When scrub finds an attr error mark shard inconsistent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15350 from mrbojangles3/master

libradosstriper: Add example code

Merge pull request #15356 from joscollin/wip-test-msgr-warning

test/msgr: silence warnings from -Wsign-compare

Merge pull request #15362 from andihit/doc-rados-xattr-commands

doc: add rados xattr commands to manpage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15366 from joscollin/wip-cleanup-test-remove-commented-code

test/old: Removed commented code

Reviewed-by: Sage Weil <sage@redhat.com>

osd: When scrub finds an attr error mark shard inconsistent

Fixes: http://tracker.ceph.com/issues/20089
Signed-off-by: David Zafman <dzafman@redhat.com>

mds: check export pin during replay

Fixes: http://tracker.ceph.com/issues/20039
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: improve debug message for subtree wait

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use generic name for export dir wait class

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14971 from SUSE/wip-mgr-py-sub-interpreter

mgr: load modules in separate python sub-interpreters

Reviewed-by: John Spray <john.spray@redhat.com>

os/bluestore: bluestore_debug_fsck_abort

Abort fsck early to get a massif result.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15259 from zealoussnow/wip-0524

build: remove ceph-disk-udev entirely

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #15285 from smithfarm/wip-kill-fcgi

build/ops: drop libfcgi build dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/old: Removed commented code

There were lot of commented code inside src/test/old/testcrush.cc. This was found while doing
the PR#15364. Removed them.

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15359 from liewegas/wip-kraken-x

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: add rados xattr commands to manpage

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@catalysts.cc>

Merge pull request #15306 from mikulely/fix-test_multi

rgw: fix test_multi.py default config file path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/upgrade/kraken-x: enable experimental for bluestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15294 from liewegas/wip-osd-tree

osd: 'osd tree in|out|up|down' to filter tree results

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15225 from smithfarm/wip-s390-rgw

build/ops: conditionalize rgw Beast frontend so it isn't built on s390x architecture

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: add up|down|in|out filters to 'osd tree'

For example, 'ceph osd tree down' will show *just* down OSDs and their
ancestors.  \o/

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: link radosgw with boost libraries

Signed-off-by: Nathan Cutler <ncutler@suse.com>

bluestore: fix a typo about bleustore

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

test/msgr: silence warnings from -Wsign-compare

The following warning appears during make:

ceph/src/test/msgr/test_msgr.cc: In member function ‘virtual void MessengerTest_MarkdownTest_Test::TestBody()’:
ceph/src/test/msgr/test_msgr.cc:1474:46: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     CHECK_AND_WAIT_TRUE(srv_dispatcher.count > last + 1);
                         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
ceph/src/test/msgr/test_msgr.cc:54:9: note: in definition of macro ‘CHECK_AND_WAIT_TRUE’
     if (expr)                           \
         ^~~~
ceph/src/test/msgr/test_msgr.cc:1475:30: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (srv_dispatcher.count == last) {
         ~~~~~~~~~~~~~~~~~~~~~^~~~~~~

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15355 from joscollin/wip-cleanup-testdenc-warning

test: test_denc.cc: silence warning from -Wsign-compare

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: test_denc.cc: silence warning from -Wsign-compare

The following warning appears during make:
ceph/src/test/test_denc.cc:669:5:   required from here
ceph/src/googletest/googletest/include/gtest/gtest.h:1530:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
ceph/src/googletest/googletest/include/gtest/gtest.h:1510:7:
   if (val1 op val2) {\

Signed-off-by: Jos Collin <jcollin@redhat.com>

os/bluestore: deep decode onode value

In particular, we want the attrs (map<string,bufferptr>) to be a deep
decode so that we do not pin this buffer, and so that any changed attr
will free the previous memory.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: use btree_set to replace set<uint64_t>

Signed-off-by: Sage Weil <sage@redhat.com>

include/cpp-btree/btree_set: add btree_set

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-bluestore-tool: init deep = false

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: rebuild Buffer buffers with too much waste

Avoid pinning extra memory by rebuilding Buffer buffers when we waste too
much.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: make wasted() const

Remove useless assert (we'll segv on the next line anyway).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: wait for kv thread to start before stopping it

Otherwise we can assert out when we try to join a thread that
hasn't started.

- move everything into _kv_start() and _kv_stop()
- separate stop bools for each thread
- wait until thread starts before signalling stop (and potentially calling
join()).

Signed-off-by: Sage Weil <sage@redhat.com>