mon/PGMonitor: assert PGMonitor stops proposing after luminous

One hacky exception here for the rebuild-mondb case.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MgrStatMonitor: keep mgrstat version ahead of pgmon

Pre-luminous, statfs and getpoolstats come from pgmon, and clients will
see and track the paxos version for that.  After the switchover, those
move to mgrstatmon.  If a clients sends a version ahead of the current
mgrstat version, it will block until the paxos version catches up, which
may be years for an aged cluster.

Fix by keeping mgrstat version ahead of pgmon's every time it updates.
This only really matters right at the transition point (we don't update
before that, and after that pgmon doesn't update).

Fixes: http://tracker.ceph.com/issues/20219
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15507 from zhangsw/fix-rgw-logtrim-bug

rgw: datalog trim and mdlog trim handles the result returned by osd incorrectly.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15102 from majianpeng/blockdevice-pmem

os/bluestore/BlockDevice: support pmem device as bluestore backend

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15577 from jcsp/wip-dashboard-cleanup

mgr: dashboard code cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: remove confusing SyncObject stuff

This was a hangover from when these wrapper classes
were borrowed from Calamari, which used these
versions/equality functions to work out when
to go fetch data from the ceph cluster.

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: fix server view on funny version format

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: fix error handling in remote_view_cache

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/dashboard: keep a global librados instance

...so that classes that need one aren't creating
their own all the time.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #15566 from baruza/wip-doc-20199

doc: Change the default values of some OSD options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15485 from ceph/wip-hdd-throttle-cost

common/config_opts.h: Lower HDD throttle cost.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15203 from Liuchang0812/wip-kill-broken-links

doc: kill some broken links

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15575 from liewegas/wip-pgnls-note

PendingReleaseNotes: notes on whiteouts vs pgnls

Merge pull request #15309 from ifed01/wip-bluestore-boost-statfs

os/bluestore: keep statfs replica in RAM to avoid expensive KV retrieval

Reviewed-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: notes on whiteouts vs pgnls

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15532 from dillaman/wip-rbd-mirror-stress

rbd-mirror: lock loss during sync should wait for in-flight copies

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #15518 from dillaman/wip-rbd-missing-default-pool

rbd: removed hardcoded default pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: introduce a debug switch to bypass kv update.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: introduce debug parameter to bypass bdev verification

E.g. this allows to backup WAL/DB volumes after preconditioning and quickly rollback to that state when needed.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: keep statfs replica in memory to avoid expensive KV access

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: move volatile_statfs struct out of TransContext

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #15433 from wjwithagen/wip-wjw-prettytable

ceph-disk: add setting for external py-modules for tox-testing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Change the default values of some OSD options

These values were changed to reflect this change a3f9cba

Fixes: http://tracker.ceph.com/issues/20199
Signed-off-by: Bara Ancincova <bara@redhat.com>

ceph-disk: use system modules if needed

 - prettytable usage was introduced in:
https://github.com/ceph/ceph/commit/3fa8bb1
   It is in the install-deps.sh file to be installed, but
   it is not per default installed in the testenvironment
   So allow tox tests to use "external" modules

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #14866 from chardan/jfw-wip-halflife_atomic_t-varan

common,test: migrate atomic_t to std::atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15520 from liewegas/wip-max-object-size

osd: reduce rados_max_object_size from 100 GB -> 128 MB

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: reduce rados_max_object_size from 100 GB -> 128 MB

I'm not sure why we had this so huge!  128 MB is a pretty reasonable size.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15230 from chardan/jfw-wip-halflife_atomic_t-xio

xio: migrate atomic_t to std::atomic<>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15516 from javacruft/i386-cmake-fixes

cmake: misc fixes for build on i386

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15525 from wjwithagen/wip-wjw-clang-testosdmap

test: osd/TestOSDMap.cc: fix Clang complain about promotion

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: removed hardcoded default rbd pool name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: removed default rbd pool name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-replay: removed default rbd pool name

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: module should depend on librbd not rbd CLI

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: new 'rbd_default_pool' configuration option

The 'rbd' pool will no longer be automatically created. Allow
the user to specify a custom default RBD pool name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: narrowed scope of exclusive lock op tracking

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: partial revert of 649b29d

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15498 from adamemerson/wip-fixation

test: Fix narrowing conversion
rgw: Support certain archaic and antiquated distributions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15436 from liewegas/wip-versions

mon: {mon,osd,mds} {versions,count-metadata}

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15501 from liewegas/wip-blocked-is-err

mon/PGMap: call blocked requests ERR not WARN

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon/PGMap: call requests blocked for 128x as long ERR not WARN

- rename the option (max -> warn)
- add an err_..._ratio multiplier
- switch to HEALTH_ERR once requests are blocked long enough
- make the error ratio high (default is 32*128s -> about an hour) so that
we don't trigger on a heavily loaded cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: Support certain archaic and antiquated distributions

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Fix narrowing conversion

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os/bluestore/BlockDevice:support pmem device as bluestore backend.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #15531 from liewegas/wip-osd-pglog

osd: rename osd -> osd_pglog; include pglog-related bufferlists

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd/osd_types: put pg_log_entry_t::extra_reqids in osd_pglog mempool

Signed-off-by: Sage Weil <sage@redhat.com>

osd: put pg_log_entry_t::snaps in osd_pglog mempool

Signed-off-by: Sage Weil <sage@redhat.com>

osd: map ObjectModDesc to osd_pglog pool

This can consume significant space for EC pools.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move pglog to osd_pglog mempool

This actually everything in the osd mempool, so it's basically a rename.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15413 from tchaikov/wip-rgw-boost-cleanup

rgw: drop asio/{yield,coroutine}.hpp replacements

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #15434 from yuyuyu101/wip-lz4

compressor: add LZ4 support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15512 from jcsp/wip-mgrmap-epoch

mon: don't prefix mgr summary with epoch number

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15437 from dachary/wip-bluestore

ceph-disk: add --filestore argument, default to --bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15547 from wjwithagen/wip-wjw-freebsd-get_device_by_fd

common/blkdev.cc: propagate get_device_by_fd to different OSes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15435 from liewegas/wip-bluestore-warning

os/bluestore: fix warning

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #15371 from liewegas/wip-mon-client-features

mon: track features from connect clients, and use it to gate set-require-min-compat-client

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/blkdev.cc: propagate get_device_by_fd to different OSes

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15530 from tchaikov/wip-jq-without-first

qa/workunits/ceph-helpers.sh: use syntax understood by jq 1.3

Reviewed-by: David Zafman <dzafman@redhat.com>

doc: kill some broken links

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #15423 from liewegas/wip-fix-indent

mon/FSCommand: fix indentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15497 from liewegas/wip-log-last

mon/LogMonitor: 'log last' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15522 from liewegas/wip-19901

ceph_test_rados_api_misc: fix LibRadosMiscConnectFailure.ConnectFailure retry

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: validate set-require-min-compat-client against connected clients

Prevent the user from setting a require_min_compat_client if a disallowed
client is currently connected to the mon cluster.  Instead, you'll get an
error like

Error EPERM: cannot set require_min_compat_client to luminous: 1 connected client(s) look like jewel (missing 0x800000000200000); add --yes-i-really-mean-it to do it anyway

Signed-off-by: Sage Weil <sage@redhat.com>

common: ceph_release_[from_]features

Map releases to client features (and back again).

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add 'features' command to show connected client features

for whole mon cluster

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15422 from liewegas/wip-osd-rotational

osd: hdd vs ssd defaults for osd op thread pool

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15517 from liewegas/wip-20197

qa/suites/rados/upgrade: restart mds

Merge pull request #15523 from ceph/wip-16674

selinux: Allow read on var_run_t

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: datalog trim and mdlog trim handles the result returned by osd
incorrectly.

Fixes: http://tracker.ceph.com/issues/20190
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

qa/workunits/ceph-helpers.sh: use syntax understood by jq 1.3

trusty still ships jq 1.3 which does not offer "first". see
https://stedolan.github.io/jq/manual/v1.3/ .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14745 from trociny/wip-18789

rbd-mirror: coordinate image syncs with leader

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15243 from yuyuyu101/wip-20049

msg: don't set msgr addr when disabing client bind

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14117 from zhangsw/fix-rgw-getobjectmetadata-bug

rgw: return the version id in get object and object metadata request.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15386 from ivancich/wip-add-readme-that-dmclock-is-a-subtree

doc: add README to dmclock subdir to inform developers it's a git subtree

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15392 from dillaman/wip-20049

msg: do not enable client-side binding by default

Reviewed-by: Haomai Wang <haomai@xsky.com>

test: osd/TestOSDMap.cc: fix Clang complain about promotion

/home/jenkins/workspace/ceph-freebsd/src/test/osd/TestOSDMap.cc:444:15: error: non-constant-expression cannot be narrowed from type 'unsigned int' to 'int' in initializer list [-Wc++11-narrowing]
    m.set(x, {i});
              ^
/home/jenkins/workspace/ceph-freebsd/src/test/osd/TestOSDMap.cc:444:15: note: insert an explicit cast to silence this issue
    m.set(x, {i});
              ^
              static_cast<int>( )
1 error generated.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15513 from Fabian-Gruenbichler/fixmgrdesc

debian: ceph-mgr: fix package description

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15375 from ivancich/wip-fix-dmclock-testing

osd/dmclock/testing: reorganize testing, building now optional

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15378 from Liuchang0812/print_ret_code_of_json_command

ceph.in: print return code when json_command failed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15496 from wjwithagen/wip-wjw-host-to-ceph-errno

core: include/types.h, introduce host_to_ceph_errno

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15500 from liewegas/wip-20135

pybind/ceph_argparse: fix empty string check

Merge pull request #15502 from xiexingguo/wip-fix-overload-extents

os/bluestore: avoid overloading extents during reshard; atomic deferred_batch_ops

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: restart mds

Signed-off-by: Sage Weil <sage@redhat.com>

selinux: Allow read on var_run_t

Fixes: http://tracker.ceph.com/issues/16674
Signed-off-by: Boris Ranto <branto@redhat.com>

ceph_test_rados_api_misc: fix LibRadosMiscConnectFailure.ConnectFailure retry

Fixes: http://tracker.ceph.com/issues/19901
Signed-off-by: Sage Weil <sage@redhat.com>

osd/dmclock/testing: reorganize testing, building now optional

Modify cmake files to take advantage or re-organization of dmclock's
cmake set-up, specifically not having dmclock's cmake files call
add_test. Remove dmclock tests from being dependencies on ceph's
"test" target. Make change so dmclock tests are not built unless
WITH_TESTS and WITH_DMCLOCK_TESTS are both set. This is so openSUSE
Leap will build correctly.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge commit '5c612d7e32a07a01b3b97f9cf58f2a3cef0e2e73'

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Squashed 'src/dmclock/' changes from d6586d7..64bcc6e

64bcc6e Remove FindGTest and FindBoost, which already exist in cmake required version 2.8.11 and later. Also clean up.
ed6dc60 Do some clean-up and remove memory leaks from tests and simulations.
360172a Allow RunEvery object to be ended before destruction via "join" member function.
135b5d0 Fix two bugs discovered by valgrind. Do some other code clean-up as well.
a26d29e Modify cmake files to match naming of cmake's upstream gtest and ceph's boost.
e23e63f Reorganize CMakeLists.txt files, so that projects that incorporate dmclock can control whether add_test is called. Now, add_test is called only at the top-level. Projects that incorporate dmclock can use add_subdirectory on subdirectories beneath the top-level to selective exactly what is included.

git-subtree-dir: src/dmclock
git-subtree-split: 64bcc6ea697c7e790ec4a4229d1bb1d492f9e803

qa/workunits/cephtool/test.sh: test 'log last [n]'

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogEntry: make LogEntryKey opaque and cache its hash value

This avoids recalculating.  We eliminate the default ctor at the same
time and make the type totally opaque (so that its fields cannot be
modified except by the ctor and decode).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15390 from liewegas/wip-osdmap-state

osd/OSDMap: make osd_state 32 bits wide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15515 from tchaikov/wip-crc32-aarch64

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

Reviewed-by: Dan Mick <dmick@redhat.com>

ceph-disk: add --filestore argument, default to --bluestore

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Loic Dachary <loic@dachary.org>

build: misc fixes for cmake build on i386

Scope SIMD CPU flag detection base on target architecture,
resolving build failures in gf-complete on i386; this is
inline with the previous autotools based feature detection.

Only compile crypto/isa-l if yasm 64 bit complier is found,
effectively limiting scope to x86_64.

Signed-off-by: James Page <james.page@ubuntu.com>

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

HAVE_ARMV8_CRC is defined by "acconfig.h". so we need to include it for
the definition.

it's a regresssion introduced by 2a3382f. in that change,
crc32c_intel_baseline.h was removed from crc32.cc, and that header
includes accconfig.h indirectly. that's why we will have a dummy
ceph_crc32c_aarch64() which always returns 0. the dummy implementation
was intended to please the compiler. but we can probably just drop it in
future as a cleanup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15511 from tchaikov/wip-doc-cartesian-product

doc: update sample explaning "%" operator in test suites

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #14074 from jecluis/wip-mon-osd-replacement

mon: replace osds with `osd destroy` and `osd new`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #15487 from liewegas/wip-no-more-stat-spam

mgr/DaemonServer: stop spamming log with pg stats

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15494 from liewegas/wip-20187

qa/workunits/cephtool/test.sh: fix osd full health detail grep

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: fix no arg check

This fixes breakage from commit a1214a702cfd8465deb91847f4dc63c0e80fb586
that caused a [] to be appended to some json argument lists.

Fixes: http://tracker.ceph.com/issues/20135
Signed-off-by: Sage Weil <sage@redhat.com>