Merge pull request #13420 from liewegas/wip-18929

osd/PG: restrict want_acting to up+acting on recovery completion

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #13479 from LiumxNL/fix-monc-rand

mon/MonClient: random all ranks then pick first_n

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13482 from idryomov/wip-qa-krbd-data-pool

qa: add workunit to test krbd data-pool support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon/MonClient: random all ranks then pick first_n

Signed-off-by: Mingxin Liu <mingxin@xsky.com>

Merge pull request #13493 from liewegas/wip-log-split-index

osd/PGLog: fix index for parent and child log on split

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13458 from tchaikov/wip-silence-gcc-warnings

include/denc, kv: silence gcc warnings

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13462 from tanghaodong25/reweight-by-pg

mon/OSDMonitor: some cleanup for reweight-by-pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13473 from tchaikov/wip-coverity-fixes

msg, messages: coverity fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12977 from weixiaowilliam/crc32c_opt

crc32c: optimize aarch64 crc32c implementation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13428 from xiexingguo/wip-flush-cache

os/bluestore: add flush_store_cache cmd

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13430 from majianpeng/tracepointprovier

common/TracepointProvider: add assert if dlopen error.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13335 from ovh/wip-clang-compilation-fix

common: Fix clang compilation

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13435 from yuyuyu101/wip-rdma-leak

msg/async/rdm: fix leak when existing failure in ip network

Reviewed-by: Adir lev <adirl@mellanox.com>

Merge pull request #13511 from tangwenjun3/wip-nvme

os/bluestore: fix bug in aio_read()

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore:fix bug in aio_read()

fixed compilation issue  in aio_read() when enable spdk

Signed-off-by: tangwenjun <tang.wenjun3@zte.com.cn>

Merge pull request #13484 from yuyuyu101/wip-rdma-perf-counter

msg/async/rdma: add perf counters to RDMA backend

Reviewed-by: Adir lev <adirl@mellanox.com>

msg/async/rdma: add perf counters to RDMA backend

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10156 from rzarzynski/wip-rgw-controllable-204

rgw: make sending Content-Length in 204 and 304 responses controllable

Merge pull request #13467 from liewegas/wip-init-ceph

init-ceph: fix ceph user args

Reviewed-by: Dan Mick <dmick@redhat.com>

osd/PGLog: avoid return by value on ginormous log

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PGLog: reindex properly on pg log split

When pg_log_t::split_out_child() runs it builds the list, which means the
old indexes are wrong (the point to bad memory), but index() will not
rebuild them because ever since b858e869e78927dccebaa350d246bd74af7f1de9
we won't rebuild them if they are already built.

Fix that by calling unindex() before the split.

Further, the new child log also needs to be indexed.  Fix that too.

Fixes: http://tracker.ceph.com/issues/18975
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13495 from ceph/revert-13452-wip-suite-repo-for-workunit

Revert "qa/tasks/workunit: use the suite repo for cloning workunit"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/perf_histogram: initialize member vars of axis_config_d

addresses CID 1400670:    (UNINIT_CTOR)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "qa/tasks/workunit: use the suite repo for cloning workunit"

Merge pull request #12370 from wido/docs-cephfs-disaster

doc: Update CephFS disaster recovery documentation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13456 from tchaikov/wip-qa-objectstore

qa/suites/rados/objectstore: enable experimental features for testing bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13437 from Liuchang0812/cleanup-rbd

librbd: use 'override' keyword instead of 'virtual'

rbd: add override in rbd subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

kv: silence the gcc warning of func foo was hidden

this silences the warnings like

ceph/ceph/src/kv/KeyValueDB.h:59:18: warning: ‘virtual void
KeyValueDB::TransactionImpl::set(const string&, const char*, size_t,
const bufferlist&)’ was hidden [-Woverloaded-\
virtual]
     virtual void set(
                  ^~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/denc: silence gcc warnings

gcc-7 complains:

ceph/ceph/src/include/denc.h:469:50: warning: enum constant in boolean
context [-Wint-in-bool-context]
 inline typename std::enable_if<traits::supported &&
                                ~~~~~~~~~~~~~~~~~~^~
           traits::featured>::type denc(
           ~~~~~~

so let's use "static constexpr bool" instead of enum.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11786 from leseb/dmcrypt-cluster-name

ceph-disk: ability to use a different cluster name with dmcrypt

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12433 from liupan1111/wip-fix-rbd-nbd-log-sock

rbd-nbd: create admin socket only for map command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

msg/async/AsyncConnection: remove no-ops

addresses CID 1400656:  Incorrect expression  (NO_EFFECT)

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/simple/Pipe: remove no-op statements

addresses CID 1400658:  Incorrect expression  (NO_EFFECT)

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDBackoff: make *_VERSION static

addresses CID 1400661:  Uninitialized variables  (UNINIT)

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/objectstore/test_fuse.sh: enable experimental features

so we can test bluestore

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/objectstore/test_fuse.sh: use portable function decl

function f() is illegal in strict POSIX shells, like dash, which is
the default /bin/sh on Ubuntu

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa/workunits/objectstore/test_fuse.sh: allow failure

This script currently has a syntax error, but still exits with
success, which is hiding that failure.  Expose it by allowing
the 'sudo' exit code to be the script's exit code.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #12797 from chendave/wip_error_message

ceph.in: Fix couple of minor issues on the messages

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13449 from xiaoxichen/mdlog_jlat

mds/MDLog.cc Fix perf counter type for jlat

rgw: make sending Content-Length in 204 and 304 controllable

This commit introduces a new configurable "rgw print prohibited
content length" to let operator decide whether RadosGW complies
to RFC 7230 (a part of the HTTP specification) or violates it
but follows the Swift's behavior.

Fixes: http://tracker.ceph.com/issues/16602
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

init-ceph: fix ceph user args

Broken by 299b7d06ac18c5cd30b8b65c7d25df9fc00287db

Signed-off-by: Sage Weil <sage@redhat.com>

qa: add workunit to test krbd data-pool support

This is based on a script that I've been using for a while for basic
smoke testing.  The matrix has exploded with the addition of data-pool
and now it's primarily a data-pool test fixture that takes minutes to
run, so turning it into a workunit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mon/OSDMonitor: some cleanup for reweight-by-pg

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #13350 from Jing-Scott/fix-swift-ver-error-code

rgw: fix error code of inexistence of versions location in swift api

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13301 from wjwithagen/wip-wjw-freebsd-manpage-location

doc: Have install put manpages in the FreeBSD correct location

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13200 from smithfarm/wip-17193-reenable-test

tests: re-enable cephfs python tests on kclient

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

doc: Have install put manpages in the FreeBSD correct location

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #13452 from tchaikov/wip-suite-repo-for-workunit

qa/tasks/workunit: use the suite repo for cloning workunit

Reviewed-by: Sage Weil <sage@redhat.com>

rbd-nbd: no need create asok file for unmap and list-mapped commands.

Fixes: http://tracker.ceph.com/issues/17951
Signed-off-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #13457 from craig08/wip-18954

ceph-disk: Fix getting wrong group name when --setgroup in bluestore

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13443 from Liuchang0812/cleanup-common

common: add override in common and misc

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk: Fix getting wrong group name when --setgroup in bluestore

ceph-disk prepare --setgroup <GROUP NAME> will be wrong when using with
bluestore

Signed-off-by: craigchi <craig10624@gmail.com>

Merge pull request #13454 from joscollin/joscollin-fix

os/FuseStore: include <functional> header in src/os/FuseStore.h for gcc 7.x

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

os/FuseStore: include <functional> header in src/os/FuseStore.h for gcc 7.x

This is done as per the Header Dependency Changes in gcc 7.
https://gcc.gnu.org/gcc-7/porting_to.html

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #13416 from dillaman/wip-rbd-mirror-cleanup

rbd-mirror: track images via global image id

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/tasks/workunit: use the suite repo for cloning workunit

as "workunits" reside in ceph/qa/workunits, it's more intuitive to
respect suite-repo option when cloning workunits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: add override for common submodule and misc

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #13406 from vumrao/wip-vumrao-18486

osd/Pool: Disallow enabling 'hashpspool' option to a pool without  '--yes-i-really-mean-it'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13370 from dreamhost/wip-18887

msg/simple/Pipe: support IPv6 QoS.

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13260 from wjwithagen/wip-wjw-freebsd-getopt-2

tools/scripts:"FreeBSD getopt is not compatible, use the one from packages"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12144 from ifed01/wip-bluestore-no-blobdepth

os/bluestore: add new garbage collector

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13390 from Liuchang0812/remove-header

os/bluestore: some cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13398 from liewegas/wip-pgnls-snaps

osd: drop support for listing objects at a given snap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #13407 from liewegas/wip-osd-msgs

osd: change a few messages at level 0 and 1; change default level to 1/5

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDLog.cc Fix perf counter type for jlat

jlat (journal latency) should be PERFCOUNTER_TIME | PERFCOUNTER_LONGRUNAVG
instead of PERFCOUNTER_U64

Current code with PERFCOUNTER_U64 make PerfCounters::tinc return
directly thus jlat will always be zero.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

os/bluestore: add flush_store_cache cmd

We want this cmd for bluestore testing, e.g., to exclude the impact of
the bluestore internal cache without rebooting host.

e.g.:
[root@tecs131 ~]# ceph --admin-daemon /run/ceph/ceph-osd.1.asok perf dump | grep "bluestore_buffers"
        "bluestore_buffers": 3166,
[root@tecs131 ~]# ceph --admin-daemon /run/ceph/ceph-osd.1.asok flush_store_cache
[root@tecs131 ~]# ceph --admin-daemon /run/ceph/ceph-osd.1.asok perf dump | grep "bluestore_buffers"
        "bluestore_buffers": 95,

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13235 from liewegas/wip-pg-split-interval

osd: have clients resend ops on pg split

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #13445 from liewegas/wip-rgw-thrash

qa/suites/rgw/thrash: add osd thrashing tests

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13434 from mikulely/cleanup-rgw-auth

rgw: cleanup unused var in rgw/rgw_rest_s3.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13412 from yuyuyu101/wip-rdma-log

msg/async/rdma: add log to show correct destruct queuepair

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rgw/thrash: add rgw with osd trashing test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw/multifs: drop redundant civetweb override

This probably breaks the frontend/apache.yaml case.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13441 from Liuchang0812/cleanup-rgw

rgw: add override in rgw subsystem

Merge pull request #13438 from Liuchang0812/cleanup-mds

mds: add override in mds subsystem

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13431 from tchaikov/wip-cython-cephfs-without-librados

pybind: cephfs should be built without librados / python-rados

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #13439 from Liuchang0812/cleanup-osd

osd: add override in osd subsystem

Reviewed-by: Sage Weil <sage@redhat.com.

Merge pull request #13436 from Liuchang0812/cleanup-mgr

mgr: add override in mgr subsystem

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13440 from Liuchang0812/cleanup-mon

mon: add override in mon subsystem

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13442 from Liuchang0812/cleanup-librados

librados: add override for librados

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13421 from dillaman/wip-18935

qa/workunits/rbd: resolve potential rbd-mirror race conditions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librados: add override for librados

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mgr: add override in mgr subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mds: add override in mds subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

common/TracepointProvider: add assert if dlopen error.

Because some reasons, if dlopen error the ceph-osd still work. But we
don't see related tracepoint by 'lttng list -u'. So add this assert to
make easily find the problem.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: add override in osd subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

msg/async/rdm: fix leak when existing failure in ip network

Signed-off-by: Haomai Wang <haomai@xsky.com>

mon: add override in mon subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: add override in rgw subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

common: Fix clang compilation

Signed-off-by: Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

Merge pull request #13317 from ktdreyer/wip-doc-ms-bind-ipv6-exclusive

doc: clarify that "ms bind ipv6" disables IPv4

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

rgw: cleanup unused var in rgw/rgw_rest_s3.cc

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

Merge pull request #13424 from dachary/wip-crush-dprintk

crush: fix dprintk compilation

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix error code of inexistence of versions location in swift api

when the versions location of container does not exist and uploading the same
object two more times to the container, it will return 412 rather than 404
in openstack swift. Meanwhile, if the dest bucket owner is not consistent with
src bucket owner, it will return 412.

http://tracker.ceph.com/issues/18880
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

pybind: cephfs: do not link against librados

otherwise python-cephfs might fail to build with followin error:

Scanning dependencies of target cython3_cephfs
/usr/bin/ld: cannot find -lrados
collect2: error: ld returned 1 exit status

Link Error: Ceph FS library not found

and apparently, cephfs *can* be built without librados. it's only
dependency is libcephfs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: cephfs: add docstring for LibCephFS's __init__()

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: fix the cstr(auth_id, ..) call

so cstr() prints the right error message if auth_id does not look right.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec: python-cephfs no longer depends on python-rados

instead, it "Recommends" python-rados. as "Recommends:" is not supported on
older distros, only enable this on SuSE.

because python-cephfs *can* be used without python-rados. the
constructor of LibCephFS accepts a ceph.conf *or* a rados.Rados
instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: fix dprintk compilation

The syntax error was not noticed because dprintk is a macro
and the code is discarded by default.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

test: fix unused function warnings due to explicit template instantiation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>