qa/suites/rgw/thrash: add rgw with osd trashing test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw/multifs: drop redundant civetweb override

This probably breaks the frontend/apache.yaml case.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13441 from Liuchang0812/cleanup-rgw

rgw: add override in rgw subsystem

Merge pull request #13438 from Liuchang0812/cleanup-mds

mds: add override in mds subsystem

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13431 from tchaikov/wip-cython-cephfs-without-librados

pybind: cephfs should be built without librados / python-rados

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #13439 from Liuchang0812/cleanup-osd

osd: add override in osd subsystem

Reviewed-by: Sage Weil <sage@redhat.com.

Merge pull request #13436 from Liuchang0812/cleanup-mgr

mgr: add override in mgr subsystem

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13440 from Liuchang0812/cleanup-mon

mon: add override in mon subsystem

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13442 from Liuchang0812/cleanup-librados

librados: add override for librados

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13421 from dillaman/wip-18935

qa/workunits/rbd: resolve potential rbd-mirror race conditions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librados: add override for librados

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mgr: add override in mgr subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mds: add override in mds subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

osd: add override in osd subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

mon: add override in mon subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: add override in rgw subsystem

Fixes: http://tracker.ceph.com/issues/18922
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #13317 from ktdreyer/wip-doc-ms-bind-ipv6-exclusive

doc: clarify that "ms bind ipv6" disables IPv4

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #13424 from dachary/wip-crush-dprintk

crush: fix dprintk compilation

Reviewed-by: Sage Weil <sage@redhat.com>

pybind: cephfs: do not link against librados

otherwise python-cephfs might fail to build with followin error:

Scanning dependencies of target cython3_cephfs
/usr/bin/ld: cannot find -lrados
collect2: error: ld returned 1 exit status

Link Error: Ceph FS library not found

and apparently, cephfs *can* be built without librados. it's only
dependency is libcephfs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: cephfs: add docstring for LibCephFS's __init__()

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: fix the cstr(auth_id, ..) call

so cstr() prints the right error message if auth_id does not look right.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec: python-cephfs no longer depends on python-rados

instead, it "Recommends" python-rados. as "Recommends:" is not supported on
older distros, only enable this on SuSE.

because python-cephfs *can* be used without python-rados. the
constructor of LibCephFS accepts a ceph.conf *or* a rados.Rados
instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush: fix dprintk compilation

The syntax error was not noticed because dprintk is a macro
and the code is discarded by default.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13089 from jcsp/wip-18600

#18600: Clear out tasks that don't make sense from multimds suite

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13093 from batrick/client-deadcode

client: remove dead log code

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13272 from jcsp/wip-18830

mds: fix bad iterator dereference reported by coverity

Reviewed by: Yan, Zheng <zyan@redhat.com>

Merge pull request #13347 from david-z/wip-straymanager-reuse-deleted-inode

mds/StrayManager: avoid reusing deleted inode in StrayManager::_purge_stray_logged

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13417 from liewegas/wip-find-dups

script/find_dups_in_pg_log: scrip to find dup requests due to short pg logs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13419 from dmick/wip-17826

ceph_common.sh: fix syntax error

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: resolve potential rbd-mirror race conditions

Fixes: http://tracker.ceph.com/issues/18935
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph_common.sh: fix syntax error

Introduced by 299b7d06ac18c5cd30b8b65c7d25df9fc00287db
Fixes: http://tracker.ceph.com/issues/17826
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #13311 from asheplyakov/master-18820

ceph-osd: --flush-journal: sporadic segfaults on exit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13307 from songbaisen/s12

osd:  add is_split check before _start_split

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13326 from xiexingguo/wip-firstn

crush: do is_out test only if we do not collide

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <loic@dachary.org>

Merge pull request #13328 from wonzhq/fix-rec-stat

osd: fix stat sum update of recovery pushing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13353 from wonzhq/wip-failed-push

osd: fix the setting of soid in sub_op_push

Reviewed-by: Sage Weil <sage@redhat.com>

script/find_dups_in_pg_log: scrip to find dup requests due to short pg logs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13389 from trociny/wip-rbd-nbd-unused_old_format

rbd-nbd: no need to check image format any more

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13248 from rzarzynski/wip-rgw-18796

rgw: improve handling of illformed Swift's container ACLs.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12810 from zhangsw/fix-rgw-datasync-errorlog

rgw: error_code in error log is not right when data sync fails.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13394 from tchaikov/wip-18838

cmake: link consumers of libclient with libcommon

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11128 from tchaikov/wip-16091

mon/MonClient: hunt monitors in parallel

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon/MonClient: remove unnecessary helper functions

refactor _reopen_session() by removing wrapper around it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: remove unnecessary include

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/monclient: hunt for multiple monitor in parallel

* add an option "mon_client_hunt_parallel" for the maxmimum number of parallel
  hunting sessions.

Fixes: http://tracker.ceph.com/issues/16091
Signed-off-by: Steven Dieffenbach <sdieffen@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: mark monc_lock a mutable

so we can label the getters of MonClient with the `const` specifier.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: use __func__ for function names

Signed-off-by: Steven Dieffenbach <sdieffen@redhat.com>

client: move monc->set_want_keys() before monc->init()

if monc's tick connect to the mon before monc.set_want_keys() is called,
monc won't ask for the key for MDS service, and hence will fail to
build_authorizer() for MDS service. this change ready us for the
feature of monc-connect-to-mon-in-parallel.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth: AuthClientHandler::init() pass parameter by const ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13149 from liewegas/wip-list-objects

librados: remove legacy object listing API, clean up newer api

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13409 from xiexingguo/wip-fix-throttler-name

os/bluestore: add "_" prefix for internal methods

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: add "_" prefix for internal methods

These 4 methods are reserved for internal use only.
Prefix them with "_" to keep pace with others.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13391 from Adirl/ibv_exp

msg/async/rdma: check if exp verbs avail

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13034 from wjwithagen/wip-wjw-brag

mailmap: Willem Jan Withagen affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13209 from wjwithagen/wip-wjw-freebsd-init-ceph

init-ceph: Make init-ceph work under FreeBSD for init-system

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13377 from wjwithagen/wip-wjw-freebsd-jemalloc

CMakeLists.txt: suppress unneeded warning about jemalloc

Merge pull request #13360 from ktdreyer/doc-firewalld-improvements

doc: improve firewalld instructions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13399 from vumrao/wip-vumrao-18919

rgw: change loglevel to 20 for 'System already converted' message

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13396 from tchaikov/wip-doc-dedup-refs

doc: remove duplicated references

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13400 from shinobu-x/wip-18840

doc: Modify Configuring Cinder section

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13405 from linuxbox2/wip-librgw-refcnt

Wip librgw refcnt

Merge pull request #13403 from MichaelEischer/master

doc: Fixes parameter name in rbd configuration on openstack havana/icehouse

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw_file: fix hiwat behavior

Removed logic to skip reclaim processing conditionally on hiwat,
this probably meant to be related to a lowat value, which does
not exist.

Having exercised the hiwat reclaim behavior, noticed that the
path which moves unreachable objects to LRU, could and probably
should remove them altogether when q.size exceeds hiwat.  Now
the max unreachable float is lane hiwat, for all lanes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: refcnt bugfixes

This change includes 3 related changes:

1. add required lock flags for FHCache updates--this is a crash
   bug under concurrent update/lookup

2. omit to inc/dec refcnt on root filehandles in 2 places--the
   root handle current is not on the lru list, so it's not
   valid to do so

3. based on observation of LRU behavior during creates/deletes,
   update (cohort) LRU unref to move objects to LRU when their
   refcount falls to SENTINEL_REFCNT--this cheaply primes the
   current reclaim() mechanism, so very significanty improves
   space use (e.g., after deletes) in the absence of scans
   (which is common due to nfs-ganesha caching)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file:  add refcount dout traces at debuglevel 17

These are helpful for checking RGWFileHandle refcnt invariants.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: add pretty-print for RGWFileHandle

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

doc: Fixes parameter name in rbd configuration on openstack havana/icehouse

The documentation for using rbd together with openstack havana/icehouse
states that the parameter libvirt_disk_cachemodes should be added to
the nova.conf file. However, this is the only parameter that has no
legacy name with a 'libvirt_' prefix. (See
https://github.com/openstack/nova/blob/icehouse-eol/nova/virt/libvirt/driver.py#L252
for the configuration option)
Thus the configured disk_cachemodes were not applied, defaulting to
no caching.

Fixes: #17978
Signed-off-by: Michael Eischer <michael.eischer@fau.de>

rgw: change log level to 20 for 'System already converted' message

Fixes: http://tracker.ceph.com/issues/18919
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Doc: Modify Configuring Cinder section

doc/rbd/rbd-openstack.rst: Add ``volume_backend_name`` in description of
cinder.conf

Fixes: http://tracker.ceph.com/issues/18840
Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #13017 from chuanhong-wang/dev_add_perfcount_for_backoff_throttle

osd: adding PerfCounters for backoff throttle

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #13274 from ifed01/wip-bluestore-perf-counters

os/bluestore: add more perf_counters to BlueStore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13321 from liewegas/wip-kill-sortbitwise-harder

osd: kill sortbitwise

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: remove duplicated references

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13303 from Jing-Scott/fix-swift-cannot-disable-object-versioning

rgw: fix swift cannot disable object versioning

@Jing-Scott  updated, addressing @rzarzynski's change request

Merge pull request #13393 from ddiss/fix_rgw_fcgi_include

rgw: set FCGI_INCLUDE_DIR for cephd_rgw_base

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-fuse: always free allocated fuse_args

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link consumers of libclient with libcommon

to avoid linking against to both libceph-common and libcommon at the same
time, because both of them will be registered as a provider of lttng
provider.

Fixes: http://tracker.ceph.com/issues/18838
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13382 from xiexingguo/wip-counter-name

os/bluestore: fix wal-queue bytes-counter to keep pace with others

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: set FCGI_INCLUDE_DIR for cephd_rgw_base

cephd_rgw_base build currently fails with fastcgi enabled:
--
In file included from /home/david/ceph/src/rgw/rgw_request.h:13:0,
from /home/david/ceph/src/rgw/rgw_main.cc:53:
/home/david/ceph/src/rgw/rgw_fcgi.h:8:21: fatal error: fcgiapp.h:
No such file or directory
^
--

This is despite the fact that fastcgi was detected and located at
configure time:
build/CMakeCache.txt:FCGI_INCLUDE_DIR:PATH=/usr/include/fastcgi

Fix this by ensuring that the cephd_rgw_base build target correctly uses
FCGI_INCLUDE_DIR.

Fixes: http://tracker.ceph.com/issues/18918
Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #13392 from Adirl/zero_inline

msg/async/rdma: zero wqe inline

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/rdma: zero wqe inline

we dont use wqe inline

issue: none

Change-Id: Ib187eec3bc79f8caf040fbbfbf0a915186cd58ba
Signed-off-by: Adir Lev <adirl@mellanox.com>

msg/async/rdma: check if exp verbs avail

issue: 975125

Change-Id: I9daa168ca0299887a7238a688508c773b98abde9
Signed-off-by: Adir Lev <adirl@mellanox.com>
Signed-off-by: Oren Duer <oren@mellanox.com>

rgw: fix swift cannot disable object versioning

we should be able to disable object verioning by removing its X-Versions-Location
metadata header by sending an empty key value. this description can be found at
No.8 in http://docs.openstack.org/user-guide/cli-swift-set-object-versions.html.

Fixes: http://tracker.ceph.com/issues/18852
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

Merge pull request #13380 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: no need to pass version and sha1 anymore

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #13368 from smithfarm/wip-doc-dev

doc: instructions for provisioning OpenStack VMs ad hoc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd-nbd: no need to check image format any more

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13367 from tchaikov/wip-qa-jewel-x-singleton

test: Thrasher: update pgp_num of all expanded pools if not yet

Reviewed-by: Sage Weil <sage@redhat.com>

script/sepia_bt.sh: no need to pass version and sha1 anymore

* try to figure out the working directory from the core dump's path
* add -C directory option, so user is able to specify the directory
  for the downloaded packages, and extracted files.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix wal-queue bytes-counter to keep pace with others

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test: Thrasher: restore changed options after done with thrash

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: Thrasher: extract _set_config() method

Signed-off-by: Kefu Chai <kchai@redhat.com>

tests: CephManager: add get_config() method

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Thrasher: update pgp_num of all expanded pools if not yet

otherwise wait_until_healthy will fail after timeout as seeing warning
like:

HEALTH_WARN pool cephfs_data pg_num 182 > pgp_num 172

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13378 from liewegas/wip-luminous

qa/rados/upgrade/jewel-x-singleton: run luminous.yaml at the end

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #13365 from xiexingguo/wip-backoff-refleak

osdc/Objecter: fix possible OSDSession leak on wrong connection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13363 from jdurgin/wip-req-luminous-error

osd: clarify REQUIRE_LUMINOUS error message

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/rados/upgrade/jewel-x-singleton: run luminous.yaml at the end

renumber too

Signed-off-by: Sage Weil <sage@redhat.com>

CMakeLists.txt: suppress unneeded warningiabout jemalloc

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc: instructions for provisioning OpenStack VMs ad hoc

Signed-off-by: Nathan Cutler <ncutler@suse.com>

osd/PrimaryLogPG: unconditionally set PGNLS next to hobject

Drop nibblewise workaround.

Signed-off-by: Sage Weil <sage@redhat.com>