Merge pull request #10643 from tchaikov/wip-cmake

cmake: fix wrong path introduced by bb163e9

Reviewed-by: Oleh Prypin <oleh@pryp.in>

Merge pull request #10557 from wjwithagen/wip-wjw-test-onexit

test: More portable use of mmap(MAP_ANON)

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix wrong path introduced by bb163e9

s/CYTHON_MODULE_LIB/CYTHON_MODULE_DIR/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9848 from dx9/wip-ac-boost

configure.ac: add _LIBS variables for boost_system and boost_iostreams

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10478 from wjwithagen/wip-wjw-freebsd-cmake-excludes

src/CMakeLists.txt: remove unneeded libraries from ceph-dencoder target

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10568 from tchaikov/wip-cmake

cmake: specify distutils build path explicitly

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10455 from wjwithagen/wip-wjw-clang-NULL

src/kv/MemDB.cc: the type of the parameter of push_back() does not match the ops's value_type

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: specify distutils build path explicitly

so we don't need to replicate the logic in distutils/command/build.py
at different places.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10635 from linuxbox2/rgw-file-aiee

rgw_file: unlock() must precede out label

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw_file: unlock() must precede out label

In lookup_handle(...).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10623 from tchaikov/wip-16940

admin/build-doc: bypass sanity check if building doc

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10417 from wjwithagen/wip-wjw-freebsd-clang

changes for Clang and yasm

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10232 from weiqiaomiao/wqm-wip-rgw-cleanbl

rgw:clean unuse bufferlist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10613 from dillaman/wip-16956

rbd-mirror: potential assertion failure during error-induced shutdown

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10622 from xiexingguo/xxg-wip-kill-cephfs-warn

libcephfs: kill compiling warning

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9744 from SUSE/wip-16345

rpm: fix shared library devel package names and dependencies

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

admin/build-doc: s/iquote /iquote/

currently this works fine, but we ran into this problem before. and
technically "-iquote<dir>" is the valid syntax, and "-iquote <dir>"
is not. see https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc,pybind: bypass sanity check if building doc

the empty .so file does not compile at all, so we need to bypass the
sanity check, if we are building docs. and what sphinx needs is just the
docstrings.

Fixes: http://tracker.ceph.com/issues/16940
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10612 from tchaikov/wip-16955

test/ceph_test_msgr: fix circular locking dependency

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10620 from dachary/wip-16961-subman

subman: use replace instead of format

Reviewed-by: Kefu Chai <kchai@redhat.com>

libcephfs: kill compiling warning

As below:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/include/cephfs/libcephfs.h:1499:20: warning: ‘struct flock’ declared inside parameter list [enabled by default]
     Fh *fh, struct flock *fl, uint64_t owner);
                    ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/include/cephfs/libcephfs.h:1499:20: warning: its scope is only this definition or declaration, which is probably not what you want [enabled by default]
/home/jenkins-build/build/workspace/ceph-pull-requests/src/include/cephfs/libcephfs.h:1501:20: warning: ‘struct flock’ declared inside parameter list [enabled by default]
     Fh *fh, struct flock *fl, uint64_t owner, int sleep);

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

subman: use replace instead of format

Otherwise all {} are assumed to be substituted. Add a test.

Fixes: http://tracker.ceph.com/issues/16961
Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10454 from SUSE/wip-quick-start

doc: add openSUSE instructions to quick-start-preflight

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw:clean unuse bufferlist

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

Merge pull request #10526 from xiexingguo/xxg-wip-misc-2016-07-30

osd: minor performance improvements and fixes

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10548 from xiexingguo/xxg-wip-osd-2016-08-02

osd: more cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10559 from xiexingguo/xxg-wip-osd-2016-08-03

osd: drop unused variables/methods

Reviewed-by: Samuel Just <sjust@redhat.com>

doc: add openSUSE instructions to quick-start-preflight

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10212 from rzarzynski/wip-rgw-cleanup-auth

rgw: clean-up in the authentication infrastructure

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10233 from weiqiaomiao/wqm-wip-ver_check

rgw: fix wrong  variable definition in cls_version_check func

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10594 from michelmno/forpull2

rpm: ExclusiveArch for suse_version

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10122 from Yan-waller/yj-wiph-rgwradoscc-0704

rgw: fix the missing return value

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10194 from Yan-waller/yj-wip-rgwrados-0708

rgw: remove unused bufferlist variable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10203 from Yan-waller/yj-wip-rgwcache-0708

rgw: merge setting flags operation together and cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10214 from theanalyst/rgw/multisite-fwd-log-cleanup

rgw: drop create_bucket in fwd_request log message

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10226 from Yan-waller/yj-wip-rgwrests3-0709

rgw: use the standard usage of string.find

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd-mirror: potential assertion failure during error-induced shutdown

Fixes: http://tracker.ceph.com/issues/16956
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10299 from Yan-waller/yj-wiph-rgwmetadata-0715

rgw: miscellaneous cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10408 from weiqiaomiao/wqm-wip-rgw-lc-set

rgw: fix wrong variable definition in rgw_cls_lc_set_entry function

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10562 from aclamk/wip-rgw-s3-keystone-cache-mutex

rgw: Got rid of recursive mutex.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10424 from theanalyst/rgw/mp-complete-error-msg

rgw: log mp upload failures due to parts mismatch

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10475 from yonghengdexin735/wip-zzz-rgw

rgw/rgw_main: fix unnecessary variables defined

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10589 from Yan-waller/yj-wip-rgwcls-0805

rgw: remove unused variables

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10611 from linuxbox2/rgw-nfs-28

Rgw nfs 28

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/ceph_test_msgr: better looking log

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/ceph_test_msgr: fix a mem leak

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/ceph_test_msgr: fix circular locking dependency

* do not acquire lock when sending message
* remove lock in session
* reduce the scope guarded by locks for better performance.

Fixes: http://tracker.ceph.com/issues/16955
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw nfs: fix write continuations

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw nfs: don't leak fh->mtx in lookup_handle()

This change fixes a serious latent locking problem, noticed after
updating the ganesha/rgw driver invalidation after renames.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw file: refuse to rename directories

The representation of paths as an aggregate involving any
number of objects argues against ever permitting such
operations.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: refuse ops on deleted targets

Detect these illegal cases, as they are indicate various incorrect
behaviors/bugs.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10598 from dillaman/wip-16882

cmake: correct ceph_test_librbd/ceph_test_rbd_mirror linkage

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9566 from ffilz/master

Add ceph_ll_setlk and ceph_ll_getlk

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10452 from jcsp/wip-16556

test: break out librados-using cephfs test

Merge pull request #10574 from dillaman/wip-16923

librbd: delay acquiring lock if image watch has failed

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10581 from dillaman/wip-16887

librbd: prevent creation of v2 image ids that are too large

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

src/CMakeLists.txt: remove unneeded libraries from ceph-dencoder target

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: correct ceph_test_librbd/ceph_test_rbd_mirror linkage

This now matches the autotools configuration and fixes the failed
assertion during test execution.

Fixes: http://tracker.ceph.com/issues/16882
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10579 from dreamhost/wip-16928

rgw: Fix civetweb IPv6

verified:  autobuild

Merge pull request #10597 from pritha-srivastava/wip_ldap_fix

rgw: Fix for incorrect access key id in reworked ldap auth code.

verified non-regression (MS AD)

rgw: Fix for incorrect access key id reworked ldap auth code.

The reworked ldap code does not use the correct access key id in get_policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #10595 from tchaikov/wip-rocksdb

rocksdb: revert the change introduced by dc41731

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rocksdb: revert the change introduced by dc41731

dc41731 reverts the submodule reference of rocksdb, and point it to an
older version which fails to support ppc64le and AArch64.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10262 from dachary/wip-16580-ceph-disk-timeout

ceph-disk: timeout ceph-disk to avoid blocking forever

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Ken Dreyer (Red Hat) <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #10491 from xiexingguo/xxg-wip-osdmon-2016-07-28

osd/OSDMonitor: misc. fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9259 from xiexingguo/xxg-wip-fix-monclientpinger

mon: MonClient may hang on pinging an unresponsive monitor

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10593 from linuxbox2/ldap-bindv3-again

rgw ldap: enforce simple_bind w/LDAPv3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9718 from songbaisen/a8

mon: clear list in better way

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10392 from stiopaa1/mon_monclient_moveCCancelMonCommand

mon/MonClient: move C_CancelMonCommand to cc file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10470 from xiexingguo/xxg-wip-osd-2016-07-28

osd: minor performance improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10539 from jcsp/wip-mds-counters

mds: add perf counters for MDLog replay and SessionMap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw ldap: enforce simple_bind w/LDAPv3

Found by Harald Klein <hklein@redhat.com>.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

ExclusiveArch for suse_version

for SLES supports only x86_64 and aarch64 targets
for openSUSE (Tumbleweed and Leap) add ppc64/ppc64le targets.

fixes: http://tracker.ceph.com/issues/16936
Signed-off-by: Michel Normand <normand@linux.vnet.ibm.com>

Merge pull request #10542 from ktdreyer/doc-rbd-libvirt-logging

doc: describe libvirt client logging

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10488 from lande1234/wip-lan-testing

rbd-mirror: add additional test scenarios

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10573 from renhwztetecs/renhw-wip-doc-cephfs

doc/mds: fixup mds doc

Reviewed-by: John Spray <john.spray@redhat.com>

mds: add perf counter for rate of replay

Useful for watching the progress of a daemon
in standby-replay or replay.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: expose perf counters for SessionMap

So that we can see how many sessions exist via
the counters rather than only by looking at status.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10512 from renhwztetecs/renhw-wip-mds-balancer

mds/MDBalancer: cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10485 from gregsfortytwo/wip-client-dump-complete

client: include COMPLETE and ORDERED states in cache dump

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10533 from gregsfortytwo/wip-mds-dispatch-refcount

mds: fix up _dispatch ref-counting semantics

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10540 from jcsp/wip-mds-multimds-standby

mds: fix daemon selection when starting ranks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: remove unused variables

remove the unused bufferlist variables

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

librbd: prevent creation of v2 image ids that are too large

The librbd API is capped at 24 characters for expressing the
object prefix for data blocks (including trailing null byte).

Fixes: http://tracker.ceph.com/issues/16887
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: Fix civetweb IPv6

Commit c38e3cbb6f7c6221209f2b512cba18c564c895a4 introduced a second
compiled version of src/civetweb/src/civetweb.c, but did not pass the
configuration header (civetweb/include/civetweb_conf.h).

As a result, USE_IPV6 was not defined when it was compiled, and that
copy was included into the radosgw binary. This caused breakage for the
civetweb frontend when used with IPv6:
  rgw frontends = civetweb port=[::]:7480

Reintroduce the header so that civetweb is compiled correctly again.

Fixes: http://tracker.ceph.com/issues/16928
Backport: jewel
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #10565 from dzafman/wip-16672

OSD crash with Hammer to Jewel Upgrade: void FileStore::init_temp_collections()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10545 from trociny/fixup-rbd-mirror-test

qa/workunits/rbd: wait for image deleted before checking health

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: describe libvirt client logging

Describe how to configure logging and the admin socket for debugging RBD
libvirt clients.

Thanks to Jason Dillman <jdillama@redhat.com> on ceph-devel for this
information.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

librbd: delay acquiring exclusive lock if watch has failed

Fixes: http://tracker.ceph.com/issues/16923
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert ImageWatcher class to template

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: More portable use of mmap(MAP_ANON)

From the Linux manual:
  MAP_ANONYMOUS
     The  mapping  is  not  backed  by any file; its contents
     are initialized to zero.  The fd and offset arguments are
     ignored; however, some implementations require fd to be -1
     if MAP_ANONYMOUS (or MAP_ANON) is specified, and portable
     applications  should  ensure  this.

FreeBSD is such a case, it wil just return an error.

 - Used in:
     src/test/on_exit.cc
     src/test/system/cross_process_sem.cc

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10566 from batrick/mds-daemon-unused

mds: remove unused MDSDaemon::objecter

Reviewed-by: John Spray <john.spray@redhat.com>

doc/mds: add generated files into the /etc/ceph/

Fixes: 16906
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

osd: add a comment for the purpose of pinned_maps

For a better understanding since it can confuse newcommers.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: remove unused MDSDaemon::objecter

Leftover from a2682edb76ceac428c686e0ac21c06511e27693e.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: use osd_target_transaction_size when batch removing objects

Hard code transaction size to be 300 is not good. Since we already
introduce osd_target_transaction_size for the same purpose, we can
use it directly here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix rare race while looking for a pg

When _have_pg() returns true, we drop the pg_map_lock
and pg->lock() simultaneously. So theoretically the
subsequent call to _lookup_lock_pg() can return NULL,
although the chance is rare.

Also, since we are going to call _lookup_lock_pg() anyway,
we can call it directly, thus the _have_pg() method is
totally unnecessary.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: kill _lookup_pg() method

It is never used and I see no chance it will be used.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: don't get transaction too big during clear_temp_objects()

Basically we don't allow a transaction to get too big, e.g., to
avoid contiguously occupation of CPU.

We break and queue the transaction here whenever we have collected
osd_target_transaction_size temp objects.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10307 from pritha-srivastava/wip_s3_auth_rework

rgw: Rework of s3 LDAP Authentication code.

verified: MS AD