Merge pull request #10145 from dillaman/wip-cmake-rbd-python

cmake: fix incorrect dependencies to librados

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10119 from renhwztetecs/renhw-wip-cdentry-dump

mds: fixup dump Formatter' type error; add path_ino and is_primary in the CDentry::dump()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10062 from renhwztetecs/renhw-wip-filer

client/filer: cleanup the redundant judgments of _write&&_fallocate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10027 from batrick/client-root-lock

client: add missing client_lock for get_root

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10100 from fullerdj/wip-djf-16144

tools/cephfs: Remove cephfs-data-scan tmap_upgrade

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9971 from fullerdj/wip-djf-16288

mds: Kill C_SaferCond in evict_sessions()

Reviewed-by: John Spray <john.spray@redhat.com>

cmake: fix incorrect dependencies to librados

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10136 from runsisi/sisi-wip-fix-missing-return

librbd: fix missing return statement if failed to get mirror image state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10094 from athanatos/wip-throttle-test

test/common/Throttle.cc: fix race in shutdown

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #10088 from pritha-srivastava/wip_s3_signedurl_errors

rgw: Fix for using port 443 with pre-signed urls.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10112 from tchaikov/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10134 from trociny/wip-run-rbd-tests-fixup

test: run-rbd-tests test cmake fixup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10133 from tchaikov/wip-fixc-cephtool-test.sh

test: cephtool/test.sh: fix expect_false() calls

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10072 from dillaman/wip-16561

librbd: failed assertion after shrinking a clone image twice

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: fix missing return statement if failed to get mirror image state

Signed-off-by: runsisi <runsisi@zte.com.cn>

test: cephtool/test.sh: fix expect_false with pipe

this fixes failures like,

/home/jenkins-build/build/workspace/ceph-pull-requests/qa/workunits/cephtool/test.sh:
line 32: ceph osd blacklist ls | grep 192.168.0.1: command not found

where the failure is not the "failure" we are expecting.

in our tests, following command

expect_false "ceph osd blacklist ls | grep 192.168.0.1"

is designed to to verify that "ceph osd blacklist ls | grep 192.168.0.1"
fails with non-zero return code. but expect_false() evaluates the command
line using plain "$@", which will send the arguments direct to the shell,
and $0 is "ceph auth get client.xx | grep caps | grep mon", which does
not exist and is not built-in command. so we need to check the grep
command instead.

for multiple piped command line, use

expect_false sh <<< "echo foo  | grep bar | grep baz"

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: run-rbd-tests test cmake fixup

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10076 from dillaman/wip-rbd-mirror-cmake

test: fix rbd-mirror workunit test cases for cmake

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9731 from badone/wip-doc-ceph-authtool-print-fix

doc: ceph-authtool man page option is --print-key not --print

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10124 from trociny/wip-cmake-tests

test: cmake related fixups for rbd tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: fix run-rbd-tests test for cmake

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: fix run-rbd-valgrind-unit-tests when built with cmake

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: ceph_test_librbd does not need rados_test_stub

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10117 from xiexingguo/xxg-wip-bluestore-2016-07-04

os/bluestore: fix incorrect pool decoding of bnode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9755 from dx9/wip-ns-name-uncompress

common/dns_resolve: use ns_name_uncompress instead of ns_name_ntop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9766 from dx9/wip-autotools-arch

configure.ac: Use uname instead of arch.

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #9772 from tchaikov/wip-close-on-exec

msgr: set close on exec flag

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9820 from dx9/wip-fix-res-query-detect

configure.ac: fix res_query detection

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9845 from dx9/wip-config-basename

common: remove basename() dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9969 from liewegas/wip-assert

include/assert: clean up ceph assertion macros

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9983 from SUSE/wip-cmake-fix-fcgi-include-dir

cmake: Fix FCGI include directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10093 from tchaikov/wip-16525

crush: reset bucket->h.items[i] when removing tree item

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9330 from dachary/wip-15176-partprobe

ceph-disk: partprobe should block udev induced BLKRRPART

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9394 from xiexingguo/xxg-wip-batchpgstates

mon/PGMonitor: batch filter pg states; add sanity check

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/CDentry: add path_ino and is_primary in the dump()

path_ino and is_primary are  main reference information in the dump(),
so add their

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mds: fixup dump Formatter' type error

fixup dump Formatter' type error,
dump_int->dump_unsigned

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

os/bluestore: fix incorrect pool decoding of bnode

"pool" is a pointer, and we shall alter its value instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: add ceph-{disk,detect-init} to ALL

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: partprobe should block udev induced BLKRRPART

Wrap partprobe with flock to stop udev from issuing BLKRRPART because
this is racy and frequently fails with a message like:

    Error: Error informing the kernel about modifications to partition
    /dev/vdc1 -- Device or resource busy.  This means Linux won't know about
    any changes you made to /dev/vdc1 until you reboot -- so you shouldn't
    mount it or use it in any way before rebooting.

Opening a device (/dev/vdc for instance) in write mode indirectly
triggers a BLKRRPART ioctl from udev (starting version 214 and up)
when the device is closed (see below for the udev release note).

However, if udev fails to acquire an exclusive lock (with
flock(fd, LOCK_EX|LOCK_NB); ) the BLKRRPART ioctl is not issued.

https://github.com/systemd/systemd/blob/045e00cf16c47bc516c0823d059b7548f3ce9c7c/src/udev/udevd.c#L1042

Acquiring an exclusive lock before running the process that opens the
device in write mode is therefore an effective way to control this
behavior.

git clone git://anonscm.debian.org/pkg-systemd/systemd.git
systemd/NEWS:
CHANGES WITH 214:

  * As an experimental feature, udev now tries to lock the
       disk device node (flock(LOCK_SH|LOCK_NB)) while it
       executes events for the disk or any of its partitions.
       Applications like partitioning programs can lock the
       disk device node (flock(LOCK_EX)) and claim temporary
       device ownership that way; udev will entirely skip all event
       handling for this disk and its partitions. If the disk
       was opened for writing, the close will trigger a partition
       table rescan in udev's "watch" facility, and if needed
       synthesize "change" events for the disk and all its partitions.
       This is now unconditionally enabled, and if it turns out to
       cause major problems, we might turn it on only for specific
       devices, or might need to disable it entirely. Device Mapper
       devices are excluded from this logic.

Fixes: http://tracker.ceph.com/issues/15176
Signed-off-by: Marius Vollmer <marius.vollmer@redhat com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #10077 from yehudasa/wip-16494

rgw: fix for issue 16494

Merge pull request #10064 from BlaXpirit/brag-py3

Port ceph-brag to Python 3 (+ small fixes)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10115 from runsisi/ss-wip-fix-rpm-spec

ceph.spec.in: fix rpm package building error

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: do not depend on sse jerasure plugins if not avaiable

should not depend on them if SSE3 and/or SSE4 is not around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link libglobal with lttng-ust

only link tracing libraries with lttng-ust, this fixes selinux denial
errors at run-time:

SELinux denials found on ubuntu@mira054.front.sepia.ceph.com: ['type=AVC
msg=audit(1467543917.770:131781): avc: denied { open } for pid=17511
comm="ceph-mon" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file', 'type=AVC
msg=audit(1467543914.518:131756): avc: denied { open } for pid=17418
comm="ceph-osd" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file', 'type=AVC
msg=audit(1467543917.770:131781): avc: denied { read } for pid=17511
comm="ceph-mon" name="lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file', 'type=AVC
msg=audit(1467543914.518:131756): avc: denied { read } for pid=17418
comm="ceph-osd" name="lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file']

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix rpm package building error as follows:

      error: Installed (but unpackaged) file(s) found:
      /usr/lib/udev/rules.d/60-ceph-by-parttypeuuid.rules

Signed-off-by: runsisi <runsisi@zte.com.cn>

cmake: refactor distutils_install_module to follow the pattern of distutils_install_cython_module

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install python packages into $CMAKE_INSTALL_PREFIX

* Distutils.cmake:
  set --prefix=${CMAKE_INSTALL_PREFIX} for python packages installed using
  setuptools. it was set to --prefix=/user only when $DESTDIR is set. so
  if user installs ceph using -DCMAKE_INSTALL_PREFIX, these python
  packages still go to /usr, which is unexpected.
* ceph-disk/CMakeLists.txt:
  install script into ${CMAKE_INSTALL_SBINDIR} instead of /usr/sbin

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not install unpackaged stuff

* doc/start/ceph.conf: it was installed as /etc/ceph.conf.example.
  but it's unexpected, and ceph.spec does not packages it.
* vstart.sh: this is for development usage. no need to package it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9885 from dachary/wip-16351-by-parttypeuuid

udev: always populate /dev/disk/by-parttypeuuid

Reviewed-by: Alexandre Derumier <aderumier@odiso.com>

Merge pull request #10095 from liewegas/wip-bluestore-trim

os/bluestore: trim cache on reads

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10080 from xiexingguo/xxg-wip-add-compress-ratio

os/bluestore: improve required compression threshold

Reviewed-by: Sage Weil <sage@

cmake: version libcephfs_jni

so we have libcephfs_jni.so.1.0.0 and libcephfs_jni.so.1, which are
expected by ceph.spec

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not install intermediate libs

rbd_replay and rbd_replay_ios are convenience libraries, and they are
not supposed to be used by user.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake install ceph-monstore-update-crush.sh into ${CMAKE_INSTALL_LIBDIR}/ceph

ceph-monstore-update-crush.sh is a user-facing script, and it is not
used internally. also ceph.spec expects it that way. technically, we
should not install arch-indep files into ${CMAKE_INSTALL_LIBDIR}/ceph.
leave this for another changeset..

this partially reverts 37f53ec

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install lib{rgw,cephfs_jni} into ${CMAKE_INSTALL_LIBDIR} not lib

rh and suse distros follows FHS and put amd64 dso libs into lib64 on
amd64 machines. so let's use ${CMAKE_INSTALL_LIBDIR} instead

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install python srcs into platform-independent dir

otherwise they will be installed into
$PREFIX/lib64/python2.7/site-packages instead of
$PREFIX/lib/python2.7/site-packages on amd64 arch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/simple: set close on exec on server sockets

mds execv() when handling the "respawn" command, to avoid fd leakage,
and enormous CLOSE_WAIT connections after respawning, we need to set
FD_CLOEXEC flag for the socket fds.

Fixes: http://tracker.ceph.com/issues/16390
Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: set close on exec on server sockets

mds execv() when handling the "respawn" command, to avoid fd leakage,
and enormous CLOSE_WAIT connections after respawning, we need to set
FD_CLOEXEC flag for the socket fds.

Fixes: http://tracker.ceph.com/issues/16390
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: add compression required ratio to enable/disable compression

Require the net gain of compression at least to be at a specified ratio,
otherwise we don't compress.

Ask for compressing at least 12.5% off, by default.

This is for the sake of performance because if the compression turns out
to be meaningless(saving little space), we can simply shut it down, as we
know the compression/decompression can be rather CPU-consuming.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/cephfs: Remove cephfs-data-scan tmap_upgrade

tmap_upgrade was only for Jewel, so remove it for Kraken.

Fixes: http://tracker.ceph.com/issues/16144
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #10096 from ceph/wip-dang-cmake

debian: set libexec dir to correct value as autotools did

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9941 from weiqiaomiao/wqm-wip-fetchobj

rgw: fix memory leak in RGWRados::fetch_remote_obj()

lgtm

debian - set libexec dir to correct value as autotools did

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

os/bluestore: trim cache on reads too

We were only trimming in the write path, which would make memory explode
on a read-only workload.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10082 from chhabaramesh/bitalloc_warn

os/bluestore: Remove bit alloc Woverloaded-virtual warnings

Merge pull request #10068 from jcsp/wip-cephfs-doc-update

doc: update CephFS "early adopters" info

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9584 from chiehtu/fix-16188

rgw: fix Swift API returns double space usage

Reviewed by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #10000 from weiqiaomiao/wqm-wip-add-userlist-cli

rgw: add "radosgw-admin user list" cli

verified

Merge pull request #9838 from xiexingguo/xxg-wip-fix-synthetic-client

client: misc fixes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10092 from ceph/wip-dang-cmake

cmake: fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10028 from ceph/wip-cmake-readme-fixes

cmake: Removed README.cmake.md, edited README.md

mds: Kill C_SaferCond in evict_sessions()

MDSRankDispatcher::evict_sessions waits on a C_SaferCond for
kill_session to complete on each of its victims. Change the
command handling flow to pass command messages all the way down
to MDSRankDispatcher. Extract the MDSDaemon's reply path into a
static function callable from a new context in the MDSRankDispatcher.

See: http://tracker.ceph.com/issues/16288
Signed-off-by: Douglas Fuller <dfuller@redhat.com>

crush: reset bucket->h.items[i] when removing tree item

* crush: so we don't see the reference after the removing, this keeps
  check_item_loc() happy, and move_bucket() use check_item_loc() to see if
  the removed bucket disappears after the removal.
* test: also add unittest_crush_wrapper::CrushWrapper.insert_item

Fixes: http://tracker.ceph.com/issues/16525
Signed-off-by: Kefu Chai <kchai@redhat.com>

init-ceph - use sbindir, rather than hard coding sbin

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

CMake - add libexecdir to the replaced variables

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #10081 from tchaikov/wip-16556

cmake: remove unnecessary linked libs from libcephfs

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10083 from trociny/wip-16539

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10065 from dillaman/wip-16558

rbd-mirror: gracefully fail if object map is unavailable

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw: Fix for using port 443 with pre-signed urls.

Fixes http://tracker.ceph.com/issues/16548

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

cmake: remove unnecessary linked libs from libcephfs

* some of the libs shares the same .cc which has static C++ variables. if
  we link against the different libs sharing the same static C++
  variables, and the dtor of the C++ variables has side-effects, among
  other things, deallocates a memory chunk. then, we are in the trouble of
  double free. so "osd" lib is removed.
* some of the libs are referenced by the linked lib, so no need to link
  against them again. for example, BLKID_LIBRARIES are linked by
  libcommon, so we can remove it from the linked libs list.
* and lib "os" and "cls_references_objs" are not used by libcephfs at all,
  so remove them.

Fixes: http://tracker.ceph.com/issues/16556
Signed-off-by: Kefu Chai <kchai@redhat.com>

Remove bit alloc Woverloaded-virtual warnings

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

rgw: fix double counting in RGWRados::update_containers_stats()

Fixes: http://tracker.ceph.com/issues/16188
Signed-off-by: Albert Tu <albert.t@inwinstack.com>

rgw: forward input data when forwarding set_bucket_version to master

Fixes: http://tracker.ceph.com/issues/16494
Needed to keep input data around to be forwarded correctly. Also, master
does not send any data back, so don't try to parse anything.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: set args when rebuilding req_info

Was missing args, so it didn't sign requests with subresources
correctly when forwarding.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc/cephfs: rename early-adopters to best-practices

...and give it a link in the TOC (in addition
to where it is referenced from the top of index.rst)

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: remove some scary warnings

...and restructure the "early adopters" page
into a "best practices" guide.  Early adopters
are now just adopters :-)

Signed-off-by: John Spray <john.spray@redhat.com>

cmake: Removed README.cmake.md, edited README.md

Deleted README.cmake.md and put any helpful
information it had into README.md.

Also a added note about which tests get executed
by ctest.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

test/common/Throttle.cc: fix race in shutdown

Previously, putters could stop before getters.  Keep putters running
until getters stop and the queue is empty.

Signed-off-by: Samuel Just <sjust@redhat.com>

qa/workunits/rbd: fix issues under cmake build environment

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

vstart: avoid using relative paths when finding cmake directories

If the "run" directory is a symbolic link (e.g. to fast storage),
this will fail.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9891 from diluga/rgw_socket_backlog_with_sign

RGW : setting socket backlog for via ceph.conf

Merge pull request #9852 from dx9/wip-rgw-ldap-link

rgw-ldap: add ldap lib to rgw lib deps based on build config

Merge pull request #9931 from Yan-waller/yj-wip-rgwmainccfix

rgw: remove uneccessary temporary variable and fix some indents

cosmetic changes only--build verified, f23

Merge pull request #9979 from Yan-waller/yj-wip-rgwrests3cc

rgw: fix potential memory leaks and remove unnecessary assignment

lgtm--should backport

Merge pull request #10031 from yehudasa/wip-16530

rgw: finish error_repo cr in stop_spawned_services()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: failed assertion after shrinking a clone image twice

Fixes: http://tracker.ceph.com/issues/16561
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: remove ceph_test_rbd_mirror_image_replay test case

Fixes: http://tracker.ceph.com/issues/16539
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9985 from Yan-waller/yj-wiph-rgwopccfixstring

rgw: use string::npos while 'string::find()' find nothing

lgtm--build verified, f23

Merge pull request #9898 from liewegas/wip-bluefs-async-flush

os/bluestore/BlueFS: do not hold internal lock while waiting for IO

Merge pull request #10002 from Yan-waller/yj-wip-rgwradosccremoveunused

rgw: remove uneccessary temporary variable 'r'

lgtm--build verified, f23