rgw: convert bucket instance listings back to metadata key format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use tenant/ for bucket instance metadata keys

to work around the ambiguity of parsing tenant: and :shard in the same
bucket instance metadata key, use tenant/ instead

to preserve backward compatibility with existing objects, new helper
function rgw_bucket_instance_key_to_oid() converts this / back to a :
before being used as an object name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add get_key() methods to format rgw_buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10128 from xiexingguo/xxg-wip-bluestore-2016-07-05

os/bluestore: introduce power 2 macros for block alignment and rounding

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10180 from theanalyst/mrgw/cmake-fixes

mrgw: search for cmake build dir.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mrgw: search for cmake build dir.

Similar to mstart and mstop, mrgw also needs to find the correct build
dir as we pass in the pidfile and asokfile which otherwise falls back to
src/run dir.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10116 from tchaikov/wip-cmake-make-check

test: make check using cmake

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10097 from yehudasa/wip-mrun-cmake

mrun, mstart.sh, mstop.sh: search for cmake build directory

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9489 from athanatos/wip-16156

ReplicatedPG: call op_applied for submit_log_entries based repops

Reviewed-by: David Zafman <dzafman@redhat.com>

qa/workunits/ceph-helpers.sh: disable test_pg_scrub()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10139 from SUSE/wip-cmake-really-fix-fcgi-include-dir

cmake: Really add FCGI_INCLUDE_DIR to include_directories for rgw

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9981 from yangdongsheng/rbd_set_vals

cls_rbd: set omap values in batch during image creation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10155 from trociny/wip-16555-workaround

qa/workunits/rbd: before removing image make sure it is not bootstrapped

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10164 from dillaman/wip-cmake-test-rbd-mirror

cmake: ceph_test_rbd_mirror does not require librados_test_stub

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Really add FCGI_INCLUDE_DIR to include_directories for rgw

Commit 3cf6c53 was incorrect.  FCGI_INCLUDE_DIR doesn't need to be
set in src/rgw/CMakeLists.txt, but it does need to be set for the
rgw_a target in src/CMakeLists.txt, as well as for the
ceph_test_librgw_file and ceph_test_librgw_file_nfsns targets in
src/test/CMakeLists.txt.  I can only assume that I must not have
done a completely clean rebuild at some point when testing a
reworked version of the earlier commit :-/

This is only a problem for distros that keep the FCGI headers in
/usr/include/fastcgi/ (e.g.: SUSE).

This commit also removes a redundant include of <fcgiapp.h>

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #9083 from badone/wip-doc-last_epoch_started

doc/dev: Fix missing code section due to no lexer for "none"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9704 from badone/wip-ceph-authtool-fixes

authtool: Enhance argument combinations validation

Reviewed-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: use "make check" for running tests

"make all" does not offer "ceph-disk" and "ceph-detect-init" for
testing. as they are solely used for testing purpose. instead, these two
python command line packages are installed by the "install" target. so
we need to use "make check" to 1) prepare the test dependencies 2)
launch ctest to perform the test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: add ceph-{disk,detect-init} to ALL"

This reverts commit 19c0731751df9642cc5b8c583ea41d12e0b5ec41.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test//osd/osd-copy-from.sh: use absolute path for "rados put"

"rados -p rbd put foo rados" does not work if "rados" is not in current
path. so change it to "rados -p rbd put foo $(which rados)"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass more env vars to osd-copy-from.sh

we should use the cmake function of add_ceph_test() to add
osd-copy-from.sh as a test. then we won't miss any env variables.
w/o this change, $CEPH_BUILD_VIRTUALENV is not passed to
osd-copy-from.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: respect $TMPDIR

create temp directory and files in $TMPDIR. the $TMPDIR is hard-wired to
/tmp before this change, we'd better respect the env variable $TMPDIR,
so it would be more consistent, and easier to do the cleanup if any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: vstart_wrapper.sh run tests in $TMPDIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: run "make check" with a unique $TMPDIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use p2 macros to simplify _do_write_small() process

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9702 from BlaXpirit/ceph-argparse-py3

ceph.in: python 3 compatibility of the ceph CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9749 from songbaisen/a10

mon: remove the redundant is_active judge in PaxosService

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9792 from stiopaa1/mon_monitorcc_addMutexLocker

mon/Monitor.cc:replce lock/unlock with Mutex:Lockr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9939 from Yan-waller/yj-wip-radosclientwaitforosd

rados/client: fix waiting on the condition variable more efficient.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9947 from yuyuyu101/wip-remove-check

osd: remove dispatch queue check since we don't queue hb message to this

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9833 from onyb/wip-rados-pypi

pybind: make rados ready for PyPI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10109 from xiexingguo/xxg-wip-bluestore-2016-07-02

os/bluestore: simplify LRUCache::trim()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8980 from stiopaa1/osd_unneededmapassign

osd/OSD.h: remove unneeded line

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8959 from hjwsm1989/hj-test

  os/filestore/LFNIndex: remove unused variable 'subdir_path'

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8687 from mslovy/wip-peering-msg-priority

message: optimization for message priority strategy

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10014 from gaowanlong/indent_caused_compile_warning

build: fix wrong indent caused compile warning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9970 from dillaman/wip-16349

rbd-mirror: gracefully handle being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

cmake: ceph_test_rbd_mirror does not require librados_test_stub

Linking both librados_test_stub and librados results in failed
test cases.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10016 from ceph/wip-cmake-rpm

rpm: build rpm with cmake

Reviewed-by: Ali Maredia <amaredia@redhat.com>

run-make-check.sh: run make check with cmake if do_cmake.sh is available

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: gracefully restart pool replayer when blacklisted

Fixes: http://tracker.ceph.com/issues/16349
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: do not clear image replayer error code after shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image deleter should use provided librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: each pool replayer should use its own librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10145 from dillaman/wip-cmake-rbd-python

cmake: fix incorrect dependencies to librados

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10119 from renhwztetecs/renhw-wip-cdentry-dump

mds: fixup dump Formatter' type error; add path_ino and is_primary in the CDentry::dump()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10062 from renhwztetecs/renhw-wip-filer

client/filer: cleanup the redundant judgments of _write&&_fallocate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10027 from batrick/client-root-lock

client: add missing client_lock for get_root

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10100 from fullerdj/wip-djf-16144

tools/cephfs: Remove cephfs-data-scan tmap_upgrade

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9971 from fullerdj/wip-djf-16288

mds: Kill C_SaferCond in evict_sessions()

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/rbd: before removing image make sure it is not bootstrapped

If an image is being bootstrapped, it implies that the rbd-mirror
daemon currently has the image open. The removal API will prevent the
removal of any image that is opened by another client.

Works-around: http://tracker.ceph.com/issues/16555
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cmake: fix incorrect dependencies to librados

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rpm: build rpm with cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10136 from runsisi/sisi-wip-fix-missing-return

librbd: fix missing return statement if failed to get mirror image state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10094 from athanatos/wip-throttle-test

test/common/Throttle.cc: fix race in shutdown

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #10088 from pritha-srivastava/wip_s3_signedurl_errors

rgw: Fix for using port 443 with pre-signed urls.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10112 from tchaikov/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10134 from trociny/wip-run-rbd-tests-fixup

test: run-rbd-tests test cmake fixup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10133 from tchaikov/wip-fixc-cephtool-test.sh

test: cephtool/test.sh: fix expect_false() calls

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10072 from dillaman/wip-16561

librbd: failed assertion after shrinking a clone image twice

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: fix missing return statement if failed to get mirror image state

Signed-off-by: runsisi <runsisi@zte.com.cn>

test: cephtool/test.sh: fix expect_false with pipe

this fixes failures like,

/home/jenkins-build/build/workspace/ceph-pull-requests/qa/workunits/cephtool/test.sh:
line 32: ceph osd blacklist ls | grep 192.168.0.1: command not found

where the failure is not the "failure" we are expecting.

in our tests, following command

expect_false "ceph osd blacklist ls | grep 192.168.0.1"

is designed to to verify that "ceph osd blacklist ls | grep 192.168.0.1"
fails with non-zero return code. but expect_false() evaluates the command
line using plain "$@", which will send the arguments direct to the shell,
and $0 is "ceph auth get client.xx | grep caps | grep mon", which does
not exist and is not built-in command. so we need to check the grep
command instead.

for multiple piped command line, use

expect_false sh <<< "echo foo  | grep bar | grep baz"

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use p2 macros to simplify the _write() process

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test: run-rbd-tests test cmake fixup

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

tests: ceph-authtool, add test for empty "--add-key="

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

authtool: Enhance argument combinations validation

Under certain circumstances ceph-authtool can create invalid key entries in
the specified keyring and behave less than intuitively. What's worse these
keys can currently cause ceph daemons to crash. This patch attempts to tighten
the behaviour and eliminate the possibility an invalid key can be created.

Fixes: http://tracker.ceph.com/issues/2904
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #10076 from dillaman/wip-rbd-mirror-cmake

test: fix rbd-mirror workunit test cases for cmake

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9731 from badone/wip-doc-ceph-authtool-print-fix

doc: ceph-authtool man page option is --print-key not --print

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10124 from trociny/wip-cmake-tests

test: cmake related fixups for rbd tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix potential access past eof during debugging prefill

1. pre-calculate r, so we don't have to do the division each loop, which
   is expensive.

2. make sure the extent to be allocated is within the block device size.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: introduce power 2 macros for block alignment and rounding

Macros for various sorts of alignment and rounding.
So we can get rid of the annoying division and mod operation and simplify
the blob/space access logic.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make sure path_fd is correctly closed during destruction

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify buffer eviction logic of LRUCache

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test: fix run-rbd-tests test for cmake

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: fix run-rbd-valgrind-unit-tests when built with cmake

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: ceph_test_librbd does not need rados_test_stub

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10117 from xiexingguo/xxg-wip-bluestore-2016-07-04

os/bluestore: fix incorrect pool decoding of bnode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9755 from dx9/wip-ns-name-uncompress

common/dns_resolve: use ns_name_uncompress instead of ns_name_ntop

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9766 from dx9/wip-autotools-arch

configure.ac: Use uname instead of arch.

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #9772 from tchaikov/wip-close-on-exec

msgr: set close on exec flag

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9820 from dx9/wip-fix-res-query-detect

configure.ac: fix res_query detection

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9845 from dx9/wip-config-basename

common: remove basename() dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9969 from liewegas/wip-assert

include/assert: clean up ceph assertion macros

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #9983 from SUSE/wip-cmake-fix-fcgi-include-dir

cmake: Fix FCGI include directory

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10093 from tchaikov/wip-16525

crush: reset bucket->h.items[i] when removing tree item

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9330 from dachary/wip-15176-partprobe

ceph-disk: partprobe should block udev induced BLKRRPART

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9394 from xiexingguo/xxg-wip-batchpgstates

mon/PGMonitor: batch filter pg states; add sanity check

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/CDentry: add path_ino and is_primary in the dump()

path_ino and is_primary are  main reference information in the dump(),
so add their

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mds: fixup dump Formatter' type error

fixup dump Formatter' type error,
dump_int->dump_unsigned

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

pybind: Prefer using $TMPDIR if available

Signed-off-by: Anirudha Bose <ani07nov@gmail.com>

os/bluestore: fix incorrect pool decoding of bnode

"pool" is a pointer, and we shall alter its value instead.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: add ceph-{disk,detect-init} to ALL

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: partprobe should block udev induced BLKRRPART

Wrap partprobe with flock to stop udev from issuing BLKRRPART because
this is racy and frequently fails with a message like:

    Error: Error informing the kernel about modifications to partition
    /dev/vdc1 -- Device or resource busy.  This means Linux won't know about
    any changes you made to /dev/vdc1 until you reboot -- so you shouldn't
    mount it or use it in any way before rebooting.

Opening a device (/dev/vdc for instance) in write mode indirectly
triggers a BLKRRPART ioctl from udev (starting version 214 and up)
when the device is closed (see below for the udev release note).

However, if udev fails to acquire an exclusive lock (with
flock(fd, LOCK_EX|LOCK_NB); ) the BLKRRPART ioctl is not issued.

https://github.com/systemd/systemd/blob/045e00cf16c47bc516c0823d059b7548f3ce9c7c/src/udev/udevd.c#L1042

Acquiring an exclusive lock before running the process that opens the
device in write mode is therefore an effective way to control this
behavior.

git clone git://anonscm.debian.org/pkg-systemd/systemd.git
systemd/NEWS:
CHANGES WITH 214:

  * As an experimental feature, udev now tries to lock the
       disk device node (flock(LOCK_SH|LOCK_NB)) while it
       executes events for the disk or any of its partitions.
       Applications like partitioning programs can lock the
       disk device node (flock(LOCK_EX)) and claim temporary
       device ownership that way; udev will entirely skip all event
       handling for this disk and its partitions. If the disk
       was opened for writing, the close will trigger a partition
       table rescan in udev's "watch" facility, and if needed
       synthesize "change" events for the disk and all its partitions.
       This is now unconditionally enabled, and if it turns out to
       cause major problems, we might turn it on only for specific
       devices, or might need to disable it entirely. Device Mapper
       devices are excluded from this logic.

Fixes: http://tracker.ceph.com/issues/15176
Signed-off-by: Marius Vollmer <marius.vollmer@redhat com>
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #10077 from yehudasa/wip-16494

rgw: fix for issue 16494

Merge pull request #10064 from BlaXpirit/brag-py3

Port ceph-brag to Python 3 (+ small fixes)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10115 from runsisi/ss-wip-fix-rpm-spec

ceph.spec.in: fix rpm package building error

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: do not depend on sse jerasure plugins if not avaiable

should not depend on them if SSE3 and/or SSE4 is not around.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link libglobal with lttng-ust

only link tracing libraries with lttng-ust, this fixes selinux denial
errors at run-time:

SELinux denials found on ubuntu@mira054.front.sepia.ceph.com: ['type=AVC
msg=audit(1467543917.770:131781): avc: denied { open } for pid=17511
comm="ceph-mon" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file', 'type=AVC
msg=audit(1467543914.518:131756): avc: denied { open } for pid=17418
comm="ceph-osd" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file', 'type=AVC
msg=audit(1467543917.770:131781): avc: denied { read } for pid=17511
comm="ceph-mon" name="lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file', 'type=AVC
msg=audit(1467543914.518:131756): avc: denied { read } for pid=17418
comm="ceph-osd" name="lttng-ust-wait-5" dev="tmpfs" ino=75199
scontext=system_u:system_r:ceph_t:s0
tcontext=system_u:object_r:tmpfs_t:s0 tclass=file']

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix rpm package building error as follows:

      error: Installed (but unpackaged) file(s) found:
      /usr/lib/udev/rules.d/60-ceph-by-parttypeuuid.rules

Signed-off-by: runsisi <runsisi@zte.com.cn>

cmake: refactor distutils_install_module to follow the pattern of distutils_install_cython_module

Signed-off-by: Kefu Chai <kchai@redhat.com>