rbd-mirror: include local pool id in resync throttle unique key

Fixes: http://tracker.ceph.com/issues/16536
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10205 from oritwas/wip-rgw-fix-upgrade

rgw: add missing master_zone when running with old default region config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10040 from xiexingguo/xxg-wip-bit-alloc-tree-configurable

os/bluestore: make zone/span size of bitmap-allocator configurable

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10075 from cbodley/wip-rgw-tenant-sync

rgw: add tenant support to multisite sync

Merge pull request #8964 from songbaisen/song24

global: log which process/command sent a signal

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10195 from stiopaa1/msg_asyncmess_removeIncFile

msg/AsyncMessenger: remove unneeded include file

Merge pull request #10224 from stiopaa1/msg_asyncmess_event_addConst

msg/async/Event.h: add const to member function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10230 from stiopaa1/msg_asyncmess_Processor_accept_changeToVoid

msg/AsyncMessenger: change return type to void

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10152 from tchaikov/wip-16277

osd: increment stats on recovery pull also

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10222 from ceph/wip-cmake

cmake changes for pypi

verfied f23

cmake: fix the detection of curl_multi_wait()

it was broken by b5bb77d5

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: remove rados.c from MANIFEST.in

this fixes the warning of

```
running egg_info
creating
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info
writing
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/PKG-INFO
writing top-level names to
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/top_level.txt
writing dependency_links to
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/dependency_links.txt
writing manifest file
'/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/SOURCES.txt'
warning: no files found matching 'rados.c'
```

in the out-of-source tree build, rados.c is not located in the same
directory of setup.py, rados.pyx, and rados.pyd. it is in the build_dir,
for example, it could be:

  ceph/build/src/pybind/rados/pyrex/rados.c

where ceph/build is the build directory of cmake.
and sdist will include it when running egg_info, because cython's
build_ext module lists `rados.c` as one of the "source_files".

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass necessary cflags to build_ext

this fixes:
```
creating
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados
creating
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados/pyrex
/usr/bin/gcc-6 -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
-fno-strict-aliasing -iquote /var/ceph/ceph/src/include -fPIC
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-
gnu/python2.7 -I/usr/include/python2.7 -c
/home/kefu/ceph/build/src/pybind/rados/pyrex/rados.c -o
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados/pyrex/ra
dos.o -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g
-fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -g
-fwrapv -O2 -Wall -Wstrict-prototypes -lpython2.
7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1
-Wl,-Bsymbolic-functions
/home/kefu/ceph/build/src/pybind/rados/pyrex/rados.c:239:28: fatal error:
rados/librados.h: No such file or directory
 #include "rados/librados.h"
                            ^
compilation terminated.
```

seems we can not pass the CFLAGS with space in it to setup.py using env variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: put cythonized .c files into ${CMAKE_CURRENT_BINARY_DIR}

it was put into the directory where .pyx is located. it's not expected
in an out-of-tree build.
* {rados,rbd,cephfs}/setup.py: use build_ext from cython if possible, fallback to
  the one from setuptools or distutils
* Distutils.cmake: pass --cython-c-in-temp --build-temp and
  --cython-include-dirs to "build_ext" command

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9138 from cxwshawn/wip-cl-fix

osd: replace ceph:atomic_t with std::atomic in osd module.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9871 from odivlad/fix-init-el7-master

master: remove SYSTEMD_RUN from initscript

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8317 from dreamhost/gentoo-python

Gentoo support for ceph-disk / ceph-detect-init; pip speedup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9085 from renhwsky/renhw-wip-osdmonitor-updatepool

osd/osdmonitor: pool of objects and bytes beyond quota should all be warn

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10234 from tchaikov/wip-drop-tmap_upgrade

ceph_test_rados_api_tmap_migrate: remove test for tmap_upgrade

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9117 from wjin/clean

osd: remove duplicated function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9252 from vumrao/wip-vumrao-15983

mon: change osdmap flags set and unset messages

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9452 from mslovy/wip-preserve-alloc-hint

osd:preserve allocation hint attribute during recovery

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9698 from LiumxNL/fix-clones-flush-evict

rados: add option to include clones when doing flush or evict

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9703 from badone/wip-cryptoKey-encrypt-segfault

cephx: Fix multiple segfaults due to attempts to encrypt or decrypt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9717 from songbaisen/a7

mon: do not recalculate 'to_remove' when it's known

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9763 from dx9/wip-pthread-np

compat: add abstractions for non portable pthread name funcs

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_api_tmap_migrate: remove test for tmap_upgrade

the tested feature was removed in ad6cc41

> tmap_upgrade was only for Jewel, so remove it for Kraken.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/AsyncMessenger: change return type to void

Change the return type of function Processor::start to void. It
doesn't make sense to return constant value(zero).

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: add support for allocating 0-sized extent

For testing, we may choose randomized length of extent to allocate,
which means we could trim down the length to 0.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix allocation of discontiguous blocks

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check for get_free() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify is_exhausted() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make zone/span size of bitmap-allocator configurable

Perviously as zone_size_block and span_size are identical, so we could use
span_size only to simplify the calcuation.

Now that the zone_size_block and span_size are both configurable, so
it is necessary to treat them respectively.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10190 from ktdreyer/doc-delete-pool

doc: more details for pool deletion

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/Event.h: add const to member function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10201 from xiexingguo/xxg-wip-bluestore-2016-07-08

os/bluestore: misc cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10157 from cbodley/wip-rgw-sync-error-repo

rgw: fix error_repo segfault in data sync

rgw: fix error_repo segfault in data sync

RGWDataSyncShardCR will only allocate an error_repo if it's doing
incremental sync, so RGWDataSyncSingleEntryCR needs to guard against a
null error_repo

also, RGWDataSyncShardCR::stop_spawned_services() was dropping the last
reference to the error_repo before calling drain_all(), which meant that
RGWDataSyncSingleEntryCR could still be holding a pointer. now uses a
boost::intrusive_ptr in RGWDataSyncSingleEntryCR to account for its
reference

Fixes: http://tracker.ceph.com/issues/16603
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10184 from ktdreyer/doc-rados-operations-systemd

doc: rm SysV instructions, add systemd

Merge pull request #9570 from zaitcev/wip-5073-multen-doc

rgw: Add documentation for the Multi-tenancy feature

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10151 from oritwas/wip-rgw-fix-cls-sync-stats

rgw: register the correct handler for cls_user_complete_stats

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove datalog keys from error repo on ENOENT

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add optional --tenant flag to test_multi.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add tenant id to GetBucketInstanceInfo

use the rgw_bucket overload of get_bucket_instance_info() so it can
supply the tenant id

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: carry tenant id with data sync

use rgw_bucket_shard to track buckets, which includes tenant id

Fixes: http://tracker.ceph.com/issues/16469
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add tenant to url for RGWRESTStreamWriteRequest

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add tenant to rgw_bucket json format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_bucket_parse_bucket_key()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add tenant name to datalog entries

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: convert bucket instance listings back to metadata key format

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use tenant/ for bucket instance metadata keys

to work around the ambiguity of parsing tenant: and :shard in the same
bucket instance metadata key, use tenant/ instead

to preserve backward compatibility with existing objects, new helper
function rgw_bucket_instance_key_to_oid() converts this / back to a :
before being used as an object name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add get_key() methods to format rgw_buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9934 from tchaikov/wip-build-on-armf

common: instantiate strict_si_cast<long> not strict_si_cast<int64_t>

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10204 from ceph/wip-cmake

cmake: fixes for pypi changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: more details for pool deletion

Prior to this change, the documention instructed administrators to
delete rulesets and users, but did not go into details regarding how to
do that.

Add example commands that admins may use to search for rulesets and
users that might reference the to-be-deleted pool.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

common: instantiate strict_si_cast<long> not strict_si_cast<int64_t>

this fixes the build on armf.

on 32bit platforms, cstdint is very likely to

 typedef long long int int64_t;

this results in compilation error like

 `common/strtol.cc:190:75: error: duplicate explicit instantiation of 'T
 strict_si_cast(const char, std::string) [with T = long long int;
 std::string = std::basic_string]'

 [-fpermissive]
 template int64_t strict_si_cast(const char *str, std::string *err);
 ^`

we can address this by instantiate the primitive type of `long long`
instead of `in64_t`.

Fixes: http://tracker.ceph.com/issues/16398
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8189 from chardan/wip-chardan-15100

librados examples: link and include from current source tree by default.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: release compressor if comp_mode turned out to be none

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop unnecessary increase of iterator

We are going to break anyway.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: assert to confirm that we don't access violation

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop unused member _bytes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: check against that we don't access violation

Some day we might want make this interface more applicable by handling
the abnormal cases more gracefully...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10137 from xiexingguo/xxg-wip-bluestore-2016-07-05-02

os/bluestore: change algorithm of compression header from string to int

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: change algorithm of compression header from string to int

The literal description of compression algorithm can vary from
various compression types and thus increases the complexity of
en/decoding, which as a result can cause chaos. Also it can be
more efficient by translating it into a fixed-length type.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: register the correct handler for cls_user_complete_stats

Fixes: http://tracker.ceph.com/issues/16624
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: add missing master_zone when running with old default region config

Fixes: http://tracker.ceph.com/issues/16627
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

pybind: put temp .o files in $TMPDIR

otherwise they will go to $PWD/$TMPDIR instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: creage egg_info files into build directory

otherwise they will go to the source tree, and "git ls-files" will list
them as untracked files, which annoy gitbuilder-ceph-tarball*-cmake
gitbuilders. like

+ echo 'error: Added files:'
error: Added files:
+ cat .git/added-files
src/pybind/rados/rados.egg-info/PKG-INFO
...

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10186 from ceph/wip-cmake-no-with-mds

cmake: remove WITH_MDS option

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/AsyncMessenger: remove unneeded include file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10163 from stiopaa1/msg_asyncmess_removeUnusedFunction_create_anon_connection

msg/AsyncMessenger: remove unused function

Merge pull request #10178 from ceph/wip-cmake

cmake: use stock Find* modules.

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: remove  option

Signed-off-by: Ali Maredia <amaredia@redhat.com>

doc: rm SysV instructions, add systemd

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10128 from xiexingguo/xxg-wip-bluestore-2016-07-05

os/bluestore: introduce power 2 macros for block alignment and rounding

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10180 from theanalyst/mrgw/cmake-fixes

mrgw: search for cmake build dir.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mrgw: search for cmake build dir.

Similar to mstart and mstop, mrgw also needs to find the correct build
dir as we pass in the pidfile and asokfile which otherwise falls back to
src/run dir.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10116 from tchaikov/wip-cmake-make-check

test: make check using cmake

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10097 from yehudasa/wip-mrun-cmake

mrun, mstart.sh, mstop.sh: search for cmake build directory

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9489 from athanatos/wip-16156

ReplicatedPG: call op_applied for submit_log_entries based repops

Reviewed-by: David Zafman <dzafman@redhat.com>

cmake: use the stock FindCURL.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use the stock FindEXPAT.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-helpers.sh: disable test_pg_scrub()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10139 from SUSE/wip-cmake-really-fix-fcgi-include-dir

cmake: Really add FCGI_INCLUDE_DIR to include_directories for rgw

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9981 from yangdongsheng/rbd_set_vals

cls_rbd: set omap values in batch during image creation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10155 from trociny/wip-16555-workaround

qa/workunits/rbd: before removing image make sure it is not bootstrapped

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10164 from dillaman/wip-cmake-test-rbd-mirror

cmake: ceph_test_rbd_mirror does not require librados_test_stub

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Really add FCGI_INCLUDE_DIR to include_directories for rgw

Commit 3cf6c53 was incorrect.  FCGI_INCLUDE_DIR doesn't need to be
set in src/rgw/CMakeLists.txt, but it does need to be set for the
rgw_a target in src/CMakeLists.txt, as well as for the
ceph_test_librgw_file and ceph_test_librgw_file_nfsns targets in
src/test/CMakeLists.txt.  I can only assume that I must not have
done a completely clean rebuild at some point when testing a
reworked version of the earlier commit :-/

This is only a problem for distros that keep the FCGI headers in
/usr/include/fastcgi/ (e.g.: SUSE).

This commit also removes a redundant include of <fcgiapp.h>

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #9083 from badone/wip-doc-last_epoch_started

doc/dev: Fix missing code section due to no lexer for "none"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9704 from badone/wip-ceph-authtool-fixes

authtool: Enhance argument combinations validation

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon : change osdmap flags set and unset message

Fixes: http://tracker.ceph.com/issues/15983
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

run-make-check.sh: use "make check" for running tests

"make all" does not offer "ceph-disk" and "ceph-detect-init" for
testing. as they are solely used for testing purpose. instead, these two
python command line packages are installed by the "install" target. so
we need to use "make check" to 1) prepare the test dependencies 2)
launch ctest to perform the test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: add ceph-{disk,detect-init} to ALL"

This reverts commit 19c0731751df9642cc5b8c583ea41d12e0b5ec41.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test//osd/osd-copy-from.sh: use absolute path for "rados put"

"rados -p rbd put foo rados" does not work if "rados" is not in current
path. so change it to "rados -p rbd put foo $(which rados)"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass more env vars to osd-copy-from.sh

we should use the cmake function of add_ceph_test() to add
osd-copy-from.sh as a test. then we won't miss any env variables.
w/o this change, $CEPH_BUILD_VIRTUALENV is not passed to
osd-copy-from.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: respect $TMPDIR

create temp directory and files in $TMPDIR. the $TMPDIR is hard-wired to
/tmp before this change, we'd better respect the env variable $TMPDIR,
so it would be more consistent, and easier to do the cleanup if any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: vstart_wrapper.sh run tests in $TMPDIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: run "make check" with a unique $TMPDIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use p2 macros to simplify _do_write_small() process

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9702 from BlaXpirit/ceph-argparse-py3

ceph.in: python 3 compatibility of the ceph CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>