cmake: link rgw_a against cls_lock_client

rgw_lc.cc is using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: exclude private symbols in public libs

we should avoid exposing non-public symbols from user facing dynamic
libraries. so pass '--exclude-libs' with appropriate argument to linker.
as libcephfs does not add the ((visibility ("default"))) specifier to
the exported symbols, we can not set the default visiblity to hidden for
libcephfs, and "-export-symbols-regex" is a libtool option, we need to
develop a way for cmake to fix the visibility of libcephfs. it's still a
TODO.

* librados
  - pass '--exclude-libs=ALL' to linker
  - add buffer.cc to librados, so we can use '--exclude-libs=ALL'.
* libcephfs: pass '--exclude-libs=libcommon.a,libclient.a,libosdc.a' to
  linker
* libcommon
  - extract common_buffer_obj from libcommon, to avoid compilation
    this source file repeatly.
* tests:
  - link against common_internal_objs explicitly if the test in
    question is using the internal symbols.
* ceph-client-debug:
  - link against client explicitly, and do not link against librados
    anymore, as it is not used in this tool.

Fixes: http://tracker.ceph.com/issues/16556
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move common_utf8 closer to where it is used

for example, json_spirit uses common_utf8, so link json_spirit against common_utf8.
because libcommon includes json_spirit, this change does not impact the
consumers of libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10457 from SUSE/wip-15497

doc: add Backporting section to Essentials chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add Backporting section to Essentials chapter

... and fix some broken links.

Fixes: http://tracker.ceph.com/issues/15497
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10414 from SUSE/wip-doc-dev

doc: add Upload logs to archive server section...

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add Upload logs to archive server section...

...to Testing in the cloud testing chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10456 from cbodley/wip-16834

rgw: use endpoints from master zone instead of zonegroup
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: use endpoints from master zone instead of zonegroup

Fixes: http://tracker.ceph.com/issues/16834
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10396 from zhouruisong/master

rgw:add a s3 API of make torrent for a object

Merge pull request #10438 from ceph/wip-arm64

ARM64 fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Tested-by: tone zhang <tone.zhang@linaro.org>

Merge pull request #10418 from wjwithagen/wip-wjw-freebsd-rocksdb

cmake: pass cmake's compiler and flags to compile RocksDB into build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9334 from yangdongsheng/rbd_clone_hiberit

rbd: inherit the parent image features when cloning an image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10087 from yangdongsheng/rbd_snap_remove_force

rbd: Allow user to remove snapshot with --force to auto flatten children

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9371 from ukernel/wip-osx-fuse

build ceph-fuse on OSX

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10436 from gregsfortytwo/wip-snapshot-doc

Add documentation about snapshots

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10399 from renhwztetecs/renhw-wip-mds-cleanup2

mds: unuse Class and cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10444 from SUSE/wip-drop-cephfs-mopup

man/Makefile-client.am: drop legacy cephfs tool

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

mds: unuse Class and cleanup

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

src/CMakeLists.txt: pass compiler and flags used to compile into RocksDB build

 - make sure the right compiler is used in CXX
 - NDEBUG if the current code is not being build with debugging

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10433 from SUSE/wip-16808

deb: merge ceph-fs-common into ceph-common

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10342 from songbaisen/b6

logmon: check is_leader() before doing any work on get_trim_to()

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #10343 from xiexingguo/xxg-wip-pipe-2016-07-19

msg/simple: fix wrong condition checking of writing TAG_CLOSE on closing

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #10345 from tchaikov/wip-dump-cmdline

global/signal_handler: dump cmdline instead of arg[0]

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10356 from stiopaa1/msg_message_changeConstructorsRemoveCodeDuplication

msg/Message.h: remove code duplication

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9290 from xiexingguo/xxg-wip-replicatedpg-cleaner

tools: do not closed stdout ; fix overload of "<" operator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9362 from xiexingguo/xxg-wip-texttable-df

tools: use TextTable for "rados df" plain output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9894 from gaowanlong/osd_recovery_max_omap_entries_per_chunk

osd: limit omap data in push op

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10369 from ceph/wip-16715

msg/async/Event: ensure not refer to member variable which may destroyed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10358 from xiexingguo/xxg-wip-bluestore-2016-07-20

kv/MemDB: fix wrong output target and add sanity checks

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10349 from xiexingguo/xxg-wip-bluestore-2016-07-19

os/bluestore: add a boundary check of cache read

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10442 from weiqiaomiao/wqm-wip-rgw-getlc

rgw: return "NoSuchLifecycleConfiguration" if lifecycle config does not exist

Merge pull request #10446 from cbodley/wip-16817

rgw: clear realm watch on failed watch_restart
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10391 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

qa/qa_scripts/cephscrub.sh: do not try to remove ceph-fs-common package

The ceph-fs-common package is being dropped.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: clear realm watch on failed watch_restart

if the realm is deleted while the gateway has a watch, it's disconnected
and watch_restart() will fail. this results in a watch handle of 0,
which leads to a segfault on ~RGWRealmWatcher when we pass it to
unwatch()

this commit cleans up the watch when watch_restart() fails, so we don't
try to unwatch() on destruction

Fixes: http://tracker.ceph.com/issues/16817
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10443 from wjwithagen/patch-13

src/CMakeLists.txt: remove double flag -Wno-invalid-offsetof

Reviewed-by: Kefu Chai <kchai@redhat.com>

man/Makefile-client.am: drop legacy cephfs tool

Somehow c76c31d312ce7a623acd99dbdedfe471c0cd445f missed this bit.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10401 from theanalyst/rgw/test-multi-nose-attr

rgw: test/multi.py add a destructive attr to tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: describe snapshots for developers, and warn about multi-fs

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/CMakeLists.txt: remove double flag -Wno-invalid-offsetof

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10375 from xiexingguo/xxg-wip-fix-bmap-ut-failure

os/bluestore: use small encoding for bluefs extent and fnode

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use small encoding for bluefs extent and fnode

Also change bdev of extent from uint16_t to uint8_t, which matches
the fnode prefer_bdev field better, and is good for efficiency.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test_librbd: add a test for newly introduced api of snap_remove2

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce no-progress for rbd snap remove.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #10440 from tchaikov/wip-16686

msg/async: remove static member variable

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

deb: merge ceph-fs-common into ceph-common

Fixes: http://tracker.ceph.com/issues/16808
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: return "NoSuchLifecycleConfiguration" if lc config does not exist

return error code "NoSuchLifecycleConfiguration" if lifecycle configuration does not exist
as AmazonS3 latest API does.

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

rgw:add a s3 API of make torrent for a object

When you execute the command gettorrent of a object, a torrent file will be produced and returned.
The torrent also will be save into a pool named default.rgw.torrent.
If the torrent of a object exists in default.rgw.torrent, it will be returned.

Signed-off-by: zhouruisong <236131368@qq.com>

Merge pull request #10430 from tchaikov/wip-cephtool-test.sh

qa/workunits/cephtool/test.sh: use absolute path for TEMP_DIR

Reviewed-by: Loic Dachary <ldachary@redhat.com>

pybind: add remove_snap2 api in python binding

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

tools/rbd: introduce an option of --force for snap remove

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: fix a incorrect input for cpp_strerror()

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: coding style: fix the indent problem

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: coding style: add space after comma

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: introduce a new flag of RBD_SNAP_REMOVE_FLATTEN

introduce RBD_SNAP_REMOVE_FLATTEN for flags of Image::snap_remove2()
to auto flatten the children of snapshot which we want to remove.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

librbd: introduce a new api of snap_remove2

currently, we only have one api for snap_remove, but if we want to
pass more options about snapshot removal, that's impossible.

This patch introduce a new api of snap_remove2 here to solve
this problem.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #10423 from ceph/wip-16801

osd: fix the mem leak of RepGather

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: move flock.cc from libmds into libcommon

as we don't need to compile the whole mds just for building the
libcephfs

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: remove static member variable

msgr is part of libcommon. and the later is linked into both libcephfs
and librados. and hence into cephfs.so and rados.so. that's why we have
double free.

Fixes: http://tracker.ceph.com/issues/16686
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10304 from ukernel/wip-16610

ObjectCacher: fix last_write check in bh_write_adjacencies()
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Update to upstream rocksdb

Signed-off-by: Dan Mick <dan.mick@redhat.com>

src/test/CMakeLists.txt: don't assume Intel (sse/sse2)

Fixes ARM64 build of ceph_perf_local test

Signed-off-by: Dan Mick <dan.mick@redhat.com>

erasure-code/jerasure/CMakeLists.txt: set NEON flags for neon objs

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10243 from jcsp/wip-cephfs-cleanup

CephFS: misc. cleanups and remove legacy cephfs tool

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #9716 from xiexingguo/xxg-wip-server_droppinlock2

Server: drop locks and auth pins if wait for pending truncate

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/cephtool/test.sh: use absolute path for TEMP_DIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10261 from songbaisen/b1

mon: remove the redundant cancel_probe_timeout function

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: fix the mem leak of RepGather

ReplicatedPG::new_repop() returns a pointer to RepGather with two refcounts,
one is held by ReplicatedPG::repop_queue, the other is supposed to be
held by the caller of this function. but it's caller
ReplicatedPG::submit_log_entries() assigns it to a
boost::intrusive_ptr<RepGather>() directly, why by default add_ref() in
its constructor. this makes the refcount 3. that's why we have a leak of
RepGather in `ReplicatedPG::new_repop(ObcLockManager&&,
boost::optional<std::function<void ()>>&&)`.

Fixes: http://tracker.ceph.com/issues/16801
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link against profiler if gperftools is found

this behaviour matches the autotools. and we should not enable profiling
unless asked to do so.

Fixes: http://tracker.ceph.com/issues/16804
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findgperftools.cmake

remove unused bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findudev.cmake

remove the crufts for handling the REQUIRED argument.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findblkid.cmake

* remove unused bits
* fix the wrong mark_as_advanced() call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix libaio detection

* we used to check the header file of "aio.h" which is part of glibc,
  the glibc async i/o, and the corresponding library is librt. but what
  we are using is libaio.
* reference libaio using AIO_LIBRARIES for consistency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include undefind paths

for example
```
include_directories(${LEVELDB_PREFIX}/include)
```
does not make sense if `LEVELDB_PREFIX` is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findbabeltrace.cmake

* we don't need the BABELTRACE_PATH_HINT setting, as packaging script
  requires the the babeltrace pacakges explicitly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add missing FindBacktrace.cmake

FindBacktrace.cmake is missing in older versions of cmake. so put a copy
in our tree.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "make uninstall" support

following https://cmake.org/Wiki/RecipeAddUninstallTarget

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "WITH_SUBMAN" option

it is used by ceph.spec.in for installing subscription manager cron job
on rhel.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup FindOpenLdap.cmake

* use find_package_handle_standard_args() and remove unused bits.
* rename OPENLDAP_LIBS to OPENLDAP_LIBRARIES, to be consistent with
  find_package() modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findtcmalloc.cmake

use find_package_handle_standard_args() and remove unused bits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not check libuuid anymore

libuuid dependency is not found in ceph.spec.in or debian/control.
and we are not using libuuid since 62bfc7a.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10319 from wjwithagen/wip-wjw-fix-posix-tests

src/test/cli/* tests: POSIX Convert grep -P to grep -E

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10308 from xiexingguo/xxg-wip-fix-osd-df

mon/osdmonitor: fix incorrect output of "osd df" due to osd out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10091 from rubenk/add-osd-id-to-service-description

systemd: add osd id to service description

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10089 from gaowanlong/omapbench_usage

test: fix usage info of omapbench

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10251 from stiopaa1/osd_watch_addConsts

osd/Watch: add consts to member functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10419 from jcsp/wip-16764

client: fix shutdown with open inodes

client: fix shutdown with open inodes

This piece of code was dereferencing an invalid
iterator (invalidated by call to erase())

Fixes: http://tracker.ceph.com/issues/16764
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #9724 from vumrao/wip-vumrao-16130

rbd: cleanup - Proxied operations shouldn't result in error messages if replayed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10123 from trociny/wip-16449

librbd: prevent creation of clone from non-primary mirrored image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10348 from trociny/wip-16717

test: use wrapper that respects RBD_FEATURES when creating rbd image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10352 from wjwithagen/wip-wjw-freebsd-cmake-2

cmake: work to get inital FreeBSD stuff

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10354 from stiopaa1/msg_message_addConst

msg/Message.h: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10413 from SUSE/wip-doc-dev

doc: add Prepare tenant section to Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add "Prepare tenant" section to "Testing in the cloud" chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10162 from gaurav36/bug-16444-radosgw-error-message

rgw: better error message when user has no bucket created yet
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

osd: fix typo in comment of store_statfs_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kv/MemDB: fill in statfs directly

Which is faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10405 from athanatos/wip-ec-pg-log

osd: fixups to explicitly persistenting missing sets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>