src/CMakeLists.txt: remove unneeded libraries from ceph-dencoder target

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge remote-tracking branch 'upstream/pull/10514/head'

* upstream/pull/10514/head:
  src/vstart.sh: stop script aborting when ceph.cfg is not there

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

src/vstart.sh: stop script aborting when ceph.cfg is not there

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10432 from dillaman/wip-16708

rbd-mirror: potential IO stall when using asok flush request

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10341 from dillaman/wip-16223

rbd-mirror: reduce memory footprint during journal replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10489 from ceph/wip-cmake

cmake changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: link libcommon against pthread

as Thread.cc lives in libcommon, and global is not using libpthread
directly. so moving libpthread linkage from libglobal to libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10177 from BlaXpirit/ceph-tests-py3

pybind: Port Python-based tests and remaining Python bindings to Python 3

Reviewed-by: Case Bodley <cbodley@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'upstream/pull/10312/head'

* upstream/pull/10312/head:
  vstart: refactor and fix forced config changes
  vstart: cleanup echoed commands and sh issues

cmake: cleanup Findfuse.cmake

* remove unused stuff, we do not pass/use FUSE_VERSION in cmake.
  the FUSE_VERSION we are using is defined by "fuse_common.h".
* use FindPackageHandleStandardArgs for handling find_package()
  arguments.
* also detect osxfuse on MacOS.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not reference undefined variables

* AIO_LIBS is now named AIO_LIBARIES, and there is not point to print
  out its path
* USE_NSS is not defined if NSS is not checked, so if(USE_NSS) is better
  here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated linked lib for libcommon

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use fullpaths of libraries

better off using the detected path, instead of using the default path
for finding the linked libaries.

see "cmake  --help-policy CMP0065" for more details.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: set policies

this makes sure we are compatible with latest and future cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove uuid

as it is replaced by the boost implementation, and not detected in
cmake anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link ec_jerasure_generic against crush

crush is not used in erasure plugins at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: exclude private symbols in librbd

missed it in 0edfad0.
* librbd: pass '--exclude-libs,ALL' to linker
* rbd: link against used libraries explicitly, rbd is accessing some
  internal symbols not exposed by librbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10483 from trociny/fixup-test-rbd-mirror

qa/workunits/rbd: small fixup and improvements for rbd-mirror tests

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>

Merge pull request #10374 from ktdreyer/wip-doc-debug-ceph-fuse

doc: add troubleshooting steps for ceph-fuse

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10332 from dillaman/wip-16689

librbd: optimize away unnecessary object map updates

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

doc: add troubleshooting steps for ceph-fuse

Thanks to John Spray <jspray@redhat.com> on ceph-devel for this
information.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10479 from batrick/standby-notes

doc: small standby doc edits

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10434 from batrick/i16743

doc: add client config ref

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10409 from weiqiaomiao/wqm-wip-rgw-reboot

rgw:fix rgw boot failed after upgrade to master latest version

Merge pull request #10379 from xiexingguo/xxg-wip-bluestore-2016-07-21

os/bluestore: fix bitmap allocating failure if max_alloc_size is 0

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10140 from ifed01/wip-bluestore-fsck-update

os/bluestore: replace interval_set with a bitset for used_block conta…

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests, however no fsck specific performance or memory tests were performed.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

doc: add historical warning for mds_standby_replay

When testing < v10.2.1 with standby replay, I was often confused why the
MDS daemons would go into replay even when this was false.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fix config separator/title size.

This makes the setting consistent with the others.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10060 from trociny/wip-16386

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10472 from tchaikov/wip-cmake

cmake: exclude non-public symbols in shared libraries

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10273 from yonghengdexin735/wip-zzz-pip-delete-no_casting_required

msg/simple/Pipe: eliminating casts for the comparing of len and recv_max_prefetch

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10477 from cbodley/wip-16839

radosgw-admin: zone[group] modify can change realm id
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9988 from ifed01/wip-bluestore-reduce-blob4

os/bluestore: get rid off blob's ref_map for non-shared objects

Mark's Comments:

This passed Jenkins checks.
This PR did not appear to have a significant impact on performance tests.
This initially failed "ceph_test_objectstore --gtest_filter=*/2" segfaulting in one of the tests.  See the attached log in the comments for the PR for details.  Subsequent reruns passed, and a run through valgrind's memcheck also passed.  Igor indicated that he has seen this particular segfault previously and believes it is related to running out of memory.  Given that the PR has passed the object store tests multiple times and the segfault has been observed in the past, I believe this PR is unrelated.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10439 from weiqiaomiao/wqm-wip-rgw-zonedump

rgw: add lc_pool when decode or encode struct RGWZoneParams

Reviewed-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: zone[group] modify can change realm id

allows the default zone and zonegroup (created with empty realm_id) to
be later added to a realm. the 'modify' command now accepts either
--realm_id=id or --rgw-realm=name

Fixes: http://tracker.ceph.com/issues/16839
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10476 from yonghengdexin735/wip-zzz-CDir

mds/CDir: remove the part of judgment for _next_dentry_on_set

Reviewed-by: John Spray <john.spray@redhat.com>

mds/CDir: remove the part of judgment for _next_dentry_on_set

Signed-off-by: zhang.zezhu <zhang.zezhu@zte.com.cn>

Merge pull request #10428 from jcsp/wip-16592

mon: validate states transmitted in beacons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon: validate states transmitted in beacons

Since FSMap was added, the state of a daemon can lead
to an entirely invalid map, but we were letting daemons
send any state they wanted.

Especially, we must not allow standby daemons to set
any state other than STANDBY.

Fixes: http://tracker.ceph.com/issues/16592
Signed-off-by: John Spray <john.spray@redhat.com>

qa/workunits/rbd: wait for image deleted before checking health

When a primiry image is being deleted, the mirrored image might
temporary be reported in error state, before deletion is propagated.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: small fixup and improvements for rbd-mirror tests

- log to stderr;
- log status if a `wait_for` function failed;
- don't needlessly sleep in `wait_for` functions after the last
  unsuccessful iteration;
- make `wait_for_pool_images` work for image removal case;
- fix `wait_for_pool_images` reset timeout (last_count set).

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cmake: link rgw_a against cls_lock_client

rgw_lc.cc is using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10429 from jcsp/wip-16807-assertion

mds: add assertion in handle_slave_rename_prep

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

cmake: exclude private symbols in public libs

we should avoid exposing non-public symbols from user facing dynamic
libraries. so pass '--exclude-libs' with appropriate argument to linker.
as libcephfs does not add the ((visibility ("default"))) specifier to
the exported symbols, we can not set the default visiblity to hidden for
libcephfs, and "-export-symbols-regex" is a libtool option, we need to
develop a way for cmake to fix the visibility of libcephfs. it's still a
TODO.

* librados
  - pass '--exclude-libs=ALL' to linker
  - add buffer.cc to librados, so we can use '--exclude-libs=ALL'.
* libcephfs: pass '--exclude-libs=libcommon.a,libclient.a,libosdc.a' to
  linker
* libcommon
  - extract common_buffer_obj from libcommon, to avoid compilation
    this source file repeatly.
* tests:
  - link against common_internal_objs explicitly if the test in
    question is using the internal symbols.
* ceph-client-debug:
  - link against client explicitly, and do not link against librados
    anymore, as it is not used in this tool.

Fixes: http://tracker.ceph.com/issues/16556
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move common_utf8 closer to where it is used

for example, json_spirit uses common_utf8, so link json_spirit against common_utf8.
because libcommon includes json_spirit, this change does not impact the
consumers of libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9925 from fullerdj/wip-djf-11171

mds: Add path filtering for dump cache

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10281 from vishalkanaujia/wip_inotable_repair

cephfs: Inotable repair during forward scrub

Reviewed-by: John Spray <john.spray@redhat.com>

test: fix librbd tests for rbd_skip_partial_discard

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Fixes: http://tracker.ceph.com/issues/16386
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rgw:fix rgw boot failed after upgrade to master latest version

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

msg/simple/Pipe: eliminating casts for the comparing of len and recv_max_prefetch

Signed-off-by: zhang.zezhu <zhang.zezhu@zte.com.cn>

doc: add client config ref

Fixes: http://tracker.ceph.com/issues/16743
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10457 from SUSE/wip-15497

doc: add Backporting section to Essentials chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add Backporting section to Essentials chapter

... and fix some broken links.

Fixes: http://tracker.ceph.com/issues/15497
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10414 from SUSE/wip-doc-dev

doc: add Upload logs to archive server section...

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add Upload logs to archive server section...

...to Testing in the cloud testing chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10456 from cbodley/wip-16834

rgw: use endpoints from master zone instead of zonegroup
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: use endpoints from master zone instead of zonegroup

Fixes: http://tracker.ceph.com/issues/16834
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10396 from zhouruisong/master

rgw:add a s3 API of make torrent for a object

Merge pull request #10438 from ceph/wip-arm64

ARM64 fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Tested-by: tone zhang <tone.zhang@linaro.org>

Merge pull request #10418 from wjwithagen/wip-wjw-freebsd-rocksdb

cmake: pass cmake's compiler and flags to compile RocksDB into build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9334 from yangdongsheng/rbd_clone_hiberit

rbd: inherit the parent image features when cloning an image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10087 from yangdongsheng/rbd_snap_remove_force

rbd: Allow user to remove snapshot with --force to auto flatten children

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9371 from ukernel/wip-osx-fuse

build ceph-fuse on OSX

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10436 from gregsfortytwo/wip-snapshot-doc

Add documentation about snapshots

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10399 from renhwztetecs/renhw-wip-mds-cleanup2

mds: unuse Class and cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10444 from SUSE/wip-drop-cephfs-mopup

man/Makefile-client.am: drop legacy cephfs tool

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

mds: unuse Class and cleanup

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

src/CMakeLists.txt: pass compiler and flags used to compile into RocksDB build

 - make sure the right compiler is used in CXX
 - NDEBUG if the current code is not being build with debugging

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10433 from SUSE/wip-16808

deb: merge ceph-fs-common into ceph-common

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10342 from songbaisen/b6

logmon: check is_leader() before doing any work on get_trim_to()

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #10343 from xiexingguo/xxg-wip-pipe-2016-07-19

msg/simple: fix wrong condition checking of writing TAG_CLOSE on closing

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>

Merge pull request #10345 from tchaikov/wip-dump-cmdline

global/signal_handler: dump cmdline instead of arg[0]

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10356 from stiopaa1/msg_message_changeConstructorsRemoveCodeDuplication

msg/Message.h: remove code duplication

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9290 from xiexingguo/xxg-wip-replicatedpg-cleaner

tools: do not closed stdout ; fix overload of "<" operator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9362 from xiexingguo/xxg-wip-texttable-df

tools: use TextTable for "rados df" plain output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9894 from gaowanlong/osd_recovery_max_omap_entries_per_chunk

osd: limit omap data in push op

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10369 from ceph/wip-16715

msg/async/Event: ensure not refer to member variable which may destroyed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10358 from xiexingguo/xxg-wip-bluestore-2016-07-20

kv/MemDB: fix wrong output target and add sanity checks

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10349 from xiexingguo/xxg-wip-bluestore-2016-07-19

os/bluestore: add a boundary check of cache read

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10442 from weiqiaomiao/wqm-wip-rgw-getlc

rgw: return "NoSuchLifecycleConfiguration" if lifecycle config does not exist

Merge pull request #10446 from cbodley/wip-16817

rgw: clear realm watch on failed watch_restart
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10391 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

qa/qa_scripts/cephscrub.sh: do not try to remove ceph-fs-common package

The ceph-fs-common package is being dropped.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: clear realm watch on failed watch_restart

if the realm is deleted while the gateway has a watch, it's disconnected
and watch_restart() will fail. this results in a watch handle of 0,
which leads to a segfault on ~RGWRealmWatcher when we pass it to
unwatch()

this commit cleans up the watch when watch_restart() fails, so we don't
try to unwatch() on destruction

Fixes: http://tracker.ceph.com/issues/16817
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10443 from wjwithagen/patch-13

src/CMakeLists.txt: remove double flag -Wno-invalid-offsetof

Reviewed-by: Kefu Chai <kchai@redhat.com>

man/Makefile-client.am: drop legacy cephfs tool

Somehow c76c31d312ce7a623acd99dbdedfe471c0cd445f missed this bit.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10401 from theanalyst/rgw/test-multi-nose-attr

rgw: test/multi.py add a destructive attr to tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: describe snapshots for developers, and warn about multi-fs

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

src/CMakeLists.txt: remove double flag -Wno-invalid-offsetof

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10375 from xiexingguo/xxg-wip-fix-bmap-ut-failure

os/bluestore: use small encoding for bluefs extent and fnode

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use small encoding for bluefs extent and fnode

Also change bdev of extent from uint16_t to uint8_t, which matches
the fnode prefer_bdev field better, and is good for efficiency.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test_librbd: add a test for newly introduced api of snap_remove2

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

rbd: introduce no-progress for rbd snap remove.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #10440 from tchaikov/wip-16686

msg/async: remove static member variable

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

deb: merge ceph-fs-common into ceph-common

Fixes: http://tracker.ceph.com/issues/16808
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: return "NoSuchLifecycleConfiguration" if lc config does not exist

return error code "NoSuchLifecycleConfiguration" if lifecycle configuration does not exist
as AmazonS3 latest API does.

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

rgw:add a s3 API of make torrent for a object

When you execute the command gettorrent of a object, a torrent file will be produced and returned.
The torrent also will be save into a pool named default.rgw.torrent.
If the torrent of a object exists in default.rgw.torrent, it will be returned.

Signed-off-by: zhouruisong <236131368@qq.com>

Merge pull request #10430 from tchaikov/wip-cephtool-test.sh

qa/workunits/cephtool/test.sh: use absolute path for TEMP_DIR

Reviewed-by: Loic Dachary <ldachary@redhat.com>

pybind: add remove_snap2 api in python binding

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>