Merge pull request #9290 from xiexingguo/xxg-wip-replicatedpg-cleaner

tools: do not closed stdout ; fix overload of "<" operator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9362 from xiexingguo/xxg-wip-texttable-df

tools: use TextTable for "rados df" plain output

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9894 from gaowanlong/osd_recovery_max_omap_entries_per_chunk

osd: limit omap data in push op

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10369 from ceph/wip-16715

msg/async/Event: ensure not refer to member variable which may destroyed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10358 from xiexingguo/xxg-wip-bluestore-2016-07-20

kv/MemDB: fix wrong output target and add sanity checks

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10349 from xiexingguo/xxg-wip-bluestore-2016-07-19

os/bluestore: add a boundary check of cache read

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10442 from weiqiaomiao/wqm-wip-rgw-getlc

rgw: return "NoSuchLifecycleConfiguration" if lifecycle config does not exist

Merge pull request #10446 from cbodley/wip-16817

rgw: clear realm watch on failed watch_restart
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10391 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: clear realm watch on failed watch_restart

if the realm is deleted while the gateway has a watch, it's disconnected
and watch_restart() will fail. this results in a watch handle of 0,
which leads to a segfault on ~RGWRealmWatcher when we pass it to
unwatch()

this commit cleans up the watch when watch_restart() fails, so we don't
try to unwatch() on destruction

Fixes: http://tracker.ceph.com/issues/16817
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10443 from wjwithagen/patch-13

src/CMakeLists.txt: remove double flag -Wno-invalid-offsetof

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10401 from theanalyst/rgw/test-multi-nose-attr

rgw: test/multi.py add a destructive attr to tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

src/CMakeLists.txt: remove double flag -Wno-invalid-offsetof

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10375 from xiexingguo/xxg-wip-fix-bmap-ut-failure

os/bluestore: use small encoding for bluefs extent and fnode

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use small encoding for bluefs extent and fnode

Also change bdev of extent from uint16_t to uint8_t, which matches
the fnode prefer_bdev field better, and is good for efficiency.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10440 from tchaikov/wip-16686

msg/async: remove static member variable

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

rgw: return "NoSuchLifecycleConfiguration" if lc config does not exist

return error code "NoSuchLifecycleConfiguration" if lifecycle configuration does not exist
as AmazonS3 latest API does.

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

Merge pull request #10430 from tchaikov/wip-cephtool-test.sh

qa/workunits/cephtool/test.sh: use absolute path for TEMP_DIR

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10423 from ceph/wip-16801

osd: fix the mem leak of RepGather

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: move flock.cc from libmds into libcommon

as we don't need to compile the whole mds just for building the
libcephfs

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: remove static member variable

msgr is part of libcommon. and the later is linked into both libcephfs
and librados. and hence into cephfs.so and rados.so. that's why we have
double free.

Fixes: http://tracker.ceph.com/issues/16686
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10304 from ukernel/wip-16610

ObjectCacher: fix last_write check in bh_write_adjacencies()
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10243 from jcsp/wip-cephfs-cleanup

CephFS: misc. cleanups and remove legacy cephfs tool

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #9716 from xiexingguo/xxg-wip-server_droppinlock2

Server: drop locks and auth pins if wait for pending truncate

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/cephtool/test.sh: use absolute path for TEMP_DIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10261 from songbaisen/b1

mon: remove the redundant cancel_probe_timeout function

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: fix the mem leak of RepGather

ReplicatedPG::new_repop() returns a pointer to RepGather with two refcounts,
one is held by ReplicatedPG::repop_queue, the other is supposed to be
held by the caller of this function. but it's caller
ReplicatedPG::submit_log_entries() assigns it to a
boost::intrusive_ptr<RepGather>() directly, why by default add_ref() in
its constructor. this makes the refcount 3. that's why we have a leak of
RepGather in `ReplicatedPG::new_repop(ObcLockManager&&,
boost::optional<std::function<void ()>>&&)`.

Fixes: http://tracker.ceph.com/issues/16801
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link against profiler if gperftools is found

this behaviour matches the autotools. and we should not enable profiling
unless asked to do so.

Fixes: http://tracker.ceph.com/issues/16804
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findgperftools.cmake

remove unused bits

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findudev.cmake

remove the crufts for handling the REQUIRED argument.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findblkid.cmake

* remove unused bits
* fix the wrong mark_as_advanced() call.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix libaio detection

* we used to check the header file of "aio.h" which is part of glibc,
  the glibc async i/o, and the corresponding library is librt. but what
  we are using is libaio.
* reference libaio using AIO_LIBRARIES for consistency.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include undefind paths

for example
```
include_directories(${LEVELDB_PREFIX}/include)
```
does not make sense if `LEVELDB_PREFIX` is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findbabeltrace.cmake

* we don't need the BABELTRACE_PATH_HINT setting, as packaging script
  requires the the babeltrace pacakges explicitly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add missing FindBacktrace.cmake

FindBacktrace.cmake is missing in older versions of cmake. so put a copy
in our tree.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "make uninstall" support

following https://cmake.org/Wiki/RecipeAddUninstallTarget

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "WITH_SUBMAN" option

it is used by ceph.spec.in for installing subscription manager cron job
on rhel.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup FindOpenLdap.cmake

* use find_package_handle_standard_args() and remove unused bits.
* rename OPENLDAP_LIBS to OPENLDAP_LIBRARIES, to be consistent with
  find_package() modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: cleanup Findtcmalloc.cmake

use find_package_handle_standard_args() and remove unused bits.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not check libuuid anymore

libuuid dependency is not found in ceph.spec.in or debian/control.
and we are not using libuuid since 62bfc7a.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10319 from wjwithagen/wip-wjw-fix-posix-tests

src/test/cli/* tests: POSIX Convert grep -P to grep -E

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10308 from xiexingguo/xxg-wip-fix-osd-df

mon/osdmonitor: fix incorrect output of "osd df" due to osd out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10091 from rubenk/add-osd-id-to-service-description

systemd: add osd id to service description

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10089 from gaowanlong/omapbench_usage

test: fix usage info of omapbench

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10251 from stiopaa1/osd_watch_addConsts

osd/Watch: add consts to member functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10419 from jcsp/wip-16764

client: fix shutdown with open inodes

client: fix shutdown with open inodes

This piece of code was dereferencing an invalid
iterator (invalidated by call to erase())

Fixes: http://tracker.ceph.com/issues/16764
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #9724 from vumrao/wip-vumrao-16130

rbd: cleanup - Proxied operations shouldn't result in error messages if replayed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10123 from trociny/wip-16449

librbd: prevent creation of clone from non-primary mirrored image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10348 from trociny/wip-16717

test: use wrapper that respects RBD_FEATURES when creating rbd image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10352 from wjwithagen/wip-wjw-freebsd-cmake-2

cmake: work to get inital FreeBSD stuff

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10354 from stiopaa1/msg_message_addConst

msg/Message.h: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10413 from SUSE/wip-doc-dev

doc: add Prepare tenant section to Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add "Prepare tenant" section to "Testing in the cloud" chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10162 from gaurav36/bug-16444-radosgw-error-message

rgw: better error message when user has no bucket created yet
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

osd: fix typo in comment of store_statfs_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

kv/MemDB: fill in statfs directly

Which is faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10405 from athanatos/wip-ec-pg-log

osd: fixups to explicitly persistenting missing sets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd_types: fix type of ChangeTracker::clean()

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: rename swap to claim, restrict to the non-tracking variant

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #10334 from ceph/wip-ec-pg-log

osd: persist the missing set explicitly

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10363 from SUSE/wip-16645

Fix libatomic_ops-devel in SUSE and specfile cleanup

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10241 from Yan-waller/yj-wip-rgwrests3-0710

rgw: add return value checking to avoid possible subsequent `parse` exception

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10331 from rzarzynski/wip-rgw-keystone-doc-fix

rgw, doc: fix formatting around Keystone-related options.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rpm: move libatomic_ops-devel to non-distro-specific section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: move gperftools-devel to non-distro-specific section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: use new name of libatomic_ops-devel

The previous form, libatomic-ops-devel, has been deprecated since May 23, 2012.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10322 from ceph/wip-debian-python-warnings

debian: python related changes

Reviewed-by: Dan Mick <dmick@redhat.com>

rgw: test/multi.py add a destructive attr to tests

Since the `test_zonegroup_remove` actually destroys a zonegroup, we
could just filter this out and run the suite as
`nosetests -a !destructive ../path/to/test-multi.py`

for provisioning a multisite mstart cluster.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10400 from tchaikov/wip-test_async_compressor

cmake: fix the build of unittest_async_compressor

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: fix the build of unittest_async_compressor

it was broken by 1e8388c

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8943 from tchaikov/wip-12287

pybind/ceph_argparse: handle non ascii unicode args

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9737 from chenji-kael/wip-rgw-lifecycle

rgw:lifecycle feature [rebased]

Merge pull request #10397 from theanalyst/wip-rgw-non-ec

wip: rgw multisite: preserve zone's extra pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10394 from tchaikov/wip-16772

cmake: disable unittest_async_compressor

Reviewed-by: Haomai Wang <haomai@xsky.com>

rgw: better error message when user has no bucket created yet

Currently when there is no bucket in the cluster and user execute
 "$radosgw-admin user  stats --uid=testid" command then rgw admin
not throwing meaningful error message.

With this fix it will show proper meaningful error message.

Fixes: http://tracker.ceph.com/issues/16444
Reported-by: Abhishek Lekshmanan <abhishek@suse.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

Merge pull request #10381 from SUSE/wip-doc-rgw-cgi-warning

doc/radosgw: rename config.rst to config-fcgi.rst
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw multisite: preserve zone's extra pool

In current multisite scenarios,if a bucket is created in master, we end
up storing multipart metadata in `$source-zone.rgw.buckets.non-ec` pool
instead of the zone's own non-ec pool, so we end up additionally
creating this pool and storing multipart metadata entries in it. Also if
a bucket is created in a secondary zone, and we initiate a multipart
upload, before mdlog sync with master, we end up getting errors during
complete multipart requests as omap entries are partly stored in the
`$zone.rgw.buckets.non-ec` as well as `$source-zone.rgw.buckets.non-ec`
pools which leads to total number of parts mismatch.

Fixes: http://tracker.ceph.com/issues/16712
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: disable unittest_async_compressor

as async compressor is not used anywhere. and haomai agrees to disable
this test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: handle non ascii unicode args

we raise UnicodeDecodeError at seeing non-ascii args if we fail to match
it with any command signatures. instead, we should use a unicode string
for representing the error in that case. please note, the exception is
not printed at all in real-world. =)

Fixes: http://tracker.ceph.com/issues/12287
Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: use python:Depends for ceph-{common,base,osd}'s deps

* ceph-base: use ${python:Depends} instead of listing the python
  dependencies manually, dh_python2 will scan the requirements
  of ceph-detect-init. and fill the subst var for us.
* ceph-common: add ${python:Depends}, as it packages ceph,
  and ceph-brag client.
* ceph-osd: it packages ceph-disk, so should add ${python:Depends}
  as its dependencies.

dh_python2 will figure them out.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: depends on python-all-dev instead of python-dev

* debian/control: Build-Depends: s/python-dev/python-all-dev/, per
    https://wiki.debian.org/Python/FAQ#Build_dependencies

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: remove ${shlibs:Depends} from python-* packages

* debian/control:
  as we have listed the linked libraries in Depends section, for example,
  python-rados depends on librados. and we don't need `dpkg-shlibdeps` to
  help figure out shared library substvar dependencies for us. by removing
  them, we can silence the warnings of
```
warning: dpkg-shlibdeps: package could avoid a useless dependency if
debian/python-rados/usr/lib/python2.7/dist-packages/rados.x86_64-linux-gnu.so
was not linked against libpthread.so.0 (it uses none of the library's
symbols)
```
  -lpthread is introduced by `python-config --ldflags` but it turns out we
  are not using any symbols from pthread in the extension directly. and
  pthread is included in glibc. so this does not added any extra
  dependency to python-* pacakges. but it's desirable to have less
  warnings.
* debian/rules: exclude python-* packages from dh_shlibdeps, as we will
  not use it to prepare the shlib deps substvars for these packages any
  more.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: do not run dh-python against non-python packages

some packages do not package python modules or scripts. so override
dh_python2 to exclude them.

this change silences warnings like:
```
warning: dpkg-gencontrol: package ceph-mon: unused substitution
variable ${python:Provides}
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8150 from wjwithagen/patch-3

.gitignore: exclude coredumps, logfiles and temporary testresults

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add op to get Lifecycle configuration

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

RGW:lifecycle feature[rebase]
As same as amazon S3 interface,"PUT Bucket lifecycle" and
"DELETE Bucket lifecycle" have been implemented,
"GET Bucket lifecycle" not realized yet as S3cmd has not
realize it also.
The feature`s main point is to remove expire file per day.
Files transfer from hot layer to cold layer is not supported.
ToDo:Maybe to transfer from replicate pool to EC pool or
from ssd to sata pool will be valuable.

Now put all buckets which should do lifecycle into shard
objects in .rgw.lc pool.

lifecycle config file format:
<LifecycleConfiguration>
    <Rule>
        <ID>sample-rule</ID>
        <Prefix></Prefix>
        <Status>enable</Status>
        <Expiration>
           <Days>1</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

Signed-off-by: Ji Chen <insomnia@139.com>

Merge pull request #9089 from SirishaGuduru/bugfix15603

doc: Modification for "TEST S3 ACCESS" section  in "INSTALL CEPH OBJECT GATEWAY" page

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10388 from yehudasa/wip-rgw-sync-err-injection

rgw: add configurables for {data,meta} sync error injection

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: add configurables for {data,meta} sync error injection

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Revert RGW torrent

Revert the RGW torrent commit for now, it was causing issues.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #10377 from chhabaramesh/extent_alloc

os/bluestore:Fix size calculation in bitallocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9589 from zhouruisong/master

rgw:add a s3 API of make torrent for a object

doc/radosgw: rename config.rst to config-fcgi.rst

This file documents how to configure RGW to use Apache/FastCGI, so rename
the file and modify the title and intro to make that clear.

Also, add a note that CGI can pose a security risk - e.g. http://httpoxy.org

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10244 from jcsp/wip-cephfs-health-doc

doc/cephfs: explain the various health messages

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc/cephfs: explain the various health messages

Signed-off-by: John Spray <john.spray@redhat.com>

os/bluestore:Fix size calculation in bitallocator

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10149 from wjwithagen/patch-12

CMakeList.txt: link ceph_objectstore_tool against fuse only if WITH_FUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/tools/CMakeLists.txt: conditionalise the FUSE usage

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10366 from ceph/wip-cmake-debian

debian: bump compat to 9

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>