rgw: clear realm watch on failed watch_restart

if the realm is deleted while the gateway has a watch, it's disconnected
and watch_restart() will fail. this results in a watch handle of 0,
which leads to a segfault on ~RGWRealmWatcher when we pass it to
unwatch()

this commit cleans up the watch when watch_restart() fails, so we don't
try to unwatch() on destruction

Fixes: http://tracker.ceph.com/issues/16817
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10401 from theanalyst/rgw/test-multi-nose-attr

rgw: test/multi.py add a destructive attr to tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10375 from xiexingguo/xxg-wip-fix-bmap-ut-failure

os/bluestore: use small encoding for bluefs extent and fnode

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use small encoding for bluefs extent and fnode

Also change bdev of extent from uint16_t to uint8_t, which matches
the fnode prefer_bdev field better, and is good for efficiency.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10440 from tchaikov/wip-16686

msg/async: remove static member variable

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10430 from tchaikov/wip-cephtool-test.sh

qa/workunits/cephtool/test.sh: use absolute path for TEMP_DIR

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10423 from ceph/wip-16801

osd: fix the mem leak of RepGather

Reviewed-by: Samuel Just <sjust@redhat.com>

msg/async: remove static member variable

msgr is part of libcommon. and the later is linked into both libcephfs
and librados. and hence into cephfs.so and rados.so. that's why we have
double free.

Fixes: http://tracker.ceph.com/issues/16686
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10304 from ukernel/wip-16610

ObjectCacher: fix last_write check in bh_write_adjacencies()
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10243 from jcsp/wip-cephfs-cleanup

CephFS: misc. cleanups and remove legacy cephfs tool

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #9716 from xiexingguo/xxg-wip-server_droppinlock2

Server: drop locks and auth pins if wait for pending truncate

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa/workunits/cephtool/test.sh: use absolute path for TEMP_DIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10261 from songbaisen/b1

mon: remove the redundant cancel_probe_timeout function

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: fix the mem leak of RepGather

ReplicatedPG::new_repop() returns a pointer to RepGather with two refcounts,
one is held by ReplicatedPG::repop_queue, the other is supposed to be
held by the caller of this function. but it's caller
ReplicatedPG::submit_log_entries() assigns it to a
boost::intrusive_ptr<RepGather>() directly, why by default add_ref() in
its constructor. this makes the refcount 3. that's why we have a leak of
RepGather in `ReplicatedPG::new_repop(ObcLockManager&&,
boost::optional<std::function<void ()>>&&)`.

Fixes: http://tracker.ceph.com/issues/16801
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10319 from wjwithagen/wip-wjw-fix-posix-tests

src/test/cli/* tests: POSIX Convert grep -P to grep -E

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10308 from xiexingguo/xxg-wip-fix-osd-df

mon/osdmonitor: fix incorrect output of "osd df" due to osd out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10091 from rubenk/add-osd-id-to-service-description

systemd: add osd id to service description

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10089 from gaowanlong/omapbench_usage

test: fix usage info of omapbench

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10251 from stiopaa1/osd_watch_addConsts

osd/Watch: add consts to member functions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10419 from jcsp/wip-16764

client: fix shutdown with open inodes

client: fix shutdown with open inodes

This piece of code was dereferencing an invalid
iterator (invalidated by call to erase())

Fixes: http://tracker.ceph.com/issues/16764
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #9724 from vumrao/wip-vumrao-16130

rbd: cleanup - Proxied operations shouldn't result in error messages if replayed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10123 from trociny/wip-16449

librbd: prevent creation of clone from non-primary mirrored image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10348 from trociny/wip-16717

test: use wrapper that respects RBD_FEATURES when creating rbd image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10352 from wjwithagen/wip-wjw-freebsd-cmake-2

cmake: work to get inital FreeBSD stuff

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10354 from stiopaa1/msg_message_addConst

msg/Message.h: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10413 from SUSE/wip-doc-dev

doc: add Prepare tenant section to Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add "Prepare tenant" section to "Testing in the cloud" chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10162 from gaurav36/bug-16444-radosgw-error-message

rgw: better error message when user has no bucket created yet
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10405 from athanatos/wip-ec-pg-log

osd: fixups to explicitly persistenting missing sets

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

osd_types: fix type of ChangeTracker::clean()

Signed-off-by: Samuel Just <sjust@redhat.com>

osd_types: rename swap to claim, restrict to the non-tracking variant

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #10334 from ceph/wip-ec-pg-log

osd: persist the missing set explicitly

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10363 from SUSE/wip-16645

Fix libatomic_ops-devel in SUSE and specfile cleanup

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10241 from Yan-waller/yj-wip-rgwrests3-0710

rgw: add return value checking to avoid possible subsequent `parse` exception

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10331 from rzarzynski/wip-rgw-keystone-doc-fix

rgw, doc: fix formatting around Keystone-related options.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rpm: move libatomic_ops-devel to non-distro-specific section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: move gperftools-devel to non-distro-specific section

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: use new name of libatomic_ops-devel

The previous form, libatomic-ops-devel, has been deprecated since May 23, 2012.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10322 from ceph/wip-debian-python-warnings

debian: python related changes

Reviewed-by: Dan Mick <dmick@redhat.com>

rgw: test/multi.py add a destructive attr to tests

Since the `test_zonegroup_remove` actually destroys a zonegroup, we
could just filter this out and run the suite as
`nosetests -a !destructive ../path/to/test-multi.py`

for provisioning a multisite mstart cluster.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10400 from tchaikov/wip-test_async_compressor

cmake: fix the build of unittest_async_compressor

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: fix the build of unittest_async_compressor

it was broken by 1e8388c

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8943 from tchaikov/wip-12287

pybind/ceph_argparse: handle non ascii unicode args

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9737 from chenji-kael/wip-rgw-lifecycle

rgw:lifecycle feature [rebased]

Merge pull request #10397 from theanalyst/wip-rgw-non-ec

wip: rgw multisite: preserve zone's extra pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10394 from tchaikov/wip-16772

cmake: disable unittest_async_compressor

Reviewed-by: Haomai Wang <haomai@xsky.com>

rgw: better error message when user has no bucket created yet

Currently when there is no bucket in the cluster and user execute
 "$radosgw-admin user  stats --uid=testid" command then rgw admin
not throwing meaningful error message.

With this fix it will show proper meaningful error message.

Fixes: http://tracker.ceph.com/issues/16444
Reported-by: Abhishek Lekshmanan <abhishek@suse.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

Merge pull request #10381 from SUSE/wip-doc-rgw-cgi-warning

doc/radosgw: rename config.rst to config-fcgi.rst
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw multisite: preserve zone's extra pool

In current multisite scenarios,if a bucket is created in master, we end
up storing multipart metadata in `$source-zone.rgw.buckets.non-ec` pool
instead of the zone's own non-ec pool, so we end up additionally
creating this pool and storing multipart metadata entries in it. Also if
a bucket is created in a secondary zone, and we initiate a multipart
upload, before mdlog sync with master, we end up getting errors during
complete multipart requests as omap entries are partly stored in the
`$zone.rgw.buckets.non-ec` as well as `$source-zone.rgw.buckets.non-ec`
pools which leads to total number of parts mismatch.

Fixes: http://tracker.ceph.com/issues/16712
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

cmake: disable unittest_async_compressor

as async compressor is not used anywhere. and haomai agrees to disable
this test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: handle non ascii unicode args

we raise UnicodeDecodeError at seeing non-ascii args if we fail to match
it with any command signatures. instead, we should use a unicode string
for representing the error in that case. please note, the exception is
not printed at all in real-world. =)

Fixes: http://tracker.ceph.com/issues/12287
Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: use python:Depends for ceph-{common,base,osd}'s deps

* ceph-base: use ${python:Depends} instead of listing the python
  dependencies manually, dh_python2 will scan the requirements
  of ceph-detect-init. and fill the subst var for us.
* ceph-common: add ${python:Depends}, as it packages ceph,
  and ceph-brag client.
* ceph-osd: it packages ceph-disk, so should add ${python:Depends}
  as its dependencies.

dh_python2 will figure them out.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: depends on python-all-dev instead of python-dev

* debian/control: Build-Depends: s/python-dev/python-all-dev/, per
    https://wiki.debian.org/Python/FAQ#Build_dependencies

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: remove ${shlibs:Depends} from python-* packages

* debian/control:
  as we have listed the linked libraries in Depends section, for example,
  python-rados depends on librados. and we don't need `dpkg-shlibdeps` to
  help figure out shared library substvar dependencies for us. by removing
  them, we can silence the warnings of
```
warning: dpkg-shlibdeps: package could avoid a useless dependency if
debian/python-rados/usr/lib/python2.7/dist-packages/rados.x86_64-linux-gnu.so
was not linked against libpthread.so.0 (it uses none of the library's
symbols)
```
  -lpthread is introduced by `python-config --ldflags` but it turns out we
  are not using any symbols from pthread in the extension directly. and
  pthread is included in glibc. so this does not added any extra
  dependency to python-* pacakges. but it's desirable to have less
  warnings.
* debian/rules: exclude python-* packages from dh_shlibdeps, as we will
  not use it to prepare the shlib deps substvars for these packages any
  more.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: do not run dh-python against non-python packages

some packages do not package python modules or scripts. so override
dh_python2 to exclude them.

this change silences warnings like:
```
warning: dpkg-gencontrol: package ceph-mon: unused substitution
variable ${python:Provides}
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8150 from wjwithagen/patch-3

.gitignore: exclude coredumps, logfiles and temporary testresults

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add op to get Lifecycle configuration

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

RGW:lifecycle feature[rebase]
As same as amazon S3 interface,"PUT Bucket lifecycle" and
"DELETE Bucket lifecycle" have been implemented,
"GET Bucket lifecycle" not realized yet as S3cmd has not
realize it also.
The feature`s main point is to remove expire file per day.
Files transfer from hot layer to cold layer is not supported.
ToDo:Maybe to transfer from replicate pool to EC pool or
from ssd to sata pool will be valuable.

Now put all buckets which should do lifecycle into shard
objects in .rgw.lc pool.

lifecycle config file format:
<LifecycleConfiguration>
    <Rule>
        <ID>sample-rule</ID>
        <Prefix></Prefix>
        <Status>enable</Status>
        <Expiration>
           <Days>1</Days>
        </Expiration>
    </Rule>
</LifecycleConfiguration>

Signed-off-by: Ji Chen <insomnia@139.com>

Merge pull request #9089 from SirishaGuduru/bugfix15603

doc: Modification for "TEST S3 ACCESS" section  in "INSTALL CEPH OBJECT GATEWAY" page

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10388 from yehudasa/wip-rgw-sync-err-injection

rgw: add configurables for {data,meta} sync error injection

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: add configurables for {data,meta} sync error injection

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Revert RGW torrent

Revert the RGW torrent commit for now, it was causing issues.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #10377 from chhabaramesh/extent_alloc

os/bluestore:Fix size calculation in bitallocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9589 from zhouruisong/master

rgw:add a s3 API of make torrent for a object

doc/radosgw: rename config.rst to config-fcgi.rst

This file documents how to configure RGW to use Apache/FastCGI, so rename
the file and modify the title and intro to make that clear.

Also, add a note that CGI can pose a security risk - e.g. http://httpoxy.org

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10244 from jcsp/wip-cephfs-health-doc

doc/cephfs: explain the various health messages

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc/cephfs: explain the various health messages

Signed-off-by: John Spray <john.spray@redhat.com>

os/bluestore:Fix size calculation in bitallocator

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10149 from wjwithagen/patch-12

CMakeList.txt: link ceph_objectstore_tool against fuse only if WITH_FUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/tools/CMakeLists.txt: conditionalise the FUSE usage

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10366 from ceph/wip-cmake-debian

debian: bump compat to 9

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #10361 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10355 from yehudasa/wip-16742

rgw: back off bucket sync on failures, don't store marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix collection of object sync errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix marker tracker completion handling

Was not tracking high markers correctly. Could only work if there was a single
hole in the completion range. Just keep a map of all the complete entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: collect() stops if error encoutered

and returns true if needed to be called again

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

debian: bump compat to 9

dh_auto_configure set -DCMAKE_BUILD_TYPE=None so the default does not
take effect at all. this is on purpose, see [1].

and dpkg-buildflags is able to produce the suggested compiling flags by
debian policy. dh_* can pass these flags to cmake only if it works at
compat>=9, see [2].

and we are safe to move to compat 9, as jewel's supported debian based
distro is jessie (with debhelper 9.20150101), ubuntu trusty (with
debhelper 9.20131227ubuntu1)

---
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=701233#35
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=653916

Fixes: http://tracker.ceph.com/issues/16744
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10368 from oritwas/wip-rgw-upgrade-period-commit

rgw: fix upgrade from old multisite to new multisite configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10360 from ceph/wip-include-include

cmake: add_subdirectory(include)

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10315 from onyb/wip-cephfs-pypi

pybind: Rework cephfs/setup.py for PyPI

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10316 from batrick/mds-standby-assertions

mds: add assertions for standby_daemons invariant

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10347 from renhwztetecs/renhw-wip-test-add-fsdump

test/mds: add fs dump in test_ceph_argparse.py

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10202 from xiaoxichen/fix_16588

mds/MDSMap default metadata pool to -1 (was: output None instead of 0 when no fs present.)

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9864 from ajarr/wip-15615

ceph_volume_client: store authentication metadata

Reviewed-by: John Spray <john.spray@redhat.com>

README.FreeBSD: Cmake instructions

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

FindNSS.cmake: find includes in /usr/local/includes/nss/nss

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

CMakeLists.txt: Add FreeBSD isms

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

do_freebsd-cmake.sh: script to cmake on FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10327 from xiexingguo/xxg-wip-bluestore-2016-07-18

os/bluestore: misc fixes/cleanups

Mark's Comments:
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR may have contributed to write performance improvements along with #10257.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10257 from chhabaramesh/extent_alloc

os/bluestore: extent alloc functionality for stupid and bitmap allocator

Mark's Comments:
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR appears to result in a large sequential write performance increase (Up to ~3.5X), and may contribute to a small random write performance increase as well.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

rgw: fix upgrade from old multisite to new multisite configuration

We need to store the updated current period after adding the old converted regions

Fixes: http://tracker.ceph.com/issues/16751
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

src/test/cli/* tests: POSIX Convert grep -P to grep -E (v2)

 - FreeBSD does not have grep -P
 - [Posix grep]: replace \t by [[:space:]]

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #9082 from stiopaa1/osd_TestOpsSocketHook_removeUnneededParam

pass string by const ref and add override to virtual function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9667 from SUSE/wip-15984

ceph-osd-prestart.sh: drop Upstart-specific code

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #10339 from tchaikov/wip-16675

selinux: allow read /proc/<pid>/cmdline

Reviewed-by: Boris Ranto <branto@redhat.com>

cmake: remove unused Findfio.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Findfcgi.cmake cleanups

FindPackageHandleStandardArgs() takes care of the find_package()
boilderplate stuff. so no need to repeat them. and remove the checkings
in env variables.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include $lrc_srcs in unittest_erasure_code_lrc

as ${lrc_srcs} is not defined in the scope where
unittest_erasure_code_lrc is added as a target. and instead we link
unittest_erasure_code_lrc against ec_lrc dynamic library. so we can
safely remove ${lrc_srcs} from unittest_erasure_code_lrc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/EXTRA_LIBS/EXTRALIBS/

it's a typo, i think. as EXTRA_LIBS is never defined anywhere.

Signed-off-by: Kefu Chai <kchai@redhat.com>