msg/async/net_handler.cc: Do not apply SO_REUSEADDR for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12297 from SUSE/wip-18134

tests: check hostname --fqdn sanity before running make check

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: check hostname --fqdn sanity before running make check

Fixes: http://tracker.ceph.com/issues/18134
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12282 from liewegas/wip-bluestore-dump

os/bluestore: clean up Allocator::dump

Merge pull request #12287 from liewegas/wip-store-kv-stats

kv: dump rocksdb stats

Merge pull request #12284 from ceph/wip-cmake-args

build: dump env during build

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #12224 from yehudasa/wip-rgw-list-all

rgw/rgw_rados: do not omap_getvals with (u64)-1 max (updated)
also approve, passed teuthology (many false positives in several classes)

Merge pull request #12187 from Jing-Scott/dev/slave-master-sync-info

rgw: metadata sync info should be shown at master zone of slave zoneg…

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: dump kv stats

Signed-off-by: Sage Weil <sage@redhat.com>

osd: dump kv stats for ObjectStores (if any)

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

kv/RocksDBStore: dump extended stats

Signed-off-by: Varada Kari <varada.kari@sandisk.com>
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

debian/rules: dump env during build

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: dump env during build

Signed-off-by: Sage Weil <sage@redhat.com>

kv: make kvdb can print itself statistics.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #12245 from ifed01/wip-fix-global-init-refcount

test/fio: fix global CephContext life cycle

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12241 from dachary/wip-17889-systemd-order

ceph-disk: enable --runtime ceph-osd systemd units

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>

os/bluestore: dump allocator state on allocation failure

This is probably noisy for an *actual* ENOSPC, but the OSD should
actually prevent those from happening in general, and we are
very interested in fixing spurious allocation failures due to
allocator problems.  See http://tracker.ceph.com/issues/18054

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12075 from ceph/wip-rgw-zonegroup-config

rgw: fix for unsafe change of rgw_zonegroup

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12280 from trociny/wip-17127

rbd-mirror: set SEQUENTIAL and NOCACHE advise flags on image sync

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: dump allocator state at lowest debug level

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused ostream arg to Allocator::dump

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12262 from cbodley/wip-17888

rgw: fix for versioned delete_multi_object
ftw

Merge pull request #12256 from liewegas/wip-18093

osd: fix memory leak from EC write workload

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12238 from gaurav36/bug-rbd-feature-enable-error

librbd: enabling/disabling rbd feature should report missing dependency

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12195 from trociny/wip-qa-rbd-wait_for_image_present

qa/workunits/rbd: use image id when probing for image presence

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11942 from wjwithagen/wip-wjw-freebsd-boost

do_freebsd.sh: Build with SYSTEM Boost on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: set SEQUENTIAL and NOCACHE advise flags on image sync

Fixes: http://tracker.ceph.com/issues/17127
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Cmake/FreeBSD: use the SYSTEM_BOOST

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12209 from tchaikov/wip-enable-unittest_dns_resolve

test: enable unittest_dns_resolve

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: enable unittest_dns_resolve

guess this test was missed when we were moving to cmake. let's re-enable
it!

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12263 from Adirl/fix_max_buffers

msg/async/rdma: don't use more buffers than what device capabilities …

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12140 from iain-buclaw-sociomantic:pybind-aioexec

python-rados: implement new aio_execute

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind: Fix comment for exec in Ioctx

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>

test/pybind: Add tests for aio_execute

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>

msg/async/rdma: don't use more buffers than what device capabilities allow

Change-Id: I6df7dd1a4d871b4dca55e784969ff459bebc6a86
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #12261 from liewegas/wip-store-rename-test

ceph_test_objectstore: fix Rename test

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12231 from yehudasa/wip-17965

rgw: set duration for lifecycle lease

rgw: fix for versioned delete_multi_object

Fixes: http://tracker.ceph.com/issues/17888
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12221 from liewegas/wip-18072

msg/simple/Pipe: handle addr decode error

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12218 from dillaman/wip-18068

librbd: diffs to clone's first snapshot should include parent diffs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

test/fio: fix global CephContext life cycle in fio pobjectstore plugin

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph_test_objectstore: fix Rename test

We were doing a no-op setattr here, even though the intenral
ENOENT used to be ignored, because it previously triggered a
bug in bluestore.  We're no longer ignoring that internal
error, though, as of 6ecb44f0c6216f3fb29b384a2d10c7d33bf1d19c,
so remove this part of the test.

The original bluestore bug we used to trigger was fixed by
0210f1fde6f8e17affdd0c49055a71a01f109447.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10086 from Yan-waller/yj-wip-rgwrests3ccstringfind

rgw: conform to the standard usage of string::find

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

ceph-disk: enable --runtime ceph-osd systemd units

If ceph-osd@.service is enabled for a given device (say /dev/sdb1 for
osd.3) the ceph-osd@3.service will race with ceph-disk@dev-sdb1.service
at boot time.

Enabling ceph-osd@3.service is not necessary at boot time because

   ceph-disk@dev-sdb1.service

calls

   ceph-disk activate /dev/sdb1

which calls

   systemctl start ceph-osd@3

The systemctl enable/disable ceph-osd@.service called by ceph-disk
activate is changed to add the --runtime option so that ceph-osd units
are lost after a reboot. They are recreated when ceph-disk activate is
called at boot time so that:

   systemctl stop ceph

knows which ceph-osd@.service to stop when a script or sysadmin wants
to stop all ceph services.

Before enabling ceph-osd@.service (that happens at every boot time),
make sure the permanent enablement in /etc/systemd is removed so that
only the one added by systemctl enable --runtime in /run/systemd
remains. This is useful to upgrade an existing cluster without creating
a situation that is even worse than before because ceph-disk@.service
races against two ceph-osd@.service (one in /etc/systemd and one in
/run/systemd).

Fixes: http://tracker.ceph.com/issues/17889
Signed-off-by: Loic Dachary <loic@dachary.org>

rgw: fix for unsafe change of rgw_zonegroup

Fixes: http://tracker.ceph.com/issues/17962
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12229 from cbodley/wip-rgw-bucket-sync-status-assert

rgw: RGWBucketSyncStatusManager uses existing async_rados

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #12254 from Adirl/disconnect_log

msg/async/rdma: fix disconnect log line

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9846 from stiopaa1/msg_asyncmess_removeUnneededInclude

msg/AsyncMessenger: remove unneeded include

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12246 from tchaikov/wip-17809

buffer: add copy(unsigned, ptr) back

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: enabling/disabling rbd feature should report missing dependency

Currently while enabling or disabling rbd feature command does not
give missing dependency for eg: attempting to enable the journaling
feature on an image that doesn't have the exclusive-lock feature
enabled should give missing dependency error message.

Fixes: http://tracker.ceph.com/issues/16985
Reported-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

Merge pull request #12206 from jdurgin/wip-librados-setxattr-overload

librados: remove new setxattr overload to avoid breaking the C++ ABI

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12234 from liewegas/wip-17942-again

osdc/Objecter: fix relock race

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12228 from ceph/wip-jlayton-umount

libcephfs and test suite fixes

osd/osd_types: avoid pinning memory with pg_log_entry_t::snaps

The pg_log_entry_t may stay in memory for a long time;
avoid pinning a larger buffer in memory.

See: http://tracker.ceph.com/issues/18093
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: prevent ObjectModDesc::bl from pinning memory

We may decode this from a large buffer (e.g., an
ECSubWrite payload) and the ObjectModDesc is part of
the pg_log_entry_t, which may stay in memory for a long
time.  Avoid pinning memory.

Fixes: http://tracker.ceph.com/issues/18093
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/rdma: fix disconnect log line

issue: none

Change-Id: I29e2a6ae78dda3cf4a1817b9ab41bea0f636bda5
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #12222 from dillaman/wip-18070

rbd-nbd: restart parent process logger after forking

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

build/ops: restart ceph-osd@.service after 20s instead of 100ms

Instead of the default 100ms pause before trying to restart an OSD, wait
20 seconds instead and retry 30 times instead of 3. There is no scenario
in which restarting an OSD almost immediately after it failed would get
a better result.

It is possible that a failure to start is due to a race with another
systemd unit at boot time. For instance if ceph-disk@.service is
delayed, it may start after the OSD that needs it. A long pause may give
the racing service enough time to complete and the next attempt to start
the OSD may succeed.

This is not a sound alternative to resolve a race, it only makes the OSD
boot process less sensitive. In the example above, the proper fix is to
enable --runtime ceph-osd@.service so that it cannot race at boot time.

The wait delay should not be minutes to preserve the current runtime
behavior. For instance, if an OSD is killed or fails and restarts after
10 minutes, it will be marked down by the ceph cluster.  This is not a
change that could break things but it is significant and should be
avoided.

Refs: http://tracker.ceph.com/issues/17889

Signed-off-by: Loic Dachary <loic@dachary.org>

buffer: add copy(unsigned, ptr) back

to be ABI-compatible with prior versions.

copy(unsigned, ptr) was renamed to copy_deep() in 6d7f748

Fixes: http://tracker.ceph.com/issues/17809
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12244 from ceph/wip-rgw-compression-doc

doc: add docs for raw compression

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12113 from ceph/wip-rgw-zone-compression

rgw: move compression config into zone placement

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12232 from liewegas/wip-rgw-lclist

rgw: need to close_section in lc list op

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove config option rgw_compression_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ops use zone placement's compression type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add --compression for zone placement add/modify

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add compression_type field to RGWZonePlacementInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: not fadvise flags in log

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: deep copy on partial bufferptrs on setattr

Otherwise we may inadvertantly pin larger buffers in
memory.

Fixes: http://tracker.ceph.com/issues/18093
Signed-off-by: Sage Weil <sage@redhat.com>

buffer: put aligned buffers in buffer_data mempool

Oops!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12105 from cbodley/wip-rgw-curl-upload

rgw: dont set CURLOPT_UPLOAD for GET requests

Merge pull request #12242 from cbodley/wip-rgw-admin-placement

radosgw-admin: more on placement configuration

doc: add docs for rgw compression

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12185 from xiexingguo/xxg-wip-bluestore-1125

os/bluestore: drop member "space" from Onode

Reviewed-by: Sage Weil <sage@redhat.com>

radosgw-admin: 'zone placement modify' doesnt require pool names

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add 'zonegroup placement default' command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

pybind: Add binding for aio_exec

This exposes to the python API for the following C methods:
* rados_aio_exec

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>

radosgw-admin: fix 'placment' typos

Fixes: http://tracker.ceph.com/issues/18078
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12230 from yehudasa/wip-18078

rgw-admin: new commands to control placement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw_admin: commands to manage placement targets

Fixes: http://tracker.ceph.com/issues/18078
added the following commands:

 radosgw-admin zone placement add
 radosgw-admin zone placement modify
 radosgw-admin zone placement rm
 radosgw-admin zone placement list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add commands to manage zonegroup placement fields

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: metadata sync info should be shown at master zone of slave zonegroup

When executing 'radosgw-admin sync status', the metadata sync info should be shown on the srceen at master zone of slave zonegroup.

Using the function store->is_meta_master() instead of 'zonegroup.is_master && zone.id == zonegroup.master_zone'

Fixes: http://tracker.ceph.com/issues/18091
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

Merge pull request #12236 from gongchuang/gc-wip-1130

msg/async/rdma: fixup memory free

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12138 from oritwas/wip-rgw-doc-upgrade-region-pool

rgw: add documentation for upgrading with rgw_region_root_pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12061 from guihecheng/my-fix

cephfs: fix missing ll_get for ll_walk

Reviewed-by: John Spray <john.spray@redhat.com>

test/libcephfs: ensure that we set the mode of the libcephfs root before testing

Patrick noticed that the libcephfs tests failed to run unless they were
run after the ceph-fuse tests had run. The ceph-fuse tests set the root
of the share to mode 01777, but the libcephfs tests don't do that even
though they rely on the directory being world-writable.

Fix this by adding a new main() function for ceph_test_libcephfs, and
calling a function (copied from access.cc) that resets the permissions
of the root before running the tests.

Fixes: http://tracker.ceph.com/issues/18013
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: fix some potential locking problems in InodeRef handling

As Zheng points out, declaring an InodeRef before you take the mutex
means that its destructor gets called after the mutex has already been
released. Handling the refcount however, requires that you hold the
mutex so this could cause refcount leaks if two threads do a
load/decrement/store at the same time. Reverse the order.

Adding a fixes line here for the currently reported bug, but it's
not yet clear whether this will fix it.

Fixes: http://tracker.ceph.com/issues/17982
Signed-off-by: Jeff Layton <jlayton@redhat.com>

msg/async/rdma: fixup memory free

Signed-off-by: gongchuang <gong.chuang@zte.com.cn>

Merge pull request #12235 from tchaikov/wip-osd_stat_t-decode

osd/osd_types: fix the osd_stat_t::decode()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/osd_types: fix the osd_stat_t::decode()

should decode the deprecated num_hb_out as vector<int> instead of uint32_t.

it's a regression introduced by 85f22ec

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: fix relock race

In commit a863ae1c0fab636eabced0979889cbb3be74bf74 we tried
to fix a race but failed because we didn't update the
session pointer.  Add a proper test for this case with
a delay injection and fix the bug.

Fixes: http://tracker.ceph.com/issues/17942
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: need to close_section in lc list op

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

msg/simple/Pipe: handle addr decode error

Fixes: http://tracker.ceph.com/issues/18072
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12194 from tchaikov/wip-unit.cc

test,cmake: turn unit.h into unit.cc to speed up compilation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephfs: fix missing ll_get for ll_walk

Fixs: http://tracker.ceph.com/issues/18086

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

rgw: set duration for lifecycle lease

Fixes: http://tracker.ceph.com/issues/17965
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove a use of g_ceph_context

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: omap_get_all() fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12178 from liupan1111/wip-fix-bug-hb-out

osd: replace hb_out and hb_in with a single hb_peers

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: RGWBucketSyncStatusManager uses existing async_rados

RGWBucketSyncStatusManager was creating its own RGWAsyncRadosProcessor
and failing to clean it up on destruction. A recent change turned
global_init()'s return value into an intrusive_ptr to guarantee that our
CephContext is freed, and this triggered an assertion that all worker
threads were shut down

Fixes: http://tracker.ceph.com/issues/18083
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use set for zonegroup placement target tags

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>