debian/rules: dump env during build

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: dump env during build

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12256 from liewegas/wip-18093

osd: fix memory leak from EC write workload

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12238 from gaurav36/bug-rbd-feature-enable-error

librbd: enabling/disabling rbd feature should report missing dependency

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12195 from trociny/wip-qa-rbd-wait_for_image_present

qa/workunits/rbd: use image id when probing for image presence

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11942 from wjwithagen/wip-wjw-freebsd-boost

do_freebsd.sh: Build with SYSTEM Boost on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Cmake/FreeBSD: use the SYSTEM_BOOST

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #12209 from tchaikov/wip-enable-unittest_dns_resolve

test: enable unittest_dns_resolve

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: enable unittest_dns_resolve

guess this test was missed when we were moving to cmake. let's re-enable
it!

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12263 from Adirl/fix_max_buffers

msg/async/rdma: don't use more buffers than what device capabilities …

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12140 from iain-buclaw-sociomantic:pybind-aioexec

python-rados: implement new aio_execute

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind: Fix comment for exec in Ioctx

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>

test/pybind: Add tests for aio_execute

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>

msg/async/rdma: don't use more buffers than what device capabilities allow

Change-Id: I6df7dd1a4d871b4dca55e784969ff459bebc6a86
Signed-off-by: Avner BenHanoch <avnerb@mellanox.com>

Merge pull request #12261 from liewegas/wip-store-rename-test

ceph_test_objectstore: fix Rename test

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12231 from yehudasa/wip-17965

rgw: set duration for lifecycle lease

Merge pull request #12221 from liewegas/wip-18072

msg/simple/Pipe: handle addr decode error

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12218 from dillaman/wip-18068

librbd: diffs to clone's first snapshot should include parent diffs

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph_test_objectstore: fix Rename test

We were doing a no-op setattr here, even though the intenral
ENOENT used to be ignored, because it previously triggered a
bug in bluestore.  We're no longer ignoring that internal
error, though, as of 6ecb44f0c6216f3fb29b384a2d10c7d33bf1d19c,
so remove this part of the test.

The original bluestore bug we used to trigger was fixed by
0210f1fde6f8e17affdd0c49055a71a01f109447.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10086 from Yan-waller/yj-wip-rgwrests3ccstringfind

rgw: conform to the standard usage of string::find

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12229 from cbodley/wip-rgw-bucket-sync-status-assert

rgw: RGWBucketSyncStatusManager uses existing async_rados

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #12254 from Adirl/disconnect_log

msg/async/rdma: fix disconnect log line

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9846 from stiopaa1/msg_asyncmess_removeUnneededInclude

msg/AsyncMessenger: remove unneeded include

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12246 from tchaikov/wip-17809

buffer: add copy(unsigned, ptr) back

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: enabling/disabling rbd feature should report missing dependency

Currently while enabling or disabling rbd feature command does not
give missing dependency for eg: attempting to enable the journaling
feature on an image that doesn't have the exclusive-lock feature
enabled should give missing dependency error message.

Fixes: http://tracker.ceph.com/issues/16985
Reported-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

Merge pull request #12206 from jdurgin/wip-librados-setxattr-overload

librados: remove new setxattr overload to avoid breaking the C++ ABI

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12234 from liewegas/wip-17942-again

osdc/Objecter: fix relock race

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12228 from ceph/wip-jlayton-umount

libcephfs and test suite fixes

osd/osd_types: avoid pinning memory with pg_log_entry_t::snaps

The pg_log_entry_t may stay in memory for a long time;
avoid pinning a larger buffer in memory.

See: http://tracker.ceph.com/issues/18093
Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: prevent ObjectModDesc::bl from pinning memory

We may decode this from a large buffer (e.g., an
ECSubWrite payload) and the ObjectModDesc is part of
the pg_log_entry_t, which may stay in memory for a long
time.  Avoid pinning memory.

Fixes: http://tracker.ceph.com/issues/18093
Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/rdma: fix disconnect log line

issue: none

Change-Id: I29e2a6ae78dda3cf4a1817b9ab41bea0f636bda5
Signed-off-by: Adir Lev <adirl@mellanox.com>

Merge pull request #12222 from dillaman/wip-18070

rbd-nbd: restart parent process logger after forking

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

buffer: add copy(unsigned, ptr) back

to be ABI-compatible with prior versions.

copy(unsigned, ptr) was renamed to copy_deep() in 6d7f748

Fixes: http://tracker.ceph.com/issues/17809
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12244 from ceph/wip-rgw-compression-doc

doc: add docs for raw compression

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12113 from ceph/wip-rgw-zone-compression

rgw: move compression config into zone placement

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12232 from liewegas/wip-rgw-lclist

rgw: need to close_section in lc list op

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove config option rgw_compression_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: ops use zone placement's compression type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add --compression for zone placement add/modify

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add compression_type field to RGWZonePlacementInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: not fadvise flags in log

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: deep copy on partial bufferptrs on setattr

Otherwise we may inadvertantly pin larger buffers in
memory.

Fixes: http://tracker.ceph.com/issues/18093
Signed-off-by: Sage Weil <sage@redhat.com>

buffer: put aligned buffers in buffer_data mempool

Oops!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12105 from cbodley/wip-rgw-curl-upload

rgw: dont set CURLOPT_UPLOAD for GET requests

Merge pull request #12242 from cbodley/wip-rgw-admin-placement

radosgw-admin: more on placement configuration

doc: add docs for rgw compression

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12185 from xiexingguo/xxg-wip-bluestore-1125

os/bluestore: drop member "space" from Onode

Reviewed-by: Sage Weil <sage@redhat.com>

radosgw-admin: 'zone placement modify' doesnt require pool names

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add 'zonegroup placement default' command

Signed-off-by: Casey Bodley <cbodley@redhat.com>

pybind: Add binding for aio_exec

This exposes to the python API for the following C methods:
* rados_aio_exec

Signed-off-by: Iain Buclaw <iain.buclaw@sociomantic.com>

radosgw-admin: fix 'placment' typos

Fixes: http://tracker.ceph.com/issues/18078
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12230 from yehudasa/wip-18078

rgw-admin: new commands to control placement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw_admin: commands to manage placement targets

Fixes: http://tracker.ceph.com/issues/18078
added the following commands:

 radosgw-admin zone placement add
 radosgw-admin zone placement modify
 radosgw-admin zone placement rm
 radosgw-admin zone placement list

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add commands to manage zonegroup placement fields

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12236 from gongchuang/gc-wip-1130

msg/async/rdma: fixup memory free

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12138 from oritwas/wip-rgw-doc-upgrade-region-pool

rgw: add documentation for upgrading with rgw_region_root_pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #12061 from guihecheng/my-fix

cephfs: fix missing ll_get for ll_walk

Reviewed-by: John Spray <john.spray@redhat.com>

test/libcephfs: ensure that we set the mode of the libcephfs root before testing

Patrick noticed that the libcephfs tests failed to run unless they were
run after the ceph-fuse tests had run. The ceph-fuse tests set the root
of the share to mode 01777, but the libcephfs tests don't do that even
though they rely on the directory being world-writable.

Fix this by adding a new main() function for ceph_test_libcephfs, and
calling a function (copied from access.cc) that resets the permissions
of the root before running the tests.

Fixes: http://tracker.ceph.com/issues/18013
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: fix some potential locking problems in InodeRef handling

As Zheng points out, declaring an InodeRef before you take the mutex
means that its destructor gets called after the mutex has already been
released. Handling the refcount however, requires that you hold the
mutex so this could cause refcount leaks if two threads do a
load/decrement/store at the same time. Reverse the order.

Adding a fixes line here for the currently reported bug, but it's
not yet clear whether this will fix it.

Fixes: http://tracker.ceph.com/issues/17982
Signed-off-by: Jeff Layton <jlayton@redhat.com>

msg/async/rdma: fixup memory free

Signed-off-by: gongchuang <gong.chuang@zte.com.cn>

Merge pull request #12235 from tchaikov/wip-osd_stat_t-decode

osd/osd_types: fix the osd_stat_t::decode()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd/osd_types: fix the osd_stat_t::decode()

should decode the deprecated num_hb_out as vector<int> instead of uint32_t.

it's a regression introduced by 85f22ec

Signed-off-by: Kefu Chai <kchai@redhat.com>

osdc/Objecter: fix relock race

In commit a863ae1c0fab636eabced0979889cbb3be74bf74 we tried
to fix a race but failed because we didn't update the
session pointer.  Add a proper test for this case with
a delay injection and fix the bug.

Fixes: http://tracker.ceph.com/issues/17942
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: need to close_section in lc list op

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

msg/simple/Pipe: handle addr decode error

Fixes: http://tracker.ceph.com/issues/18072
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12194 from tchaikov/wip-unit.cc

test,cmake: turn unit.h into unit.cc to speed up compilation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephfs: fix missing ll_get for ll_walk

Fixs: http://tracker.ceph.com/issues/18086

Signed-off-by: Gui Hecheng <guihecheng@cmss.chinamobile.com>

rgw: set duration for lifecycle lease

Fixes: http://tracker.ceph.com/issues/17965
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove a use of g_ceph_context

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12178 from liupan1111/wip-fix-bug-hb-out

osd: replace hb_out and hb_in with a single hb_peers

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

rgw: RGWBucketSyncStatusManager uses existing async_rados

RGWBucketSyncStatusManager was creating its own RGWAsyncRadosProcessor
and failing to clean it up on destruction. A recent change turned
global_init()'s return value into an intrusive_ptr to guarantee that our
CephContext is freed, and this triggered an assertion that all worker
threads were shut down

Fixes: http://tracker.ceph.com/issues/18083
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use set for zonegroup placement target tags

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only set CURLOPT_UPLOAD for PUT/POST requests

when set on GET requests, curl sends a 'Transfer-encoding: chunked'
header, but doesn't do the actual encoding to terminate the message

Fixes: http://tracker.ceph.com/issues/17822
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12220 from liewegas/wip-coverity

librbd/ImageCtx: no need for virtual dtor

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12219 from zealoussnow/wip-1130

osdc: pass a string reference type to "osdmap->lookup_pg_pool_name"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12166 from liewegas/wip-osdmap-encoding

osd: condition OSDMap encoding on features

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12202 from liewegas/wip-bluestore-coverity

os/bluestore: misc coverity fixes/cleanups

Merge pull request #12047 from rzarzynski/wip-rgw-17931

rgw: add support for the prefix parameter in account listing of Swift API

Merge pull request #11671 from linuxbox2/rgw_file-17663

rgw_file:  apply missed base64 try-catch

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11751 from yehudasa/wip-17788

rgw: extract host name from host:port string

Merge pull request #12217 from dzafman/wip-tests-off

Turn off tests again due to Jenkins failures

Reviewed-by: Samuel Just <sjust@redhat.com>

rbd-nbd: restart parent process logger after forking

Fixes: http://tracker.ceph.com/issues/18070
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12204 from liewegas/wip-bluestore-17824

os/bluestore/KernelDevice: fix race in aio_thread vs aio_wait

Merge pull request #12130 from liupan1111/wip-osd-hearteat

osd: remove redudant call of heartbeat_check

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12169 from liewegas/wip-rados-bench-hints

rados: add hints to rados bench

Reviewed-by: Kefu Chai <kchai@redhat.com>

osdc: pass a string reference type to "osdmap->lookup_pg_pool_name"

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

librbd: diffs to clone's first snapshot should include parent diffs

If the clone has a backing object created after the snapshot that
overwrites an extent of the parent, the parent diffs within that
extent are not included in the result.

Fixes: http://tracker.ceph.com/issues/18068
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: Turn off test-erasure-eio.sh due to Jenkins failures

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Turn off osd-scrub-repair.sh due to Jenkins failures

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11764 from bassamtabbara/wip-embedded

cmake: Support for embedding Ceph Daemons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12097 from ceph/wip-17837

src/mds: fix MDSMap upgrade decoding

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12090 from wjwithagen/wip-wjw-do_freebsd.sh

FreeBSD: do_freebsd.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: replace hb_out and hb_in with a single hb_peers

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #12200 from ddiss/tracker18049_ceph_disk_trigger_flock_timeout

systemd/ceph-disk: reduce ceph-disk flock contention

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12180 from tchaikov/wip-silence-get_timeout_delays

tests: disable the echo when running get_timeout_delays()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa/workunits/rbd: use image id when probing for image presence

This fixes a race in resync tests leading to false negative results.

Fixes: http://tracker.ceph.com/issues/18048
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

embedded: Add RADOS classes to embedded cephd library

RADOS classes can now be statically compiled and added to
the embedded cephd library.

The RADOS ClassHandler now has an option to skip calling dlclose
just like PluginRegistry.

All RADOS classes where changed to use a CLS_INIT macro that
will either use __cls_init or classname_cls_init. this enables
the static compiling of all RADOS classes in a single library. Also
global method definitions where moved to inside cls_init.

Also added a few aconfig defines including WITH_EMBEDDED, WITH_CEPHFS,
WITH_RBD, and WITH_KVS. Note that WITH_RBD was not defined before
and the ceph-dencoder was broken when it was turned on.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: Add librbd and librados to embedded ceph

This commit add librados, librbd to the embedded ceph static library. This is needed to support daemons built with libcephd that want to run mon commands, create rbd volumes, snapshots etc.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: Support for running MON and OSD from libcephd

Added supported for running MON and OSD daemon code almost
untouched directly from libcephd.

Also added two API functions for generating an FSID and secret
key that can be used to bootstrap a new cluster.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: add compression and EC plugins to libcephd

Compression and erasure coding plugins are now statically compiled
into libcephd. A new method is added to load them into the
respective registry.

The static libraries are only built when WITH_EMBEDDED is enabled
and existing plugins are unaffected.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>