test / librbd: create few empty objects during flatten

Fixes: http://tracker.ceph.com/issues/15028
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test / librbd: Write ones instead of zeroes to test images

Signed-off-by: Douglas Fuller <dfuller@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd: Create few empty objects during copyup

This is based out of Doug's (@fullerdj) work (PR #9329)
as an attempt to avoid creating empty objects when
flattening an image and otherwise whenever unnecessary.

This gives good optimization benefit when a parent image
is sparsely populated. Moreover, this change is required
for correct behavior when checking disk usage of a clone
(which used to report fully allocated image due to all,
including empty objects being created during flatten).

Signed-off-by: Douglas Fuller dfuller@redhat.com
Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd: make has_parent() prone to callers from copyup

This is required when CopyupRequest would need to invoke
pre_object_map_update() as part of upcoming changes to
create fewer child image objects whenever possible.

CopyupRequest constructor accepts image extents as an
rvalue forcing the caller to transfer ownership to it
and leaving the original variable in an unspecified
stated making has_parent() return incorrect state when
invoked from CopyupRequest. Therefore, introduce a
private tracking state that can be used in place of
checking emptiness of parent image extents.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #12991 from dillaman/wip-17447

librbd: avoid possible recursive lock when racing acquire lock

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12982 from dillaman/wip-18524

librbd: remove unnecessary dependencies of ManagedLock

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13010 from liewegas/wip-bluestore-allocator

os/bluestore: fix Allocator::allocate() int truncation

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #12908 from tchaikov/wip-rocksdb

rocksdb: build with ppc64

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12830 from renhwztetecs/renhw-wip-libcephfs-misc

libcephfs: cleanups

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12994 from ukernel/wip-18487

mds: add authority check for delay dirfrag split

Reviewed-by: John Spray <john.spray@redhat.com>

os/bluestore: fix Allocator::allocate() int truncation

An allocation of size 0x80000000 gets truncated to 0.  Larger values can
break things in similar ways.

Introduced in 5ab034345d7320fbc86a2133c0c29ec1aca4b71a.

Fixes: http://tracker.ceph.com/issues/18595
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12947 from smatzek/fix_18538

ceph-disk: use correct user in check_journal_req

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #12865 from ceph/wip-rgw-18476

rgw: fix handling RGWUserInfo::system in RGWHandler_REST_SWIFT.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12968 from liewegas/wip-kraken-notes

doc/release-notes: final kraken notes

Merge pull request #12963 from liewegas/wip-bluestore-extents

os/bluestore: fix reclaim_blocks and clean up Allocator interface

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12550 from stiopaa1/B9935_segfaultInRmDir

client/Client.cc: prevent segfaulting

Reviewed-by: John Spray <john.spray@redhat.com>

mds: add authority check for delay dirfrag split

the dirfrag can be migareted to other mds while waiting in the
timer.

Fixes: http://tracker.ceph.com/issues/18487
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #12988 from SUSE/wip-submitting-patches

doc: clarify SubmittingPatches.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: avoid possible recursive lock when racing acquire lock

Fixes: http://tracker.ceph.com/issues/17447
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12492 from yehudasa/wip-func-oid-tracing

osd, messenger, librados: lttng oid tracing

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: clean up pre-release lock handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: separate testing of exclusive lock from managed lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: removed ManagedLock dependency from ancillary classes

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12972 from linuxbox2/wip-librgw-shut

rgw: librgw shut

doc: clarify SubmittingPatches.rst

This commit clarifies that folks opening bugfix PRs that need backporting to
one or more stable branches should always target master, and that the
backporting itself takes place only after the master PR is merged.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12907 from theanalyst/wip-18064

rgw: sync status compares the current master period

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12357 from dongbula/fix-versioned-bucket-datasync-fail

rgw: fix versioned bucket data sync fail when upload is busy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12945 from jimmyway/wip-add-cmd-snap-set-limit-prompt

rbd: add error prompt when input command 'snap set limit' is incomplete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12886 from rjfd/wip-shared-managed-lock

librbd: support for shared locking in ManagedLock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix handling RGWUserInfo::system in RGWHandler_REST_SWIFT.

Before this patch the flag was wrongly handled in the Swift API
implementation. In rare conditions this might result in setting
req_state::system_request.

This may happen only if both of those conditions are fulfilled:
 * RadosGW is running in a multi-site configuration (at least
   one user with the system flag turned on is present),
 * the "rgw_swift_account_in_url" configurable has been switched
   to true. The value is false by default and our documentation
   doesn't actually mention about the option.

The issue doesn't affect Jewel nor any previous release.

Fixes: http://tracker.ceph.com/issues/18476
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

librgw: remove call to crypto::shutdown

The call is cognate to removed code in radosgw, and triggers an
assert that the crypto module has already been shutdown.  With this
and preceding changes, I see clean shutdown on abnormal termination
of nfs-ganesha during startup.

Fixes: http://tracker.ceph.com/issues/18585
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:  s/static std::map/static pointer/ for ext_mime_map

In some librgw shutdown cases, it appears that the shared
object runtime has partially destructed the static map when
our shutdown code is running.

Fixes: http://tracker.ceph.com/issues/18585
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file:  signal RGWLibFrontend shutdown correctly

The shutdown member of RGWLibProcess should be set.

Fixes: http://tracker.ceph.com/issues/18585
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12927 from rjfd/wip-fix-rpath-suse

cmake: fix rpath on shared libraries and binaries targets

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12750 from zhangsw/feature-rgw-lc-validatei

rgw: add lifecycle validation according to S3.

os/bluestore/BlueFS: dump allocator freelist on failure in reclaim_blocks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: handle failure to reclaim blocks without crashing

We shouldn't fail to reclaim space in general, but if we do, do not treat
it as a fatal error.  Log loudly.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12827 from Liuchang0812/wip-18459

doc: correct S3 lifecycle support explain

unitest_bit_alloc, unittest_alloc: fixes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prealloc/reserve extent vector for common alloc path

No need to worry about the gift/reclaim path--those are very rare.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/Allocator: drop unused and goofy release_extents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop useless count arg to allocate

The vector<> has a size.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: return blocks allocated from allocate()

Instead of having a separate output argument with the number of
blocks allocated, just return it via the return value.  Simplifies
the calling convention.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: manage vector from ExtentList

ExtentList was previous relying the caller to preallocate/size the
vector to be large enough for the worst case allocation of extents,
and keeping it's own manual count of the extent list size.  Instead,
manage that from ExtentList, and remove the preallocation from the
callers.

Fixes: http://tracker.ceph.com/issues/18573
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12681 from xiexingguo/xxg-wip-bluestore-1227

os/bluestore: generate same onode extent-shard keys in a more efficient way

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12696 from xiexingguo/xxg-wip-improve-loops

os/bluestore: miscellaneous fixes to BitAllocator

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #12719 from xiexingguo/xxg-wip-bluestore-1229

os/bluestore/BitmapFreelistManager: readability improvements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12942 from Liuchang0812/cleanup-bluestore

os/bluestore: avoid unnecessary memory copy, use variable reference in BlockDevice::Open

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12946 from ifed01/wip-bluestore-pextent-vector-mempooled

os/bluestore: Add bluestore pextent vector to mempool

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: fix rpath on shared libraries and executables

The problem was due to the attempt of CMake to rewrite the RPATH of
the executables, or shared libraries, that didn't have any RPATH linked
during the building phase.

Currently INSTALL_RPATH is already set for all binaries. This patch
sets an empty INSTALL_RPATH for all binaries that don't depend on
any internal (also being built) target.

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc: correct S3 lifecycle support explain

Fixes: http://tracker.ceph.com/issues/18459
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

doc/release-notes: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: Support for shared locking in ManagedLock

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc/release-notes: final kraken notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12964 from jcsp/wip-18574

qa: update remaining ceph.com to download.ceph.com

Merge pull request #12962 from dillaman/wip-rbd-boost-crash

rbd: remove direct linking to static boost libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: update remaining ceph.com to download.ceph.com

Fixes: http://tracker.ceph.com/issues/18574
Signed-off-by: John Spray <john.spray@redhat.com>

rbd: remove direct linking to static boost libraries

The boost libraries are included in libceph-common. If linked
directly, boost::regex will abort during shutdown since it uses
a global static variable and it will be destructed twice.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12885 from rjfd/wip-18325

librbd: allow to open an image without opening the parent image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12922 from trociny/wip-managed_lock_refactoring

librbd: managed lock refactoring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12953 from dillaman/wip-18542

qa/tasks/qemu: update default image url after ceph.com redesign

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12954 from dillaman/wip-rbd-coverity

rbd: correct coverity warnings

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12467 from bassam/pr-external-rocksdb

cmake: support for external rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: correct coverity warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9028 from tchaikov/wip-bt-dump

common/BackTrace: add operator<<

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/qemu: update default image url after ceph.com redesign

Fixes: http://tracker.ceph.com/issues/18542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12807 from runsisi/wip-fix-pg-ls

mon/PGMonitor: fix description for ceph pg ls

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: modify error code in lifecycle to be compatible with S3.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

mon/PGMonitor: ceph pg ls-by-pool only supports string type of pool arg

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #12928 from tmuthamizhan/wip-ansible-fs-suite

qa/suites/ceph-ansible: removing fs workunit

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

librbd: helper methods to query and break lock

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: managed_lock: make AcquireRequest use GetLockRequest and BreakRequest

Initially this was implemented for exclusive_lock (03533b9,
23f60fe) but was lost when merging managed_lock refactoring.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd, messenger, librados: lttng enhancements

Few critical functions have been instrumented with function tracing
and oid tracing calls to help routing performance analysis.
Oid tracing events are heavily dependent on functional changes and may need
to be revised as and when core data flow logic changes.

Signed-off-by: Anjaneya Chagam <anjaneya.chagam@intel.com>

common: extend lttng tracing infrastructure

New function and oid lttng tracing enhancements that provide function
and oid level latency breakdown. Requires WITH_EVENTTRACE=ON option to enable.

Signed-off-by: Anjaneya Chagam <anjaneya.chagam@intel.com>

Merge pull request #12915 from jcsp/wip-18361

client: populate metadata during mount

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12107 from mslovy/wip-monmsg-minwait

mon: no delay for single message MSG_ALIVE and MSG_PGTEMP

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11955 from batrick/log-2-to-1-write

log: use one write system call per message

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9924 from Yan-waller/yj-wiph-commonentitynamecc

common/config: fix return type of string::find and use string::npos

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12925 from liewegas/wip-crypto-shutdown-fixes

common: misc fixes detected by crypto shutdown assert

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12944 from liewegas/wip-centos-latest

qa/distros: make centos_latest 7.3

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #12841 from cbodley/wip-18407

rgw multisite: fix ref counting of completions

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12605 from cbodley/wip-18300

rgw: RGWMetaSyncShardCR drops stack refs on destruction

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: allow to open an image without opening parent image

Fixes: http://tracker.ceph.com/issues/18325
Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: support for external rocksdb

add support for building with an external rocksdb.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Ceph-disk to use correct user in check_journal_req

The ceph-disk tool calls ceph-osd to check the journal requirements
using OSD id 0. This creates a log file for osd-0 on the system
using the current user/group for file permissions.  When ceph-disk
is run as root this makes the file owned by root which makes
the osd daemon for osd.0 unable to write to its own log file.

This commit changes the journal reqs calls of ceph-osd to pass
the ceph user and group so ceph-osd creates the log file with the
appropriate permissions.

Fixes: http://tracker.ceph.com/issues/18538
Signed-off-by: Samuel Matzek <smatzek@us.ibm.com>

Merge pull request #12798 from wjwithagen/wip-wjw-krbd-optional

rbd: enable rbd on FreeBSD (without KRBD)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: add pextent vector to mempool

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/objectstore: add bufferptr & blouestore_blob_t::extents to "sizeof" UT

Signed-off-by: Igor Fedotvo <ifedotov@mirantis.com>

Merge pull request #12930 from chendave/wip_cephx_refactor

msg: Revert the change from assert(0)-> ceph_abort() where is not applicable

Reviewed-by: Sage Weil <sage@redhat.com>

qa/distros: make centos_latest 7.3

This will fix smithi runs and break vps runs until we get a 7.3 cloud-init
image ready.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: add error prompt when input command 'snap set limit' is incomplete

Signed-off-by: Tang Jin <tang.jin@istuary.com>

os/bluestore: avoid unnecessary memory copy, use variable reference in BlockDevice::Open

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12939 from johnlinp/fix-nspr-include

cmake: search for nspr include files for both suffixes: nspr4 and nspr

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: search for nspr include files for both suffixes: nspr4 and nspr

Fixes: http://tracker.ceph.com/issues/18535
Signed-off-by: John Lin <johnlinp@gmail.com>

Merge pull request #12936 from badone/wip-pg-states-doc-fix

doc: min_size advice is not helpful

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: generate same onode extent-shard keys in a more efficient way

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: min_size advice is not helpful

When we are in the incomplete state adjusting min_size will not help with recovery.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12247 from YankunLi/fix_bf_append_parameter

bufferlist: cleanup semantical wrong for bufferlist::append

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12345 from rook/wip-embedded-add-rgw-mds

cmake: add RGW and MDS to libcephd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/BackTrace: add operator<<

replace BackTrace::print() with the operator<< where the former is used.

Signed-off-by: Kefu Chai <kchai@redhat.com>