rados: use bare omap_get_keys op

This handles the client-side looping on 'more' if the OSD limits
the response size.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: pass nullptr for pmore for omap fetch

Assume OSD limits are hugh enough.

Signed-off-by: Sage Weil <sage@redhat.com>

rados: omap bench: pass null for pmore for omap fetch

Assume OSD limits are high enough for us.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_*: use new omap methods

Make these tests pass nullptr for pmore.  This codifies our
assumption that the OSD limits are high enough.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add omap_get_{keys,vals}2 with pmore output arg

Expose public methods that include a new output argument to indicate
whether there are more keys to fetch or not.

Mark the old interfaces deprecated.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: infer ptruncated on old OSDs via max_entries

If we do not get an explicit 'more' value from the OSD, infer it by
checking whether we got the max requested entries.  On old OSDs, which
don't enforce a limit, this will work.  On new OSDs, we will get the
explicit result.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: add pmore argument to omap_get_{keys,vals}

Note that the MDS callers have new #warnings indicating that they
are not providing the pmore argument and are thus broken.  (They
were already broken.)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: tell client if we truncate results

If we truncate the results of the omap read commands,
provide a flag so that the caller knows there is more
to be read.  We don't need to provide the name of the
next key because the interface is defined as
"start_after" (not "start_with"), allowing them to use
the last key they received.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12865 from ceph/wip-rgw-18476

rgw: fix handling RGWUserInfo::system in RGWHandler_REST_SWIFT.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12968 from liewegas/wip-kraken-notes

doc/release-notes: final kraken notes

Merge pull request #12963 from liewegas/wip-bluestore-extents

os/bluestore: fix reclaim_blocks and clean up Allocator interface

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12550 from stiopaa1/B9935_segfaultInRmDir

client/Client.cc: prevent segfaulting

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12988 from SUSE/wip-submitting-patches

doc: clarify SubmittingPatches.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12492 from yehudasa/wip-func-oid-tracing

osd, messenger, librados: lttng oid tracing

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12972 from linuxbox2/wip-librgw-shut

rgw: librgw shut

doc: clarify SubmittingPatches.rst

This commit clarifies that folks opening bugfix PRs that need backporting to
one or more stable branches should always target master, and that the
backporting itself takes place only after the master PR is merged.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #12907 from theanalyst/wip-18064

rgw: sync status compares the current master period

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12357 from dongbula/fix-versioned-bucket-datasync-fail

rgw: fix versioned bucket data sync fail when upload is busy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12945 from jimmyway/wip-add-cmd-snap-set-limit-prompt

rbd: add error prompt when input command 'snap set limit' is incomplete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12886 from rjfd/wip-shared-managed-lock

librbd: support for shared locking in ManagedLock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix handling RGWUserInfo::system in RGWHandler_REST_SWIFT.

Before this patch the flag was wrongly handled in the Swift API
implementation. In rare conditions this might result in setting
req_state::system_request.

This may happen only if both of those conditions are fulfilled:
 * RadosGW is running in a multi-site configuration (at least
   one user with the system flag turned on is present),
 * the "rgw_swift_account_in_url" configurable has been switched
   to true. The value is false by default and our documentation
   doesn't actually mention about the option.

The issue doesn't affect Jewel nor any previous release.

Fixes: http://tracker.ceph.com/issues/18476
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

librgw: remove call to crypto::shutdown

The call is cognate to removed code in radosgw, and triggers an
assert that the crypto module has already been shutdown.  With this
and preceding changes, I see clean shutdown on abnormal termination
of nfs-ganesha during startup.

Fixes: http://tracker.ceph.com/issues/18585
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw:  s/static std::map/static pointer/ for ext_mime_map

In some librgw shutdown cases, it appears that the shared
object runtime has partially destructed the static map when
our shutdown code is running.

Fixes: http://tracker.ceph.com/issues/18585
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file:  signal RGWLibFrontend shutdown correctly

The shutdown member of RGWLibProcess should be set.

Fixes: http://tracker.ceph.com/issues/18585
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #12927 from rjfd/wip-fix-rpath-suse

cmake: fix rpath on shared libraries and binaries targets

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12750 from zhangsw/feature-rgw-lc-validatei

rgw: add lifecycle validation according to S3.

os/bluestore/BlueFS: dump allocator freelist on failure in reclaim_blocks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: handle failure to reclaim blocks without crashing

We shouldn't fail to reclaim space in general, but if we do, do not treat
it as a fatal error.  Log loudly.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12827 from Liuchang0812/wip-18459

doc: correct S3 lifecycle support explain

unitest_bit_alloc, unittest_alloc: fixes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: prealloc/reserve extent vector for common alloc path

No need to worry about the gift/reclaim path--those are very rare.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/Allocator: drop unused and goofy release_extents

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop useless count arg to allocate

The vector<> has a size.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: return blocks allocated from allocate()

Instead of having a separate output argument with the number of
blocks allocated, just return it via the return value.  Simplifies
the calling convention.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: manage vector from ExtentList

ExtentList was previous relying the caller to preallocate/size the
vector to be large enough for the worst case allocation of extents,
and keeping it's own manual count of the extent list size.  Instead,
manage that from ExtentList, and remove the preallocation from the
callers.

Fixes: http://tracker.ceph.com/issues/18573
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12681 from xiexingguo/xxg-wip-bluestore-1227

os/bluestore: generate same onode extent-shard keys in a more efficient way

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12696 from xiexingguo/xxg-wip-improve-loops

os/bluestore: miscellaneous fixes to BitAllocator

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #12719 from xiexingguo/xxg-wip-bluestore-1229

os/bluestore/BitmapFreelistManager: readability improvements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12942 from Liuchang0812/cleanup-bluestore

os/bluestore: avoid unnecessary memory copy, use variable reference in BlockDevice::Open

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12946 from ifed01/wip-bluestore-pextent-vector-mempooled

os/bluestore: Add bluestore pextent vector to mempool

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: fix rpath on shared libraries and executables

The problem was due to the attempt of CMake to rewrite the RPATH of
the executables, or shared libraries, that didn't have any RPATH linked
during the building phase.

Currently INSTALL_RPATH is already set for all binaries. This patch
sets an empty INSTALL_RPATH for all binaries that don't depend on
any internal (also being built) target.

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc: correct S3 lifecycle support explain

Fixes: http://tracker.ceph.com/issues/18459
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

doc/release-notes: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: Support for shared locking in ManagedLock

Signed-off-by: Ricardo Dias <rdias@suse.com>

doc/release-notes: final kraken notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12964 from jcsp/wip-18574

qa: update remaining ceph.com to download.ceph.com

Merge pull request #12962 from dillaman/wip-rbd-boost-crash

rbd: remove direct linking to static boost libraries

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: update remaining ceph.com to download.ceph.com

Fixes: http://tracker.ceph.com/issues/18574
Signed-off-by: John Spray <john.spray@redhat.com>

rbd: remove direct linking to static boost libraries

The boost libraries are included in libceph-common. If linked
directly, boost::regex will abort during shutdown since it uses
a global static variable and it will be destructed twice.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12885 from rjfd/wip-18325

librbd: allow to open an image without opening the parent image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12922 from trociny/wip-managed_lock_refactoring

librbd: managed lock refactoring

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12953 from dillaman/wip-18542

qa/tasks/qemu: update default image url after ceph.com redesign

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12954 from dillaman/wip-rbd-coverity

rbd: correct coverity warnings

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #12467 from bassam/pr-external-rocksdb

cmake: support for external rocksdb

Reviewed-by: Kefu Chai <kchai@redhat.com>

rbd: correct coverity warnings

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9028 from tchaikov/wip-bt-dump

common/BackTrace: add operator<<

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

qa/tasks/qemu: update default image url after ceph.com redesign

Fixes: http://tracker.ceph.com/issues/18542
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12807 from runsisi/wip-fix-pg-ls

mon/PGMonitor: fix description for ceph pg ls

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: modify error code in lifecycle to be compatible with S3.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

mon/PGMonitor: ceph pg ls-by-pool only supports string type of pool arg

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #12928 from tmuthamizhan/wip-ansible-fs-suite

qa/suites/ceph-ansible: removing fs workunit

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

librbd: helper methods to query and break lock

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: managed_lock: make AcquireRequest use GetLockRequest and BreakRequest

Initially this was implemented for exclusive_lock (03533b9,
23f60fe) but was lost when merging managed_lock refactoring.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

osd, messenger, librados: lttng enhancements

Few critical functions have been instrumented with function tracing
and oid tracing calls to help routing performance analysis.
Oid tracing events are heavily dependent on functional changes and may need
to be revised as and when core data flow logic changes.

Signed-off-by: Anjaneya Chagam <anjaneya.chagam@intel.com>

common: extend lttng tracing infrastructure

New function and oid lttng tracing enhancements that provide function
and oid level latency breakdown. Requires WITH_EVENTTRACE=ON option to enable.

Signed-off-by: Anjaneya Chagam <anjaneya.chagam@intel.com>

Merge pull request #12915 from jcsp/wip-18361

client: populate metadata during mount

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12107 from mslovy/wip-monmsg-minwait

mon: no delay for single message MSG_ALIVE and MSG_PGTEMP

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11955 from batrick/log-2-to-1-write

log: use one write system call per message

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9924 from Yan-waller/yj-wiph-commonentitynamecc

common/config: fix return type of string::find and use string::npos

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12925 from liewegas/wip-crypto-shutdown-fixes

common: misc fixes detected by crypto shutdown assert

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12944 from liewegas/wip-centos-latest

qa/distros: make centos_latest 7.3

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #12841 from cbodley/wip-18407

rgw multisite: fix ref counting of completions

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12605 from cbodley/wip-18300

rgw: RGWMetaSyncShardCR drops stack refs on destruction

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

librbd: allow to open an image without opening parent image

Fixes: http://tracker.ceph.com/issues/18325
Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: support for external rocksdb

add support for building with an external rocksdb.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #12798 from wjwithagen/wip-wjw-krbd-optional

rbd: enable rbd on FreeBSD (without KRBD)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: add pextent vector to mempool

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/objectstore: add bufferptr & blouestore_blob_t::extents to "sizeof" UT

Signed-off-by: Igor Fedotvo <ifedotov@mirantis.com>

Merge pull request #12930 from chendave/wip_cephx_refactor

msg: Revert the change from assert(0)-> ceph_abort() where is not applicable

Reviewed-by: Sage Weil <sage@redhat.com>

qa/distros: make centos_latest 7.3

This will fix smithi runs and break vps runs until we get a 7.3 cloud-init
image ready.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: add error prompt when input command 'snap set limit' is incomplete

Signed-off-by: Tang Jin <tang.jin@istuary.com>

os/bluestore: avoid unnecessary memory copy, use variable reference in BlockDevice::Open

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #12939 from johnlinp/fix-nspr-include

cmake: search for nspr include files for both suffixes: nspr4 and nspr

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: search for nspr include files for both suffixes: nspr4 and nspr

Fixes: http://tracker.ceph.com/issues/18535
Signed-off-by: John Lin <johnlinp@gmail.com>

Merge pull request #12936 from badone/wip-pg-states-doc-fix

doc: min_size advice is not helpful

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: generate same onode extent-shard keys in a more efficient way

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: min_size advice is not helpful

When we are in the incomplete state adjusting min_size will not help with recovery.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #12247 from YankunLi/fix_bf_append_parameter

bufferlist: cleanup semantical wrong for bufferlist::append

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12345 from rook/wip-embedded-add-rgw-mds

cmake: add RGW and MDS to libcephd

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common/BackTrace: add operator<<

replace BackTrace::print() with the operator<< where the former is used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12784 from LiumxNL/fix-wrongly-delete-routed-op

mon: fix wrongly delete routed pgstats op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12651 from liewegas/wip-mempool-string

os/bluestore: put strings in mempool

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12661 from xiexingguo/xxg-wip-cleanup-bitalloc

os/bluestore: cleanup BitAllocator

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #12700 from ifed01/wip-bluestore-collection-everywhere

os/bluestore: various onode changes to reduce its in-memory footprint.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12876 from xiexingguo/xxg-wip-be-for-cache

os/bluestore: use Best-Effort policy when evicting onode from cache

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12877 from xiexingguo/xxg-wip-compress-opt

os/bluestore: conditionally load crr option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12900 from XinzeChi/wip-bluestore-valid

os/bluestore: nullptr in OmapIteratorImpl::valid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12844 from athanatos/wip-17831

osd/ReplicatedBackend: take read locks for clone sources during recovery

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert the change from assert(0)-> ceph_abort() where is not applicable

Signed-off-by: Dave Chen <wei.d.chen@intel.com>

removing fs workunit

removing fs workunit for now until tracker issue 18528 is resolved.

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>