os/bluestore: exercise handle when trying to get through throttlers

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: confirm buffer is empty before moving it into AOUT queue

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: avoid reset cache_private if we haven't to

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix sanity check of object key length

We now encode shard-id into exactly one byte.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11109 from dmick/master

unittest_chain_xattr: account for existing xattrs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9569 from stiopaa1/osdc_objectCacher_const

osdc/ObjectCacher.h: add const to member functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9701 from stiopaa1/osd_removeUnneededReturn

OSD.cc: remove unneeded return

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9932 from stiopaa1/test_simple_dispatcher_removeUnusedVariable

test/simple_dispatcher.cc: remove unused variable

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10071 from stiopaa1/compatset_removeUnneededInline_mc

CompatSet.h: remove unneeded inline

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10503 from stiopaa1/msg_queuestrategy_addOverrideNew

msg/QueueStrategy: add override to virtual functs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10575 from stiopaa1/mon_osdmonitor_removeUnneededCasts

mon/OSDMonitor.cc: remove unneeded casts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10599 from stiopaa1/mon_monclient_deleteCopyConstrAndAssing

mon/MonClient.h: delete copy constr and assing op

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10618 from stiopaa1/mon_configkeyservice_passStringByConstRef

mon/ConfigKeyService: pass strings by const ref

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10639 from stiopaa1/common_trackedop_moveShardedTrackingData

common/TrackedOp: move ShardedTrackingData to .cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10721 from stiopaa1/mon_logmonitor_moveCLogStruct

mon/LogMonitor: move C_Log struct to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10549 from stiopaa1/mon_healthmonitor_addOverride

mon/HealthMonitor: add override to virtual functs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10529 from stiopaa1/mon_paxosservice_moveClassesTocc

mon/PaxosService: move classes to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10422 from stiopaa1/msg_accepter_addOverride

msg/Accepter.h: add override to virtual function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10416 from stiopaa1/mon_elector_moveCElectionExpire

mon/Elector:move C_ElectionExpire class to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10383 from stiopaa1/mon_monclient_moveCTickClass

mon/MonClient: move C_Tick struct to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10441 from stiopaa1/msg_accepter_moveIncludeToCC

msg/Accepter: move include to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10513 from stiopaa1/mon_monitor_moveCScrubCScrubTimeout

mon/Monitor: move C_Scrub, C_ScrubTimeout to .cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10482 from stiopaa1/msg_faststrategy_addOverride

msg/FastStrategy.h: add override to virtual funct

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10605 from stiopaa1/mon_monclient_passStringsByRef

mon/MonClient.h: pass strings by const reference

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10734 from stiopaa1/osdc_objecter_moveRequestStatHook

osdc/Objecter: move RequestStateHook class to .cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10833 from stiopaa1/os_memstore_moveBufferlistObject

os/MemStore: move BufferlistObject to .cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11099 from stiopaa1/osd_osd_finalInitInitializeVar

osd/OSD.cc: initialise variable at definition

Merge pull request #11114 from stiopaa1/osd_osd_addConsts

osd/OSD.h: add const to member functions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11117 from xiexingguo/xxg-wip-mon-0918

tools/rebuild_mondb: kill comipling warning and other fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rebuild_mondb: kill compiling warning

As follow:

[ 72%] Building CXX object src/tools/CMakeFiles/ceph-objectstore-tool.dir/RadosDump.cc.o
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc: In function ‘int update_mon_db(ObjectStore&, OSDSuperblock&, const string&, const string&)’:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc:289:22: warning: ‘crc’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         if (have_crc && osdmap.get_crc() != crc) {
                      ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc:238:14: note: ‘crc’ was declared here
     uint32_t crc;

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/ceph_objectstore_tool: set error code if path of mon-db does not exist

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/rebuild_mondb: return error if ondisk version of pg_info is incompatible

In this case "r" will be >= 0 and caller will be able to catch
this kind of error.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

tools/rebuild_mondb: avoid unnecessary result code cast

In general we return negative codes for error cases, so there is
no need perform the cast here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8558 from branch-predictor/bp-mark-down-on-perm-rst

msg: mark daemons down on RST + ECONNREFUSED

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11050 from xiexingguo/xxg-wip-os-0912

os/ObjectStore: fix return code of collection_empty() method

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11112 from dillaman/wip-17289

librbd: corrected use-after-free in ImageWatcher

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

osd/OSD.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10933 from tchaikov/wip-rebuild-monstore

tools: rebuild monstore

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

librbd: corrected use-after-free in ImageWatcher

Fixes: http://tracker.ceph.com/issues/17289
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

unittest_chain_xattr: account for existing xattrs

On CentOS with selinux enabled, files always have an
xattr 'security.selinux'.  Account for that when adding/
removing xattrs.

Fixes: http://tracker.ceph.com/issues/16025
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #11107 from stiopaa1/msg_stack_deletecopyAssign

msg/Stack.h: delete copy constr and assign op

msg/Stack.h: delete copy constr and assign op

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11106 from liewegas/wip-bluestore-clone-range

os/bluestore: make clone_range copy-on-write

Merge pull request #11101 from andihit/master

doc: fix description for rsize and rasize

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #10267 from cbodley/wip-fio-objectstore

fio engine for objectstore

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore: do not dup Blob::id in dup()

This field is local to a particular onode!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add cow clone_range via _do_clone_range

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: factor _do_clone_range out of _clone

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: CLONE implicitly creates the target

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: EINVAL bad clone_range args

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10841 from fiskn/master

udev: add krbd readahead placeholder

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11075 from badone/wip-run-cmake-check-ctest

run-cmake-check.sh: Actually run the tests

udev: add krbd readahead placeholder

Signed-off-by: Nick Fisk <nick@fisk.me.uk>

Merge pull request #10687 from rzarzynski/wip-rgw-is_admin-fix

rgw: fix is_admin handling in RGWLDAPAuthEngine and introduce acct_privilege_t
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11102 from ceph/wip-17279

rgw: revert a commit that broke s3 signature validation
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

run-cmake-check.sh: Actually run the tests

Actually run the tests from run-cmake-check.sh

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11098 from ktdreyer/wip-17280-help-orphan-stale-secs

radosgw-admin: add "--orphan-stale-secs" to --help

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: fix description for rsize and rasize

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@gmail.com>

Revert "rgw: Fix Host->bucket fallback logic inversion"

This reverts commit 0e0a4bc3dcb61c2e8dfe65daefe207c62b36306c.

this commit, while logically correct, causes us to add a / to the end of
the request_uri. this breaks signature verification, because the initial
signature was generated without the /

Fixes: http://tracker.ceph.com/issues/17279
Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add "--orphan-stale-secs" to --help

The radosgw-admin --help did not include the description of the
`--orphan-stale-secs` option of the `orphans find` command. The option
sets the number of seconds to wait before declaring an object to be an
orphan.

Fixes: http://tracker.ceph.com/issues/17280
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

osd/OSD.cc: initialise variable at definition

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11097 from ktdreyer/wip-17280-doc-orphan-stale-secs

doc: add "--orphan-stale-secs" to radosgw-admin(8)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10415 from jcsp/wip-16255

ceph-create-keys: fix existing-but-different case

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: add "--orphan-stale-secs" to radosgw-admin(8)

The radosgw-admin(8) manual page did not include the description of the
`--orphan-stale-secs` option of the `orphans find` command. The option sets
the number of seconds to wait before declaring an object to be an
orphan.

Fixes: http://tracker.ceph.com/issues/17280
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11095 from liewegas/wip-bluefs

os/bluestore/BlueFS: prevent concurrent async compaction

os/bluestore/BlueFS: add assert for sanity/clarity

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11096 from ifed01/wip-bluestore-compression-test-fix

test/store_test: fix statfs results check to consider SSD min_alloc_size

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: prevent concurrent async compaction

Tried to fix this in dbe23c94c0074358380a40d47a417f7999920696
but got the condition wrong (log_flushing is for
normal fsync).

Signed-off-by: Sage Weil <sage@redhat.com>

test/store_test: fix statfs results check to consider SSD min_alloc_size

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11081 from ifed01/wip-plugin-registry-fix

common/PluginRegistry: improve error output for shared library load fa…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11092 from badone/wip-dry-run-run-make-check

run-make-check.sh: Make DRY_RUN actually do a dry run

Merge pull request #11091 from badone/wip-submodule-init-run-cmake-check

run-cmake-check.sh: Init submodules

Merge pull request #11087 from liewegas/wip-bluefs-allocator

os/bluestore/BlueFS: use StupidAllocator; fix async compaction bug

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #11088 from tchaikov/do-freebsd-without-autotools

do_freebsd*.sh: rename do_freebsd-cmake.sh to do_freebsd.sh

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

run-make-check.sh: Make DRY_RUN actually do a dry run

The "cd build" command will cause an error during a dry run if the directory
does not exist.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: add rados/operations/disaster-recovery.rst

document the process to recover from leveldb corruption.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph_monstore_tool: add "rebuild" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph-objectstore-tool: add "update-mon-db" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/AuthMonitor: make AuthMonitor::IncType public

so ceph-objectstore-tool is able to use it when rebuilding monitor
db.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

run-cmake-check.sh: Init submodules

If submodules are not initialised run-cmake-check.sh can fail.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

do_freebsd*.sh: rename do_freebsd-cmake.sh to do_freebsd.sh

as autotools is dropped. and remove trailing spaces.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11072 from dmick/wip-17268

ceph-disk tests: Let missing python interpreters be non-fatal

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #11076 from badone/wip-gitignore-pyc

.gitignore: Add .pyc files globally

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11042 from somnathr/wip-verbose-log

kv/rocksdb: dump transactions on error

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: add safety checks on ino 0|1 op_file_update

We only want to log updates on ino 1 in very specific
cases, and never on ino 0.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not dirty compacting log ino 0

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: use hex for debug

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: config option for allocator (default stupid)

StupidAllocator is probably a more appropriate choice for bluefs.
Either way, it needn't be tied to what BlueStore is using.

Signed-off-by: Sage Weil <sage@redhat.com>

Logging proper message during read error from disk

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Logging rocksdb transaction in ceph log

The rocksdb transaction is now been logged into ceph log based
on debug level. If transaction is failed, the error code,
error string and the entire transaction is dumped in the log.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #11085 from stiopaa1/osd_osd_moveUnderPrivate

osd/OSD.h: make some members private

osd/OSD.h: make some members private

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11067 from stiopaa1/common_timer_replaceLongTypeswithauto

common/Timer.cc: replace long types with auto

Merge pull request #11068 from liewegas/wip-bluestore-leaks

os/bluestore: fix a few leaks

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: clear SharedBlob cache in dtor

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make Cache::lock a recursive_mutex

We need to remove items from the cache LRUs in their
destructors, and it is very awkward to ensure that all
releasing paths either hold the cache lock or have
already cleared the cache items out.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix SharedBlob leaks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix Extent leak from ~ExtentMap

Signed-off-by: Sage Weil <sage@redhat.com>

common/PluginRegistry: improve error output forshared library load failure.
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11080 from ifed01/wip-compression-store-improvement

test/store_test: extend Bluestore compression test to verify compress…