doc: manual instructions to set up mds daemon

Signed-off-by: Peter Maloney <peter.maloney@brockmann-consult.de>

Merge pull request #8558 from branch-predictor/bp-mark-down-on-perm-rst

msg: mark daemons down on RST + ECONNREFUSED

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11050 from xiexingguo/xxg-wip-os-0912

os/ObjectStore: fix return code of collection_empty() method

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11112 from dillaman/wip-17289

librbd: corrected use-after-free in ImageWatcher

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10933 from tchaikov/wip-rebuild-monstore

tools: rebuild monstore

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

librbd: corrected use-after-free in ImageWatcher

Fixes: http://tracker.ceph.com/issues/17289
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11107 from stiopaa1/msg_stack_deletecopyAssign

msg/Stack.h: delete copy constr and assign op

msg/Stack.h: delete copy constr and assign op

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11106 from liewegas/wip-bluestore-clone-range

os/bluestore: make clone_range copy-on-write

Merge pull request #11101 from andihit/master

doc: fix description for rsize and rasize

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #10267 from cbodley/wip-fio-objectstore

fio engine for objectstore

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore: do not dup Blob::id in dup()

This field is local to a particular onode!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add cow clone_range via _do_clone_range

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: factor _do_clone_range out of _clone

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: CLONE implicitly creates the target

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: EINVAL bad clone_range args

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10841 from fiskn/master

udev: add krbd readahead placeholder

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #11075 from badone/wip-run-cmake-check-ctest

run-cmake-check.sh: Actually run the tests

udev: add krbd readahead placeholder

Signed-off-by: Nick Fisk <nick@fisk.me.uk>

Merge pull request #10687 from rzarzynski/wip-rgw-is_admin-fix

rgw: fix is_admin handling in RGWLDAPAuthEngine and introduce acct_privilege_t
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11102 from ceph/wip-17279

rgw: revert a commit that broke s3 signature validation
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

run-cmake-check.sh: Actually run the tests

Actually run the tests from run-cmake-check.sh

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11098 from ktdreyer/wip-17280-help-orphan-stale-secs

radosgw-admin: add "--orphan-stale-secs" to --help

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: fix description for rsize and rasize

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@gmail.com>

Revert "rgw: Fix Host->bucket fallback logic inversion"

This reverts commit 0e0a4bc3dcb61c2e8dfe65daefe207c62b36306c.

this commit, while logically correct, causes us to add a / to the end of
the request_uri. this breaks signature verification, because the initial
signature was generated without the /

Fixes: http://tracker.ceph.com/issues/17279
Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: add "--orphan-stale-secs" to --help

The radosgw-admin --help did not include the description of the
`--orphan-stale-secs` option of the `orphans find` command. The option
sets the number of seconds to wait before declaring an object to be an
orphan.

Fixes: http://tracker.ceph.com/issues/17280
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11097 from ktdreyer/wip-17280-doc-orphan-stale-secs

doc: add "--orphan-stale-secs" to radosgw-admin(8)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10415 from jcsp/wip-16255

ceph-create-keys: fix existing-but-different case

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: add "--orphan-stale-secs" to radosgw-admin(8)

The radosgw-admin(8) manual page did not include the description of the
`--orphan-stale-secs` option of the `orphans find` command. The option sets
the number of seconds to wait before declaring an object to be an
orphan.

Fixes: http://tracker.ceph.com/issues/17280
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11095 from liewegas/wip-bluefs

os/bluestore/BlueFS: prevent concurrent async compaction

os/bluestore/BlueFS: add assert for sanity/clarity

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11096 from ifed01/wip-bluestore-compression-test-fix

test/store_test: fix statfs results check to consider SSD min_alloc_size

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: prevent concurrent async compaction

Tried to fix this in dbe23c94c0074358380a40d47a417f7999920696
but got the condition wrong (log_flushing is for
normal fsync).

Signed-off-by: Sage Weil <sage@redhat.com>

test/store_test: fix statfs results check to consider SSD min_alloc_size

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11081 from ifed01/wip-plugin-registry-fix

common/PluginRegistry: improve error output for shared library load fa…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11092 from badone/wip-dry-run-run-make-check

run-make-check.sh: Make DRY_RUN actually do a dry run

Merge pull request #11091 from badone/wip-submodule-init-run-cmake-check

run-cmake-check.sh: Init submodules

Merge pull request #11087 from liewegas/wip-bluefs-allocator

os/bluestore/BlueFS: use StupidAllocator; fix async compaction bug

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #11088 from tchaikov/do-freebsd-without-autotools

do_freebsd*.sh: rename do_freebsd-cmake.sh to do_freebsd.sh

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

run-make-check.sh: Make DRY_RUN actually do a dry run

The "cd build" command will cause an error during a dry run if the directory
does not exist.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: add rados/operations/disaster-recovery.rst

document the process to recover from leveldb corruption.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph_monstore_tool: add "rebuild" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph-objectstore-tool: add "update-mon-db" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/AuthMonitor: make AuthMonitor::IncType public

so ceph-objectstore-tool is able to use it when rebuilding monitor
db.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>

run-cmake-check.sh: Init submodules

If submodules are not initialised run-cmake-check.sh can fail.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

do_freebsd*.sh: rename do_freebsd-cmake.sh to do_freebsd.sh

as autotools is dropped. and remove trailing spaces.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11072 from dmick/wip-17268

ceph-disk tests: Let missing python interpreters be non-fatal

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #11076 from badone/wip-gitignore-pyc

.gitignore: Add .pyc files globally

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11042 from somnathr/wip-verbose-log

kv/rocksdb: dump transactions on error

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: add safety checks on ino 0|1 op_file_update

We only want to log updates on ino 1 in very specific
cases, and never on ino 0.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not dirty compacting log ino 0

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: use hex for debug

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: config option for allocator (default stupid)

StupidAllocator is probably a more appropriate choice for bluefs.
Either way, it needn't be tied to what BlueStore is using.

Signed-off-by: Sage Weil <sage@redhat.com>

Logging proper message during read error from disk

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Logging rocksdb transaction in ceph log

The rocksdb transaction is now been logged into ceph log based
on debug level. If transaction is failed, the error code,
error string and the entire transaction is dumped in the log.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #11085 from stiopaa1/osd_osd_moveUnderPrivate

osd/OSD.h: make some members private

osd/OSD.h: make some members private

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11067 from stiopaa1/common_timer_replaceLongTypeswithauto

common/Timer.cc: replace long types with auto

Merge pull request #11068 from liewegas/wip-bluestore-leaks

os/bluestore: fix a few leaks

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: clear SharedBlob cache in dtor

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make Cache::lock a recursive_mutex

We need to remove items from the cache LRUs in their
destructors, and it is very awkward to ensure that all
releasing paths either hold the cache lock or have
already cleared the cache items out.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix SharedBlob leaks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix Extent leak from ~ExtentMap

Signed-off-by: Sage Weil <sage@redhat.com>

common/PluginRegistry: improve error output forshared library load failure.
Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11080 from ifed01/wip-compression-store-improvement

test/store_test: extend Bluestore compression test to verify compress…

Merge pull request #11074 from badone/wip-dry-run

run-*make-check.sh: Make DRY_RUN actually mean a dry run

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/store_test: extend Bluestore compression test to verify compression statistics and hence compression workability

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10965 from ceph/wip-crypto-init

common: only call crypto::init once per CephContext
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: add ExtentMap::add() helper

Matches rm().

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix Extent leak from _wctx_finish

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix Extent leak on ExtentMap removal

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10998 from ceph/wip-16695

rgw: work around curl_multi_wait bug with non-blocking reads

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11069 from liewegas/wip-rocksdb

rocksdb: pull up to master (4.12 + a few patches)

Merge pull request #11079 from ifed01/wip-fix-zlib

compressor/zlib: zlib wrapper fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11077 from xiexingguo/xxg-wip-bluestore-0914

os/bluestore: add checks and kill unreachable code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11014 from chhabaramesh/extent_alloc

os/bluestore: make assert conditional with macro for allocator

Reviewed-by: Sage Weil <sage@redhat.com>

compressor/zlis: improve buffer allocation to avoid space waste

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

common/compressor: fix improper length value passed to zlib compressor

Signed-off-by: Igor Fedotov <ifedotovo@miranitis.com>

test/objectstore: extend compression test case to use both snappy & zlib

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

.gitignore: Add .pyc files globally

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11071 from dmick/wip-17267

Stop hiding errors from run-tox.sh

Reviewed-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: drop result check of OP_TRY_RENAME

Because _rename() never returns -ENOENT, it only
returns -EEXIST on failure.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

run-*make-check.sh: Make DRY_RUN actually mean a dry run

DRY_RUN does not currently do an actual dry run as it actually runs an install
command.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: add sanity check for compress()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Stop hiding errors from run-tox.sh

The grep -v should have been grep, but, why even bother?
Tox doesn't output that much; just dump it, and exit with
tox's error code.

Fixes: http://tracker.ceph.com/issues/17267
Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-disk tests: Let missing python interpreters be non-fatal

Let tox ignore missing interpreters, and add py35 and py36 as well.
This lets tests work on Xenial, for example, which only has py35.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10826 from xiexingguo/xxg-wip-fix-objecter-0823

osdc/objecter: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10859 from xiexingguo/xxg-wip-fix-osd-08-25

osd/osd_type: check if pool is gone during check_new_interval()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11059 from markhpc/wip-bluefs-wal-buffered

bluestore/BlueFS: revert direct IO for WRITER_WAL

Merge pull request #11030 from amoxic/amoxic-wip-fix-do-write-small

os/bluestore: fix offset bug in _do_write_small.

Reviewed-by: Sage Weil <sage@redhat.com>

rocksdb: pull up to master (4.12 + a few patches)

This includes Somnath's fix for log recycling.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc: remove MOSDFailure include

Doesn't seem to be necessary anymore.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

test/osd: add test for fast mark down functionality

Test checks both async and simple messenger and also checks whether
disabling "osd fast fail on connection refused" option restores old
behavior.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

msg/async: implement ECONNREFUSED detection

This commit adds code that detects ECONNREFUSED and dispatches appropriate
event further in Async messenger.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

messages/MOSDFailure.h: distinguish between timeout and immediate failure

Change "is_failed" field to "flags" and use it to distinguish between timeout
and immediate, known OSD failure. Then use that in OSD and MON, and make sure
"min_reporters" don't affect known failures by actually going around failure
heuristic code.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

OSD: Implement ms_handle_refused

Added implementation of ms_handle_refused in OSD code, so it sends
MOSDFailure message in case the peer connection fails with ECONNREFUSED
*and* it is known to be up and new option "osd fast fail on connection
refused" which enables or disables new behavior.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

msg/simple: add ms_handle_refused callback

Added new callback (ms_handle_refused) to dispatchers. It is called
once connection attempt fails with ECONNREFUSED.
Also added dummy ms_handle_refused handlers across codebase.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

common/Timer.cc: replace long types with auto

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

fio: use ref counting for Engine instance

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11046 from stiopaa1/common_timer_deleteCopyConstrAndAssign

common/Timer.h: delete copy constr and assign op

Reviewed-by: Kefu Chai <kchai@redhat.com>