test/store_test: fix statfs results check to consider SSD min_alloc_size

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11092 from badone/wip-dry-run-run-make-check

run-make-check.sh: Make DRY_RUN actually do a dry run

Merge pull request #11091 from badone/wip-submodule-init-run-cmake-check

run-cmake-check.sh: Init submodules

Merge pull request #11087 from liewegas/wip-bluefs-allocator

os/bluestore/BlueFS: use StupidAllocator; fix async compaction bug

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #11088 from tchaikov/do-freebsd-without-autotools

do_freebsd*.sh: rename do_freebsd-cmake.sh to do_freebsd.sh

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

run-make-check.sh: Make DRY_RUN actually do a dry run

The "cd build" command will cause an error during a dry run if the directory
does not exist.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

run-cmake-check.sh: Init submodules

If submodules are not initialised run-cmake-check.sh can fail.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

do_freebsd*.sh: rename do_freebsd-cmake.sh to do_freebsd.sh

as autotools is dropped. and remove trailing spaces.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11072 from dmick/wip-17268

ceph-disk tests: Let missing python interpreters be non-fatal

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #11076 from badone/wip-gitignore-pyc

.gitignore: Add .pyc files globally

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #11042 from somnathr/wip-verbose-log

kv/rocksdb: dump transactions on error

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: add safety checks on ino 0|1 op_file_update

We only want to log updates on ino 1 in very specific
cases, and never on ino 0.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not dirty compacting log ino 0

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/StupidAllocator: use hex for debug

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: config option for allocator (default stupid)

StupidAllocator is probably a more appropriate choice for bluefs.
Either way, it needn't be tied to what BlueStore is using.

Signed-off-by: Sage Weil <sage@redhat.com>

Logging proper message during read error from disk

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Logging rocksdb transaction in ceph log

The rocksdb transaction is now been logged into ceph log based
on debug level. If transaction is failed, the error code,
error string and the entire transaction is dumped in the log.

Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #11085 from stiopaa1/osd_osd_moveUnderPrivate

osd/OSD.h: make some members private

osd/OSD.h: make some members private

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11067 from stiopaa1/common_timer_replaceLongTypeswithauto

common/Timer.cc: replace long types with auto

Merge pull request #11068 from liewegas/wip-bluestore-leaks

os/bluestore: fix a few leaks

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: clear SharedBlob cache in dtor

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make Cache::lock a recursive_mutex

We need to remove items from the cache LRUs in their
destructors, and it is very awkward to ensure that all
releasing paths either hold the cache lock or have
already cleared the cache items out.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix SharedBlob leaks

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix Extent leak from ~ExtentMap

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11080 from ifed01/wip-compression-store-improvement

test/store_test: extend Bluestore compression test to verify compress…

Merge pull request #11074 from badone/wip-dry-run

run-*make-check.sh: Make DRY_RUN actually mean a dry run

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/store_test: extend Bluestore compression test to verify compression statistics and hence compression workability

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10965 from ceph/wip-crypto-init

common: only call crypto::init once per CephContext
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: add ExtentMap::add() helper

Matches rm().

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix Extent leak from _wctx_finish

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix Extent leak on ExtentMap removal

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10998 from ceph/wip-16695

rgw: work around curl_multi_wait bug with non-blocking reads

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11069 from liewegas/wip-rocksdb

rocksdb: pull up to master (4.12 + a few patches)

Merge pull request #11079 from ifed01/wip-fix-zlib

compressor/zlib: zlib wrapper fix

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11077 from xiexingguo/xxg-wip-bluestore-0914

os/bluestore: add checks and kill unreachable code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11014 from chhabaramesh/extent_alloc

os/bluestore: make assert conditional with macro for allocator

Reviewed-by: Sage Weil <sage@redhat.com>

compressor/zlis: improve buffer allocation to avoid space waste

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

common/compressor: fix improper length value passed to zlib compressor

Signed-off-by: Igor Fedotov <ifedotovo@miranitis.com>

test/objectstore: extend compression test case to use both snappy & zlib

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

.gitignore: Add .pyc files globally

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11071 from dmick/wip-17267

Stop hiding errors from run-tox.sh

Reviewed-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: drop result check of OP_TRY_RENAME

Because _rename() never returns -ENOENT, it only
returns -EEXIST on failure.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

run-*make-check.sh: Make DRY_RUN actually mean a dry run

DRY_RUN does not currently do an actual dry run as it actually runs an install
command.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: add sanity check for compress()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Stop hiding errors from run-tox.sh

The grep -v should have been grep, but, why even bother?
Tox doesn't output that much; just dump it, and exit with
tox's error code.

Fixes: http://tracker.ceph.com/issues/17267
Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph-disk tests: Let missing python interpreters be non-fatal

Let tox ignore missing interpreters, and add py35 and py36 as well.
This lets tests work on Xenial, for example, which only has py35.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10826 from xiexingguo/xxg-wip-fix-objecter-0823

osdc/objecter: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10859 from xiexingguo/xxg-wip-fix-osd-08-25

osd/osd_type: check if pool is gone during check_new_interval()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11059 from markhpc/wip-bluefs-wal-buffered

bluestore/BlueFS: revert direct IO for WRITER_WAL

Merge pull request #11030 from amoxic/amoxic-wip-fix-do-write-small

os/bluestore: fix offset bug in _do_write_small.

Reviewed-by: Sage Weil <sage@redhat.com>

rocksdb: pull up to master (4.12 + a few patches)

This includes Somnath's fix for log recycling.

Signed-off-by: Sage Weil <sage@redhat.com>

common/Timer.cc: replace long types with auto

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11046 from stiopaa1/common_timer_deleteCopyConstrAndAssign

common/Timer.h: delete copy constr and assign op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11057 from liewegas/wip-os-fuse

qa/workunits/objectstore/test_fuse.sh: make test_fuse.sh work with filestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11064 from ilc/make-srpm

make-srpm.sh: A simple script to make the srpm for ceph.

Merge pull request #11061 from xiexingguo/xxg-wip-bs-0913

os/bluestore: fix shard_info::dump()

Reviewed-by: Sage Weil <sage@redhat.com>

make-srpm.sh: A simple script to make the srpm for ceph.

This is a simple script to make the srpm for ceph based
on make-dist.

Yes, it is short, but it will stop many people from
having to solve the same problem.

Signed-off-by: Ira Cooper <ira@redhat.com>

qa/workunits/objecstore/test_fuse.sh: make it work with filestore

Signed-off-by: Sage Weil <sage@redhat.com>

Delete unnecessary parentheses.

Signed-off-by: feng yu amoxic@126.com

Merge pull request #11016 from wido/snap-max-uid

mds: Set mds_snap_max_uid to 4294967294

Reviewed-by: John Spray <john.spray@redhat.com>

min_min_alloc_size to accommodate changing min_alloc_size

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: fix dump of shard_info

which shall include the "extents" field too.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10873 from dreamhost/wip-17136

rgw: Fix Host->bucket fallback logic inversion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: detect and work around a curl_multi_wait bug

Fixes: http://tracker.ceph.com/issues/15915
Fixes: http://tracker.ceph.com/issues/16695
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use non-blocking reads for clear_signal

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: factored clear_signal out of do_curl_wait

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10918 from tianshan/fix-bucket-decode-v

rgw: fix rgw_bucket_dir_entry decode v

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

bluestore/BlueFS: revert direct IO for WRITER_WAL

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #11040 from xiexingguo/xxg-wip-bluestore-0910

os/bluestore: fix leak of result-checking of _fsck_check_extents

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11047 from tchaikov/wip-long-live-readdir

ceph_mon: use readdir() as readdir_r() is deprecated

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11033 from xiexingguo/xxg-wip-bluestore-0909

os/bluestore: fix potential access violation during rename

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11044 from dillaman/wip-17227

librbd: ignore partial refresh error when acquiring exclusive lock

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11037 from dillaman/wip-17254

librbd: possible deadlock if cluster connection closed after image

Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph_mon: use readdir() as readdir_r() is deprecated

see https://lwn.net/Articles/696469/, readdir_r() is deprecated by glibc
since 2.24. so let's use readdir() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10941 from ceph/8.31.16-fix-install-deps

install-deps: exit non-zero when we cannot match distro

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11043 from stiopaa1/osd_osd_removeUnneededReturns

osd/OSD.cc: remove unneeded returns

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSD.cc: remove unneeded returns

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

common/Timer.h: delete copy constr and assign op

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: ignore partial refresh error when acquiring exclusive lock

Fixes: http://tracker.ceph.com/issues/17227
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11041 from stiopaa1/osd_replicatedpg_removeClassRedeclaration

osd/ReplicatedPG: remove class redeclaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10952 from codeyz/dev

crush/CrushCompiler.cc:884

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10986 from stiopaa1/osd_osd_remove_repeated_searching_of_map

osd/OSD.cc: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11000 from stiopaa1/osd_osd_removeUnneededSearchMap

osd/OSD.cc: remove unneeded searching of map

Reviewed-by: Jun Huang <hjwsm1989@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10935 from branch-predictor/bp-optimized-crush-ln

crush: don't normalize input of crush_ln iteratively

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10776 from ceph/wip-offline-split

ceph-objectstore-tool: add a way to split filestore directories offline

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10011 from songbaisen/a16

 mon: remove the redundant list swap in paxos commit_proposal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10872 from rjfd/wip-16776

Enable builds without ceph-test subpackage

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: remove class redeclaration

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: possible deadlock if cluster connection closed after image

Fixes: http://tracker.ceph.com/issues/17254
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: drop bluestore_blob_id_t

which is not used anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill fill() method from bluestore_extent_ref_map_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check during extent loading

Otherwise there are potential extent leaks.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11034 from dillaman/wip-17251

librbd: potential seg fault when blacklisting an image client

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: fix leak of result-checking of _fsck_check_extents

We shall not ignore the result of _fsck_check_extents() durint fsck().

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: using is_spanning() for spanning checking

Otherwise is_spanning() has no caller and shall be deleted.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix race condition

The empty() method also needs to be called under the protection of lock.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify extent_shard_key a little

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: return onode_key to caller

The get_key_extent_shard() method shall decompose extent_shard_key
into onode_key and offset, but currently we don't fill onode_key
and return it to caller.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make extent key generation a little faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>