rocksdb: pull up to master (4.12 + a few patches)

This includes Somnath's fix for log recycling.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11046 from stiopaa1/common_timer_deleteCopyConstrAndAssign

common/Timer.h: delete copy constr and assign op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11057 from liewegas/wip-os-fuse

qa/workunits/objectstore/test_fuse.sh: make test_fuse.sh work with filestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11064 from ilc/make-srpm

make-srpm.sh: A simple script to make the srpm for ceph.

Merge pull request #11061 from xiexingguo/xxg-wip-bs-0913

os/bluestore: fix shard_info::dump()

Reviewed-by: Sage Weil <sage@redhat.com>

make-srpm.sh: A simple script to make the srpm for ceph.

This is a simple script to make the srpm for ceph based
on make-dist.

Yes, it is short, but it will stop many people from
having to solve the same problem.

Signed-off-by: Ira Cooper <ira@redhat.com>

qa/workunits/objecstore/test_fuse.sh: make it work with filestore

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11016 from wido/snap-max-uid

mds: Set mds_snap_max_uid to 4294967294

Reviewed-by: John Spray <john.spray@redhat.com>

os/bluestore: fix dump of shard_info

which shall include the "extents" field too.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10873 from dreamhost/wip-17136

rgw: Fix Host->bucket fallback logic inversion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10918 from tianshan/fix-bucket-decode-v

rgw: fix rgw_bucket_dir_entry decode v

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11040 from xiexingguo/xxg-wip-bluestore-0910

os/bluestore: fix leak of result-checking of _fsck_check_extents

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11047 from tchaikov/wip-long-live-readdir

ceph_mon: use readdir() as readdir_r() is deprecated

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11033 from xiexingguo/xxg-wip-bluestore-0909

os/bluestore: fix potential access violation during rename

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11044 from dillaman/wip-17227

librbd: ignore partial refresh error when acquiring exclusive lock

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11037 from dillaman/wip-17254

librbd: possible deadlock if cluster connection closed after image

Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph_mon: use readdir() as readdir_r() is deprecated

see https://lwn.net/Articles/696469/, readdir_r() is deprecated by glibc
since 2.24. so let's use readdir() instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10941 from ceph/8.31.16-fix-install-deps

install-deps: exit non-zero when we cannot match distro

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11043 from stiopaa1/osd_osd_removeUnneededReturns

osd/OSD.cc: remove unneeded returns

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/OSD.cc: remove unneeded returns

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

common/Timer.h: delete copy constr and assign op

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: ignore partial refresh error when acquiring exclusive lock

Fixes: http://tracker.ceph.com/issues/17227
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11041 from stiopaa1/osd_replicatedpg_removeClassRedeclaration

osd/ReplicatedPG: remove class redeclaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10952 from codeyz/dev

crush/CrushCompiler.cc:884

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10986 from stiopaa1/osd_osd_remove_repeated_searching_of_map

osd/OSD.cc: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11000 from stiopaa1/osd_osd_removeUnneededSearchMap

osd/OSD.cc: remove unneeded searching of map

Reviewed-by: Jun Huang <hjwsm1989@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10935 from branch-predictor/bp-optimized-crush-ln

crush: don't normalize input of crush_ln iteratively

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10776 from ceph/wip-offline-split

ceph-objectstore-tool: add a way to split filestore directories offline

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10011 from songbaisen/a16

 mon: remove the redundant list swap in paxos commit_proposal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10872 from rjfd/wip-16776

Enable builds without ceph-test subpackage

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: remove class redeclaration

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librbd: possible deadlock if cluster connection closed after image

Fixes: http://tracker.ceph.com/issues/17254
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: drop bluestore_blob_id_t

which is not used anymore.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: kill fill() method from bluestore_extent_ref_map_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check during extent loading

Otherwise there are potential extent leaks.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11034 from dillaman/wip-17251

librbd: potential seg fault when blacklisting an image client

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore: fix leak of result-checking of _fsck_check_extents

We shall not ignore the result of _fsck_check_extents() durint fsck().

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: using is_spanning() for spanning checking

Otherwise is_spanning() has no caller and shall be deleted.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix race condition

The empty() method also needs to be called under the protection of lock.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify extent_shard_key a little

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: return onode_key to caller

The get_key_extent_shard() method shall decompose extent_shard_key
into onode_key and offset, but currently we don't fill onode_key
and return it to caller.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make extent key generation a little faster

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix potential access violation during rename

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

install-deps: exit non-zero when we cannot match distro

Signed-off-by: Gregory Meno <gmeno@redhat.com>

Merge pull request #11001 from stiopaa1/client_barrier_moveCBlockSyncClass

client/barrier: move C_Block_Sync class to .cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10606 from jcsp/wip-mds-standby-trim

mds: trim null dentries proactively

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-objectstore-tool: add a way to split filestore directories offline

Use the usual split code, and split each dir that meets the
usual split criteria.

This can be run with lower than usual split settings, to avoid costly
online splits. To make sure the directories are not merged again, use
a load merge threshold (e.g. filestore merge threshold = 1), and
adjust the split multiplier accordingly.

Fixes: http://tracker.ceph.com/issues/17220
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10985 from dillaman/wip-16921

rbd-nbd: fix kernel deadlock during teuthology testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

deb: Moved ceph-client-debug from ceph-base to ceph-test package

Signed-off-by: Ricardo Dias <rdias@suse.com>

rpm: Added ceph_test_package bcond flag to control the build of ceph-test package

Fixes: http://tracker.ceph.com/issues/16776
Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: Use WITH_TESTS flag to guard the build of ceph-test scripts/binaries

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: potential seg fault when blacklisting an image client

Fixes: http://tracker.ceph.com/issues/17251
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ignore cache busy errors when shrinking an image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: invalidate cache before trimming image

Any potential writeback outside the extents of a shrunk image
would result in orphaned objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: mask out-of-bounds IO errors caused by image shrink

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: fix kernel deadlock during teuthology testing

Fixes: http://tracker.ceph.com/issues/16921
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rpm: Renamed bcond tests flag to make_check

Signed-off-by: Ricardo Dias <rdias@suse.com>

rpm: Moved ceph-client-debug from ceph-base to ceph-test package

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #10723 from ceph/wip-os-fuse

os/FuseStore: fix several FuseStore issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9767 from dx9/wip-safe-io

safe_io: Improve portability by replacing loff_t type usage with off_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11028 from ceph/wip-17450

cmake: set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Fixes: http://tracker.ceph.com/issues/17250
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10302 from stiopaa1/msg_asyncconn_addConstToMemFun

msg/AsyncConnection: add const to mem functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10114 from stiopaa1/msg_asyncmess_get_proto_version_addConst

msg/AsyncMessenger: add const to function

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11015 from tchaikov/wip-optional-py3

cmake: make py3 a nice-to-have

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'rgwfs-fix-rgw-lookup' of https://github.com/cmgitdream/ceph into cmgitdream-rgwfs-fix-rgw-lookup

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

osd/OSD.cc: remove unneeded searching of map

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

client/barrier: move C_Block_Sync class to .cc

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncMessenger: add const to function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncConnection: add const to mem functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11003 from ceph/wip-cmake

cmake: include(SIMDExt) in src/CMakeLists.txt

Merge pull request #11023 from liewegas/wip-bluefs-extent-append

os/bluestore/BlueFS: only extend extent on same bdev

os/bluestore/BlueFS: only extend extent on same bdev

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10822 from Yan-waller/yj-wip-rgwfile-0823

rgw_file: fix return value signedness (rgw_readdir)

verified: f23

Merge pull request #10321 from stiopaa1/async_EventSelect_addoverride

msg/EventSelect.h: add override to virtual funct

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10318 from stiopaa1/async_eventkqueue_addOverride

msg/EventKqueue.h: add override to virtual func

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10314 from stiopaa1/async_eventpoll_addOverride

msg/EventPoll.h: add override to virtual functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10882 from stiopaa1/rgw_rgw_coroutine_moveWaitContextClass

rgw/rgw_coroutine: move WaitContext to .cc file

verified: f23

Merge pull request #10929 from atheism/rgw-lc-fix-typo

rgw: fix typo of "minite" to "minute" in rgw_lc.cc

verified: f23

Merge pull request #11010 from liewegas/wip-bluefs-compaction-race

os/bluestore/BlueFS: do not start racing async compaction

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #11013 from xiexingguo/xxg-wip-fix-cmake-check

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Merge pull request #11012 from majianpeng/bluefs-buffered-io

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11009 from liewegas/wip-bluestore-keys

os/bluestore: make onode keys more efficient (and sort correctly)

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: make py3 a nice-to-have

python3 is not a hard requirement to build ceph, so make it optional.
add an option named "WITH_PYTHON3" which accepts ON, OFF, or CHECK.

Fixes: http://tracker.ceph.com/issues/17103
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11017 from ifed01/wip-bluestore-cleanup

os/bluestore: remove some copy-pastes

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove some copy-pastes

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11011 from liewegas/wip-bluestore-inline-bl

os/bluestore: fix a few memory utilization leaks and wasters

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

mds: Set mds_snap_max_uid to 4294967294

Since kernel version 2.6 the Linux kernel supports 32-bit integers
and thus the limit is no longer 65536.

By setting this to a higher default value we make sure that all users
will be allowed to create snapshots in the future by default.

Signed-off-by: Wido den Hollander <wido@42on.com>

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Now it use bluefs_buffered_io to control whether use buffer or directio
when write. But in fact for logs of rocksdb & bluefs, whether
bluefs_buffer_io is true or false, the logs only need directio.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #10957 from ukernel/wip-17172

client: properly set inode number of created inode in replay request

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9936 from onyb/wip-ceph-disk-py3

ceph-disk: Compatibility fixes for Python 3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Introduced by https://github.com/ceph/ceph/pull/11007.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/FuseStore: only flush if dirty

No need to rewrite the object unless it was changed.

This partially fixes truncate.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore: fix test_fuse.sh

We switched from @ to # a while back.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not waste memory on cached encoded blobs

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement buffer::list::reserve(n)

Make sure we have N bytes of append_buffer reserved. On
a new or cleared list, this allocates exactly that much
runway, allowing us to control memory usage.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix Dir memory leak

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not start racing async compaction

Compaction is triggred from sync_metadata.  If one compaction is
in progress and another thread also calls sync_metadata, do not
trigger a second async compaction!

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: resurrect blob and extent_map unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bits for unit tests

Signed-off-by: Sage Weil <sage@redhat.com>