os/bluestore: add sanity check during extent loading

Otherwise there are potential extent leaks.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix leak of result-checking of _fsck_check_extents

We shall not ignore the result of _fsck_check_extents() durint fsck().

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11001 from stiopaa1/client_barrier_moveCBlockSyncClass

client/barrier: move C_Block_Sync class to .cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10606 from jcsp/wip-mds-standby-trim

mds: trim null dentries proactively

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10985 from dillaman/wip-16921

rbd-nbd: fix kernel deadlock during teuthology testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: ignore cache busy errors when shrinking an image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: invalidate cache before trimming image

Any potential writeback outside the extents of a shrunk image
would result in orphaned objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: mask out-of-bounds IO errors caused by image shrink

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: fix kernel deadlock during teuthology testing

Fixes: http://tracker.ceph.com/issues/16921
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10723 from ceph/wip-os-fuse

os/FuseStore: fix several FuseStore issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9767 from dx9/wip-safe-io

safe_io: Improve portability by replacing loff_t type usage with off_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11028 from ceph/wip-17450

cmake: set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Fixes: http://tracker.ceph.com/issues/17250
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10302 from stiopaa1/msg_asyncconn_addConstToMemFun

msg/AsyncConnection: add const to mem functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10114 from stiopaa1/msg_asyncmess_get_proto_version_addConst

msg/AsyncMessenger: add const to function

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11015 from tchaikov/wip-optional-py3

cmake: make py3 a nice-to-have

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'rgwfs-fix-rgw-lookup' of https://github.com/cmgitdream/ceph into cmgitdream-rgwfs-fix-rgw-lookup

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

client/barrier: move C_Block_Sync class to .cc

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncMessenger: add const to function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncConnection: add const to mem functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11003 from ceph/wip-cmake

cmake: include(SIMDExt) in src/CMakeLists.txt

Merge pull request #11023 from liewegas/wip-bluefs-extent-append

os/bluestore/BlueFS: only extend extent on same bdev

os/bluestore/BlueFS: only extend extent on same bdev

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10822 from Yan-waller/yj-wip-rgwfile-0823

rgw_file: fix return value signedness (rgw_readdir)

verified: f23

Merge pull request #10321 from stiopaa1/async_EventSelect_addoverride

msg/EventSelect.h: add override to virtual funct

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10318 from stiopaa1/async_eventkqueue_addOverride

msg/EventKqueue.h: add override to virtual func

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10314 from stiopaa1/async_eventpoll_addOverride

msg/EventPoll.h: add override to virtual functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10882 from stiopaa1/rgw_rgw_coroutine_moveWaitContextClass

rgw/rgw_coroutine: move WaitContext to .cc file

verified: f23

Merge pull request #10929 from atheism/rgw-lc-fix-typo

rgw: fix typo of "minite" to "minute" in rgw_lc.cc

verified: f23

Merge pull request #11010 from liewegas/wip-bluefs-compaction-race

os/bluestore/BlueFS: do not start racing async compaction

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #11013 from xiexingguo/xxg-wip-fix-cmake-check

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Merge pull request #11012 from majianpeng/bluefs-buffered-io

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11009 from liewegas/wip-bluestore-keys

os/bluestore: make onode keys more efficient (and sort correctly)

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: make py3 a nice-to-have

python3 is not a hard requirement to build ceph, so make it optional.
add an option named "WITH_PYTHON3" which accepts ON, OFF, or CHECK.

Fixes: http://tracker.ceph.com/issues/17103
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11017 from ifed01/wip-bluestore-cleanup

os/bluestore: remove some copy-pastes

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove some copy-pastes

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11011 from liewegas/wip-bluestore-inline-bl

os/bluestore: fix a few memory utilization leaks and wasters

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Now it use bluefs_buffered_io to control whether use buffer or directio
when write. But in fact for logs of rocksdb & bluefs, whether
bluefs_buffer_io is true or false, the logs only need directio.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #10957 from ukernel/wip-17172

client: properly set inode number of created inode in replay request

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9936 from onyb/wip-ceph-disk-py3

ceph-disk: Compatibility fixes for Python 3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Introduced by https://github.com/ceph/ceph/pull/11007.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/FuseStore: only flush if dirty

No need to rewrite the object unless it was changed.

This partially fixes truncate.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore: fix test_fuse.sh

We switched from @ to # a while back.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not waste memory on cached encoded blobs

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement buffer::list::reserve(n)

Make sure we have N bytes of append_buffer reserved. On
a new or cleared list, this allocates exactly that much
runway, allowing us to control memory usage.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix Dir memory leak

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not start racing async compaction

Compaction is triggred from sync_metadata.  If one compaction is
in progress and another thread also calls sync_metadata, do not
trigger a second async compaction!

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: resurrect blob and extent_map unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bits for unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove faulted debug hackery

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: small put_ref cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10978 from chhabaramesh/extent_alloc

os/bluestore: Hint based allocation in bitmap Allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11007 from liewegas/wip-autotools-must-die

remove autotools

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: encode shard id in single byte

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix key+name sort wonkiness

We want to unconditoinally start with the name or key, *then* do < > or =
(and if not =, a trailing name).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unnecessary separator between fields

We have fixed length/order for integer fields and use !
to terminate string fields, so there is no need to use
any extra separators, which is simpler as well as faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove never reachable asserts

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: don't dirty onode if its size is already at desired offset

for truncate operation.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: tunable prealloc size for ExtentMap inline_bl

Otherwise we eat 4KB for every Onode.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: clear append_buffer on clear()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11008 from liewegas/wip-libaio

test/objectstore/CMakeLists.txt: fix libaio conditional

test/objectstore/CMakeLists.txt: fix libaio conditional

Signed-off-by: Sage Weil <sage@redhat.com>

remove autotools

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10963 from liewegas/wip-bluestore-sharded-extent-map

os/bluestore: shard extent map

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

os/bluestore: assert shared blob cache cleared on split

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump some stats after fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument onode reshard events

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument transaction count

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument big/small writes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make blob_t unused helpers use logical length

These were taking min_alloc_size, but this can change
across mounts; better to use the logical blob length
instead (that's what we want anyway!).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use block_size for allocator unit

We need to handle objects written during previous mounts
that may have had a smaller min_alloc_size.  Use
block_size, which is a safe lower bound.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fsck used_block bitmap

This has to be block_size bits because min_alloc_size
can vary over mounts.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: optimize compress_extent_map

Only examine the range we just wrote to (and to the left
and right).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make nid and blobid allocation less racy

We could bump the _max value for a TransContext in it's
prepare state, have it wait for a long time on IO, and
let another txc allocate and commit something with
an id higher than the previous max.

Fix this first by pushing the max ids into the
TransContext where we can deal with them at commit time,
and then making _kv_sync_thread bump the committed
max in a safe way.

Note that this will need to change if/when we do
these commits in parallel.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: shard extent map

Rewrote much of the persistence of onode metadata.  The
highlights:

 - extents and blobs stored together (the blob with the
   first referencing extent).
 - extents sharded across multiple k/v keys
 - if a blob if referenced from multiple blobs, it's
   stored in the onode key (called a "spanning blob").
 - when we clone a blob we copy the metadata, but mark
   it shared and put (just) the ref_map on the underlying
   blocks in a shared_blob key.  at this point we also
   assign a globally unique id (sbid = shared blob id)
   so the key has a unique name.
 - we instantiate a SharedBlob in memory regardless of
   whether we need to load the ref_map (which is only
   needed for deallocations!).  the BufferSpace is
   attached to this SharedBlob so we get unified caching
   across clones.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10973 from wjwithagen/wip-wjw-freebsd-cmake-excludes-2

cmake: FreeBSD specific excludes in CMakeLists.txt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Hint argument in alloc_blocks + test case changes

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

remove wrap argument and handle in wrappers

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

hint in extent_alloc code

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10996 from jcsp/wip-16973

mds: log path with CDir damage messages

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10928 from stiopaa1/rbd_mirror_imagesyncthrottler

rbd_mirror/ImageSynceThrottler: move struct to .cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10947 from atheism/radosgw-admin-help

rgw: right parenthesis is missing in radosgw-admin help message on caps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10954 from stiopaa1/rgw_rgwrados_removeUnneededClassCTick

rgw/rgw_rados.h: remove unneeded class C_Tick

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10967 from stiopaa1/osd_replicatedbackend_moveClasses

osd/ReplicatedBackend: move classes to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10971 from stiopaa1/osd_replicatedpg_moveContextClasses

osd/ReplicatedPG: move classes to .cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10979 from wjwithagen/wip-wjw-cephtool-test

qa/workunits/cephtool/test.sh: fix omission of ceph-command

Merge pull request #10988 from jazeltq/fix-compile-error

autogen: Fix rocksdb error when make dist

mds: log path with CDir damage messages

Previously you just got the inode number, which
wasn't terribly useful for e.g. a missing fragment
object, as you couldn't readily resolve the parent
path.

Fixes: http://tracker.ceph.com/issues/16973
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10930 from atheism/rgw-lc-fix-sleep-time

rgw/rgw_lc.cc: fix sleep time according to the error message

cmake: include(SIMDExt) in src/CMakeLists.txt

as we need to check HAVE_ARMV8_CRC in src/CMakeLists.txt.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: trim null dentries proactively

Instead of leaving null dentries (e.g. left
behind from unlinks) in the cache until they
fall out of the LRU, actively push them
to the bottom of the LRU and then consume
all nulls at the bottom in trim() even if
the cache is not oversized yet.

This fixes the case where standby replay daemons
would otherwise accumulate a cache full of
null dentries resulting from unlinks, and it
makes the behaviour of active daemons more
deterministic.

Fixes: http://tracker.ceph.com/issues/16919
Signed-off-by: John Spray <john.spray@redhat.com>

src/test/objectstore/CMakeLists.txt: conditionaly build tests

 - Honor WITH_AIO tests that depend on bluestore parts

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/tools/CMakeLists.txt: conditionaly build ceph-client-debug

 - Honor WITH_LIBCEPHFS

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/pybind/CMakeLists.txt: conditionaly build cython tools

 - honor the WITH_RBD and WITH_LIBCEPHFS flags

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/CMakeLists.txt: exclude parts not buildable on FreeBSD

 - FreeBSD does not have keyutils or any kernel type security store
   So using secret.c is not really relevant here
 - Have common/secret.c depend on KEYUTILS_FOUND
 - This also excludes building mount.cephfs,
   Next to the fact that this is 100% linux bases mount stuff
 - krbd is the Linux kernel version for RBD based disks, but again
   completely Linux based and porting to FreeBSD, if any, has to follow
   a different route

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Makfile: Fix rocksdb error when make dist

Rocksdb delete some files, so we should change another way
to get rocksdb's files.

Currently, if we do not run configure in googletest/googlemock,
we also will get a error when make dist. So fix it too.

Signed-off-by: tianqing <tianqing@unitedstack.com>

Merge pull request #10972 from wjwithagen/wip-wjw-cmake-resolv

CmakeLists.txt: use LIB_RESOLV instead of resolv.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10378 from trociny/wip-14738

librbd: optionally unregister "laggy" journal clients

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph_test_objectstore: test shards for longer

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: occasional umount/fsck/mount

Signed-off-by: Sage Weil <sage@redhat.com>