Merge pull request #11041 from stiopaa1/osd_replicatedpg_removeClassRedeclaration

osd/ReplicatedPG: remove class redeclaration

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10952 from codeyz/dev

crush/CrushCompiler.cc:884

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10986 from stiopaa1/osd_osd_remove_repeated_searching_of_map

osd/OSD.cc: remove repeated searching of map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11000 from stiopaa1/osd_osd_removeUnneededSearchMap

osd/OSD.cc: remove unneeded searching of map

Reviewed-by: Jun Huang <hjwsm1989@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10935 from branch-predictor/bp-optimized-crush-ln

crush: don't normalize input of crush_ln iteratively

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10776 from ceph/wip-offline-split

ceph-objectstore-tool: add a way to split filestore directories offline

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10011 from songbaisen/a16

 mon: remove the redundant list swap in paxos commit_proposal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10872 from rjfd/wip-16776

Enable builds without ceph-test subpackage

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: remove class redeclaration

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11034 from dillaman/wip-17251

librbd: potential seg fault when blacklisting an image client

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #11001 from stiopaa1/client_barrier_moveCBlockSyncClass

client/barrier: move C_Block_Sync class to .cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10606 from jcsp/wip-mds-standby-trim

mds: trim null dentries proactively

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-objectstore-tool: add a way to split filestore directories offline

Use the usual split code, and split each dir that meets the
usual split criteria.

This can be run with lower than usual split settings, to avoid costly
online splits. To make sure the directories are not merged again, use
a load merge threshold (e.g. filestore merge threshold = 1), and
adjust the split multiplier accordingly.

Fixes: http://tracker.ceph.com/issues/17220
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10985 from dillaman/wip-16921

rbd-nbd: fix kernel deadlock during teuthology testing

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

deb: Moved ceph-client-debug from ceph-base to ceph-test package

Signed-off-by: Ricardo Dias <rdias@suse.com>

rpm: Added ceph_test_package bcond flag to control the build of ceph-test package

Fixes: http://tracker.ceph.com/issues/16776
Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: Use WITH_TESTS flag to guard the build of ceph-test scripts/binaries

Signed-off-by: Ricardo Dias <rdias@suse.com>

librbd: potential seg fault when blacklisting an image client

Fixes: http://tracker.ceph.com/issues/17251
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ignore cache busy errors when shrinking an image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: invalidate cache before trimming image

Any potential writeback outside the extents of a shrunk image
would result in orphaned objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: mask out-of-bounds IO errors caused by image shrink

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-nbd: fix kernel deadlock during teuthology testing

Fixes: http://tracker.ceph.com/issues/16921
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rpm: Renamed bcond tests flag to make_check

Signed-off-by: Ricardo Dias <rdias@suse.com>

rpm: Moved ceph-client-debug from ceph-base to ceph-test package

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #10723 from ceph/wip-os-fuse

os/FuseStore: fix several FuseStore issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9767 from dx9/wip-safe-io

safe_io: Improve portability by replacing loff_t type usage with off_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11028 from ceph/wip-17450

cmake: set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Set ARM_CRC_FLAGS from the CRC test rather than ARM_NEON_FLAGS

Fixes: http://tracker.ceph.com/issues/17250
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #10302 from stiopaa1/msg_asyncconn_addConstToMemFun

msg/AsyncConnection: add const to mem functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10114 from stiopaa1/msg_asyncmess_get_proto_version_addConst

msg/AsyncMessenger: add const to function

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11015 from tchaikov/wip-optional-py3

cmake: make py3 a nice-to-have

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge branch 'rgwfs-fix-rgw-lookup' of https://github.com/cmgitdream/ceph into cmgitdream-rgwfs-fix-rgw-lookup

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

osd/OSD.cc: remove unneeded searching of map

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

client/barrier: move C_Block_Sync class to .cc

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncMessenger: add const to function

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

msg/AsyncConnection: add const to mem functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #11003 from ceph/wip-cmake

cmake: include(SIMDExt) in src/CMakeLists.txt

Merge pull request #11023 from liewegas/wip-bluefs-extent-append

os/bluestore/BlueFS: only extend extent on same bdev

os/bluestore/BlueFS: only extend extent on same bdev

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10822 from Yan-waller/yj-wip-rgwfile-0823

rgw_file: fix return value signedness (rgw_readdir)

verified: f23

Merge pull request #10321 from stiopaa1/async_EventSelect_addoverride

msg/EventSelect.h: add override to virtual funct

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10318 from stiopaa1/async_eventkqueue_addOverride

msg/EventKqueue.h: add override to virtual func

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10314 from stiopaa1/async_eventpoll_addOverride

msg/EventPoll.h: add override to virtual functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10882 from stiopaa1/rgw_rgw_coroutine_moveWaitContextClass

rgw/rgw_coroutine: move WaitContext to .cc file

verified: f23

Merge pull request #10929 from atheism/rgw-lc-fix-typo

rgw: fix typo of "minite" to "minute" in rgw_lc.cc

verified: f23

Merge pull request #11010 from liewegas/wip-bluefs-compaction-race

os/bluestore/BlueFS: do not start racing async compaction

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #11013 from xiexingguo/xxg-wip-fix-cmake-check

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Merge pull request #11012 from majianpeng/bluefs-buffered-io

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11009 from liewegas/wip-bluestore-keys

os/bluestore: make onode keys more efficient (and sort correctly)

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: make py3 a nice-to-have

python3 is not a hard requirement to build ceph, so make it optional.
add an option named "WITH_PYTHON3" which accepts ON, OFF, or CHECK.

Fixes: http://tracker.ceph.com/issues/17103
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11017 from ifed01/wip-bluestore-cleanup

os/bluestore: remove some copy-pastes

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove some copy-pastes

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11011 from liewegas/wip-bluestore-inline-bl

os/bluestore: fix a few memory utilization leaks and wasters

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Now it use bluefs_buffered_io to control whether use buffer or directio
when write. But in fact for logs of rocksdb & bluefs, whether
bluefs_buffer_io is true or false, the logs only need directio.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #10957 from ukernel/wip-17172

client: properly set inode number of created inode in replay request

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9936 from onyb/wip-ceph-disk-py3

ceph-disk: Compatibility fixes for Python 3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Introduced by https://github.com/ceph/ceph/pull/11007.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/FuseStore: only flush if dirty

No need to rewrite the object unless it was changed.

This partially fixes truncate.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/objectstore: fix test_fuse.sh

We switched from @ to # a while back.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not waste memory on cached encoded blobs

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement buffer::list::reserve(n)

Make sure we have N bytes of append_buffer reserved. On
a new or cleared list, this allocates exactly that much
runway, allowing us to control memory usage.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix Dir memory leak

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not start racing async compaction

Compaction is triggred from sync_metadata.  If one compaction is
in progress and another thread also calls sync_metadata, do not
trigger a second async compaction!

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: resurrect blob and extent_map unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bits for unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove faulted debug hackery

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: small put_ref cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10978 from chhabaramesh/extent_alloc

os/bluestore: Hint based allocation in bitmap Allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11007 from liewegas/wip-autotools-must-die

remove autotools

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: encode shard id in single byte

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix key+name sort wonkiness

We want to unconditoinally start with the name or key, *then* do < > or =
(and if not =, a trailing name).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unnecessary separator between fields

We have fixed length/order for integer fields and use !
to terminate string fields, so there is no need to use
any extra separators, which is simpler as well as faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove never reachable asserts

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: don't dirty onode if its size is already at desired offset

for truncate operation.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: tunable prealloc size for ExtentMap inline_bl

Otherwise we eat 4KB for every Onode.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: clear append_buffer on clear()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11008 from liewegas/wip-libaio

test/objectstore/CMakeLists.txt: fix libaio conditional

test/objectstore/CMakeLists.txt: fix libaio conditional

Signed-off-by: Sage Weil <sage@redhat.com>

remove autotools

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10963 from liewegas/wip-bluestore-sharded-extent-map

os/bluestore: shard extent map

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

os/bluestore: assert shared blob cache cleared on split

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump some stats after fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument onode reshard events

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument transaction count

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument big/small writes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make blob_t unused helpers use logical length

These were taking min_alloc_size, but this can change
across mounts; better to use the logical blob length
instead (that's what we want anyway!).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use block_size for allocator unit

We need to handle objects written during previous mounts
that may have had a smaller min_alloc_size.  Use
block_size, which is a safe lower bound.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fsck used_block bitmap

This has to be block_size bits because min_alloc_size
can vary over mounts.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: optimize compress_extent_map

Only examine the range we just wrote to (and to the left
and right).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make nid and blobid allocation less racy

We could bump the _max value for a TransContext in it's
prepare state, have it wait for a long time on IO, and
let another txc allocate and commit something with
an id higher than the previous max.

Fix this first by pushing the max ids into the
TransContext where we can deal with them at commit time,
and then making _kv_sync_thread bump the committed
max in a safe way.

Note that this will need to change if/when we do
these commits in parallel.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: shard extent map

Rewrote much of the persistence of onode metadata.  The
highlights:

 - extents and blobs stored together (the blob with the
   first referencing extent).
 - extents sharded across multiple k/v keys
 - if a blob if referenced from multiple blobs, it's
   stored in the onode key (called a "spanning blob").
 - when we clone a blob we copy the metadata, but mark
   it shared and put (just) the ref_map on the underlying
   blocks in a shared_blob key.  at this point we also
   assign a globally unique id (sbid = shared blob id)
   so the key has a unique name.
 - we instantiate a SharedBlob in memory regardless of
   whether we need to load the ref_map (which is only
   needed for deallocations!).  the BufferSpace is
   attached to this SharedBlob so we get unified caching
   across clones.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10973 from wjwithagen/wip-wjw-freebsd-cmake-excludes-2

cmake: FreeBSD specific excludes in CMakeLists.txt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Hint argument in alloc_blocks + test case changes

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

remove wrap argument and handle in wrappers

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

hint in extent_alloc code

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10996 from jcsp/wip-16973

mds: log path with CDir damage messages

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10928 from stiopaa1/rbd_mirror_imagesyncthrottler

rbd_mirror/ImageSynceThrottler: move struct to .cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10947 from atheism/radosgw-admin-help

rgw: right parenthesis is missing in radosgw-admin help message on caps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10954 from stiopaa1/rgw_rgwrados_removeUnneededClassCTick

rgw/rgw_rados.h: remove unneeded class C_Tick

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10967 from stiopaa1/osd_replicatedbackend_moveClasses

osd/ReplicatedBackend: move classes to cc file

Reviewed-by: Sage Weil <sage@redhat.com>