Merge pull request #10314 from stiopaa1/async_eventpoll_addOverride

msg/EventPoll.h: add override to virtual functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10882 from stiopaa1/rgw_rgw_coroutine_moveWaitContextClass

rgw/rgw_coroutine: move WaitContext to .cc file

verified: f23

Merge pull request #10929 from atheism/rgw-lc-fix-typo

rgw: fix typo of "minite" to "minute" in rgw_lc.cc

verified: f23

Merge pull request #11010 from liewegas/wip-bluefs-compaction-race

os/bluestore/BlueFS: do not start racing async compaction

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #11013 from xiexingguo/xxg-wip-fix-cmake-check

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Merge pull request #11012 from majianpeng/bluefs-buffered-io

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11009 from liewegas/wip-bluestore-keys

os/bluestore: make onode keys more efficient (and sort correctly)

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11017 from ifed01/wip-bluestore-cleanup

os/bluestore: remove some copy-pastes

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: remove some copy-pastes

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11011 from liewegas/wip-bluestore-inline-bl

os/bluestore: fix a few memory utilization leaks and wasters

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore/BlueFS: For logs of rocksdb & bluefs only use directio.

Now it use bluefs_buffered_io to control whether use buffer or directio
when write. But in fact for logs of rocksdb & bluefs, whether
bluefs_buffer_io is true or false, the logs only need directio.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #10957 from ukernel/wip-17172

client: properly set inode number of created inode in replay request

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9936 from onyb/wip-ceph-disk-py3

ceph-disk: Compatibility fixes for Python 3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

build: drop dryrun of autogen.sh from run-cmake-check.sh script

Introduced by https://github.com/ceph/ceph/pull/11007.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: do not waste memory on cached encoded blobs

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: implement buffer::list::reserve(n)

Make sure we have N bytes of append_buffer reserved. On
a new or cleared list, this allocates exactly that much
runway, allowing us to control memory usage.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix Dir memory leak

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: do not start racing async compaction

Compaction is triggred from sync_metadata.  If one compaction is
in progress and another thread also calls sync_metadata, do not
trigger a second async compaction!

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: resurrect blob and extent_map unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bits for unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove faulted debug hackery

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: small put_ref cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10978 from chhabaramesh/extent_alloc

os/bluestore: Hint based allocation in bitmap Allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11007 from liewegas/wip-autotools-must-die

remove autotools

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: encode shard id in single byte

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix key+name sort wonkiness

We want to unconditoinally start with the name or key, *then* do < > or =
(and if not =, a trailing name).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unnecessary separator between fields

We have fixed length/order for integer fields and use !
to terminate string fields, so there is no need to use
any extra separators, which is simpler as well as faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove never reachable asserts

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: don't dirty onode if its size is already at desired offset

for truncate operation.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: tunable prealloc size for ExtentMap inline_bl

Otherwise we eat 4KB for every Onode.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: clear append_buffer on clear()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11008 from liewegas/wip-libaio

test/objectstore/CMakeLists.txt: fix libaio conditional

test/objectstore/CMakeLists.txt: fix libaio conditional

Signed-off-by: Sage Weil <sage@redhat.com>

remove autotools

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10963 from liewegas/wip-bluestore-sharded-extent-map

os/bluestore: shard extent map

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

os/bluestore: assert shared blob cache cleared on split

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump some stats after fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument onode reshard events

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument transaction count

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument big/small writes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make blob_t unused helpers use logical length

These were taking min_alloc_size, but this can change
across mounts; better to use the logical blob length
instead (that's what we want anyway!).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use block_size for allocator unit

We need to handle objects written during previous mounts
that may have had a smaller min_alloc_size.  Use
block_size, which is a safe lower bound.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fsck used_block bitmap

This has to be block_size bits because min_alloc_size
can vary over mounts.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: optimize compress_extent_map

Only examine the range we just wrote to (and to the left
and right).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make nid and blobid allocation less racy

We could bump the _max value for a TransContext in it's
prepare state, have it wait for a long time on IO, and
let another txc allocate and commit something with
an id higher than the previous max.

Fix this first by pushing the max ids into the
TransContext where we can deal with them at commit time,
and then making _kv_sync_thread bump the committed
max in a safe way.

Note that this will need to change if/when we do
these commits in parallel.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: shard extent map

Rewrote much of the persistence of onode metadata.  The
highlights:

 - extents and blobs stored together (the blob with the
   first referencing extent).
 - extents sharded across multiple k/v keys
 - if a blob if referenced from multiple blobs, it's
   stored in the onode key (called a "spanning blob").
 - when we clone a blob we copy the metadata, but mark
   it shared and put (just) the ref_map on the underlying
   blocks in a shared_blob key.  at this point we also
   assign a globally unique id (sbid = shared blob id)
   so the key has a unique name.
 - we instantiate a SharedBlob in memory regardless of
   whether we need to load the ref_map (which is only
   needed for deallocations!).  the BufferSpace is
   attached to this SharedBlob so we get unified caching
   across clones.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10973 from wjwithagen/wip-wjw-freebsd-cmake-excludes-2

cmake: FreeBSD specific excludes in CMakeLists.txt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Hint argument in alloc_blocks + test case changes

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

remove wrap argument and handle in wrappers

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

hint in extent_alloc code

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10996 from jcsp/wip-16973

mds: log path with CDir damage messages

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10928 from stiopaa1/rbd_mirror_imagesyncthrottler

rbd_mirror/ImageSynceThrottler: move struct to .cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10947 from atheism/radosgw-admin-help

rgw: right parenthesis is missing in radosgw-admin help message on caps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10954 from stiopaa1/rgw_rgwrados_removeUnneededClassCTick

rgw/rgw_rados.h: remove unneeded class C_Tick

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10967 from stiopaa1/osd_replicatedbackend_moveClasses

osd/ReplicatedBackend: move classes to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10971 from stiopaa1/osd_replicatedpg_moveContextClasses

osd/ReplicatedPG: move classes to .cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10979 from wjwithagen/wip-wjw-cephtool-test

qa/workunits/cephtool/test.sh: fix omission of ceph-command

Merge pull request #10988 from jazeltq/fix-compile-error

autogen: Fix rocksdb error when make dist

mds: log path with CDir damage messages

Previously you just got the inode number, which
wasn't terribly useful for e.g. a missing fragment
object, as you couldn't readily resolve the parent
path.

Fixes: http://tracker.ceph.com/issues/16973
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10930 from atheism/rgw-lc-fix-sleep-time

rgw/rgw_lc.cc: fix sleep time according to the error message

src/test/objectstore/CMakeLists.txt: conditionaly build tests

 - Honor WITH_AIO tests that depend on bluestore parts

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/tools/CMakeLists.txt: conditionaly build ceph-client-debug

 - Honor WITH_LIBCEPHFS

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/pybind/CMakeLists.txt: conditionaly build cython tools

 - honor the WITH_RBD and WITH_LIBCEPHFS flags

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/CMakeLists.txt: exclude parts not buildable on FreeBSD

 - FreeBSD does not have keyutils or any kernel type security store
   So using secret.c is not really relevant here
 - Have common/secret.c depend on KEYUTILS_FOUND
 - This also excludes building mount.cephfs,
   Next to the fact that this is 100% linux bases mount stuff
 - krbd is the Linux kernel version for RBD based disks, but again
   completely Linux based and porting to FreeBSD, if any, has to follow
   a different route

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Makfile: Fix rocksdb error when make dist

Rocksdb delete some files, so we should change another way
to get rocksdb's files.

Currently, if we do not run configure in googletest/googlemock,
we also will get a error when make dist. So fix it too.

Signed-off-by: tianqing <tianqing@unitedstack.com>

Merge pull request #10972 from wjwithagen/wip-wjw-cmake-resolv

CmakeLists.txt: use LIB_RESOLV instead of resolv.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10378 from trociny/wip-14738

librbd: optionally unregister "laggy" journal clients

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph_test_objectstore: test shards for longer

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: occasional umount/fsck/mount

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add SyntheticMatrixSharding

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10870 from ceph/wip_build_cls_rbd

cmake: add dependency from ceph_smalliobenchrbd to cls libraries

Reviewed-by: Jason Dillaman <jdillama@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10912 from stiopaa1/journal_journaltimer_moveCRemoveSet

journal: move JournalTrimmer::C_RemoveSet struct

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Add dependencies from ceph_smalliobenchrbd on cls_rbd, cls_journal,
and cls_lock.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #10984 from ceph/wip-kfsx-exclusive-lock

test/librbd/fsx: enable exclusive-lock feature in krbd mode

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10964 from liewegas/wip-bluefs-compaction

os/bluestore/BlueFS: fix async compaction logging bug

Merge pull request #10966 from liewegas/wip-bluefs-runway

os/bluestore/BlueFS: factor unflushed log into runway calculation

Reviewed-by: Varada Kari <varada.kari@sandisk.com>
Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #10970 from xiexingguo/xxg-wip-honour-alloweio-flag

os/bluestore: honour allow-eio flag; use global compressor if possible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10974 from dillaman/wip-17210

librbd: potential double-unwatch of watch handle upon error

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mds: s/used_ions/used_inos/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10774 from renhwztetecs/renhw-wip-cephfs-python-umount

libcephfs: add unmount function in cephfs.pyx

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #10914 from batrick/mds-no-maxmds

mds: remove max_mds config option

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10922 from ceph/wip-jlayton-statx

First pile of statx patches

test/cephfs: add mount&&unmount test

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #10981 from SUSE/wip-fix-install-deps

build/ops: fix /etc/os-release parsing in install-deps.sh

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

build/ops: fix /etc/os-release parsing in install-deps.sh

85a370e35fc42031a7f7e24dea9e50a649c0f309 introduced the DISTRO variable whose
value was obtained by parsing /etc/os-release like this:

DISTRO=$(grep  "^ID=" /etc/os-release | sed "s/ID=//")

This unfortunately picks up the double-quotes, so on a CentOS system DISTRO
will be equal to '"centos"'.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: use global compressor if possible

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10969 from xiexingguo/xxg-wip-fix-cache-logger

os/bluestore: replace store with logger in Cache

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

qa/workunits/cephtool/test.sh: fix omision of ceph-command

 - The script fails with these commands, but not for the correct reason.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd-mirror: option to automatically resync after journal client disconnect

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: stop replay when client is disconnected

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: resync was possible only when image replayer start had succeeded

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: decode_client_meta should return false on error

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: new command to disconnect journal client

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: optionally flag "laggy" journal clients disconnected

Fixes: http://tracker.ceph.com/issues/14738
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: allow to trim journal for "laggy" clients

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/journal: add async client_update_state method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9592 from trociny/wip-15632

librbd: API methods to directly acquire and release the exclusive lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: potential double-unwatch of watch handle upon error

Fixes: http://tracker.ceph.com/issues/17210
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9758 from SUSE/wip-btrfs-not-ideal

doc: remove btrfs contradiction

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10469 from wjwithagen/wip-wjw-compileflag

cmake: supress more warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>