os/bluestore: encode shard id in single byte

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix key+name sort wonkiness

We want to unconditoinally start with the name or key, *then* do < > or =
(and if not =, a trailing name).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unnecessary separator between fields

We have fixed length/order for integer fields and use !
to terminate string fields, so there is no need to use
any extra separators, which is simpler as well as faster.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove never reachable asserts

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: don't dirty onode if its size is already at desired offset

for truncate operation.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11008 from liewegas/wip-libaio

test/objectstore/CMakeLists.txt: fix libaio conditional

test/objectstore/CMakeLists.txt: fix libaio conditional

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10963 from liewegas/wip-bluestore-sharded-extent-map

os/bluestore: shard extent map

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

os/bluestore: assert shared blob cache cleared on split

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump some stats after fsck

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument onode reshard events

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument transaction count

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: instrument big/small writes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make blob_t unused helpers use logical length

These were taking min_alloc_size, but this can change
across mounts; better to use the logical blob length
instead (that's what we want anyway!).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use block_size for allocator unit

We need to handle objects written during previous mounts
that may have had a smaller min_alloc_size.  Use
block_size, which is a safe lower bound.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix fsck used_block bitmap

This has to be block_size bits because min_alloc_size
can vary over mounts.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: optimize compress_extent_map

Only examine the range we just wrote to (and to the left
and right).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make nid and blobid allocation less racy

We could bump the _max value for a TransContext in it's
prepare state, have it wait for a long time on IO, and
let another txc allocate and commit something with
an id higher than the previous max.

Fix this first by pushing the max ids into the
TransContext where we can deal with them at commit time,
and then making _kv_sync_thread bump the committed
max in a safe way.

Note that this will need to change if/when we do
these commits in parallel.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: shard extent map

Rewrote much of the persistence of onode metadata.  The
highlights:

 - extents and blobs stored together (the blob with the
   first referencing extent).
 - extents sharded across multiple k/v keys
 - if a blob if referenced from multiple blobs, it's
   stored in the onode key (called a "spanning blob").
 - when we clone a blob we copy the metadata, but mark
   it shared and put (just) the ref_map on the underlying
   blocks in a shared_blob key.  at this point we also
   assign a globally unique id (sbid = shared blob id)
   so the key has a unique name.
 - we instantiate a SharedBlob in memory regardless of
   whether we need to load the ref_map (which is only
   needed for deallocations!).  the BufferSpace is
   attached to this SharedBlob so we get unified caching
   across clones.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10973 from wjwithagen/wip-wjw-freebsd-cmake-excludes-2

cmake: FreeBSD specific excludes in CMakeLists.txt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10996 from jcsp/wip-16973

mds: log path with CDir damage messages

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10928 from stiopaa1/rbd_mirror_imagesyncthrottler

rbd_mirror/ImageSynceThrottler: move struct to .cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10947 from atheism/radosgw-admin-help

rgw: right parenthesis is missing in radosgw-admin help message on caps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10954 from stiopaa1/rgw_rgwrados_removeUnneededClassCTick

rgw/rgw_rados.h: remove unneeded class C_Tick

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10967 from stiopaa1/osd_replicatedbackend_moveClasses

osd/ReplicatedBackend: move classes to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10971 from stiopaa1/osd_replicatedpg_moveContextClasses

osd/ReplicatedPG: move classes to .cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10979 from wjwithagen/wip-wjw-cephtool-test

qa/workunits/cephtool/test.sh: fix omission of ceph-command

Merge pull request #10988 from jazeltq/fix-compile-error

autogen: Fix rocksdb error when make dist

mds: log path with CDir damage messages

Previously you just got the inode number, which
wasn't terribly useful for e.g. a missing fragment
object, as you couldn't readily resolve the parent
path.

Fixes: http://tracker.ceph.com/issues/16973
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10930 from atheism/rgw-lc-fix-sleep-time

rgw/rgw_lc.cc: fix sleep time according to the error message

src/test/objectstore/CMakeLists.txt: conditionaly build tests

 - Honor WITH_AIO tests that depend on bluestore parts

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/tools/CMakeLists.txt: conditionaly build ceph-client-debug

 - Honor WITH_LIBCEPHFS

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/pybind/CMakeLists.txt: conditionaly build cython tools

 - honor the WITH_RBD and WITH_LIBCEPHFS flags

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/CMakeLists.txt: exclude parts not buildable on FreeBSD

 - FreeBSD does not have keyutils or any kernel type security store
   So using secret.c is not really relevant here
 - Have common/secret.c depend on KEYUTILS_FOUND
 - This also excludes building mount.cephfs,
   Next to the fact that this is 100% linux bases mount stuff
 - krbd is the Linux kernel version for RBD based disks, but again
   completely Linux based and porting to FreeBSD, if any, has to follow
   a different route

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Makfile: Fix rocksdb error when make dist

Rocksdb delete some files, so we should change another way
to get rocksdb's files.

Currently, if we do not run configure in googletest/googlemock,
we also will get a error when make dist. So fix it too.

Signed-off-by: tianqing <tianqing@unitedstack.com>

Merge pull request #10972 from wjwithagen/wip-wjw-cmake-resolv

CmakeLists.txt: use LIB_RESOLV instead of resolv.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10378 from trociny/wip-14738

librbd: optionally unregister "laggy" journal clients

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph_test_objectstore: test shards for longer

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: occasional umount/fsck/mount

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: add SyntheticMatrixSharding

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10870 from ceph/wip_build_cls_rbd

cmake: add dependency from ceph_smalliobenchrbd to cls libraries

Reviewed-by: Jason Dillaman <jdillama@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10912 from stiopaa1/journal_journaltimer_moveCRemoveSet

journal: move JournalTrimmer::C_RemoveSet struct

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Add dependencies from ceph_smalliobenchrbd on cls_rbd, cls_journal,
and cls_lock.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #10984 from ceph/wip-kfsx-exclusive-lock

test/librbd/fsx: enable exclusive-lock feature in krbd mode

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10964 from liewegas/wip-bluefs-compaction

os/bluestore/BlueFS: fix async compaction logging bug

Merge pull request #10966 from liewegas/wip-bluefs-runway

os/bluestore/BlueFS: factor unflushed log into runway calculation

Reviewed-by: Varada Kari <varada.kari@sandisk.com>
Reviewed-by: Somnath Roy <somnath.roy@sandisk.com>

Merge pull request #10970 from xiexingguo/xxg-wip-honour-alloweio-flag

os/bluestore: honour allow-eio flag; use global compressor if possible

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10974 from dillaman/wip-17210

librbd: potential double-unwatch of watch handle upon error

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mds: s/used_ions/used_inos/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10774 from renhwztetecs/renhw-wip-cephfs-python-umount

libcephfs: add unmount function in cephfs.pyx

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #10914 from batrick/mds-no-maxmds

mds: remove max_mds config option

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10922 from ceph/wip-jlayton-statx

First pile of statx patches

test/cephfs: add mount&&unmount test

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #10981 from SUSE/wip-fix-install-deps

build/ops: fix /etc/os-release parsing in install-deps.sh

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

build/ops: fix /etc/os-release parsing in install-deps.sh

85a370e35fc42031a7f7e24dea9e50a649c0f309 introduced the DISTRO variable whose
value was obtained by parsing /etc/os-release like this:

DISTRO=$(grep  "^ID=" /etc/os-release | sed "s/ID=//")

This unfortunately picks up the double-quotes, so on a CentOS system DISTRO
will be equal to '"centos"'.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore: use global compressor if possible

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10969 from xiexingguo/xxg-wip-fix-cache-logger

os/bluestore: replace store with logger in Cache

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

qa/workunits/cephtool/test.sh: fix omision of ceph-command

 - The script fails with these commands, but not for the correct reason.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rbd-mirror: option to automatically resync after journal client disconnect

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: stop replay when client is disconnected

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: resync was possible only when image replayer start had succeeded

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: decode_client_meta should return false on error

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd: new command to disconnect journal client

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: optionally flag "laggy" journal clients disconnected

Fixes: http://tracker.ceph.com/issues/14738
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

journal: allow to trim journal for "laggy" clients

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls/journal: add async client_update_state method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9592 from trociny/wip-15632

librbd: API methods to directly acquire and release the exclusive lock

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: potential double-unwatch of watch handle upon error

Fixes: http://tracker.ceph.com/issues/17210
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9758 from SUSE/wip-btrfs-not-ideal

doc: remove btrfs contradiction

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10469 from wjwithagen/wip-wjw-compileflag

cmake: supress more warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: move classes to .cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

CmakeLists.txt: use LIB_RESOLV instead of resolv.

 - FreeBSD does not have a separate libresolv, but keeps it in
   libc

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: honour allow-eio flag during read

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: check iterator before using during fsck

Also drop redundant checking.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: replace store with logger in Cache

As what we really want is to log cache-related statistics.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/ReplicatedBackend: move classes to cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore/BlueFS: factor unflushed log into runway calculation

Include the size of the in-memory buffer we haven't yet flushed
into the runway calculation when deciding whether to allocate
more log space.

Reported-by: Somnath Roy <somnath.roy@sandisk.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10945 from dillaman/wip-17188

librbd: deadlock when replaying journal during image open

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

SQUASH: client: request all shared caps if ctime was requested

Much like the change_attr, ctime changes can potentially be cached on
clients. Request all shared caps if the want mask specifies the ctime.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #10746 from ceph/wip-async-default

msg: make async backend default

scripts/bdev_grep: parse bluefs style extents too

(they use + instead of ~)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: fix async compaction txn

Our transaction writes are labeled with a seq and uuid
to avoid replaying over garbage.

Two bugs, one real, one potential.

1) The second async compaction transactoin didn't have
its seq and uuid set, so replay always stopped.

2) We were writing two separate transactions, one with
all the new metadata, and the next one with a jump to
the new log offset.  If the first write completed but
it was torn and the second transaction didn't hit disk,
we might see an old transaction with seq == 2 and the
same uuid and replay that instead.

Fix both of these by making the async log txn one single
transaction that jumps directly to the new log offset.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10938 from liewegas/wip-bluestore-csum-error

os/bluestore: fix/improve csum error message

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #10961 from xiexingguo/xxg-wip-bluestore-09-02

os/bluestore: add cache-related stats

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10953 from liewegas/wip-bluestore-fixes

os/bluestore: misc fixes

Reviewed- by: Igor Fedotov ifedotov@mirantis.com

test/librbd/fsx: enable exclusive-lock feature in krbd mode

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

SQUASH: mds/client: add routines to copy to/from ceph_mds_request_head_legacy

Signed-off-by: Jeff Layton <jlayton@redhat.com>

os/bluestore: trivial cleanups

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: don't overwrite type if it does already exist and at desired type

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

src/CMakeLists.txt: -Woverloaded-virtual is already included in -Wall on Clang

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10956 from ceph/wip-cmake

cmake: pass -DINTEL* to gf-complete cflags

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: conditional checking label of bluefs devices

As NVMe devices may not support to do so.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

cmake: pass -DINTEL* to gf-complete cflags

* in addition to the -march=foobar, we also need to pass -DFOOBAR to
  gf-complete sources' cflags, so that the SIMD instructions can be
  generated as expected.
* also extract the compiler support for instruction extensions detections
  into SIMDExt.cmake.
* and only check the compiler support if CMAKE_SYSTEM_PROCESSOR matches
  with the instruction-set/arch to be checked.

Signed-off-by: tone.zhang <tone.zhang@linaro.org>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10934 from tchaikov/wip-kill-warnings

Wip kill warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

os/bluestore: end scope of std::hex properly

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add onode/buffer cache statistics

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

SQUASH: client: querying for stx_version field requires getting all shared caps

Otherwise, someone could potentially query for just CEPH_STATX_VERSION,
and see it as unchanged, even when there are changes buffered up on
other clients.

By doing this, I don't think we'll incur any perf hit in the common use
case which is ganesha querying for all attributes. We are adding Xs
here, but unless there is a lot of xattr activity I don't think that
will generally cause a lot of exclusive cap recalls.

Also, we don't actually need AUTH caps to fetch CEPH_STATX_RDEV, remove
that from statx_to_mask. PIN is sufficient there.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

rgw/rgw_rados.h: remove unneeded class C_Tick

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

SQUASH: mds/client: don't get clever with ceph_mds_request_head_legacy encode/decode

As requested by Greg...

Declare a legacy object on the stack, and do an extra copy to/from it.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

SQUASH: client: always take a change_attr update from the server if it's newer

I think this is more correct in the client. If we see a newer
change_attr, then we always want to take note of that fact, even if
we have the right caps.

Signed-off-by: Jeff Layton <jlayton@redhat.com>