cmake: add gtest to ceph_test_snap_mapper

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: test_rbd_mirror_image_replay missing ContextCompletion.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: link libradostriper with librados_objs

create an OBJECT target for libradostriper, so it is able to access the
internal symbols of librados.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove visibility flags from libcephfs,libradosstriper

because libradosstriper does not put the `__attribute__ ((visibility
("default")))` specifier on the exported class/methods. we cannot
add the visibility to this library yet. the same applies to libcephfs.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move rados_snap_set_diff_obj into librbd_internal

ceph_test_librbd use it and links directly against librbd_internal
instead of against librbd

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9207 from rjfd/wip-15670

rbd-mirror: image resync

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9626 from liewegas/wip-bluestore-2q

os/bluestore: implement 2Q buffer cache policy

ceph_test_objectstore: define matrixes over config options and write sizes

These replace a few of the ad hoc synthetic tests.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: handle bnode removal race

One thread might try to drop a bnode while another does a lookup and
retakes a ref:

 1- drop bnode ref (-> 0)
 2- take bnode_set lock, do lookup
 2- take bnode ref (-> 1)
 2- drop lock
 1- take bnode_set lock
 1- remove from set
 1- drop lock

Fix this race by rechecking the ref count while we are holding the lock.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "msg: default ms type = async"

Not ready quite yet, causing too many test failures.

This reverts commit 928887a09c2cb9944531165fcf4aa9026b49a3d9.
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9451 from s09816/master

librbd: enable/disable of features is not allowed when already enabled/disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9715 from trociny/wip-16321

librbd: re-register watch on old format image rename

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9874 from athanatos/wip-16185

OSDMonitor::prepare_pgtemp: only update up_thru if newer

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9875 from athanatos/wip-16127

OSDMonitor: drop pg temp from sources other than the current primary

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9850 from dillaman/wip-16404

librbd: flag image as updated after proxying maintenance op

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8370 from jack-changtao/pg_del

osd/PG: cleanup: delete find_best_info again

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9802 from yuyuyu101/wip-16378

msg/async: fix test stuck into STATE_WAIT

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8195 from tchaikov/wip-cmake-visibility

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

os/bluestore: drop unneeded conditional in LRUCache trim

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: protect BnodeSet with mutex

The onodes (and thus bnodes) may not get destroyed with the collection
lock held, which means we cannot rely on that lock to protected the
bnodeset structure.  Give BnodeSet it's own mutex to protect itself.

While we are at it, make dummy a member so we can avoid constructing it
each time.  This is now safe since we are under the new lock.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove bluestore_cache_tails

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix 8-bit csum

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: default to 512MB cache, buffered reads

Test with small cache to exercise trimming.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: 2q by default

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: implement 2Q buffer cache policy

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move merge operator

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: buffer cache_private field, returned by discard

2Q (and likly MQ?) need to know when we re-add a block that was previously
evicted.  We track these as STATE_EMPTY buffers, but we throw them out
with _discard before we add the (re)read data into the BufferSpace. Add
a Buffer field called cache_private that is returned by discard (actually,
a max over everything that is discarded) that a cache implementation can
use to discover it has re-read a recently evicted item.

The Cache implemention needs to set cache_private values such that the MAX
over them is what it actually wants...

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: include buffer * in operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cache: simplify lru trim

Everything in the LRU is clean.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make Cache implementation configurable

For now we only do "lru" (LRUCache).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cache: define 'empty' buffer

This is a buffer with no data.  It's a placeholder that is there to be used
by cache replacement algorithms like 2Q and MQ that needs history for
trimmed buffers.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cache: keep writing buffers out of LRU

Don't bother putting a writing buffer in the LRU until it is clean (and
trimmable).  Otherwise, we end up with writing items near the end of the
LRU that we have to iterate over on every trim() invocation.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clean up Cache::trim

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cache: control insert point of new onodes

Not used yet.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cache: control insertion point of new buffers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make buffer lru private

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make onode lru private

Add cache->onode_lru.push_back(*po->second).  Make LRU private.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix locking for _dump_blob_map

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: use ${CEPH_SHARED} when adding libradosstriper

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9827 from liewegas/wip-bluestore-invalidate

os/bluestore: discard unreferenced buffers as they are overwritten

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9868 from liewegas/wip-bluestore-lba

include/small_encoding: fix lba encoding with many low zeros

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9863 from trociny/fixup-comp-warn

test: fix compilation warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9821 from dx9/wip-ac-alpine

configure.ac: add --with-reentrant-strsignal and --with-thread-safe-r…

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: discard unreferenced buffers as they are overwritten

Signed-off-by: Sage Weil <sage@redhat.com>

include/small_encoding: fix lba encoding with many low zeros

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: recover lock to continue later

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9865 from liewegas/wip-bluestore-encoding

os/bluestore: fix blob_t encoding test cases, and clean up initialization

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

configure.ac: add --with-reentrant-strsignal and --with-thread-safe-res-query options for Alpine support

Signed-off-by: John Coyle <dx9err@gmail.com>

test: fix compilation warnings

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

os/bluestore/bluestore_types: fix blob initializers, test instances

- initialize members in c++11 style
- drop pextent ctor
- fix up test instances
- make csum_order default to 0, so it matches the
  CSUM_NONE decoding logic.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: drop unused var

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9813 from jcsp/wip-kernel-doc

doc/cephfs: add advice about kernel versions

Merge pull request #9823 from dillaman/wip-16362

rbd-mirror: keep events from different epochs independent

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

cmake: include Group.cc in rbd cli

to match with automake, and the "group" command is missing in the cmake
build of "rbd" cli tool.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install dso into ${CMAKE_INSTALL_LIBDIR} instead of "lib"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add visibility flags for exported dso

* no regression found, and there's a substantial reduction in the size of
  generated librados library (both stripped and not).
* also remove the duplicated set_target_properties(librbd...) command

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move snap_set_diff.cc out of librados

snap_set_diff.cc is used internally and is not exposed via librados. we
cannot use the function implemented in it by linking against librados. so
compile it as an OBJECT target and link against it individualy if
necessary.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge branch 'wip-dmick-install-deps-master'

install-deps.sh: use mk-build-deps instead of processing control

Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

install-deps.sh: use mk-build-deps instead of processing control

mk-build-deps is designed to handle installing build dependencies;
use that instead, so '|' indications are handled properly.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #9791 from ceph/wip-cmake-compile-flags

cmake: Wip cmake compile flags

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9481 from yehudasa/wip-rgw-hold-lease-1

rgw: stop bucket lease only after draining object sync operations

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: CMAKE_BUILD_TYPE defaulted to RelWithDebInfo

Default build type is RelWithDebInfo unless
CMAKE_BUILD_TYPE is specified when running the
`cmake` command.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9066 from yangdongsheng/rbd_bench

rbd: introduce a simple bench for read

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: flag image as updated after proxying maintenance op

Fixes: http://tracker.ceph.com/issues/16404
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cmake: eliminate FORTIFY_SOURCE warnings

"-werr" was added to CHECK_C_COMPILER_FLAG to
make sure FORTIFY_SOURCE is only turned on if
the flag generates no warnings.

Added "-U_FORTIFY_SOURCE" to CMAKE_C_FLAGS to
mimic PR #9761.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9728 from liewegas/wip-bluestore-encoding

os/bluestore: space efficient int encoding

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

rbd-mirror: keep events from different epochs independent

Fixes: http://tracker.ceph.com/issues/16362
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore/bluestore_types: delta encode for ref_map offsets

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make blob csum info encoding conditional

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use FLAG_CSUM to indicate if checksums are present

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: print intervals in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use varint for pextent count

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: varint or lba encode pextent, lextent, ref_map, onode

Certainly not done here, but this is a start.

Signed-off-by: Sage Weil <sage@redhat.com>

include/small_encoding: small encode/decode helpers

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9819 from dillaman/wip-16363

librbd: fix lockdep issue when duplicate event detected

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9816 from dillaman/wip-16198

librbd: potential race when replaying journal ops

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9788 from dillaman/wip-16230

rbd-mirror: potential race condition accessing local image journal

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9787 from batrick/mds-rename-enotempty

mds: improve comments for ENOTEMPTY checks

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9840 from xiexingguo/xxg-wip-fix-bmap-fm

os/bluestore: minor improvements for bitmap-freelist-manager

Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/AsyncConnection: shorter handshake step deps

Originally connect side will wait for accept side sending banner, actually
we don't need this logic to ensure anything.

So connect side will issue banner after tcp handshake straightway. It will
help for reducing session handshake period, then it avoid peer side drop this
connection in low level but connect side don't know right now.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9815 from xiexingguo/xxg-wip-fix-blue-cache

os/bluestore: cache relevant cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: make bytes_per_key a global member

So we don't have to figure it out everywhere.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix a typo(nonexistant->nonexistent)

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify BufferSpace::read() a little

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify BufferSpace::read() a little

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: change type of length of region_t from uint32_t to uint64_t

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check when try to rm_buffer

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9828 from liewegas/wip-bluestore-read-before-write

os/bluestore: limit read-amp on overwrite

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #9826 from liewegas/wip-bluefs

os/bluestore/BlueFS: instrument which file types we write to

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #9834 from tchaikov/wip-silence-warnings

test, bluestore: silence warnings

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #9635 from liewegas/wip-make-dist

make-dist: include src/.git_version and src/ceph_ver.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9836 from dx9/wip-hardcoded-tcmalloc

cmake: fix hard coded tcmalloc lib

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix hard coded tcmalloc lib

Signed-off-by: John Coyle <dx9err@gmail.com>

Merge pull request #9670 from ceph/wip-debian-clean

debian: silence couple warnings

Reviewed-by: Dan Mick <dmick@redhat.com>

bluestore/BitAllocator: silence warnings of method hidden

is_allocated(int64_t *blocks, int64_t num_blocks, int blk_off) only gets
called in BitMapAreaIN and its child classes. so it's safe to move this
method down to BitMapAreaIN.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph_test_objectstore: remove unused function

its caller was removed in 8a42e61

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9832 from ceph/wip-cmake-librbd-unittest-fix

cmake: unittest_librbd missing a source file

cmake: unittest_librbd missing a source file

src/test/librbd/test_ConsistencyGroups.cc missing
from ${unittest_librbd_srcs}.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9531 from yehudasa/wip-16175

rgw: data sync debug logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9676 from efirs/fix_bitmap_freelist

os/bluestore: Fix enumeration when only single key in BitmapFreelistManager