rbd-mirror: ensure replay status formatter has completed before stopping

Fixes: http://tracker.ceph.com/issues/16352
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9753 from liewegas/wip-readme

README: update with vstart, test, and make check instructions

Reviewed-by: Ali Maredia <amaredia@redhat.com>

README.md: note where make check test log output goes

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: document running a test cluster

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: simplify cmake instructions

Assume build is subdir of the ceph.git checkout.

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: drop obsolete comment about debian squeeze

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: get source first (including submodules!)

Signed-off-by: Sage Weil <sage@redhat.com>

README.md: document how to run unit tests (with autotools)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9151 from fullerdj/wip-djf-15706

rbd: add methods to set and get snapshot limits

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9292 from marcan/pyrbd-aio

pybind: AIO bindings for RBD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9333 from VictorDenisov/consistency_groups_only_crud

rbd: initial operations for consistency groups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9535 from xinxinsh/wip-bug

librbd: object_may_exist always return true when you write an empty object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9682 from yuyuyu101/wip-fix-task-finisher

librbd: cancel all tasks should wait until finisher is done

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9567 from liewegas/wip-bluestore-blobcache

os/bluestore: convert cache to be blob-based (vs onode-based)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: include device offsets in csum error message

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove Onode's BufferSpace

(now unused)

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: change _do_read to use blob cache

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9748 from Yan-waller/yj-wip-rgwstoreuserinfo

rgw/user: remove needless bufferlist encoding

build verfiied, there can be no side effect

Merge pull request #9720 from dillaman/wip-16202

qa/workunits: rados cppool now requires --yes-i-really-really-mean-it

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw/user: remove needless bufferlist encoding

info encoded but never be used.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

mailmap: Victor Denisov affiliation

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Add cg create, cg list, cg remove commands

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Add cg_create, cg_list, cg_remove

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Add cg_create, cg_dir_add, cg_dir_remove, cg_dir_list

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

Merge pull request #9708 from tchaikov/wip-make_rocksdb_makefile-without-rdb

script/make_rocksdb_makefile.sh: do not include rocksdb/tools/rdb/*

Reviewed-by: Dan Mick <dmick@redhat.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

Merge pull request #9713 from xiexingguo/xxg-wip-server_droppin

server: drop locks and auth pins before waiting for trunc

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9710 from xiexingguo/xxg-wip-server_duplock

Server: avoid duplicated call to acquire_locks()

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9706 from xiexingguo/xxg-wip-server_dcr

mds/server: avoid side-effect of more() method

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9647 from xiexingguo/xxg-wip-server_hcr

server: fix potential access violation

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8737 from david-z/wip-mds-slow-req-output

mds: ceph status outputs mds slow request for better monitoring

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

os/bluestore: inline and simplify _read_whole_blob

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use _dump_blob_map helper

Unify bnode and onode dump output.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do writes into blob buffer cache

No change to the read-side yet.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: use single bl_eq helper

Use a single bl comparator helper that dumps buffers if they do not match.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: clean up csum read error checks

Fix warning message and return error code.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use Blob* in WriteContext::write_item

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: switch to intrusive_set for blobs

Switch from a map of bare bluestore_blob_t's to an intrusive_set of
Blob structs that includes id, bluestore_blob_t, and BufferSpace.  The
buffer cache portion isn't used yet, but it is properly initialized with
the Cache parent.

There is no real functional change here except that during clone we move
the existing Blob from the Onode's BlobMap to the Bnode's without any
copying.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: move blob_map out of onode_t

Move the blob_map index out of the onode_t proper.  None of the (important)
onode_t methods use the blob_map, which is expected since blobs may exist
in the onode or bnode map.  Instead, move it into Onode, parallel the
Bnode counterpart.

This will allow us to create a blob map that includes the buffer cache
with an encode/decode and lifecycle independent of the onode_t.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9526 from liewegas/wip-bluestore-csum2

os/bluestore: vary csum chunk sizes based on hints

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add crc32c_16 and crc32c_8

This is much faster than a slice-by-8 crc16, perhaps even without the
intel instructions.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/snappy: style cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/snappy: compress to page-aligned memory buffer

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove unused ext_offset region_t field

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify spare read path, remove pextent constraint

The previous read code had the constraint that a physical extent had to
be a multiple of the csum chunk size.  This isn't needed: we might have
a csum_block of 1MB and min_alloc_size of 4KB and that's okay.

Collapse the two helpers into a single loop that uses the blob_t::map()
method to do the pextent part of the read.  This is simpler and avoids
the temporary extents2read structure.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: csum_block -> csum_chunk

Be consistent with terminology.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ensure blob_t::put_ref respects csum chunk size

We can't deallocate part of a csum chunk.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test larger writes from Synthetic

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: randomly vary object size and write size hints

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: choose better csum_order in _do_alloc_write

Try to use the wctx hint, but set a floor based on the buffer length.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make preferred csum order a function of expected_write_size

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use larger csum blocks for sequential writes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_alloc_write compress condition

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: set csum_order via WriteContext

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: maintain min_alloc_size_order

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: test alloc hints

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: print alloc hint flags as a string

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add ceph_osd_alloc_hint_flag_string helper

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _setallochint -> _set_alloc_hint

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not print bnode key to debug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9607 from ifed01/wip-bluestore-statfs-fsck

os/bluestore: add statfs result consistency checking to fsck

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9524 from liewegas/wip-ctz

common: add cbits/ctz/clz bit op helpers, replace open-coded calc_bits_for helpers

Reviewed-by: Samuel Just <sjust@redhat.com>

os/bluestore/StupidAllocator: use cbits instead of calculating shift manually

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: pg_pool_t::calc_bits_of -> cbits

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9696 from ifed01/wip-bluestore-csum-fixes

os/bluestore: fix for unaligned writes and improve test coverage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9604 from ifed01/wip-bluestore-maxalloc

os/bluestore: add an option to limit max allocation size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9378 from xiexingguo/xxg-wip-fix-piderrorcode

global: negative error return code for pidfh::write()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9397 from xiexingguo/xxg-wip-fixsocketfd-leak

common/admin_socket: fix socket fd leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8679 from prallabh/master

rgw: Have a flavor of bucket deletion to bypass GC.

it gets the OSDs substantially hotter (CPU), but is a large speedup (I measured 54% on a small setup, some distortion from debugging)

TaskFinisher: cancel all tasks wait until finisher done

Otherwise, caller may think pending task won't be executed but actually
finisher may execute callback which may cause refer to freed object.

Signed-off-by: Haomai Wang <haomai@xsky.com>

os/bluestore: fix exceeding blob number after random 4K writes

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/store_test: add an additional bluestore test case to verify for exceeding blob number

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9666 from ceph/wip-ceph_test_objectstore

test: ceph_test_objectstore: do not override plugin-dir if not necessary

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

vstart.sh: set plugin-dir for cmake

test: ceph_test_objectstore: do not override plugin-dir if not necessary

there is chance that we launch the test with the plugins installed in
`/usr/${lib}/ceph'. and we don't have ".lib" or $CEPH_LIB for
ceph_test_objectstore, in this case, we should leave plugin-dir
unchanged. it will work just fine if ceph-base or ceph-common is
installed.

Fixes: http://tracker.ceph.com/issues/16254
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits: rados cppool now requires --yes-i-really-really-mean-it

Fixes: http://tracker.ceph.com/issues/16202
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9709 from trociny/wip-16289

qa/workunits/rbd: respect RBD_CREATE_ARGS environment variable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: fixes assert on unaligned writes when csum verification is disabled

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9442 from houlinfei/wip-modify-parlistseq

ceph-disk: modify the disk partition sequence of ceph-disk list

Reviewed-by: Loic Dachary <ldachary@redhat.com>

objectstore/store_test: improves test coverage for Bluestore - adds no csum and many 4K writes cases. Fix corresponding issues in testing framework.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9354 from dreamhost/wip-15975

rgw: Fallback to Host header for bucket name.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9681 from xiexingguo/xxg-wip-server_ts

server: set terminating_sessions flag correctly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9693 from xiexingguo/xxg-wip-server_hco2

server: negative error code when responding to client

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9472 from xiexingguo/xxg-wip-fix-divisionbyzero

mds: fix potential division-by-zero error

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9445 from xiexingguo/xxg-wip-fix-mdsmemoryleak

mds: avoid duplicated call of context; fix potential memory leak

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9494 from renhwztetecs/renhw-wip-cdir-total

mds/cdir: fix the type of dentry in the judgment; cleanup undefined functions

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9649 from xiexingguo/xxg-wip-server_hco

server: return after respond_to_request() for EROFS

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8218 from jupiturliu/kstore_r/w_flush

os/kstore: we have flush_txns is to make sure read after write

Reviewed-by: Sage Weil <sage@redhat.com>

server: drop locks and auth pins before waiting for trunc

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9639 from jjhuo/rocksdb-directio

os/bluestore: Add an option to choose bluefs direct or buffered io mode

Reviewed-by: Sage Weil <sage@redhat.com>

doc/release-notes: v10.2.2

Signed-off-by: Sage Weil <sage@redhat.com>

Server: avoid duplicated call to acquire_locks()

For snapped case we may call acquire_locks() twice,
which is not necessary.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: respect RBD_CREATE_ARGS environment variable

Fixes: http://tracker.ceph.com/issues/16289
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Server: use req directly

We have req pointed to mdr->client_request already, so use
it directly to keep the bothering of a long dereference series of
pointer(which also results in a longer line) away.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds/Mutation: avoid side-effect of slave_did_prepare() method

We shall check _more does exist first.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds/Mutation: use c++11-style null pointer check

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Ceph status outputs mds slow request for better monitoring

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

script/make_rocksdb_makefile.sh: do not include rocksdb/tools/rdb/*

Fixes: http://tracker.ceph.com/issues/13554
Signed-off-by: Kefu Chai <kchai@redhat.com>

server: avoid side-effect of assert

The more() method will implicitly allocate a new internal _more memeber
if it does not have and we shall avoid this.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9697 from jcsp/wip-16298

mds: fix MDLog recovery rare case