test/osd/TestRados.cc: silence a compiler warning

fix following warning by returning `nullptr` after an assert(0).
```
ceph/src/test/osd/TestRados.cc: In member function ‘TestOp*
WeightedTestGenerator::gen_op(RadosTestContext&, TestOpType)’:
ceph/src/test/osd/TestRados.cc:241:3: warning: control reaches
end of non-void function [-Wreturn-type]
   }
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9438 from ceph/wip-cmake

cmake: rbd_mirror not linking rbd_test_mock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9060 from mikulely/fix-rgw-multiste-doc

doc: fix rgw multiste config

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #9392 from yehudasa/wip-16072

rgw/rgw_common.cc: modify the end check in RGWHTTPArgs::sys_get

Tested-by: Orit Wasserman <owasserm@redhat.com>
http://pulpito.ceph.com/owasserm-2016-06-01_20:43:36-rgw:verify-wip-orit-testing---basic-mira/

cmake: rbd_mirror not linking rbd_test_mock

Added rbd_test_mock lib, also changed name of
journal_test to journal_test_mock to mimic
automake naming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9209 from yehudasa/wip-15939

rgw: aws4 fixes

verified

Merge pull request #9084 from dzafman/wip-dz-misc

Wip dz misc

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9426 from linuxbox2/cmake-mds

cmake: change libmds back to a static library

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: restore static linkage (libmds)

Required by ceph-mds.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9385 from ceph/wip-cmake-kefu

cmake: more fixes

fixes make install workflow

Merge pull request #9228 from liewegas/wip-bluestore-write

os/bluestore: new write path (checksums and compression)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9106 from SUSE/wip-15869

rpm: unconditionally set ceph user's primary group to ceph (SUSE)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

cmake: install cython modules

* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
  FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
  with existing targets like "rbd" or "rados". so we can not reuse the
  name in ${name}.pyx. and instead, we should specify the target name
  explicitly.
* add distutils_install_cython_module() function to build and install
  cython modules.
* we can split build and install of cython module, but the install phase
  always tries to build the module. so keep it this way. will look at it
  later on.
* move the variables initializations into the Distutils.cmake module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install compressor plugins into ${pkglibdir/compressor

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install erasure plugins into ${pkglibdir}/erasure-code

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated file from ceph-osd

objclass/class_api.cc is already included in libosd

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix dependencies on tracing headers

group the header dependencies by its tp .so, so the traced target can
depend on them by the name of ${name}-tp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: libradosstriper's OUTPUT_NAME should be radosstriper

and s/libradosstriper/radosstriper/ otherwise the created .so
filename would be liblibradosstriper.so with the default prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install the artifacts the packaging requires

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install init script to etc/init.d

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add the autoconf path variables back

* partially revert 7a602ec.
* the directory variables created by automake, like "prefix", "bindir",
  and "libdir", are used for generating configuration_file() for substitution,
  and they should have the same names with ones from autotools.
* also fix the ${pkglibdir}, it should be the ${libdir}/${PACKAGE}. so
  the plugins are not installed into ${libdir}, in which the installed shared
  objects are supposed to be shared with other applications.
* install shared libraries into ${CMAKE_INSTALL_LIBDIR} instead of
  ${prefix}/lib. this complies to what ceph.spec.in requires:
  ceph.spec.in expects the shared libraries to be installed into
  ${_libdir}, and ${_libdir} is /usr/lib64 on an amd64 machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-post-file

and related pubkey for sftp

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph-brag

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: project name should be "ceph"

so it would be easier to figure out paths, also
CMAKE_INSTALL_DOCDIR is composed using the PROJECT_NAME also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-{detect-init,disk}

add a cmake module named Distutils.cmake for setting up python modules
using setup.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph_rgw_{json,multi}parser

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile and install ceph-bluefs-tool

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/ceph_test_xattr_bench/ceph_xattr_bench/

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename ceph-psim to ceph_psim

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix bash_completion install path

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fsck: check for dup overlay keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: use common helper to verify blobs and refs

The checks are the same (or should be--we had missed a few).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add FLAG_NOCACHE flag; do not cache unbuffered writes

Add a Buffer flag to mark that a buffer should not be cached once it is
stable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not use buffered bdev in write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not rely on bdev buffered reads in read path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: no buffered write in wal path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: populate buffer cache on read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep intrusive_list of WRITING buffers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simple per-collection lru for buffers

Size these using a global config.  This is only a starting point--we'll
obviously have to rework this to share memory across collections.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use bufferptr for csum_data

encode/decode of vector<char> is not optimized.  Bufferptr is a more
natural type here anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objecstore: Adds a test case for compression stuff verification (incomplete)

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes configuration observation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Cosmetic fixes in bluestore logging

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Enables cow for cloning at bluestore for store test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes Bnode serialization/deserialization and removes legacy Bnode::ref_map

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph_test_objectstore: extends SimpleObjectTest with the case where write happens for neighboring csum blocks to verify for potential alignment issue

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Removes legacy block_size retrieval

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use WriteContext and do_alloc_write for _do_write_small

Kill some mostly-duplicated code

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: consolidate WriteContext items into a write_item

Also include b_off in there.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid unnecessary write_onode calls

_wctx_finish callers always write the onode; we only need to worry about
our changes to the bnode.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_* methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_zeros args

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix offset skew check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ~0x -> ~

e.g., 0x432da000~1000 instead of 0x432da000~0x1000

I think it's sufficiently clear that the value after ~ should have the same
base as the first bit, and it's easier to read.  And less text.

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: Extends decompressor interface to be able to provide compressed data length.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: compress on write

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not partially deallocate compressed blobs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write_big: limit size of blobs based on compression mode

We may want to compress in smaller chunks based on hints/policy.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track new compression config options

Class-wide Compressor, compression mode, and options.  For now these are
global, although later we'll do them per-Collection so they can be pool-
specific.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add length to the compression_header_t

Snappy fails to decompress if there are extra zeros in the input buffer.
So, store the length explicitly in the header to avoid feeding them into
the decompressor.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix BufferSpace::read()

- we weren't reading from 'clean' buffers
- restructured loop a bit chasing another bug (but it ended up being
  in the caller)

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add COMPRESSIBLE and INCOMPRESSIBLE alloc hints

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: add a get_type() method to Compressor interface

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read cached vs read result assembly

We weren't handling the case of

 read block 0~300
 cache bloc 100~100

where the result is read(head) + cached + read(tail). Restructure the
loop to handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read read out of buffer cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix up _set_csum helper

- make it thread-safe
- call during mount

Signed-off-by: Sage Weil <sage@redhat.com>

os/store_test: Fixes dump_mismatch_bl to avoid assert on lengths mismatch. Starts using it for BufferCacheTest

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use bdev_block_size instead of min_alloc_size for allocators

min_alloc_size is more dynamic; we just need the block size unit here.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: min_alloc_size options for different media types

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: Fixes duplicate blob move when cloning

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: avoid passing overlapping allocated/released sets to fm

BitmapFreelistManager doesn't like overlapping allocated+released sets
when the debug option is enabled, because it does a read to verify the
op is valid and that may not have been applied to the kv store yet.

This makes bluestore ObjectStore/StoreTest.SimpleCloneTest/2 pass with
bluestore_clone_cow = false and bluestore_freelist_type = bitmap.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitmapFreelistManager: drop newline on hex dumps

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add no-newline hexdump option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitmapFreelistManager: use hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop warning

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix BufferCacheReadTest

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_onode crcs in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove obsolete tail cache

The buffer cache will cover this in a much more general way.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Fixes improper length calculation in BufferSpace::read + adds simplified test case to highlight an issue for append to existing blob

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: drop min_alloc_size locals

We have this in the class, now.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix min_alloc_size global

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: release partial extents

Use the blob put_ref helper so that we can deallocate blobs partially
(instead of always waiting until they are completely unused).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only write into a blob region that is allocated

We're only worried about direct writes and wal overwrites; the other write
paths are to freshly allocated blobs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t: add tracking for released extents

We reference count which parts of the blob are used (by lextents), but
currently we only release our space back to the system when all references
go away.  That is a problem if the blob is large (say, 4MB), and we, say,
truncate off most (but not all) of it.

Unfortunately, we can't simply deallocate anything that doesn't have a
reference, because the logical refs are on byte boundaries, and allocation
happens in larger units (min_alloc_size).  A one byte logical punch_hole
might be responsible for the release of a larger block of storage.

To resolve this, we keep track of which portions of the blob have been
released by poisoning the offset in the extents vector.  We expect that
this vector will be almost always short, so we do not bother with a
indexed structure, since iterating a blob offset to determine if it is
still allocated is likely faster.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add poison offset to pextent_t

This is a "magic" offset that we can use to indicate an invalid extent
(vs, say, an extent at offset 0 that might clobber real data if it were
used).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove dead _txc_release

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only direct write into unused blob space

We can only do a direct write into an already-allocated blob once, if that
range hasn't yet been used.  Once it has been used, it is much to complex
to keep track of when all references to it have committed to disk before
reusing it, so we don't try to handle that case at all.

Since the range has never been used, we can assert that there are no
references to it.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: mark used range on partial blob writes

- writing into unreferenced blob space
- wal blob writes

both need to update the blob used map.  The full blob writes generates
blobs that are always full, so no change is needed there.  New partial
blob creations need to indicate which parts aren't yet used.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add blob_t unused

Keep track of which ranges of this blob have *never* been used.  We do
this as a negative so that the common case of a fully-written blob is an
empty set.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: benchmark different csum methods

crc32c wins on my laptop.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bluestore_types: run csum tests on all algorithms

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t: add xxhash64

Signed-off-by: Sage Weil <sage@redhat.com>

common/Checksummer: add xxhash64

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop old Checksummer

blob_t uses it directly via the static methods.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use blob_t csum methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: simpler {calc,verify}_csum methods

This keeps the CSUM_* definitions local to blob_t, and avoids passing
arguments around.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: defer csum calcuations sometimes

When we are doing a partial chunk overwrite, we need to defer the csum_data
update.  Otherwise, another write in the same transaction might need to
read part of the chunk, not find the data in the buffer cache, read it
from disk, and fail the csum check.

This patch defers the calculation until after we've build the transaction
and are about to commit to the kv store.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/dev/bluestore: update based on Igor's feedback

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Fixes some issues when using Buffer Cache from _do_read and improves test coverage

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes invalid assert in  Buffer::truncate

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>