librbd: object-map rebuild fail, add error information

Signed-off-by: Na Xie <xie.na@h3c.com>

Merge pull request #9349 from Adirl/fix_compiltaion_entity_addr_t4

msg: add const overload to entity_addr_t functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9295 from cbodley/wip-rgw-rados-vector

rgw: use vector for librados handles

Merge pull request #9454 from tanghaodong25/bluestore-write-path

os/bluestore: no 'head_read' when check ondisk length.

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: no 'head_read' when check ondisk length.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #9458 from jimwright/master

librados: Added declaration for rados_aio_get_version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9462 from athanatos/wip-15952

osd/ReplicatedPG: adjust num_pinned in _delete_oid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9236 from athanatos/wip-15943

osd/PG: update PGPool to detect map gaps and reset cached_removed_snaps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9460 from pritha-srivastava/wip-cmake-vstart

cmake: detection scripts use lowercase project name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: detection scripts use lowercase project name

Changing the detection scripts to use the new project name, changed in commit
c1a37101ea87a973068f2dc9858487f83ae57f88 (cmake: project name should be "ceph")

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

librados: Added declaration for rados_aio_get_version

Was missing from librados.h and hence had an incorrect
visability.

Fixes: #15535

Merge pull request #9449 from ifed01/wip-bluestore-crc32

os/bluestore: Some improvement for crc32c calculations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9448 from tchaikov/wip-fix-warning

test/osd/TestRados.cc: silence a compiler warning

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rados/loadgen-mostly-read.sh: half as much data (512mb)

Use less space so we can run on openstack.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9037 from rzarzynski/wip-rgw-swift-quota

rgw: add support for account and container quotas of Swift API
Reviewed-by" Orit Wasserman <owasserm@redhat.com>

os/bluestore: dec->hex for verify_csum error reporting

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

common/Checksummer: Eliminates data copying in crc32c calculation

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9367 from Yan-waller/yj-wip-rgwfixhextobuf

rgw/common: fix error return value of hex_to_buf.
Reviewed by: Orit Wasserman <owasserm@redhat.com>

common/bufferlist: Adds test cases for bufferlist::iterator::crc32c

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

common/bufferlist: Fixes lack of offset increment in get_ptr_and_advance

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

buffer: add iterator crc32c method

Calculate crc from an iterator position.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/TestRados.cc: silence a compiler warning

fix following warning by returning `nullptr` after an assert(0).
```
ceph/src/test/osd/TestRados.cc: In member function ‘TestOp*
WeightedTestGenerator::gen_op(RadosTestContext&, TestOpType)’:
ceph/src/test/osd/TestRados.cc:241:3: warning: control reaches
end of non-void function [-Wreturn-type]
   }
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9438 from ceph/wip-cmake

cmake: rbd_mirror not linking rbd_test_mock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9060 from mikulely/fix-rgw-multiste-doc

doc: fix rgw multiste config

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #9392 from yehudasa/wip-16072

rgw/rgw_common.cc: modify the end check in RGWHTTPArgs::sys_get

Tested-by: Orit Wasserman <owasserm@redhat.com>
http://pulpito.ceph.com/owasserm-2016-06-01_20:43:36-rgw:verify-wip-orit-testing---basic-mira/

cmake: rbd_mirror not linking rbd_test_mock

Added rbd_test_mock lib, also changed name of
journal_test to journal_test_mock to mimic
automake naming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9209 from yehudasa/wip-15939

rgw: aws4 fixes

verified

Merge pull request #9084 from dzafman/wip-dz-misc

Wip dz misc

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9426 from linuxbox2/cmake-mds

cmake: change libmds back to a static library

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: restore static linkage (libmds)

Required by ceph-mds.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9385 from ceph/wip-cmake-kefu

cmake: more fixes

fixes make install workflow

Merge pull request #9228 from liewegas/wip-bluestore-write

os/bluestore: new write path (checksums and compression)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9106 from SUSE/wip-15869

rpm: unconditionally set ceph user's primary group to ceph (SUSE)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

cmake: install cython modules

* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
  FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
  with existing targets like "rbd" or "rados". so we can not reuse the
  name in ${name}.pyx. and instead, we should specify the target name
  explicitly.
* add distutils_install_cython_module() function to build and install
  cython modules.
* we can split build and install of cython module, but the install phase
  always tries to build the module. so keep it this way. will look at it
  later on.
* move the variables initializations into the Distutils.cmake module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install compressor plugins into ${pkglibdir/compressor

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install erasure plugins into ${pkglibdir}/erasure-code

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated file from ceph-osd

objclass/class_api.cc is already included in libosd

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix dependencies on tracing headers

group the header dependencies by its tp .so, so the traced target can
depend on them by the name of ${name}-tp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: libradosstriper's OUTPUT_NAME should be radosstriper

and s/libradosstriper/radosstriper/ otherwise the created .so
filename would be liblibradosstriper.so with the default prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install the artifacts the packaging requires

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install init script to etc/init.d

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add the autoconf path variables back

* partially revert 7a602ec.
* the directory variables created by automake, like "prefix", "bindir",
  and "libdir", are used for generating configuration_file() for substitution,
  and they should have the same names with ones from autotools.
* also fix the ${pkglibdir}, it should be the ${libdir}/${PACKAGE}. so
  the plugins are not installed into ${libdir}, in which the installed shared
  objects are supposed to be shared with other applications.
* install shared libraries into ${CMAKE_INSTALL_LIBDIR} instead of
  ${prefix}/lib. this complies to what ceph.spec.in requires:
  ceph.spec.in expects the shared libraries to be installed into
  ${_libdir}, and ${_libdir} is /usr/lib64 on an amd64 machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-post-file

and related pubkey for sftp

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph-brag

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: project name should be "ceph"

so it would be easier to figure out paths, also
CMAKE_INSTALL_DOCDIR is composed using the PROJECT_NAME also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-{detect-init,disk}

add a cmake module named Distutils.cmake for setting up python modules
using setup.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph_rgw_{json,multi}parser

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile and install ceph-bluefs-tool

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/ceph_test_xattr_bench/ceph_xattr_bench/

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename ceph-psim to ceph_psim

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix bash_completion install path

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fsck: check for dup overlay keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: use common helper to verify blobs and refs

The checks are the same (or should be--we had missed a few).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add FLAG_NOCACHE flag; do not cache unbuffered writes

Add a Buffer flag to mark that a buffer should not be cached once it is
stable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not use buffered bdev in write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not rely on bdev buffered reads in read path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: no buffered write in wal path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: populate buffer cache on read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep intrusive_list of WRITING buffers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simple per-collection lru for buffers

Size these using a global config.  This is only a starting point--we'll
obviously have to rework this to share memory across collections.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use bufferptr for csum_data

encode/decode of vector<char> is not optimized.  Bufferptr is a more
natural type here anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objecstore: Adds a test case for compression stuff verification (incomplete)

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes configuration observation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Cosmetic fixes in bluestore logging

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Enables cow for cloning at bluestore for store test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes Bnode serialization/deserialization and removes legacy Bnode::ref_map

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph_test_objectstore: extends SimpleObjectTest with the case where write happens for neighboring csum blocks to verify for potential alignment issue

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Removes legacy block_size retrieval

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use WriteContext and do_alloc_write for _do_write_small

Kill some mostly-duplicated code

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: consolidate WriteContext items into a write_item

Also include b_off in there.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid unnecessary write_onode calls

_wctx_finish callers always write the onode; we only need to worry about
our changes to the bnode.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_* methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_zeros args

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix offset skew check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ~0x -> ~

e.g., 0x432da000~1000 instead of 0x432da000~0x1000

I think it's sufficiently clear that the value after ~ should have the same
base as the first bit, and it's easier to read.  And less text.

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: Extends decompressor interface to be able to provide compressed data length.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: compress on write

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not partially deallocate compressed blobs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write_big: limit size of blobs based on compression mode

We may want to compress in smaller chunks based on hints/policy.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track new compression config options

Class-wide Compressor, compression mode, and options.  For now these are
global, although later we'll do them per-Collection so they can be pool-
specific.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add length to the compression_header_t

Snappy fails to decompress if there are extra zeros in the input buffer.
So, store the length explicitly in the header to avoid feeding them into
the decompressor.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix BufferSpace::read()

- we weren't reading from 'clean' buffers
- restructured loop a bit chasing another bug (but it ended up being
  in the caller)

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add COMPRESSIBLE and INCOMPRESSIBLE alloc hints

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: add a get_type() method to Compressor interface

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read cached vs read result assembly

We weren't handling the case of

 read block 0~300
 cache bloc 100~100

where the result is read(head) + cached + read(tail). Restructure the
loop to handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read read out of buffer cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix up _set_csum helper

- make it thread-safe
- call during mount

Signed-off-by: Sage Weil <sage@redhat.com>

os/store_test: Fixes dump_mismatch_bl to avoid assert on lengths mismatch. Starts using it for BufferCacheTest

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use bdev_block_size instead of min_alloc_size for allocators

min_alloc_size is more dynamic; we just need the block size unit here.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: min_alloc_size options for different media types

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: Fixes duplicate blob move when cloning

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: avoid passing overlapping allocated/released sets to fm

BitmapFreelistManager doesn't like overlapping allocated+released sets
when the debug option is enabled, because it does a read to verify the
op is valid and that may not have been applied to the kv store yet.

This makes bluestore ObjectStore/StoreTest.SimpleCloneTest/2 pass with
bluestore_clone_cow = false and bluestore_freelist_type = bitmap.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitmapFreelistManager: drop newline on hex dumps

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add no-newline hexdump option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitmapFreelistManager: use hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop warning

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix BufferCacheReadTest

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_onode crcs in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove obsolete tail cache

The buffer cache will cover this in a much more general way.

Signed-off-by: Sage Weil <sage@redhat.com>