Merge pull request #9462 from athanatos/wip-15952

osd/ReplicatedPG: adjust num_pinned in _delete_oid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9236 from athanatos/wip-15943

osd/PG: update PGPool to detect map gaps and reset cached_removed_snaps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9460 from pritha-srivastava/wip-cmake-vstart

cmake: detection scripts use lowercase project name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: detection scripts use lowercase project name

Changing the detection scripts to use the new project name, changed in commit
c1a37101ea87a973068f2dc9858487f83ae57f88 (cmake: project name should be "ceph")

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #9449 from ifed01/wip-bluestore-crc32

os/bluestore: Some improvement for crc32c calculations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9448 from tchaikov/wip-fix-warning

test/osd/TestRados.cc: silence a compiler warning

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rados/loadgen-mostly-read.sh: half as much data (512mb)

Use less space so we can run on openstack.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9037 from rzarzynski/wip-rgw-swift-quota

rgw: add support for account and container quotas of Swift API
Reviewed-by" Orit Wasserman <owasserm@redhat.com>

os/bluestore: dec->hex for verify_csum error reporting

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

common/Checksummer: Eliminates data copying in crc32c calculation

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9367 from Yan-waller/yj-wip-rgwfixhextobuf

rgw/common: fix error return value of hex_to_buf.
Reviewed by: Orit Wasserman <owasserm@redhat.com>

common/bufferlist: Adds test cases for bufferlist::iterator::crc32c

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

common/bufferlist: Fixes lack of offset increment in get_ptr_and_advance

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

buffer: add iterator crc32c method

Calculate crc from an iterator position.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/TestRados.cc: silence a compiler warning

fix following warning by returning `nullptr` after an assert(0).
```
ceph/src/test/osd/TestRados.cc: In member function ‘TestOp*
WeightedTestGenerator::gen_op(RadosTestContext&, TestOpType)’:
ceph/src/test/osd/TestRados.cc:241:3: warning: control reaches
end of non-void function [-Wreturn-type]
   }
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9438 from ceph/wip-cmake

cmake: rbd_mirror not linking rbd_test_mock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9060 from mikulely/fix-rgw-multiste-doc

doc: fix rgw multiste config

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #9392 from yehudasa/wip-16072

rgw/rgw_common.cc: modify the end check in RGWHTTPArgs::sys_get

Tested-by: Orit Wasserman <owasserm@redhat.com>
http://pulpito.ceph.com/owasserm-2016-06-01_20:43:36-rgw:verify-wip-orit-testing---basic-mira/

cmake: rbd_mirror not linking rbd_test_mock

Added rbd_test_mock lib, also changed name of
journal_test to journal_test_mock to mimic
automake naming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #9209 from yehudasa/wip-15939

rgw: aws4 fixes

verified

Merge pull request #9084 from dzafman/wip-dz-misc

Wip dz misc

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9426 from linuxbox2/cmake-mds

cmake: change libmds back to a static library

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: restore static linkage (libmds)

Required by ceph-mds.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9385 from ceph/wip-cmake-kefu

cmake: more fixes

fixes make install workflow

Merge pull request #9228 from liewegas/wip-bluestore-write

os/bluestore: new write path (checksums and compression)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9106 from SUSE/wip-15869

rpm: unconditionally set ceph user's primary group to ceph (SUSE)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

cmake: install cython modules

* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
  FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
  with existing targets like "rbd" or "rados". so we can not reuse the
  name in ${name}.pyx. and instead, we should specify the target name
  explicitly.
* add distutils_install_cython_module() function to build and install
  cython modules.
* we can split build and install of cython module, but the install phase
  always tries to build the module. so keep it this way. will look at it
  later on.
* move the variables initializations into the Distutils.cmake module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install compressor plugins into ${pkglibdir/compressor

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install erasure plugins into ${pkglibdir}/erasure-code

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated file from ceph-osd

objclass/class_api.cc is already included in libosd

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix dependencies on tracing headers

group the header dependencies by its tp .so, so the traced target can
depend on them by the name of ${name}-tp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: libradosstriper's OUTPUT_NAME should be radosstriper

and s/libradosstriper/radosstriper/ otherwise the created .so
filename would be liblibradosstriper.so with the default prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install the artifacts the packaging requires

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install init script to etc/init.d

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add the autoconf path variables back

* partially revert 7a602ec.
* the directory variables created by automake, like "prefix", "bindir",
  and "libdir", are used for generating configuration_file() for substitution,
  and they should have the same names with ones from autotools.
* also fix the ${pkglibdir}, it should be the ${libdir}/${PACKAGE}. so
  the plugins are not installed into ${libdir}, in which the installed shared
  objects are supposed to be shared with other applications.
* install shared libraries into ${CMAKE_INSTALL_LIBDIR} instead of
  ${prefix}/lib. this complies to what ceph.spec.in requires:
  ceph.spec.in expects the shared libraries to be installed into
  ${_libdir}, and ${_libdir} is /usr/lib64 on an amd64 machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-post-file

and related pubkey for sftp

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph-brag

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: project name should be "ceph"

so it would be easier to figure out paths, also
CMAKE_INSTALL_DOCDIR is composed using the PROJECT_NAME also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-{detect-init,disk}

add a cmake module named Distutils.cmake for setting up python modules
using setup.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph_rgw_{json,multi}parser

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile and install ceph-bluefs-tool

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/ceph_test_xattr_bench/ceph_xattr_bench/

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename ceph-psim to ceph_psim

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix bash_completion install path

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fsck: check for dup overlay keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: use common helper to verify blobs and refs

The checks are the same (or should be--we had missed a few).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add FLAG_NOCACHE flag; do not cache unbuffered writes

Add a Buffer flag to mark that a buffer should not be cached once it is
stable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not use buffered bdev in write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not rely on bdev buffered reads in read path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: no buffered write in wal path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: populate buffer cache on read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep intrusive_list of WRITING buffers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simple per-collection lru for buffers

Size these using a global config.  This is only a starting point--we'll
obviously have to rework this to share memory across collections.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use bufferptr for csum_data

encode/decode of vector<char> is not optimized.  Bufferptr is a more
natural type here anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objecstore: Adds a test case for compression stuff verification (incomplete)

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes configuration observation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Cosmetic fixes in bluestore logging

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Enables cow for cloning at bluestore for store test

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes Bnode serialization/deserialization and removes legacy Bnode::ref_map

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

ceph_test_objectstore: extends SimpleObjectTest with the case where write happens for neighboring csum blocks to verify for potential alignment issue

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Removes legacy block_size retrieval

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use WriteContext and do_alloc_write for _do_write_small

Kill some mostly-duplicated code

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: consolidate WriteContext items into a write_item

Also include b_off in there.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: avoid unnecessary write_onode calls

_wctx_finish callers always write the onode; we only need to worry about
our changes to the bnode.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_* methods

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop unused _pad_zeros args

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix offset skew check

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ~0x -> ~

e.g., 0x432da000~1000 instead of 0x432da000~0x1000

I think it's sufficiently clear that the value after ~ should have the same
base as the first bit, and it's easier to read.  And less text.

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: Extends decompressor interface to be able to provide compressed data length.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: compress on write

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not partially deallocate compressed blobs

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _do_write_big: limit size of blobs based on compression mode

We may want to compress in smaller chunks based on hints/policy.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: track new compression config options

Class-wide Compressor, compression mode, and options.  For now these are
global, although later we'll do them per-Collection so they can be pool-
specific.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add length to the compression_header_t

Snappy fails to decompress if there are extra zeros in the input buffer.
So, store the length explicitly in the header to avoid feeding them into
the decompressor.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix BufferSpace::read()

- we weren't reading from 'clean' buffers
- restructured loop a bit chasing another bug (but it ended up being
  in the caller)

Signed-off-by: Sage Weil <sage@redhat.com>

librados: add COMPRESSIBLE and INCOMPRESSIBLE alloc hints

Signed-off-by: Sage Weil <sage@redhat.com>

compressor: add a get_type() method to Compressor interface

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read cached vs read result assembly

We weren't handling the case of

 read block 0~300
 cache bloc 100~100

where the result is read(head) + cached + read(tail). Restructure the
loop to handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix _do_read read out of buffer cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix up _set_csum helper

- make it thread-safe
- call during mount

Signed-off-by: Sage Weil <sage@redhat.com>

os/store_test: Fixes dump_mismatch_bl to avoid assert on lengths mismatch. Starts using it for BufferCacheTest

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: use bdev_block_size instead of min_alloc_size for allocators

min_alloc_size is more dynamic; we just need the block size unit here.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: min_alloc_size options for different media types

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: Fixes duplicate blob move when cloning

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: avoid passing overlapping allocated/released sets to fm

BitmapFreelistManager doesn't like overlapping allocated+released sets
when the debug option is enabled, because it does a read to verify the
op is valid and that may not have been applied to the kv store yet.

This makes bluestore ObjectStore/StoreTest.SimpleCloneTest/2 pass with
bluestore_clone_cow = false and bluestore_freelist_type = bitmap.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitmapFreelistManager: drop newline on hex dumps

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add no-newline hexdump option

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BitmapFreelistManager: use hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: drop warning

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: fix BufferCacheReadTest

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: _dump_onode crcs in hex

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: remove obsolete tail cache

The buffer cache will cover this in a much more general way.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: Fixes improper length calculation in BufferSpace::read + adds simplified test case to highlight an issue for append to existing blob

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: drop min_alloc_size locals

We have this in the class, now.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix min_alloc_size global

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: release partial extents

Use the blob put_ref helper so that we can deallocate blobs partially
(instead of always waiting until they are completely unused).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: only write into a blob region that is allocated

We're only worried about direct writes and wal overwrites; the other write
paths are to freshly allocated blobs.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: blob_t: add tracking for released extents

We reference count which parts of the blob are used (by lextents), but
currently we only release our space back to the system when all references
go away.  That is a problem if the blob is large (say, 4MB), and we, say,
truncate off most (but not all) of it.

Unfortunately, we can't simply deallocate anything that doesn't have a
reference, because the logical refs are on byte boundaries, and allocation
happens in larger units (min_alloc_size).  A one byte logical punch_hole
might be responsible for the release of a larger block of storage.

To resolve this, we keep track of which portions of the blob have been
released by poisoning the offset in the extents vector.  We expect that
this vector will be almost always short, so we do not bother with a
indexed structure, since iterating a blob offset to determine if it is
still allocated is likely faster.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: add poison offset to pextent_t

This is a "magic" offset that we can use to indicate an invalid extent
(vs, say, an extent at offset 0 that might clobber real data if it were
used).

Signed-off-by: Sage Weil <sage@redhat.com>