os/bluestore: paranoid checks on onode_map vs lru

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: replace broken get_next with map_any

The get_next implementation assumed the lru was local to the OnodeSpace,
but it includes the whole cache now.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: better audit_lru debugging

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: shard the cache

Signed-off-by: Sage Weil <sage@redhat.com>

osd: communicate shard count to ObjectStore

Signed-off-by: Sage Weil <sage@redhat.com>

osd: use spg_t method to hash pg to (wq) shard

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: unify lrus into single Cache class

This will be sharded soon, but for now there's just one instance.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: unique type names or onode and buffer lrus

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: take collection lock when trimming collection cache

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: audit LRU at various points

This is purely for debugging.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: don't print bnode key

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix buffercache discard bug

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9465 from liewegas/wip-bluestore-zero

os/bluestore: remove aio_zero from bdev interface

Merge pull request #9417 from chhabaramesh/master

os/bluestore: Tree based bitmapalloc

Merge pull request #9466 from liewegas/wip-rocksdb

rocksdb: pull up to latest master

Makefile-rocksdb.am: update

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9457 from yehudasa/wip-14534

rgw: can set negative max_buckets on RGWUserInfo
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

unittest_bluefs: fix more

- wait for aio to finish
- shut down bdev, which stops the aio completion thread
- *then* deallocate the IOContexts

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9399 from mikulely/update-ceph-deploy-link

doc: update broken link for deploy ceph by Chef

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9441 from xinxinsh/wip-16706

librbd: remove invalid object map flag when object map is disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9437 from x11562/object-map

librbd: add object-map rebuild fail error log

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8652 from JiYou/fix-cephfs

cephfs: should close file handler before exit in cephfs.cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8925 from ukernel/wip-15723

client: fstat should take CEPH_STAT_CAP_INODE_ALL

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9421 from renhwztetecs/renhw-wip-cdir

mds/cdir: limit max snap id in lookup()

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #9418 from xiexingguo/xxg-wip-fix-mdsmonitor

mon/MDSMonitor: fix wrong positive result of jewel flag check

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8965 from ukernel/wip-lock-starve

mds: finish lock waiters in the same order that they were added.

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #8695 from renhwztetecs/renhw-wip-mds-dispatch

 mds: dispatch assert cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

os/bluestore:Tree based bitmapalloc

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

librbd: object-map rebuild fail, add error information

Signed-off-by: Na Xie <xie.na@h3c.com>

remove invalid objectmap flag when objectmap is disabled
Fixes: http://tracker.ceph.com/issues/16076
Signed-off-by: xinxin shu <shuxinxin@chinac.com>

Merge pull request #9349 from Adirl/fix_compiltaion_entity_addr_t4

msg: add const overload to entity_addr_t functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: remove aio_zero from BlockDevice interface

No users.  Not needed.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9295 from cbodley/wip-rgw-rados-vector

rgw: use vector for librados handles

Merge pull request #9454 from tanghaodong25/bluestore-write-path

os/bluestore: no 'head_read' when check ondisk length.

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: no 'head_read' when check ondisk length.

Signed-off-by: Haodong Tang <haodong.tang@intel.com>

Merge pull request #9458 from jimwright/master

librados: Added declaration for rados_aio_get_version

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9462 from athanatos/wip-15952

osd/ReplicatedPG: adjust num_pinned in _delete_oid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9236 from athanatos/wip-15943

osd/PG: update PGPool to detect map gaps and reset cached_removed_snaps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9460 from pritha-srivastava/wip-cmake-vstart

cmake: detection scripts use lowercase project name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: detection scripts use lowercase project name

Changing the detection scripts to use the new project name, changed in commit
c1a37101ea87a973068f2dc9858487f83ae57f88 (cmake: project name should be "ceph")

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

librados: Added declaration for rados_aio_get_version

Was missing from librados.h and hence had an incorrect
visability.

Fixes: #15535

Merge pull request #9449 from ifed01/wip-bluestore-crc32

os/bluestore: Some improvement for crc32c calculations

Reviewed-by: Sage Weil <sage@redhat.com>

rocksdb: pull up to latest master

This includes our replay fixes, which are now upstream.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #9448 from tchaikov/wip-fix-warning

test/osd/TestRados.cc: silence a compiler warning

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rados/loadgen-mostly-read.sh: half as much data (512mb)

Use less space so we can run on openstack.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: can set negative max_buckets on RGWUserInfo

This can be used to disable user's bucket creation

Fixes: http://tracker.ceph.com/issues/14534
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9037 from rzarzynski/wip-rgw-swift-quota

rgw: add support for account and container quotas of Swift API
Reviewed-by" Orit Wasserman <owasserm@redhat.com>

os/bluestore: dec->hex for verify_csum error reporting

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: reuse populate_with_generic_attrs() across rgw_op.cc.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

common/Checksummer: Eliminates data copying in crc32c calculation

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9367 from Yan-waller/yj-wip-rgwfixhextobuf

rgw/common: fix error return value of hex_to_buf.
Reviewed by: Orit Wasserman <owasserm@redhat.com>

common/bufferlist: Adds test cases for bufferlist::iterator::crc32c

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

common/bufferlist: Fixes lack of offset increment in get_ptr_and_advance

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

buffer: add iterator crc32c method

Calculate crc from an iterator position.

Signed-off-by: Sage Weil <sage@redhat.com>

test/osd/TestRados.cc: silence a compiler warning

fix following warning by returning `nullptr` after an assert(0).
```
ceph/src/test/osd/TestRados.cc: In member function ‘TestOp*
WeightedTestGenerator::gen_op(RadosTestContext&, TestOpType)’:
ceph/src/test/osd/TestRados.cc:241:3: warning: control reaches
end of non-void function [-Wreturn-type]
   }
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9438 from ceph/wip-cmake

cmake: rbd_mirror not linking rbd_test_mock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9060 from mikulely/fix-rgw-multiste-doc

doc: fix rgw multiste config

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #9392 from yehudasa/wip-16072

rgw/rgw_common.cc: modify the end check in RGWHTTPArgs::sys_get

Tested-by: Orit Wasserman <owasserm@redhat.com>
http://pulpito.ceph.com/owasserm-2016-06-01_20:43:36-rgw:verify-wip-orit-testing---basic-mira/

cmake: rbd_mirror not linking rbd_test_mock

Added rbd_test_mock lib, also changed name of
journal_test to journal_test_mock to mimic
automake naming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

mon/MDSMonitor: fix wrongly set expiration time of blacklist

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/MDSMonitor: fix assert

Belowe here "version" will never equal to "fsmap.epoch", otherwise we shall
return already.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/MDSMonitor: drop redundant assignment

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9209 from yehudasa/wip-15939

rgw: aws4 fixes

verified

Merge pull request #9084 from dzafman/wip-dz-misc

Wip dz misc

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9426 from linuxbox2/cmake-mds

cmake: change libmds back to a static library

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: restore static linkage (libmds)

Required by ceph-mds.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #9385 from ceph/wip-cmake-kefu

cmake: more fixes

fixes make install workflow

Merge pull request #9228 from liewegas/wip-bluestore-write

os/bluestore: new write path (checksums and compression)

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #9106 from SUSE/wip-15869

rpm: unconditionally set ceph user's primary group to ceph (SUSE)

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

cmake: install cython modules

* fix CYTHON_ADD_MODULE() macro. because python_add_module() offered by
  FindPythonLibs.cmake creates a target with name of ${name}, which conflicts
  with existing targets like "rbd" or "rados". so we can not reuse the
  name in ${name}.pyx. and instead, we should specify the target name
  explicitly.
* add distutils_install_cython_module() function to build and install
  cython modules.
* we can split build and install of cython module, but the install phase
  always tries to build the module. so keep it this way. will look at it
  later on.
* move the variables initializations into the Distutils.cmake module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install compressor plugins into ${pkglibdir/compressor

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install erasure plugins into ${pkglibdir}/erasure-code

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated file from ceph-osd

objclass/class_api.cc is already included in libosd

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix dependencies on tracing headers

group the header dependencies by its tp .so, so the traced target can
depend on them by the name of ${name}-tp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: libradosstriper's OUTPUT_NAME should be radosstriper

and s/libradosstriper/radosstriper/ otherwise the created .so
filename would be liblibradosstriper.so with the default prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install the artifacts the packaging requires

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install init script to etc/init.d

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add the autoconf path variables back

* partially revert 7a602ec.
* the directory variables created by automake, like "prefix", "bindir",
  and "libdir", are used for generating configuration_file() for substitution,
  and they should have the same names with ones from autotools.
* also fix the ${pkglibdir}, it should be the ${libdir}/${PACKAGE}. so
  the plugins are not installed into ${libdir}, in which the installed shared
  objects are supposed to be shared with other applications.
* install shared libraries into ${CMAKE_INSTALL_LIBDIR} instead of
  ${prefix}/lib. this complies to what ceph.spec.in requires:
  ceph.spec.in expects the shared libraries to be installed into
  ${_libdir}, and ${_libdir} is /usr/lib64 on an amd64 machine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-post-file

and related pubkey for sftp

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph-brag

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: project name should be "ceph"

so it would be easier to figure out paths, also
CMAKE_INSTALL_DOCDIR is composed using the PROJECT_NAME also.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: install ceph-{detect-init,disk}

add a cmake module named Distutils.cmake for setting up python modules
using setup.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add ceph_rgw_{json,multi}parser

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: compile and install ceph-bluefs-tool

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/ceph_test_xattr_bench/ceph_xattr_bench/

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename ceph-psim to ceph_psim

to match with automake and packager

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix bash_completion install path

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fsck: check for dup overlay keys

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: use common helper to verify blobs and refs

The checks are the same (or should be--we had missed a few).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: add FLAG_NOCACHE flag; do not cache unbuffered writes

Add a Buffer flag to mark that a buffer should not be cached once it is
stable.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not use buffered bdev in write path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not rely on bdev buffered reads in read path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: no buffered write in wal path

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: populate buffer cache on read

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: keep intrusive_list of WRITING buffers

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simple per-collection lru for buffers

Size these using a global config.  This is only a starting point--we'll
obviously have to rework this to share memory across collections.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: use bufferptr for csum_data

encode/decode of vector<char> is not optimized.  Bufferptr is a more
natural type here anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objecstore: Adds a test case for compression stuff verification (incomplete)

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Fixes configuration observation.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: Cosmetic fixes in bluestore logging

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>