tests: base gmock class support for librbd

Created mock classes to represent a few central librbd classes
and a basic gmock test fixture for future gmock-based unit tests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support templating of ImageCtx for async state machines

This will facilitate the creation of unit test mocks to verify
non-librados actions. The templates (by default) will be flagged
as extern to avoid the translation unit bloat.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: add mock class for IoCtx operations

Unit tests can now use gmock to simulate responses from the OSDs
via a mocked librados library.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: add reference counting to pools

This will ensure that in-flight operations can't access freed
pool memory.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

gtest: enable use of TR1 tuples

Since the TR1 tuple is already in use within the Ceph project,
this flag needs to be enabled to avoid gmock link errors within
gtest.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5808 from ceph/wip-simplify-logrotate

logrotate.conf: Simplify log files reopening after log rotation

Reviewed-by: Milan Broz <mbroz@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5771 from ceph/wip-fix-i386-build

memstore: fix the build on i386

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5804 from tchaikov/wip-fix-build-boost-1.59

mon: fix the build with boost 1.59

Reviewed-by: Loic Dachary <ldachary@redhat.com>

logrotate.conf: Simplify log files reopening after log rotation

SIGHUP (1) signal makes ceph services reopen its log files so we do not
need to call any init scripts magic, here.

Additionally, the old approach caused problems with rhel-based systems
since it found the service command and tried to reload ceph.service
instead of ceph.target.

This also caused problems with SELinux as processes with context
logrotate_t are not allowed to issue systemctl reload on a process.

This patch is also in sync with other logrotate postrotate scripts which
do exactly the same thing (send SIGHUP to the daemon).

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #5802 from liewegas/wip-coverity

a few coverity fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon: fix the build with boost 1.59

cmd_vartype is a
boost::variant<std::string, bool, int64_t, double,
               std::vector<std::string>>, so we can not get a
int8_t from it.

Fixes: #12922
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: ProxyWriteOp::reqid should not be a ref

CID 1322778 (#1 of 1): Pointer to local outside scope (RETURN_LOCAL)
1. escape_local_addr: Returning, through this->reqid, the address of stack variable _reqid.
    2. return: Returning here.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: init coll_t::removal_seq in all ctors

CID 1322784 (#1 of 1): Uninitialized scalar variable (UNINIT)
2. uninit_use_in_call: Using uninitialized value coll.removal_seq when calling coll_t. [show details]

Signed-off-by: Sage Weil <sage@redhat.com>

test_cls_numops: fix iterator use

CID 1322828 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
28. use_after_free: Using invalidated internal representation of local it.
CID 1322827 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
25. use_after_free: Using invalidated internal representation of local it.
CID 1322826 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
31. use_after_free: Using invalidated internal representation of local it.
CID 1322825 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
31. use_after_free: Using invalidated internal representation of local it.

Signed-off-by: Sage Weil <sage@redhat.com>

debian: package radosgw-object-expirer in radosgw deb

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: package new rgw files

   /usr/bin/radosgw-object-expirer
   /usr/lib64/rados-classes/libcls_timeindex.so

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5799 from liewegas/wip-cmake

cmake: fix build (newstore issues)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

CMakeLists.txt: add newstore files

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: ignore build (usually used by cmake)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/master' into infernalis

rgw/Makefile.am: ship rgw_object_expirer_core.h

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5744 from ceph/wip-12909

cmake: update FUSE_INCLUDE_DIRS to match autoconf

Merge pull request #5610 from ceph/wip-cmake

cmake: make check

ceph.spec: build requires cmake

Signed-off-by: Sage Weil <sage@redhat.com>

debian/control: build requires cmake

Signed-off-by: Sage Weil <sage@redhat.com>

debian/control: build-requires libboost-regex-dev

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5692 from ceph/wip-rgw-swift-expiration

Wip rgw swift expiration

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: install crushtool to destdir/bin

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cmake: add blkid as dependency to libcommon

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: Changed name of crc32 target to crc32c

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Added shell script tests

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Fixed HAVE_BETTER_YASM_ELF64 variable

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed trailing spaces from isa .s files

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Uncommented erasure-code/shec conditional

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed traces of CDS, minor cmake fixes

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Fixed rbd_replay build issue

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed scripts, check_PROGRAMS included

Removed the unittest scripts for the time being.
Built unittests included in check_PROGRAMS target.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Cleaned up syntax for make check targets

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: check_TESTPROGRAMS tests running

Make check working, accept rocksdb tests. Clean up coming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

README.md: Add basic CMake instructions

README.md: Fixed spacing, trimmed cmake section

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #5792 from ceph/wip-vstart-rgw

vstart: add -c argument to radosgw-admin commands

Reviewed-by: Sage Weil <sage@redhat.com>

vstart: add -c argument to radosgw-admin commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #5590 from majianpeng/mds

Mds: add osdmap epoch for setxattr.

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: don't copy delete_at attr, unless it's intra region copy

We don't want to keep the expiration value of a copied object, unless
we're doing a copy within the same zone group.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: objexp shards index by key

Not by time. This should provide better concurrency.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: delete-at and delete-after also on obj put / copy

And potentially later we could use also the S3 api, so it
could work with multipart upload, and POST obj.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5775 from dachary/wip-do-autogen

tools: fix do_autogen.sh -R

Merge pull request #5712 from yuyuyu101/wip-12801

Mon: Make ceph osd metadata support dump all osds

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

tools: fix do_autogen.sh -R

The R letter was missing from the getopts flags. Also sort the flags
alphabetically to make it easier to spot that kind of lossage.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

rocksdb: fix 32-bit build

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5752 from wonzhq/doc-write-recency

doc: add the doc for min_write_recency_for_promote

Reviewed-by: Loic Dachary <ldachary@redhat.com>

memstore: fix the build on i386

on i386, uint64 is `unsigned long long`, while size_t is `unsigned int`.
std::min(uint64, size_t) can not be resolved.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5739 from ceph/wip-12776

mds: fix shutdown while in standby

doc: add the doc for min_write_recency_for_promote

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #5736 from tianshan/wip-12864

rados: make 'rados bench' support json format output

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5748 from liewegas/wip-warnings

fix newstore warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-osd-prestart.sh: fix osd data dir ownership check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5749 from liewegas/wip-async-ms

msg/async: log tx/rx at level 1

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

vstart.sh: enable all experimental features for vstart

Signed-off-by: Sage Weil <sage@redhat.com>

ms/async: log message tx/rx at level 1

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5722 from cxwshawn/vs-fix

vstart.sh: add --mon_num --osd_num --mds_num --rgw_port option

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5693 from tchaikov/wip-12730

common/SubProcess: silence compiler warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5643 from dreamhost/wip-make-check-makeopt

make-check: support MAKEOPTS overrides.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5299 from hjwsm1989/pgmonitor-const

  mon: added const to dump_* functions in PGMonitor

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5156 from rubenk/fix-indentation

Fix indentation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5275 from tchaikov/wip-12287

pybind/ceph_argparse: do not choke on non-ascii prefix

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5702 from Sandy4999/wip-doc-sandy

doc:radosgw: correct typos of the command removing a subuser

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #5747 from ceph/wip-user

fix ceph-disk

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5742 from dachary/wip-user

tests: ceph-disk: dmcrypt simplification

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5746 from ceph/wip-fix-doc-build

doc: fix the code-block in ruby.rst

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix the code-block in ruby.rst

* and add the link to library homepage in the section titles

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update FUSE_INCLUDE_DIRS to match autoconf

client/fuse_ll.cc is now including <fuse.h> and <fuse_lowlevel.h>
instead of <fuse/fuse.h> and <fuse/fuse_lowlevel.h>, so we need to add
the fuse directory to the FUSE_INCLUDE_DIRS variable

using find_path() with just fuse.h was finding a /usr/include/fuse.h
instead of the one in /usr/include/fuse/. looking for fuse_common.h and
fuse_lowlevel.h first causes it to generate the correct
FUSE_INCLUDE_DIRS=/usr/include/fuse

Fixes: #12909
Signed-off-by: Casey Bodley <cbodley@redhat.com>

os/newstore: fix swarning

os/newstore/NewStore.cc: In member function 'int NewStore::_zero(NewStore::TransContext*, NewStore::CollectionRef&, const ghobject_t&, uint64_t, size_t)':
os/newstore/NewStore.cc:3693:32: warning: ignoring return value of 'int ftruncate(int, __off64_t)', declared with attribute warn_unused_result [-Wunused-result]
       ::ftruncate(fd, f.length);
                                ^

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5578 from ceph/wip-newstore

osd: newstore (experimental)

ceph_test_keyvaluedb: add simple commit latency benchmark

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: update todo

Signed-off-by: Sage Weil <sage@redhat.com>

do_autogen.sh: build static rocksdb by default

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: update alt dist rule

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: make OMapIterator test work with FileStore

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: enable newstore tests

Signed-off-by: Sage Weil <sage@redhat.com>

rocksdb: update to 3.11.2

Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDBStore: make other rmkey match

No need for Slice() here; it can take a string.

Signed-off-by: Sage Weil <sage@redhat.com>

os/RocksDBStore: fix rmkey()

This took way too long to debug!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_keyvaluedb: some simple KeyValueDB unit tests

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix end bound on collection_list

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: flush object before doing omap reads

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: add 'newstore backend options' to pass options to e.g. rocksdb

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: change escaping chars

# is lowest besides space and !, except for " (which would be too
confusing).

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: trim overlay when zeroing extent

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: tolerate null pnext to collection_list()

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: fix collection range for temp objects

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: Implement fiemap

For simplicity we ignore holes inside an fragment now.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

os/newstore: make sync/async submit_transaction optional

It seems doing this synchronously may be better for SSDs?

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: renamed TransContext::fds -> sync_items

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: queue kv transactions in kv_sync_thread

It appears that db->submit_transaction() will block if there is a sync
commit that is in progress instead of simply queueing the new txn for
later.  To work around this, submit these to the backend in the
kv_sync_thread prior to the synchronous submit_transaction_sync().

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: process multiple aio completions at a time

This isn't affecting things for a slow disk, but it will matter for faster
backends.

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: clean up kv commit debug output

Signed-off-by: Sage Weil <sage@redhat.com>

os/newstore: only ftruncate if i_size is incorrect

Even a no-op ftruncate can block in the kernel.  Prior to this change I
could frequently see ftruncate wait for an aio completion on the same
file.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "os/newstore: avoid sync append for small ios"

This reverts commit 69baab2f7eaca7688ce1d45802a82fc3539cd906.

This is slower.  :(

os/newstore: avoid sync append for small ios

An append is expensive in terms of latency (write, fdatasync, kv commit),
while a wal write is just the kv commit and the write and fdatasync are
async.  For small IOs doing the wal may improve performance.

Signed-off-by: Sage Weil <sage@redhat.com>