cmake: remove Boost libraries from EXTRALIBS

* and remove ${Boost_SYSTEM_LIBRARY} from test_rados_api_tier

Signed-off-by: Kefu Chai <kchai@redhat.com>

log: remove "boost/asio.hpp" from Log.h

graylog uses boost/asio.hpp which introduces the link-time dependency on
libboost_system to the compilation units which includes Log.h and
LogClient.h. so it appears that perfglue/heap_profiler.cc is referencing
libboost_system.so, and fails the cmake build of all tests which links
against tcmalloc.

in this change, we:

* remove unnecessary #includes from Graylog.h
* forward declare Graylog class, so that "Graylog.h" is not included in
  any header files to avoid the link-time dependency pollution

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable warnings introduced by 457f023

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: add libboost_system to EXTRALIBS"

This reverts commit 21438a6773a3153556de07096919ab46c4540eee.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7496 from xiexingguo/xxg-wip-fixnvme

osd: BlueStore/NVMEDevice: fix compiling and fd leak

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7349 from yuyuyu101/add-pwritev

osd: FileStore: use pwritev instead of lseek+writev

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7504 from renhwsky/renhw-wip-mon-shutdown

mon: unregister command on shutdown

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7550 from tchaikov/wip-osd-cleanup

osd: a fix for HeartbeatDispatcher and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7553 from XinzeChi/wip-osdmap-affinity

osd: OSDMap: reset osd_primary_affinity shared_ptr when deepish_copy_from

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7560 from tchaikov/wip-fix-icl

crush/CrushTester: workaround a bug in boost::icl

Merge pull request #7568 from liewegas/wip-bluestore-mutex

osd: bluestore/blockdevice: use std::mutex et al

Merge pull request #7578 from ceph/fix-stripper

Stripper: Fix incorrect push_front -> append_zero change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Stripper: Fix incorrect push_front -> append_zero change

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7444 from dreamhost/wip-rgw-static-website-robbat2

rgw: approximate AmazonS3 HostId error field.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

v10.0.3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7572 from ceph/wip-cmake-2-8-16

cmake: Added new unittests to make check

Merge pull request #7574 from SUSE/wip-spec-license

ceph.spec.in: add license declaration

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add license declaration

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: Added new unittests to make check

Added unittest_shunique_lock & unittest_mutex_debug
to make check, and made them link ALLOC_LIBS

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #7569 from SUSE/wip-copyright2

ceph.spec.in: add copyright notice

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add copyright notice

http://tracker.ceph.com/issues/14694 Fixes: #14694

Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore/BlockDevice: std::atomic_int

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: use std::mutex and std::condition_variable

Pull aio wait code into an IOContext method too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7513 from athanatos/wip-ceph-test-rados-2

ceph_test_rados: use less CPU

Merge pull request #5931 from ceph/wip-cxx11concurrency

librados: move to c++11 concurrency types

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge branch 'print-blocked-op' of git://github.com/majianpeng/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6608 from esnyder/wip-6634

osd: remove unused SnapContext member snapc from MOSDSubOp message

Reviewed-by: Xiaoxi Chen <xiaoxi.chen@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' of git://github.com/joconcepts/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7181 from songbaisen/song5

crush: reply quickly from get_immediate_parent

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7437 from Ved-vampir/wip_zlib_plugin

common: add zlib compression plugin

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7527 from liewegas/wip-no-rgw-prestart

systemd/ceph-radosgw-prestart.sh: remove

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7547 from yuyuyu101/wip-14679

AsyncConnection: avoid debug log in cleanup_handler

crush/CrushTester: workaround a bug in boost::icl

the boost::icl in boost v1.55 has a bug when working with libc++,

> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
> function gap_insert, file
> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.

see https://svn.boost.org/trac/boost/ticket/9501 and
    https://svn.boost.org/trac/boost/ticket/9987

the bug was fixed in boost v1.56. as a workaround we use the the
`set` from `boost::container` instead of the `std::set` from libc++.
we should drop this workaround once we bump the supported boost
version from 1.42 to 1.56.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7554 from tchaikov/wip-ceph-disk-test

tests: ceph-disk.sh should compare with the resolved path

Reviewed-by: Loic Dachary <ldachary@redhat.com>

OSDMap: reset osd_primary_affinity shared_ptr when deepish_copy_from

Base on current logic, OSDMonitor may call create_pending and
encode_pending twice for the some epoch.

In encode_pending:

tmp.deepish_copy_from(osdmap);
tmp.apply_incremental(pending_inc);

This Op would change the tmp osd_primary_affinity, but the osd_primary_affinity
is declared as ceph::shared_ptr, so this would change the osdmap too. When this
round encode_pending is proposed fail. We may call encode_pending again, but the
osdmap is changed last round, so the pending_inc would be wrong.

Fixes: #14686
Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #7551 from JevonQ/update-libcephfs-version

libcephfs: update LIBCEPHFS_VERSION to indicate the interface was changed

Reviewed-by: Sage Weil <sage@redhat.com>

tests: ceph-disk.sh should compare with the resolved path

Signed-off-by: Kefu Chai <kchai@redhat.com>

libcephfs: Update LIBCEPHFS_VERSION to indicate the interface was changed

Signed-off-by: Jevon Qiao <scaleqiao@gmail.com>

AsyncMessenger: remove unused variables

* remove AsyncMessenger::listen_sd
* remove `ee` in KqueueDriver::del_event()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crypto: remove unused Digest::sec_type

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: remove unused ObjectPlayer::m_fetch_interval

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: remove unused C_DiffObject::m_request_num

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: replace atomic_t with std::atomic<int>

it fails to compile with clang, and turns out it should be
`ceph::atomic_t`, but I take this chance to replace it
with std::atomic<> to phase out atomic_t.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ECUtil: remove unused stripe_info_t::stripe_size

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: init HeartbeatDispatcher using OSD->cct

its base class 'Dispatcher' is uninitialized when used here to access
'Dispatcher::cct', so we'd better use OSD::cct to initialize it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7549 from tchaikov/wip-fix-ceph-disk-tests

ceph-disk/test: fix test_prepare.py::TestPrepare tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk/test: fix test_prepare.py::TestPrepare tests

actually we are verifying if the variable is an instance of
specified class. for example, the `prepare.data` should be
a `PrepareFilestoreData` if `--bluestore` is not specified.

Signed-off-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: avoid debug log in cleanup_handler

local connection will be stop and call cleanup_handler after messenger
is down
introduced in
commit(https://github.com/ceph/ceph/commit/9da2fffd31562ed5d0b795d7862b3ebec66aba40)

Fix #14679

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7540 from dillaman/wip-librados-test-stub-unwatch

librados_test_stub: protect against notify/unwatch race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7529 from dillaman/wip-14412

librbd: journal framework for tracking exclusive lock transitions

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7377 from liewegas/wip-datadir-search

config: add $data_dir/config to config search path

Merge pull request #7466 from adamemerson/wip-bsd-time-second-try

common: time: have skewing-now call non-skewing now

Merge pull request #7463 from dachary/wip-13422-pid-file

global: do not start two daemons with a single pid-file (part 2)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7460 from xiexingguo/xxg-wip-block

osd: blockdevice: avoid implicit cast and add guard

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7530 from xiexingguo/xxg-wip-nid

osd: bluestore: assign nid for newly created onode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7521 from liewegas/wip-memstore

osd: memstore: set Collection::cid on create

Merge pull request #7486 from ceph/wip-bluestore-oref

osd: bluestore: fix two wal bugs; reuse OnodeRefs when applying txc

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7531 from jjhuo/master

osd: bluestore: remove unused intrusive member hook in enode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7535 from liewegas/wip-bluefs-mkfs

osd: bluestore/BlueFS: initialize super block_size earlier in mkfs

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #7536 from liewegas/wip-bluestore-block

osd: bluestore: fix block device file creation

Merge pull request #7541 from linuxbox2/cmake-feb5

Cmake feb5

osdc: Update to use C++11 concurrency

The only externally visible change is that Objecter::get_osdmap_read
and Objedcter::put_osdmap_read have been removed in favor of
Objecter::with_osdmap.

It can be used like:

objecter->with_osdmap([&](const OSDMap& o) { o.do_stuff(); };
int x = objecter->with_osdmap([&](const OSDMap& o) {
    return get_thing(id); });
objecter->with_osdmap(std::mem_fn(&OSDMap::do_stuff));
int x = objecter->with_osdmap(std::mem_fn(&OSDMap::get_thing),
                              id);

The choice between the style of the first two examples or the second two
is arbitrary and depends on what you prefer.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cmake: add libboost_system to EXTRALIBS

This concisely fixes several unittest builds, and reflects the
fact that this library dependency has moved into several areas
of the codebase (libcephfs, librbd, librgw).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cmake: don't try to install src/ceph-disk directory

The proper cmake boilerplate to call ceph-disk/ceph-disk's
python installer is not present, FYI.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cmake: don't add_test cephtool_test_osd twice

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

librados_test_stub: protect against notify/unwatch race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

concurrency: Add shunique_lock

Locker class tracking shared/unique/unlocked state of SharedMutex
concept.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

librbd: partial implementation of journal client / tag allocation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initial version of journal client and tag metadata

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: added tag support methods

librbd, for example, will allocate a new tag after acquiring
the exclusive lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: new get_client / client_update methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: client registration should hold opaque data structure

The opaque structure will support journal client applications to
store and retrieve complex state.

Fixes: #13298
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: new tag management methods and handling

In the case of librbd, a new tag will be allocated when the
exclusive lock is acquired.  All tags for the same dataset
(e.g. librbd image) will belong to the same class.  Tags are
automatically pruned on tag create / client unregister
if no other clients' commit position would require the tags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: switched entry tags to use id instead of string

Later commits will add the ability to allocate tags and
associate them with registered clients.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

concurrency: make C++11 style debugging mutices

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #7474 from jack-changtao/wip-cmake2

CMake: For CMake version <= 2.8.11, use LINK_PRIVATE and LINK_PUBLIC

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7108 from jdurgin/wip-rbd-mirroring

rbd-mirror: daemon skeleton

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

time: Remove constexpr from non-literal type in test

This fixes a compile error under clang and fixes another FreeBSD porting
issue.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

time: Have skewing-now call non-skewing now

For the real-time clocks, Ceph's testing infrastructure likes to be able to
inject a skew. To avoid pulling CephContext into ceph_time.h these are moved to
ceph_time.cc. The original way this was done called clock_gettime in both
places.

This is an unnecessary duplication and apparently error-prone. So only call
clock_gettime from one place.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #7538 from tchaikov/wip-fix-exe-test

tests: mon: "chmod +x mon-created-time.sh"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: mon: "chmod +x mon-created-time.sh"

otherwise it always fails to run.

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/KernelDevice: use _exit(1) for failure injection

This way we don't generate a core file, which spams the log and trips up
teuthology.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fix block device file creation

Just make a separate flag to indicate whether we create a block
file.  This lets us drop the weird touch in vstart.sh, and default
to creating a token 'block' file on --mkfs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7519 from xiexingguo/xxg-wip-finisher

librados: shutdown finisher in a more graceful way

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7506 from liewegas/wip-bluestore-block

osd: bluestore: set default wal size; improve weird implicit block file creation logic

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7503 from tchaikov/wip-cmake-spdk

build: spdk submodule; cmake

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7218 from dachary/wip-13942-ceph-disk

ceph-disk: support bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueFS: initialize super block_size earlier in mkfs

Reported-by: Dan Mick <dmick@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>

os/kstore: kill dead code

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/kstore: fix race condition

The get_omap_iterator shall check the existence of onode too,
which may race with the rename method.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix race condition

The get_omap_iterator shall check the existence of onode too, which may
race with the rename method.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: remove unused intrusive member hook in enode

Enode only uses intrusive unordered_set base hook, member hook is not neccesary.
This should save some memory space, when we have tons of enodes.

Signed-off-by: Jianjian Huo <samuel.huo@gmail.com>

global: do not start two daemons with a single pid-file (part 2)

Fixes the following bugs:

* the fd is open(O_WRONLY) and cannot be read from, safe_read
  always fails and never removes the pid file.

* pidfile_open(g_conf) is close(STDOUT_FILENO) and there is a risk that
  pidfile_open gets STDOUT_FILENO only to have it closed and redirected
  to /dev/null.

* Before writing the file, ftruncate it so that overriding a file
  containing the pid 1234 with the pid 89 does not end up being
  a file with 8934.

* Before reading the file, lseek back to offset 0 otherwise it
  will read nothing.

* tests_pidfile was missing an argument when failing
  TEST_without_pidfile and killed all process with ceph in their name,
  leading to chaos and no useful error message.

* lstat(fd) cannot possibly return a result different from the one
  obtained right after the file was open, stat(path) must be used
  instead.

In addition to fixing the bugs above, refactor the pidfile.cc
implementation to:

* be systematic about error reporting (using cerr for when removing
  the pidfile because derr is not available at this point and derr
  when creating the pidfile).

* replace pidfile_open / pidfile_write with just pidfile_write since
  there never is a case when they are not used together.

More test cases are added to test_pidfile to verify the bugs above are
fixed.

http://tracker.ceph.com/issues/13422 Fixes: #13422

Signed-off-by: Loic Dachary <loic@dachary.org>

os/kstore: assign nid for newly created onode

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: assign nid for newly created onode

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7522 from ktdreyer/wip-admin-build-doc-deps

admin/build-doc: depend on zlib1g-dev and graphviz

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: unregister command on shutdown

unregister command on shutdown
include: "quorum enter" && "quorum exit"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

systemd/ceph-radosgw-prestart.sh: remove

This is unpackaged and unused.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_config: cope with $data_dir not being expanded on non-daemons

Signed-off-by: Sage Weil <sage@redhat.com>