Merge pull request #7350 from ddiss/tracker14499_enotconn_rados_monitor_log

librados: check connection state in rados_monitor_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7584 from ifed01/wip-14511

osd: fix lack of object unblock when flush fails

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7593 from ifed01/wip-no-blockedpromote-return-fix

osd: fix return value from maybe_handle_cache_detail()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7587 from cbodley/wip-pageset-align

osd: memstore: fix alignment of Page for test_pageset

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #7595 from liewegas/wip-hobject-stringify

osd: ghobject_t: use ! instead of @ as a separator

Merge pull request #7596 from dachary/wip-erasure-code-corpus

tests: sync ceph-erasure-code-corpus for mktemp -d

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7608 from liewegas/wip-warnings

tests: fix a few build warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7626 from tchaikov/wip-fix-automake

Makefile: workaround an automake bug for "make check"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7351 from fmeppo/master

Add --setuser and --setgroup options for ceph-disk

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Makefile: workaround an automake bug for "make check"

override the automake variable of am__is_gnu_make. it is a workaround
for http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18744. this bug was
fixed in automake 1.15, but automake 1.13 is supported. so we can not
just require 1.15 using `AM_INIT_AUTOMAKE`. this only happens when we
have lots of source files *and* we are using automake v1.14 or lower.

Fixes: #14723
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7605 from dachary/wip-python-wheelhouse

python: use pip instead of python setup.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk,ceph-detect-init: clean the build directory

b030d8fc113736b479e84246b6b36d5f698a2da3 should have kept the build
directory in .gitignore and the clean target. Re-adding.

Signed-off-by: Loic Dachary <loic@dachary.org>

python: use pip instead of python setup.py

python setup.py develop may try to pull dependencies from the net and
has no way to collect them from the wheelhouse that was populated by
install-deps.sh. Use pip install -e instead

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7528 from objoo/YD-jewel-mailmap-updates-take2

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7609 from dillaman/wip-librbd-flatten-test

test: new librbd flatten test case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7583 from dillaman/wip-librbd-journal-fsx

librbd: integrate journal replay with fsx testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7613 from wjwithagen/patch-1

osd: add missing newline to usage message

Reviewed-by: Sage Weil <sage@redhat.com>

src/ceph_osd.cc Add missing newline to usage message

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common: fix build error with Graylog

Graylog includes boost stuff, which clobbers our assert macros.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6920 from oritwas/wip-mstart-cmake

ajdust mstart and mstop script to run with cmake build

Merge pull request #7523 from tchaikov/wip-fix-cmake

cmake: fix the build of tests

Merge pull request #7607 from FlorentCoppint/master

mount.fuse.ceph: better parsing of arguments passed to mount.fuse.ceph by mount command

Reviewed-by: Sage Weil <sage@redhat.com>

test: new librbd flatten test case

AIO operations after a flatten operation were previously
hanging during the close of the parent image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

unittest_[ceph_]crypto: fix warnings

test/crypto.cc:20:31: warning: ‘crypto_env’ defined but not used [-Wunused-variable]
 ::testing::Environment* const crypto_env = ::testing::AddGlobalTestEnvironment(n
                               ^
test/ceph_crypto.cc:12:31: warning: ‘crypto_env’ defined but not used [-Wunused-variable]
 ::testing::Environment* const crypto_env = ::testing::AddGlobalTestEnvironment(n
                               ^

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_compression_[*_][plugin]: fix warnings

test/compressor/test_compression_plugin.cc: In function ‘int main(int, char**)’:
test/compressor/test_compression_plugin.cc:55:38: warning: ignoring return value of ‘int system(const char*)’, declared with attribute warn_unused_result [-Wunused-result]
   system("mkdir -p .libs/compressor");
                                      ^
test/compressor/test_compression_plugin.cc:56:59: warning: ignoring return value of ‘int system(const char*)’, declared with attribute warn_unused_result [-Wunused-result]
   system("cp .libs/libceph_example.so* .libs/compressor/");
                                                           ^

Signed-off-by: Sage Weil <sage@redhat.com>

Better parsing of arguments passed to mount.fuse.ceph by mount command.

Signed-off-by: Florent Bautista <florent@coppint.com>

Merge pull request #5961 from majianpeng/small-fix

osd: clean up CMPXATTR checks

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7533 from Cdiscount/master

debian: include cpio in build-requiers

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_libcephfs: fix warnings

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7537 from ifed01/wip-no-promote-for-delete-fix

osd: fix unnecessary object promotion when deleting from cache pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7567 from branch-predictor/bp-slimdown-conns

msg: significantly reduce minimal memory usage of connections

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7579 from javacruft/wip-systemd-escaping

systemd: correctly escape block device paths

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7581 from jcsp/wip-asok-lockdep

librados: mix lock cycle (un)registering asok commands

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7604 from ceph/wip-selinux-update-radosgw-log

selinux: allow log files to be located in /var/log/radosgw

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7606 from tchaikov/wip-fix-ceph-disk

tests: ceph-disk.sh: use "readlink -f" instead for fullpath

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh: use "readlink -f" instead for fullpath

if $PATH has ".." in it, and the program happen to be located in
"..", `which program` will print `../program` instead of its fullpath,
so we should always use `readlink -f` for the fullpath.

Signed-off-by: Kefu Chai <kchai@redhat.com>

selinux: Allow log files to be located in /var/log/radosgw

We do suggest users to put their logs in /var/log/radosgw in the
documentation at times. We should also label that directory with
ceph_var_log_t so that ceph daemons can also write there.

The commit also updates the man page for this policy. This man page is
automatically generated by

* sepolicy manpage -p . -d ceph_t

and have not been reloaded in a while. Hence, it contains few more
changes than the new radosgw directory.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #7601 from jjhuo/master

os/bluestore: fix a typo in SPDK path parsing

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore: fix a typo in SPDK path parsing

SPDK block path with bluestore is something like spdk:55cd2e404bd73932.
The size of the prefix to compare should be 5.
sizeof(SPDK_PREFIX)-1 returns 5, while sizeof(SPDK_PREFIX-1) returns 8.

Signed-off-by: Jianjian Huo <samuel.huo@gmail.com>

librbd: protect journal replay against overlapping writes

If fsx issues a back-to-back synchronous write, these will be
replayed as AIO writes.  If object map is enabled, it's possible
for the two writes to commit to disk out-of-order if the first
write required an object map update.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track in-flight AIO flush requests during journal replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: possible librbd journal replay flush race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7588 from dmick/master

common/page.cc: _page_mask has too many bits

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7413 from YankunLi/dev

rgw: remove duplicated code in RGWRados::get_bucket_info()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge branch 'wip-11642-0203'

rgw: make function parameter 'errordoc_key' passed by reference

Signed-off-by: Bo Cai <cai.bo@h3c.com>

Merge branch 'master' of github.com:ceph/ceph

Merge pull request #7544 from ceph/wip-rh-covscan

rgw/rgw_orphan: check the return value of save_state

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7586 from bsedyang/master

rgw: user quota may not adjust on bucket removal

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc/release-notes: v10.0.3

Finalized release notes.

Signed-off-by: Sage Weil <sage@redhat.com>

OSD/ReplicatedPG: Adding test case to catch issue #14511

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Add better documentation of --setuser and --setgroup options for ceph-disk.

Signed-off-by: Mike Shuey <shuey@purdue.edu>

Add --setuser and --setgroup options to ceph-disk(8).

Signed-off-by: Mike Shuey <shuey@purdue.edu>

Add support to override ceph-disk's user and group choices

Signed-off-by: Mike Shuey <shuey@purdue.edu>

Merge pull request #7592 from dachary/wip-release-notes

release-notes: draft v10.0.3 release notes

tests: sync ceph-erasure-code-corpus for mktemp -d

Because it's posix and --directory is not.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7594 from tchaikov/fix-ceph-disk-test

tests: ceph-disk.sh: should use "readlink -f" instead

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ghobject_t: use ! instead of @ as a separator

@ confuses less's search (/ then @ does not do what you want).  !
behaves as expected.

Signed-off-by: Sage Weil <sage@redhat.com>

tests: ceph-disk.sh: should use "readlink -f" instead

turns out trusty does not have `realpath` unless the package named
`realpath` is installed.

Reported-by: Joao Eduardo Luis <joao@suse.de>
Signed-off-by: Kefu Chai <kchai@redhat.com>

release-notes: draft v10.0.3 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

scripts: ceph-release-notes fix merge messages handling

Strip the title line if it is the same as the pr title instead of
ignoring the message entirely.

Signed-off-by: Loic Dachary <loic@dachary.org>

cmake: remove Boost libraries from EXTRALIBS

* and remove ${Boost_SYSTEM_LIBRARY} from test_rados_api_tier

Signed-off-by: Kefu Chai <kchai@redhat.com>

log: remove "boost/asio.hpp" from Log.h

graylog uses boost/asio.hpp which introduces the link-time dependency on
libboost_system to the compilation units which includes Log.h and
LogClient.h. so it appears that perfglue/heap_profiler.cc is referencing
libboost_system.so, and fails the cmake build of all tests which links
against tcmalloc.

in this change, we:

* remove unnecessary #includes from Graylog.h
* forward declare Graylog class, so that "Graylog.h" is not included in
  any header files to avoid the link-time dependency pollution

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable warnings introduced by 457f023

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: add libboost_system to EXTRALIBS"

This reverts commit 21438a6773a3153556de07096919ab46c4540eee.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/page.cc: _page_mask has too many bits

It's not used anywhere, but in case it is one day...

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test_rados_api_misc: disconnected rados_monitor_log test

Piggy-back on LibRadosMiscConnectFailure to confirm that
rados_monitor_log() against a disconnected cluster context fails with
-ENOTCONN, instead of aborting.

Signed-off-by: David Disseldorp <ddiss@suse.de>

memstore: fix alignment of Page for test_pageset

test_pageset creates Pages with size=1, which messes up alignment of the
Page contents and leads to a bus error on arm64

Page::create() now adds padding to make sure Page is properly aligned,
and test_pageset verifies that alignment

Reported-by: Dan Mick <dmick@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: user quota may not adjust on bucket removal

Description:
If the user/admin removes a bucket using --force/--purge-objects options with s3cmd/radosgw-admin respectively, the user stats will continue to reflect the deleted objects for quota purposes, and there seems to be no way to reset them. User stats need to be sync'ed prior to bucket removal.

Solution:
Sync user stats before removing a bucket.

Fixes: #14507
Signed-off-by: Edward Yang eyang@us.fujitsu.com

librados: check connection state in rados_monitor_log

rados_monitor_log() may abort the calling process if called with a
disconnected rados cluster context. Return -ENOTCONN up to the caller,
rather than aborting in such cases.

Fixes: #14499
Signed-off-by: David Disseldorp <ddiss@suse.de>

OSD/ReplicatedPG: Fixing lack of object unblock when flush fails.
See: http://tracker.ceph.com/issues/14511

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test: fsx now exercises librbd journal replay

write, discard, snap create, snap protect, flatten, and resize journal
events are tests when using the new '-j' option.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: missing lock on maint operation replay callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not fire flush completions before associated write

Prevents a race condition in journal replay when committing events
to the journal.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

OSD/ReplicatePG: Fixes return value from maybe_handle_cache_detail() depending on maybe_promote() result

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com

Merge pull request #7496 from xiexingguo/xxg-wip-fixnvme

osd: BlueStore/NVMEDevice: fix compiling and fd leak

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7349 from yuyuyu101/add-pwritev

osd: FileStore: use pwritev instead of lseek+writev

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7504 from renhwsky/renhw-wip-mon-shutdown

mon: unregister command on shutdown

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7550 from tchaikov/wip-osd-cleanup

osd: a fix for HeartbeatDispatcher and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7553 from XinzeChi/wip-osdmap-affinity

osd: OSDMap: reset osd_primary_affinity shared_ptr when deepish_copy_from

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7560 from tchaikov/wip-fix-icl

crush/CrushTester: workaround a bug in boost::icl

Merge pull request #7568 from liewegas/wip-bluestore-mutex

osd: bluestore/blockdevice: use std::mutex et al

debian: fix build-depends

Signed-off-by: Rémi BUISSON <remi.buisson@cdiscount.com>

Merge pull request #7578 from ceph/fix-stripper

Stripper: Fix incorrect push_front -> append_zero change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osdc/Objecter: drop lock to unregister asok hooks

unregister blocks until any running hook has
completed, and that running hook might
be one that takes the objecter lock, so you
can't hold it while unregistering.

Signed-off-by: John Spray <john.spray@redhat.com>

common: asok should drop lock to call hook

Otherwise, hooks cannot take any locks which
were held during registration of hooks, because
it creates a cycle.

Signed-off-by: John Spray <john.spray@redhat.com>

Correctly escape block device paths for systemd

Ensure that the leading / is stripped from block device
paths before escaping for using in systemd unit names.

Fixes: #14706
Signed-off-by: James Page <james.page@ubuntu.com>

mailmap: Matt Benjamin affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Emile Snyder affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Jonas Keidel affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap:  Alan Grosskurth affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Alexey Sheplyakov affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Benoît Knecht affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Chuanhong Wang affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Gu Zhongyan affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Ivan Grcic affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Jeffrey Lu affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: JP François affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Kevin Jones affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>