rgw: add RGWSyncLogTrimThread to RGWRados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWDataLogTrimCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWRadosTimelogTrimCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: enable async calls to time_log_trim

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rest handlers to query sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use separate cr manager for read_sync_status

RGWCoroutinesManager::run() is not reentrant, so concurrent users of
read_sync_status() must use different managers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: change read_sync_status interface

RGWDataSyncStatusManager::read_sync_status() now operates on the given
parameter, rather than its internal member variable. this allows
multiple concurrent readers, which is needed for the rest interface

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: expose sync managers through RGWRados

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: don't ignore ENOENT in RGWRemoteDataLog::read_sync_status()

rest handlers for sync status need to return ENOENT errors. the only
other callers are in radosgw-admin, so the ENOENT errors are ignored at
those call sites instead

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add json decoders for data sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add dout_prefix for rgw_data_sync.cc

and took out the redundant 'rgw' from 'rgw meta sync:'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use RGWShardCollectCR for RGWReadDataSyncStatusCoroutine

this allows us to limit the number of outstanding requests for shard
markers

there also appeared to be issues with spawning the shard CRs
from RGWReadDataSyncStatusCoroutine::handle_data(), because
handle_data() was returning before the shard CRs completed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add empty_on_enoent flag to RGWSimpleRadosReadCR

RGWSimpleRadosReadCR won't currently fail with ENOENT, but instead
passes an empty object to handle_data(). add an empty_on_enoent flag to
the constructor, defaulting to true, to make this behavior optional for
callers that do want to fail on ENOENT

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Revert RGW torrent

Revert the RGW torrent commit for now, it was causing issues.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #10377 from chhabaramesh/extent_alloc

os/bluestore:Fix size calculation in bitallocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9589 from zhouruisong/master

rgw:add a s3 API of make torrent for a object

Merge pull request #10244 from jcsp/wip-cephfs-health-doc

doc/cephfs: explain the various health messages

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc/cephfs: explain the various health messages

Signed-off-by: John Spray <john.spray@redhat.com>

os/bluestore:Fix size calculation in bitallocator

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10149 from wjwithagen/patch-12

CMakeList.txt: link ceph_objectstore_tool against fuse only if WITH_FUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/tools/CMakeLists.txt: conditionalise the FUSE usage

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10366 from ceph/wip-cmake-debian

debian: bump compat to 9

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #10361 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10355 from yehudasa/wip-16742

rgw: back off bucket sync on failures, don't store marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix collection of object sync errors

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix marker tracker completion handling

Was not tracking high markers correctly. Could only work if there was a single
hole in the completion range. Just keep a map of all the complete entries.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: collect() stops if error encoutered

and returns true if needed to be called again

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

debian: bump compat to 9

dh_auto_configure set -DCMAKE_BUILD_TYPE=None so the default does not
take effect at all. this is on purpose, see [1].

and dpkg-buildflags is able to produce the suggested compiling flags by
debian policy. dh_* can pass these flags to cmake only if it works at
compat>=9, see [2].

and we are safe to move to compat 9, as jewel's supported debian based
distro is jessie (with debhelper 9.20150101), ubuntu trusty (with
debhelper 9.20131227ubuntu1)

---
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=701233#35
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=653916

Fixes: http://tracker.ceph.com/issues/16744
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10368 from oritwas/wip-rgw-upgrade-period-commit

rgw: fix upgrade from old multisite to new multisite configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10360 from ceph/wip-include-include

cmake: add_subdirectory(include)

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10315 from onyb/wip-cephfs-pypi

pybind: Rework cephfs/setup.py for PyPI

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10316 from batrick/mds-standby-assertions

mds: add assertions for standby_daemons invariant

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10347 from renhwztetecs/renhw-wip-test-add-fsdump

test/mds: add fs dump in test_ceph_argparse.py

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10202 from xiaoxichen/fix_16588

mds/MDSMap default metadata pool to -1 (was: output None instead of 0 when no fs present.)

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9864 from ajarr/wip-15615

ceph_volume_client: store authentication metadata

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10327 from xiexingguo/xxg-wip-bluestore-2016-07-18

os/bluestore: misc fixes/cleanups

Mark's Comments:
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR may have contributed to write performance improvements along with #10257.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10257 from chhabaramesh/extent_alloc

os/bluestore: extent alloc functionality for stupid and bitmap allocator

Mark's Comments:
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR appears to result in a large sequential write performance increase (Up to ~3.5X), and may contribute to a small random write performance increase as well.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

rgw: fix upgrade from old multisite to new multisite configuration

We need to store the updated current period after adding the old converted regions

Fixes: http://tracker.ceph.com/issues/16751
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9082 from stiopaa1/osd_TestOpsSocketHook_removeUnneededParam

pass string by const ref and add override to virtual function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9667 from SUSE/wip-15984

ceph-osd-prestart.sh: drop Upstart-specific code

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #10339 from tchaikov/wip-16675

selinux: allow read /proc/<pid>/cmdline

Reviewed-by: Boris Ranto <branto@redhat.com>

cmake: remove unused Findfio.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Findfcgi.cmake cleanups

FindPackageHandleStandardArgs() takes care of the find_package()
boilderplate stuff. so no need to repeat them. and remove the checkings
in env variables.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not include $lrc_srcs in unittest_erasure_code_lrc

as ${lrc_srcs} is not defined in the scope where
unittest_erasure_code_lrc is added as a target. and instead we link
unittest_erasure_code_lrc against ec_lrc dynamic library. so we can
safely remove ${lrc_srcs} from unittest_erasure_code_lrc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/EXTRA_LIBS/EXTRALIBS/

it's a typo, i think. as EXTRA_LIBS is never defined anywhere.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/HAVE_FUSE/WITH_FUSE/ s/HAVE_LIBFUSE/WITH_FUSE/

we don't define HAVE_FUSE, and HAVE_LIBFUSE is defined to be compatible
with autotools. so use WITH_FUSE in cmake whenever possible.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove $UNITTEST_CXX_FLAGS from ceph-kvstore-tool's linked libs

it does not make sense to link against CXX_FLAGS.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: CMAKE_MODULE_PATH is a list and empty by default

we should not reference it using ${CMAKE_MODULE_PATH} before assigning
to it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add_subdirectory(include)

fixes the packaging failure introduced by 741f990

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/PTHREAD_LIBS/CMAKE_THREAD_LIBS_INIT/

${CMAKE_THREAD_LIBS_INIT} is defined by find_package(Thread), while
PTHREAD_LIBS is defined by the acx_pthread.m4.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use HAVE_ARMV8_CRC not ${HAVE_ARMV8_CRC}

as this variable could be undefined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use FindBacktrace instead of Findexecinfo

always better to use upstream cmake modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add "WITH_PROFILER" option

this option matches '--with-profiler' option in autoconf. and it is off
by default. we should not link against libprofiler unless asked to do
so. this change fixes this problem.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10353 from rjfd/wip-fix-ceph-mount

rpm: Fix creation of mount.ceph symbolic link for SUSE distros

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10351 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: back off bucket sync on failures, don't store marker

Fixes: http://tracker.ceph.com/issues/16742
If we fail on any single entry in bucket, skip updating the marker tracker
so that next time we'll go over that entry, and back off. This will trigger
a report to the data sync error repo and eventually a retry on the failing
object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10295 from xiexingguo/xxg-wip-bluestore-2016-07-14

kv/MemDB: misc fixes and cleanups

Mark's Comments:
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10294 from xiexingguo/xxg-wip-bluestore-2016-07-13

os/bluestore: use BE for gifting and reclaiming from bluefs

Mark's Comments:
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10277 from xiexingguo/xxg-wip-fix-posix-fallocate

os/bluestore: fix error handling of posix_fallocate()

Mark's Comments:
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10249 from dillaman/wip-16622

rbd-mirror: snap rename does not properly replicate to peers

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10313 from ceph/objclass-perm-feedback

Objclass perm feedback

Reviewed-by: Kefu Chai <kchai@redhat.com>

rpm: Fix creation of mount.ceph symbolic link for SUSE distros

Signed-off-by: Ricardo Dias <rdias@suse.com>

cmake: disable VTA for osdcap.cc

this silences the warning of:
```
In file included from
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:58:0,
                 from
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/osd/osdcap.cc:20:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/osd/osdcap.cc:
In member function ‘virtual void
OSDCap_AllowClassMulti_Test::TestBody()’:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/osd/osdcap.cc:766:6:
note: variable tracking size limit exceeded with
-fvar-tracking-assignments, retrying without
 TEST(OSDCap, AllowClassMulti) {
      ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/internal/gtest-internal.h:1211:3:
note: in definition of macro ‘GTEST_TEST_CLASS_NAME_’
   test_case_name##_##test_name##_Test
   ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2181:3:
note: in expansion of macro ‘GTEST_TEST_’
   GTEST_TEST_(test_case_name, test_name, \
   ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2187:42:
note: in expansion of macro ‘GTEST_TEST’
 # define TEST(test_case_name, test_name) GTEST_TEST(test_case_name,
 # test_name)
                                          ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/osd/osdcap.cc:766:1:
note: in expansion of macro ‘TEST’
 TEST(OSDCap, AllowClassMulti) {
 ^
```

see also b668051

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove util.cc from lib{rados,cephfs},ceph-objectstore-tool

util.cc is included by both librados and libcephfs, the `lvm` static
variable in `lsb_release_parse()` will be free twice by them. this
could lead to double free issue. and util.cc is not used by client at all, so
remove it from them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: restructure src/CMakeLists.txt in a more hierarchical way

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: preprocess journal events prior to applying

Fixes: http://tracker.ceph.com/issues/16622
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: event preprocessor to handle snap rename operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improve journaling debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: separate journal event decoding and processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: record original snap name in snap rename journal event

Remote peers need a key to map snapshot ids between clusters.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simple duplicate op checks for all maintenance operations

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: exercise snapshot renames within rbd-mirror test

Snapshot rename operations utilize the (cluster) unique snapshot
sequence to prevent attempts at replays. When mirroring to a
different cluster, these sequences will not align.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10250 from gregsfortytwo/wip-sessionmap-committing

mds: Return "committing" rather than "committed" member in get_committing

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10346 from tchaikov/wip-make-check

test: ceph-disk.sh do not kill all daemons

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test/mds: add fs dump in test_ceph_argparse.py

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Revert "run-make-check.sh: run tests in two steps"

This reverts commit cd0dfc305d3861b9baf332a1cc13f1c716d8017b.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: label some tests with "Racing""

This reverts commit 3a21d0040a7a96aefcf0f4876ee4006c4225019c.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph-disk.sh: do not kill all daemons

this causes mysterious "racing" issues when running ceph-helper tests in
parallel.

Fixes: http://tracker.ceph.com/issues/16729
Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind: Replace -I with -iquote in cephfs test

Signed-off-by: Anirudha Bose <ani07nov@gmail.com>

selinux: allow read /proc/<pid>/cmdline

we read /proc/<pid>/cmdline to figure out who is terminating us.

Fixes: http://tracker.ceph.com/issues/16675
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10317 from wjwithagen/wip-wjw-fix-dl-lib

Cmake: fix using CMAKE_DL_LIBS instead of dl

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDSMap output -1 instead of 0 when no fs present.

When no fs present, the mdsmap::dump will output metadata_pool: 0,
which is misleading.

Change the initial value of metadata_pool from 0 to -1.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

Merge pull request #10147 from SUSE/wip-16598

rpm: move mount.ceph from ceph-base to ceph-common and add symlink in /sbin for SUSE

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com

Merge pull request #9411 from ceph/wip-rocksdb-perf

os/RocksDBStore: use effective Get API instead of iterator api

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10130 from yonghengdexin735/wip-zzz-add-

src/common/buffer.cc fix judgment for lseek

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10197 from songbaisen/b0

mon: write fsid use the right return value

Reviewed-by: Sage Weil <sage@redhat.com>

pybind: Rework cephfs/setup.py for PyPI

Also use `set_include_dirs' in distutils.ccompiler instead of `add_include_dir'
for supporting non-standard include directories.

Signed-off-by: Anirudha Bose <ani07nov@gmail.com>

Merge pull request #9291 from trociny/wip-15715

rbd-nbd does not properly handle resize notifications

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9878 from neurodrone/rbd_api_shrink_check_on_resize

librbd: add explicit shrink check while resizing images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #9955 from ukernel/wip-mds-snap-failover

mds: snap failover fixes

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10283 from ceph/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

build/ops: build mount.ceph and mount.fuse.ceph as client binaries

Signed-off-by: Nathan Cutler <ncutler@suse.com>

extent alloc functionality for stupid and bitmap allocator

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

os/bluestore: use p2 macros to simplify bit-allocator block alignment

Mark's comments:

This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Closes #10253

os/bluestore: require block_size to be power of 2 aligned

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: use ISP2 macro for zone/span size checking

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix potential uninitialized nid of onode

The _zero() process may implicitly create a new onode,
thus we shall call _assign_nid() to initialize the nid
properly. And if the onode already has one, _assign_nid()
does nothing.

So it is proper to call _assign_nid() here under any case.

Mark's comments:

This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Closes #10236

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: check against we don't overflow

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: try to reap as many collections as we can

So if there is one collection getting contiguously stucking,
we don't abort at the same point each time.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: make device size of BitFreelistManager is block-size aligned

Otherwise if we try to set past-eof blocks as allocated durint create(),
the call to _xor() will trigger the firing of the following assert:

   assert((length & block_mask) == length);

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: end scope of std::hex properly; convert csum error to EIO

Mark's comments:

This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Closes #10225

os/bluestore: end scope of std::hex properly

To avoid side-effects by accident.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: convert csum error to EIO

The verify_csum() method either returns -1 or -EOPNOTSUPP, which
is not very proper and difficult for user understanding.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: assert lextent is shared

Otherwise we are risking of accessing violation.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: drop duplicated assignment of result code

These two methods never fail actually.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: improve _do_read() a little

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
os/bluestore: assert decoding of shard of key to be successful

Otherwise we are risking of acessing null pointer.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rpm: move mount.ceph from ceph-base to ceph-common

Ceph clients use mount.ceph to mount CephFS filesystems, and
ceph-base is not expected to be installed on client systems.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm: create mount.ceph symlink in /sbin (SUSE only)

Fixes: http://tracker.ceph.com/issues/16598
Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph_volume_client: version on-disk metadata

Version on-disk metadata with two attributes,
'compat version', the minimum CephFSVolume Client
version that can decode the metadata, and
'version', the version that encoded the metadata.

Signed-off-by: Ramana Raja <rraja@redhat.com>

ceph_volume_client: add versioning

Add class attributes to CephFSVolumeClient to version
its capabilities.

'version' attribute stores the current version number
of CephFSVolumeClient.

'compat_version' attribute stores the earliest version
number of CephFSVolumeClient that the current version is
compatible with.

Fixes: http://tracker.ceph.com/issues/15406
Signed-off-by: Ramana Raja <rraja@redhat.com>