test/librados: add test that requires correct dup error detection

This only works reliably with the
objecter_retry_writes_after_first_reply setting, so make it part of
the test setup.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

test/librados: add a way to pass ceph config options

This way individual tests or testcases can change settings

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Objecter: add option for testing osd dup handling

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG: require kraken feature bit on osdmap to record write errors in the pg log

Older OSDs can't handle the error entries.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

OSDMonitor: add kraken feature bit

This is needed to turn on persisting write errors, since older OSDs
won't be able to handle them.

Other features for kraken could potentially use this as well.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG: return error codes for dups from the pg log

This prevents reordering guarded writes or deletes.

Without this, the following sequence:

delete foo -> -ENOENT
write foo -> success
(client connection fails)
resend delete foo -> success, object deleted
resend write foo -> success - dup op, so no write performed

results in the object not existing, instead of containing data. After
this change, both delete and write are detected as dups and the
original ordering is preserved.

Fixes: http://tracker.ceph.com/issues/14468
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG: skip stat invalidation when recording write errors

This is only needed for the lost/unfound use of submit_log_entries() etc.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG: persist write errors in the pg log

This is required to prevent re-ordering of guarded writes or deletes
in the presence of network failures and resends.

Use the existing submit_log_entries() method to initiate a repop that
only updates the pg log.

Keep the write error semantics close to the existing implementation -
if we have a buffer, return it, but do not persist the buffer for now.

Refs: http://tracker.ceph.com/issues/14468
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PGLog: skip indexing errors by object

Dup detection only needs them indexed by version, and keeping them out
of the object index prevents error entries from contributing to the
missing set during recovery.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

PGLog: ignore error entries when constructing the missing set

Errors should only be used for dup detection.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd_types: add an error pg log entry type

This will store write error codes for use in dup op detection.
A few places use checks assuming is_update() or is_delete() are
opposites - fix those to ignore or consider errors, as appropriate.

Refs: http://tracker.ceph.com/issues/14468
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG: removed unused obc check

The one place this was set was removed by e7edf20fb7264bfaeeb30ac0185e9edc925878b6

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG: removed unused OnComplete struct

Added but never used in e7edf20fb7264bfaeeb30ac0185e9edc925878b6

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG: remove unused mark_object_lost()

The only caller was removed in e7edf20fb7264bfaeeb30ac0185e9edc925878b6

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10201 from xiexingguo/xxg-wip-bluestore-2016-07-08

os/bluestore: misc cleanups

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10157 from cbodley/wip-rgw-sync-error-repo

rgw: fix error_repo segfault in data sync

rgw: fix error_repo segfault in data sync

RGWDataSyncShardCR will only allocate an error_repo if it's doing
incremental sync, so RGWDataSyncSingleEntryCR needs to guard against a
null error_repo

also, RGWDataSyncShardCR::stop_spawned_services() was dropping the last
reference to the error_repo before calling drain_all(), which meant that
RGWDataSyncSingleEntryCR could still be holding a pointer. now uses a
boost::intrusive_ptr in RGWDataSyncSingleEntryCR to account for its
reference

Fixes: http://tracker.ceph.com/issues/16603
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10184 from ktdreyer/doc-rados-operations-systemd

doc: rm SysV instructions, add systemd

Merge pull request #9570 from zaitcev/wip-5073-multen-doc

rgw: Add documentation for the Multi-tenancy feature

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10151 from oritwas/wip-rgw-fix-cls-sync-stats

rgw: register the correct handler for cls_user_complete_stats

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9934 from tchaikov/wip-build-on-armf

common: instantiate strict_si_cast<long> not strict_si_cast<int64_t>

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10204 from ceph/wip-cmake

cmake: fixes for pypi changes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common: instantiate strict_si_cast<long> not strict_si_cast<int64_t>

this fixes the build on armf.

on 32bit platforms, cstdint is very likely to

 typedef long long int int64_t;

this results in compilation error like

 `common/strtol.cc:190:75: error: duplicate explicit instantiation of 'T
 strict_si_cast(const char, std::string) [with T = long long int;
 std::string = std::basic_string]'

 [-fpermissive]
 template int64_t strict_si_cast(const char *str, std::string *err);
 ^`

we can address this by instantiate the primitive type of `long long`
instead of `in64_t`.

Fixes: http://tracker.ceph.com/issues/16398
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8189 from chardan/wip-chardan-15100

librados examples: link and include from current source tree by default.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: release compressor if comp_mode turned out to be none

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop unnecessary increase of iterator

We are going to break anyway.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: assert to confirm that we don't access violation

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop unused member _bytes

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: check against that we don't access violation

Some day we might want make this interface more applicable by handling
the abnormal cases more gracefully...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10137 from xiexingguo/xxg-wip-bluestore-2016-07-05-02

os/bluestore: change algorithm of compression header from string to int

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: change algorithm of compression header from string to int

The literal description of compression algorithm can vary from
various compression types and thus increases the complexity of
en/decoding, which as a result can cause chaos. Also it can be
more efficient by translating it into a fixed-length type.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: register the correct handler for cls_user_complete_stats

Fixes: http://tracker.ceph.com/issues/16624
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

pybind: put temp .o files in $TMPDIR

otherwise they will go to $PWD/$TMPDIR instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: creage egg_info files into build directory

otherwise they will go to the source tree, and "git ls-files" will list
them as untracked files, which annoy gitbuilder-ceph-tarball*-cmake
gitbuilders. like

+ echo 'error: Added files:'
error: Added files:
+ cat .git/added-files
src/pybind/rados/rados.egg-info/PKG-INFO
...

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10186 from ceph/wip-cmake-no-with-mds

cmake: remove WITH_MDS option

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10163 from stiopaa1/msg_asyncmess_removeUnusedFunction_create_anon_connection

msg/AsyncMessenger: remove unused function

Merge pull request #10178 from ceph/wip-cmake

cmake: use stock Find* modules.

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: remove  option

Signed-off-by: Ali Maredia <amaredia@redhat.com>

doc: rm SysV instructions, add systemd

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10128 from xiexingguo/xxg-wip-bluestore-2016-07-05

os/bluestore: introduce power 2 macros for block alignment and rounding

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10180 from theanalyst/mrgw/cmake-fixes

mrgw: search for cmake build dir.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mrgw: search for cmake build dir.

Similar to mstart and mstop, mrgw also needs to find the correct build
dir as we pass in the pidfile and asokfile which otherwise falls back to
src/run dir.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #10116 from tchaikov/wip-cmake-make-check

test: make check using cmake

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10097 from yehudasa/wip-mrun-cmake

mrun, mstart.sh, mstop.sh: search for cmake build directory

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9489 from athanatos/wip-16156

ReplicatedPG: call op_applied for submit_log_entries based repops

Reviewed-by: David Zafman <dzafman@redhat.com>

cmake: use the stock FindCURL.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use the stock FindEXPAT.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-helpers.sh: disable test_pg_scrub()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10139 from SUSE/wip-cmake-really-fix-fcgi-include-dir

cmake: Really add FCGI_INCLUDE_DIR to include_directories for rgw

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9981 from yangdongsheng/rbd_set_vals

cls_rbd: set omap values in batch during image creation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10155 from trociny/wip-16555-workaround

qa/workunits/rbd: before removing image make sure it is not bootstrapped

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10164 from dillaman/wip-cmake-test-rbd-mirror

cmake: ceph_test_rbd_mirror does not require librados_test_stub

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Really add FCGI_INCLUDE_DIR to include_directories for rgw

Commit 3cf6c53 was incorrect.  FCGI_INCLUDE_DIR doesn't need to be
set in src/rgw/CMakeLists.txt, but it does need to be set for the
rgw_a target in src/CMakeLists.txt, as well as for the
ceph_test_librgw_file and ceph_test_librgw_file_nfsns targets in
src/test/CMakeLists.txt.  I can only assume that I must not have
done a completely clean rebuild at some point when testing a
reworked version of the earlier commit :-/

This is only a problem for distros that keep the FCGI headers in
/usr/include/fastcgi/ (e.g.: SUSE).

This commit also removes a redundant include of <fcgiapp.h>

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #9083 from badone/wip-doc-last_epoch_started

doc/dev: Fix missing code section due to no lexer for "none"

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9704 from badone/wip-ceph-authtool-fixes

authtool: Enhance argument combinations validation

Reviewed-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: use "make check" for running tests

"make all" does not offer "ceph-disk" and "ceph-detect-init" for
testing. as they are solely used for testing purpose. instead, these two
python command line packages are installed by the "install" target. so
we need to use "make check" to 1) prepare the test dependencies 2)
launch ctest to perform the test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: add ceph-{disk,detect-init} to ALL"

This reverts commit 19c0731751df9642cc5b8c583ea41d12e0b5ec41.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test//osd/osd-copy-from.sh: use absolute path for "rados put"

"rados -p rbd put foo rados" does not work if "rados" is not in current
path. so change it to "rados -p rbd put foo $(which rados)"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass more env vars to osd-copy-from.sh

we should use the cmake function of add_ceph_test() to add
osd-copy-from.sh as a test. then we won't miss any env variables.
w/o this change, $CEPH_BUILD_VIRTUALENV is not passed to
osd-copy-from.sh.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: respect $TMPDIR

create temp directory and files in $TMPDIR. the $TMPDIR is hard-wired to
/tmp before this change, we'd better respect the env variable $TMPDIR,
so it would be more consistent, and easier to do the cleanup if any.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: vstart_wrapper.sh run tests in $TMPDIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: run "make check" with a unique $TMPDIR

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: use p2 macros to simplify _do_write_small() process

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #9702 from BlaXpirit/ceph-argparse-py3

ceph.in: python 3 compatibility of the ceph CLI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9749 from songbaisen/a10

mon: remove the redundant is_active judge in PaxosService

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9792 from stiopaa1/mon_monitorcc_addMutexLocker

mon/Monitor.cc:replce lock/unlock with Mutex:Lockr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9939 from Yan-waller/yj-wip-radosclientwaitforosd

rados/client: fix waiting on the condition variable more efficient.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9947 from yuyuyu101/wip-remove-check

osd: remove dispatch queue check since we don't queue hb message to this

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9833 from onyb/wip-rados-pypi

pybind: make rados ready for PyPI

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10109 from xiexingguo/xxg-wip-bluestore-2016-07-02

os/bluestore: simplify LRUCache::trim()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8980 from stiopaa1/osd_unneededmapassign

osd/OSD.h: remove unneeded line

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8959 from hjwsm1989/hj-test

  os/filestore/LFNIndex: remove unused variable 'subdir_path'

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #8687 from mslovy/wip-peering-msg-priority

message: optimization for message priority strategy

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10014 from gaowanlong/indent_caused_compile_warning

build: fix wrong indent caused compile warning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9970 from dillaman/wip-16349

rbd-mirror: gracefully handle being blacklisted

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

cmake: ceph_test_rbd_mirror does not require librados_test_stub

Linking both librados_test_stub and librados results in failed
test cases.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

msg/AsyncMessenger: remove unused function

Remove create_anon_connection() function.

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10016 from ceph/wip-cmake-rpm

rpm: build rpm with cmake

Reviewed-by: Ali Maredia <amaredia@redhat.com>

run-make-check.sh: run make check with cmake if do_cmake.sh is available

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: gracefully restart pool replayer when blacklisted

Fixes: http://tracker.ceph.com/issues/16349
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: do not clear image replayer error code after shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: image deleter should use provided librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: each pool replayer should use its own librados connection

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10145 from dillaman/wip-cmake-rbd-python

cmake: fix incorrect dependencies to librados

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10119 from renhwztetecs/renhw-wip-cdentry-dump

mds: fixup dump Formatter' type error; add path_ino and is_primary in the CDentry::dump()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10062 from renhwztetecs/renhw-wip-filer

client/filer: cleanup the redundant judgments of _write&&_fallocate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10027 from batrick/client-root-lock

client: add missing client_lock for get_root

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10100 from fullerdj/wip-djf-16144

tools/cephfs: Remove cephfs-data-scan tmap_upgrade

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9971 from fullerdj/wip-djf-16288

mds: Kill C_SaferCond in evict_sessions()

Reviewed-by: John Spray <john.spray@redhat.com>

qa/workunits/rbd: before removing image make sure it is not bootstrapped

If an image is being bootstrapped, it implies that the rbd-mirror
daemon currently has the image open. The removal API will prevent the
removal of any image that is opened by another client.

Works-around: http://tracker.ceph.com/issues/16555
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cmake: fix incorrect dependencies to librados

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rpm: build rpm with cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10136 from runsisi/sisi-wip-fix-missing-return

librbd: fix missing return statement if failed to get mirror image state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10094 from athanatos/wip-throttle-test

test/common/Throttle.cc: fix race in shutdown

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #10088 from pritha-srivastava/wip_s3_signedurl_errors

rgw: Fix for using port 443 with pre-signed urls.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10112 from tchaikov/wip-cmake

cmake changes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10134 from trociny/wip-run-rbd-tests-fixup

test: run-rbd-tests test cmake fixup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10133 from tchaikov/wip-fixc-cephtool-test.sh

test: cephtool/test.sh: fix expect_false() calls

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10072 from dillaman/wip-16561

librbd: failed assertion after shrinking a clone image twice

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

librbd: fix missing return statement if failed to get mirror image state

Signed-off-by: runsisi <runsisi@zte.com.cn>