librados: do not choke on asphyxiate doc generation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph_test_rados_api_io: fix new test

From 0c802e11d9f0ee993994d391d1e00aeb76965d6a.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/master' into wip-librados-fadvise

Conflicts:
src/test/librbd/test_librbd.cc

Merge pull request #3141 from ceph/wip-librbd-test-cleanup

tests: Minor cleanup to librbd test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3160 from xylv/wip-8562-rebase

rgw: Conditional PUT on ETag

Reviewed-by: Yehuda Sadeh <yehuda@rehat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3153 from ceph/wip-10296

osd: fix block device discard detection

Reviewed-by: Loic Dachary <loic@dachary.org>
Tested-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3144 from ceph/wip-aarch64

fixes for aarch64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

rgw: Conditional PUT on ETag

Fixes: #8562
Signed-off-by: Ray Lv <raylv@yahoo-inc.com>

test/librados: Add test case for librados function which can handle fadvise flas.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/librbd: Add test case for librbd function which can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

FileStore: Implement fadvise handle for read-operation.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ECBackend: Make parity data don't cache.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ObjectStore/Transaction: Add fadvise_flags to track write fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PGTransaction: Modify append/write function can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: Make async-read can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

include: Remove unused code: class triple.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

types: Add operator<< for type boost::tuple<A, B, C>.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

encoding: Add type boost::tuple<A,B,C> encoding/decode func.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PGBackend: Make objects_read_sync can handle fadvise flags

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ObjectStore: Modify read function can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: print OSDOp flags

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: increment version for new functions rbd_read2/write2,rbd_aio_read2/write2.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: add new read/write api for librbd C++ interface which handle op flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: add new read/write api for librbd C interface which handle op
flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Add set_op_flags in AioWrite to set op_flags for rbd write.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Add m_op_flags in AioRead to track op flags for rbd read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd/internal: Modify read/write api to handle op flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados/ObjectOperation: add new function set_op_flags2

After adding four read/write fadvise flags OP_FADVISE_*, those flags
can set multi flags at the same time. For compatibility, add new
function. The old func set_op_flags will be deprecated.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados: Add read/write fadvise op flags CEPH_OSD_OP_FLAG_FADVISE_*.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados: Add read/write fadvise op flags in ObjectOperationFlags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados.h: Add read/write op fadvise flags  LIBRADOS_OP_FLAG_FADVISE_*.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

unittest_blkdev: test an abbreviated /sys/block dir

Note that we assume unittest_blkdev is run from the current directory (or
rather that test/common/test_blkdev_sys_block exists).

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add simple sandboxing function for testing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3155 from athanatos/wip-8011

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Fixes: #8011
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3154 from athanatos/wip-10059

Wip 10059

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3156 from athanatos/wip-10150

ReplicatedPG::scan_range: an object can disappear between the list and t...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3157 from athanatos/wip-10157

PGLog: include rollback_info_trimmed_to in (read|write)_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3158 from athanatos/wip-10259

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o...

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::scan_range: an object can disappear between the list and the attr get

The first item in the range is often last_backfill, upon which writes
can be occuring.  It's trimmed off on the primary side anyway.

Fixes: 10150
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #2579 from yuandong1222/cache-hboject-filestore

Cache hobject filestore_key to avoid massive _reverse_nibbles calls

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3018 from tchaikov/wip-fix-safe_read_file

safe_io: do not set ending \0 in safe_read_file()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3070 from ceph/wip-10080

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

Reviewed-by: Sage Weil <sage@redhat.com>

debian: enable libgoogle-perftools-dev on arm64

These binaries haven't landed in Ubuntu, but they are in
sid and jessie for arm64. On Saucy I've installed them by
hand from ports.ubuntu.com and things seems pretty swell.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3150 from ceph/wip-10288

mon: fix `fs ls` on peons

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common/blkdev: fix block device discard check

- fix base name calculation (do not assume sda)
- reverse sense of check (it was returning false when true before?)
- add a generic helper to get other properties, too

Fixes: #10296
Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: get_block_device_base

And a test.  Ideally the test would get run on a machine with weirdo
devices like /dev/cciss/c0d1.

This mirrors the logic in ceph-disk, which has been tested.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3152 from dachary/wip-10296-journal-discard

os: do not attempt to detect discard by default

Reviewed-by: Sage Weil <sage@redhat.com>

tests: do not ignore ceph-disk.sh::test_activate_dev failure

The status of the test commands must be tested instead of being
silently ignored.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

os: do not attempt to detect discard by default

If the journal_discard option is false, the journal should no try to
guess if the device has discard capabilities.

http://tracker.ceph.com/issues/10296 Refs: #10296

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mon: fix `fs ls` on peons

This was incorrectly using pending_mdsmap instead
of mdsmap.  We didn't notice in test because of
single-mon configurations.

Fixes: #10288
Backport: giant

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3149 from ceph/wip-typo

mds: fix log typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: fix log typo

s/trvserse/traverse/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3137 from thesues/fix-strtol

Fix libstriprados::remove, use strtoll insdead of strtol

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3038 from ceph/wip-mds-readonly

Wip mds readonly

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3145 from XinzeChi/wip-finisher

Finisher: call cond signal if necessary in function queue

Reviewed-by: Sage Weil <sage@redhat.com>

Finisher: call signal if necessary in function queue

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

lttng: add int type definitions

The normal path through #include <lttng/tracepoint.h> doesn't
appear to include int defintions like uint64_t that are used
in Ceph so we add our definitions file.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

cycles: add reader for i386 and aarch64

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3119 from ceph/wip-8797

rados.py: remove Rados.__del__(); it just causes problems

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Call Rados.shutdown() explicitly before exit

This is mostly a demonstration of good behavior, as the resources will
be reclaimed on exit anyway.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rados.py: remove Rados.__del__(); it just causes problems

Recent versions of Python contain a change to thread shutdown that
causes ceph to hang on exit; see http://bugs.python.org/issue21963.
As it turns out, this is relatively easy to avoid by not spawning
threads on exit, as Rados.__del__() will certainly do by calling
shutdown(); I suspect, but haven't proven, that the problem is
that shutdown() tries to start() a threading.Thread() that never
makes it all the way back to signal start().

Also add a PendingReleaseNote and extra doc comments to clarify.

Fixes: #8797
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3129 from ceph/wip-gfcomplete-dirty

gf-complete: update submodule to newest upstream

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3071 from ceph/wip-10164

mds: drop dirty dentries in deleted directory

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

check endpoints is not empty before use it.

Signed-off-by: VRan Liu <gliuwr@gmail.com>
(cherry picked from commit 9d5e4edc15678de675d7a77badcbace0b710d553)

Merge pull request #3135 from XinzeChi/wip-3

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher

Reviewed-by: Sage Weil <sage@redhat.com>

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher.

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3140 from leseb/sample-ceph.conf

ceph.conf: update sample

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.conf: update sample

This commit adds:

* client admin socket
* change some default
* re-indent

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

tests: Minor cleanup to librbd test

The tests can now be repeated to increase the chance of hitting
edge condition failures.  Also added additional logic to immediately
fail IO tests when an error is encountered.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Fix libstriprados::remove, use strtoll insdead of strtol

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #3120 from ceph/wip-10010

ceph-osd: don't close stderr if not daemonizing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: set dirfrag version when fetching dirfrag is skipped

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph-osd: remove extra close of stderr

Otherwise, one loses log messages when running with -f or -d.  When
daemonizing, stderr is already closed in global_init_postfork_finish.

Fixes: #10010, #10113, #9810
Signed-off-by: Dan Mick <dan.mick@redhat.com>

gf-complete: update submodule to newest upstream

Keep the tree cleaner!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3073 from cholcombe973/get_pool_values_docs

Updating docs for Get Pool Value

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Modifying the docs to add the Get pool commands to match the CLI.
Signed-off-by: Chris Holcombe <chris.holcombe@nebula.com>

Merge pull request #3106 from danderson/asan-fixes

Asan fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3113 from wonzhq/name-norm

mailmap: Zhiqiang Wang name normalization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Zhiqiang Wang name normalization

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Fix stack buffer overflow reported by ASan.

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (new[] vs. delete).

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (malloc vs. C++ delete).

Signed-off-by: David Anderson <dave@natulte.net>

Merge pull request #3103 from danderson/ceph-dev-mode-python-envvar

ceph: respect the PYTHON environment variable for dev mode.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph: respect the PYTHON environment variable for dev mode.

On OSes where `python` is python3, dev mode's re-exec makes the
ceph tool fail. The standard way to fix this is by exporting
the PYTHON envvar pointing to the python2 interpreter.

Signed-off-by: Dave Anderson <dave@natulte.net>

Merge pull request #3099 from sileht/sileht/python-rados-object-lock

python-rados: Add object lock support

Reviewed-by: Loic Dachary <ldachary@redhat.com>

python-rados: Add object lock support

This change adds to the python binding the support of:
 - rados_lock_exclusive
 - rados_lock_shared
 - rados_unlock

http://tracker.ceph.com/issues/6114 Refs: #6114

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #3035 from dachary/wip-10197-arch-arm

arch: add support for HW_CAP based neon runtime detection

Reviewed-by: Sage Weil <sage@redhat.com>

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::operator==

Fixes: 10259
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>

common/ceph_context: don't import std namespace

This was broken by 7fed5dee4f96a83d1d6914f6fc0895bba2d15b99

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3020 from dachary/wip-10192-objectstore-tool-lookup

objectstore tool lookup by name & --op list filter

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: Accept json object with --pgid instead of array

It isn't anticipated that anyone would use this but keeps backward compatible

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Improve object spec parsing error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Fix errors messages in newer code

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Remove extraneous endl on error throw messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Add --format and --pretty-format support

--pretty-format defaults true
Add --format so xml output can be requested
--op list defaults to single line of json per object
To override this more human readable output use --pretty-format=false
Add testing of --op list special handling

Signed-off-by: David Zafman <dzafman@redhat.com>

.gitignore: add ceph_perf_objectstore

Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2987 from yuyuyu101/wip-9888

AsyncMessenger: Async event threads can shared by all AsyncMessenger

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3043 from yuyuyu101/objectstore-perf

Objectstore::Transaction Perf Program

Reviewed-by: Sage Weil <sage@redhat.com>