tests: fix signedness compilation warnings

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3172 from dachary/wip-10146-retrigger-udev

ceph-disk: test re-using an existing journal partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: LOG.info instead of print

When a non fatal error happens with parted, use LOG.info to display it
instead of print.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: test re-using an existing journal partition

Add a ceph-disk test to first setup an OSD with a separate journal
block device, then tear down the OSD (simulating a failure) and create
a new OSD which re-uses the same journal device.

Add create_dev / destroy_dev helpers that encapsulate the operations
that ensure the partition table is up to date in the kernel and the
symlinks are created as expected. In particular it makes sure the kernel
is aware that the partition table of a newly created device is
empty. If the device previously existed and the kernel was not informed
of the latest partition table updates via partprobe / partx, it may
have cached an old partition table which can create all sorts of
unexpected behaviors such as a failure to create the by-partuuid
symbolic links as described in http://tracker.ceph.com/issues/10146
Refs: #10146

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: fix get_partition_uuid for loop/cciss

Loopback and cciss devices have an extra 'p' before the partition
number. Add exceptional handling of these devices.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

ceph-disk: don't change the journal partition uuid

We observe that the new /dev/disk/by-partuuid/<journal_uuid>
symlink is not always created by udev when reusing a journal
partition. Fix by not changing the uuid of a journal partition
in this case -- instead we can reuse the existing uuid (and
journal_symlink) instead. We also now assert that the symlink
exists before further preparing the OSD.

Fixes: #10146
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Tested-by: Dan van der Ster <daniel.vanderster@cern.ch>

documentation: simplify running make check

Encapsulate the compilation steps (install dependencies, autogen.sh,
configure, make check) in the run-make-check.sh script. Update the
developer documentation to point to this script instead of multiple
steps.

It is intended as a tool to help new developer make sure their patch is
sane, it focuses on efficiency (runs make check in parallel if possible)
and coverage (enables docker based tests if possible).

http://tracker.ceph.com/issues/10265 Fixes: #10265

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install parted in centos container

Because ceph-disk depends on it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install hdparm in containers

The ceph-disk tests use hdparm, install it in containers that run
them.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: update centos docker file to epel 7.5

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: docker tests only need a workdir, not a clone

Instead of cloning the repository, create a work directory that has
symbolic links to the original .git, using

https://github.com/git/git/blob/master/contrib/workdir/git-new-workdir

It resolves the problem of fetching a commit that is not attached to any
ref, which is apparently not implemented in the git protocol (discussed
on irc.freenode.net#git).

http://tracker.ceph.com/issues/10264 Fixes: #10264

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3126 from ceph/wip-cleanup-removal

osd: Remove dead code related to old pg removal mechanism

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3141 from ceph/wip-librbd-test-cleanup

tests: Minor cleanup to librbd test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3160 from xylv/wip-8562-rebase

rgw: Conditional PUT on ETag

Reviewed-by: Yehuda Sadeh <yehuda@rehat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3153 from ceph/wip-10296

osd: fix block device discard detection

Reviewed-by: Loic Dachary <loic@dachary.org>
Tested-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3144 from ceph/wip-aarch64

fixes for aarch64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

rgw: Conditional PUT on ETag

Fixes: #8562
Signed-off-by: Ray Lv <raylv@yahoo-inc.com>

unittest_blkdev: test an abbreviated /sys/block dir

Note that we assume unittest_blkdev is run from the current directory (or
rather that test/common/test_blkdev_sys_block exists).

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add simple sandboxing function for testing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3155 from athanatos/wip-8011

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Fixes: #8011
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3154 from athanatos/wip-10059

Wip 10059

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3156 from athanatos/wip-10150

ReplicatedPG::scan_range: an object can disappear between the list and t...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3157 from athanatos/wip-10157

PGLog: include rollback_info_trimmed_to in (read|write)_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3158 from athanatos/wip-10259

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o...

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::scan_range: an object can disappear between the list and the attr get

The first item in the range is often last_backfill, upon which writes
can be occuring.  It's trimmed off on the primary side anyway.

Fixes: 10150
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #2579 from yuandong1222/cache-hboject-filestore

Cache hobject filestore_key to avoid massive _reverse_nibbles calls

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3018 from tchaikov/wip-fix-safe_read_file

safe_io: do not set ending \0 in safe_read_file()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3070 from ceph/wip-10080

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

Reviewed-by: Sage Weil <sage@redhat.com>

debian: enable libgoogle-perftools-dev on arm64

These binaries haven't landed in Ubuntu, but they are in
sid and jessie for arm64. On Saucy I've installed them by
hand from ports.ubuntu.com and things seems pretty swell.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3150 from ceph/wip-10288

mon: fix `fs ls` on peons

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common/blkdev: fix block device discard check

- fix base name calculation (do not assume sda)
- reverse sense of check (it was returning false when true before?)
- add a generic helper to get other properties, too

Fixes: #10296
Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: get_block_device_base

And a test.  Ideally the test would get run on a machine with weirdo
devices like /dev/cciss/c0d1.

This mirrors the logic in ceph-disk, which has been tested.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3152 from dachary/wip-10296-journal-discard

os: do not attempt to detect discard by default

Reviewed-by: Sage Weil <sage@redhat.com>

tests: do not ignore ceph-disk.sh::test_activate_dev failure

The status of the test commands must be tested instead of being
silently ignored.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

os: do not attempt to detect discard by default

If the journal_discard option is false, the journal should no try to
guess if the device has discard capabilities.

http://tracker.ceph.com/issues/10296 Refs: #10296

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mon: fix `fs ls` on peons

This was incorrectly using pending_mdsmap instead
of mdsmap.  We didn't notice in test because of
single-mon configurations.

Fixes: #10288
Backport: giant

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3149 from ceph/wip-typo

mds: fix log typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: fix log typo

s/trvserse/traverse/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3137 from thesues/fix-strtol

Fix libstriprados::remove, use strtoll insdead of strtol

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3038 from ceph/wip-mds-readonly

Wip mds readonly

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3145 from XinzeChi/wip-finisher

Finisher: call cond signal if necessary in function queue

Reviewed-by: Sage Weil <sage@redhat.com>

Finisher: call signal if necessary in function queue

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

lttng: add int type definitions

The normal path through #include <lttng/tracepoint.h> doesn't
appear to include int defintions like uint64_t that are used
in Ceph so we add our definitions file.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

cycles: add reader for i386 and aarch64

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3119 from ceph/wip-8797

rados.py: remove Rados.__del__(); it just causes problems

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Call Rados.shutdown() explicitly before exit

This is mostly a demonstration of good behavior, as the resources will
be reclaimed on exit anyway.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rados.py: remove Rados.__del__(); it just causes problems

Recent versions of Python contain a change to thread shutdown that
causes ceph to hang on exit; see http://bugs.python.org/issue21963.
As it turns out, this is relatively easy to avoid by not spawning
threads on exit, as Rados.__del__() will certainly do by calling
shutdown(); I suspect, but haven't proven, that the problem is
that shutdown() tries to start() a threading.Thread() that never
makes it all the way back to signal start().

Also add a PendingReleaseNote and extra doc comments to clarify.

Fixes: #8797
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3129 from ceph/wip-gfcomplete-dirty

gf-complete: update submodule to newest upstream

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3071 from ceph/wip-10164

mds: drop dirty dentries in deleted directory

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

check endpoints is not empty before use it.

Signed-off-by: VRan Liu <gliuwr@gmail.com>
(cherry picked from commit 9d5e4edc15678de675d7a77badcbace0b710d553)

Merge pull request #3135 from XinzeChi/wip-3

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher

Reviewed-by: Sage Weil <sage@redhat.com>

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher.

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3140 from leseb/sample-ceph.conf

ceph.conf: update sample

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.conf: update sample

This commit adds:

* client admin socket
* change some default
* re-indent

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

tests: Minor cleanup to librbd test

The tests can now be repeated to increase the chance of hitting
edge condition failures.  Also added additional logic to immediately
fail IO tests when an error is encountered.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Fix libstriprados::remove, use strtoll insdead of strtol

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #3120 from ceph/wip-10010

ceph-osd: don't close stderr if not daemonizing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: set dirfrag version when fetching dirfrag is skipped

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph-osd: remove extra close of stderr

Otherwise, one loses log messages when running with -f or -d.  When
daemonizing, stderr is already closed in global_init_postfork_finish.

Fixes: #10010, #10113, #9810
Signed-off-by: Dan Mick <dan.mick@redhat.com>

gf-complete: update submodule to newest upstream

Keep the tree cleaner!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: Remove dead code related to old pg removal mechanism

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #3073 from cholcombe973/get_pool_values_docs

Updating docs for Get Pool Value

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Modifying the docs to add the Get pool commands to match the CLI.
Signed-off-by: Chris Holcombe <chris.holcombe@nebula.com>

Merge pull request #3106 from danderson/asan-fixes

Asan fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3113 from wonzhq/name-norm

mailmap: Zhiqiang Wang name normalization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Zhiqiang Wang name normalization

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Fix stack buffer overflow reported by ASan.

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (new[] vs. delete).

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (malloc vs. C++ delete).

Signed-off-by: David Anderson <dave@natulte.net>

Merge pull request #3103 from danderson/ceph-dev-mode-python-envvar

ceph: respect the PYTHON environment variable for dev mode.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph: respect the PYTHON environment variable for dev mode.

On OSes where `python` is python3, dev mode's re-exec makes the
ceph tool fail. The standard way to fix this is by exporting
the PYTHON envvar pointing to the python2 interpreter.

Signed-off-by: Dave Anderson <dave@natulte.net>

Merge pull request #3099 from sileht/sileht/python-rados-object-lock

python-rados: Add object lock support

Reviewed-by: Loic Dachary <ldachary@redhat.com>

python-rados: Add object lock support

This change adds to the python binding the support of:
 - rados_lock_exclusive
 - rados_lock_shared
 - rados_unlock

http://tracker.ceph.com/issues/6114 Refs: #6114

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #3035 from dachary/wip-10197-arch-arm

arch: add support for HW_CAP based neon runtime detection

Reviewed-by: Sage Weil <sage@redhat.com>

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::operator==

Fixes: 10259
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>

common/ceph_context: don't import std namespace

This was broken by 7fed5dee4f96a83d1d6914f6fc0895bba2d15b99

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3020 from dachary/wip-10192-objectstore-tool-lookup

objectstore tool lookup by name & --op list filter

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_objectstore_tool: Accept json object with --pgid instead of array

It isn't anticipated that anyone would use this but keeps backward compatible

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Improve object spec parsing error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Fix errors messages in newer code

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Remove extraneous endl on error throw messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Add --format and --pretty-format support

--pretty-format defaults true
Add --format so xml output can be requested
--op list defaults to single line of json per object
To override this more human readable output use --pretty-format=false
Add testing of --op list special handling

Signed-off-by: David Zafman <dzafman@redhat.com>

.gitignore: add ceph_perf_objectstore

Signed-off-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2987 from yuyuyu101/wip-9888

AsyncMessenger: Async event threads can shared by all AsyncMessenger

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3043 from yuyuyu101/objectstore-perf

Objectstore::Transaction Perf Program

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3087 from dachary/wip-9785-dmcrypt-keys-permissions

ceph-disk: dmcrypt file permissions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3094 from yuandong1222/bufferlist-get_contiguous

common: bufferlist::get_contiguous return 0 when param len == 0

Reviewed-by: Sage Weil <sage@redhat.com>

common: bufferlist::get_contiguous return 0 when param len == 0

Change-Id: I3eaee45b12c634687cd1b294b3c264aed8cab03a
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

ceph_objectstore_tool: Strip _* (always _head) from pgid in list entry output

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: BUG: --op list wasn't including snapshots

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: For terminal output of list one object per line

Instead of a parsable array make it easier to cut and paste listed objects

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: In error case umount objectstore

Signed-off-by: David Zafman <dzafman@redhat.com>

objectstore_tool: test --op list variants

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: parse new object description format

The object format changed from

    {json object}

to

    [pgid,{json object}]

The parser is updated accordingly. If the --pgid is present, check that
it equals the pgid from the object description.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: filter --op list and explore all PGs

The positional object name is used to filter the output of --op list and
only show the objects with a matching name. If both the object name and
the pgid are omitted, all objects from all PGs are displayed.

The output format is changed from

    {"oid":"GROUP","key":"","snapid":-2,
     "hash":2659194943,"max":0,"pool":0,"namespace":""}

to

    [["0.7_head",{"oid":"GROUP","key":"","snapid":-2,
                  "hash":2659194943,"max":0,"pool":0,
                  "namespace":""}]]

where the first member is the pgid where the object is found.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: lookup objects by name

If the object is not a parsable JSON string, assume an object name and
look it up in all the PGs. If multiple objects have the same name, only
apply the command to one of them. It is primarily useful in a test
environment where the names of the tests objects are known and only a
small number of objects exists. It replaces the following:

    path='--data-path dev/osd0 --journal-path dev/osd0.journal'
    for pgid in $(./ceph_objectstore_tool $path --op list-pgs) ; do
      object=$(./ceph_objectstore_tool $path --pgid $pgid --op list |
               grep '"oid":"NAME"')
      test -n "$object" && break
    done
    ./ceph_objectstore_tool $path --pgid $pgid "$object" remove

with:

    ./ceph_objectstore_tool $path NAME remove

http://tracker.ceph.com/issues/10192 Fixes: #10192

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: refactor list-lost and fix-lost

Abstract out the PG exploration loops and encapsulate the list-lost and
fix-lost semantic in a callable object.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

objectstore_tool: update usage strings

Signed-off-by: Loic Dachary <ldachary@redhat.com>