osdc: continuously subscribe to osd map on pause

We were doing this on FULL, should also have done it
on PAUSERD and PAUSEWR, to avoid paused requests
staying that way forever.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Objecter: add have_map method

This is for places we're going to call wait_for_map, so
that we can easily check if we will *probably* get
a true from wait_for_map, and thereby avoid
allocating Contexts we won't need.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Objecter: add op_cancel_writes

This is for use by Client when it encounters
the FULL flag and wants to abort operations
rather than having them pause.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/ObjectCacher: invoke flush_set_callback on purge_set

For the benefit of Client, so that it can get its
caps released when using purge_set to deal with
an ENOSPC condition.  Does not affect librbd because
librbd doesn't provide flush_set_callback.

Signed-off-by: John Spray <john.spray@redhat.com>

osdc/Objecter: respect honor_osdmap_full setting

With this, it no longer needs to special case the MDS,
as the MDS can just call unset_honor_osdmap_full.

Signed-off-by: John Spray <john.spray@redhat.com>

os: free space tracking for MemStore

Allow users to set an artificial upper bound
on size of the memstore OSD, and report actual
usage information in fsstat.

This is a useful tool for simulating nearly-full
systems.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3075 from theanalyst/doc/v0.89/mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Blaine Gardner affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Lei Dong affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Sandon Van Ness name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Chendi Xue name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Conflicts:
.mailmap

mailmap: Ding Dinghua affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Dongmao Zhang affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Kefu Chai affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Michal Jarzabek affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Xinze Chi affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Karl Eichwalder affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Dan Mick name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Adam Spiers affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Nilamdyuti Goswami affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #3175 from ceph/wip-test-huge-tickets

qa: add script to test how libceph handles huge auth tickets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3174 from dachary/wip-warning

tests: fix signedness compilation warnings

Merge pull request #2986 from ceph/wip-9998

crush: fix weight underfloat issue

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa: add script to test how libceph handles huge auth tickets

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #3112 from xiaoxichen/cleanup_getheader

Cleanup:Use get_type()instead of get_header().type

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: fix signedness compilation warnings

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3172 from dachary/wip-10146-retrigger-udev

ceph-disk: test re-using an existing journal partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: LOG.info instead of print

When a non fatal error happens with parted, use LOG.info to display it
instead of print.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: test re-using an existing journal partition

Add a ceph-disk test to first setup an OSD with a separate journal
block device, then tear down the OSD (simulating a failure) and create
a new OSD which re-uses the same journal device.

Add create_dev / destroy_dev helpers that encapsulate the operations
that ensure the partition table is up to date in the kernel and the
symlinks are created as expected. In particular it makes sure the kernel
is aware that the partition table of a newly created device is
empty. If the device previously existed and the kernel was not informed
of the latest partition table updates via partprobe / partx, it may
have cached an old partition table which can create all sorts of
unexpected behaviors such as a failure to create the by-partuuid
symbolic links as described in http://tracker.ceph.com/issues/10146
Refs: #10146

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: fix get_partition_uuid for loop/cciss

Loopback and cciss devices have an extra 'p' before the partition
number. Add exceptional handling of these devices.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

ceph-disk: don't change the journal partition uuid

We observe that the new /dev/disk/by-partuuid/<journal_uuid>
symlink is not always created by udev when reusing a journal
partition. Fix by not changing the uuid of a journal partition
in this case -- instead we can reuse the existing uuid (and
journal_symlink) instead. We also now assert that the symlink
exists before further preparing the OSD.

Fixes: #10146
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Tested-by: Dan van der Ster <daniel.vanderster@cern.ch>

documentation: simplify running make check

Encapsulate the compilation steps (install dependencies, autogen.sh,
configure, make check) in the run-make-check.sh script. Update the
developer documentation to point to this script instead of multiple
steps.

It is intended as a tool to help new developer make sure their patch is
sane, it focuses on efficiency (runs make check in parallel if possible)
and coverage (enables docker based tests if possible).

http://tracker.ceph.com/issues/10265 Fixes: #10265

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install parted in centos container

Because ceph-disk depends on it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install hdparm in containers

The ceph-disk tests use hdparm, install it in containers that run
them.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: update centos docker file to epel 7.5

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: docker tests only need a workdir, not a clone

Instead of cloning the repository, create a work directory that has
symbolic links to the original .git, using

https://github.com/git/git/blob/master/contrib/workdir/git-new-workdir

It resolves the problem of fetching a commit that is not attached to any
ref, which is apparently not implemented in the git protocol (discussed
on irc.freenode.net#git).

http://tracker.ceph.com/issues/10264 Fixes: #10264

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3126 from ceph/wip-cleanup-removal

osd: Remove dead code related to old pg removal mechanism

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3141 from ceph/wip-librbd-test-cleanup

tests: Minor cleanup to librbd test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3160 from xylv/wip-8562-rebase

rgw: Conditional PUT on ETag

Reviewed-by: Yehuda Sadeh <yehuda@rehat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3153 from ceph/wip-10296

osd: fix block device discard detection

Reviewed-by: Loic Dachary <loic@dachary.org>
Tested-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3144 from ceph/wip-aarch64

fixes for aarch64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

rgw: Conditional PUT on ETag

Fixes: #8562
Signed-off-by: Ray Lv <raylv@yahoo-inc.com>

unittest_blkdev: test an abbreviated /sys/block dir

Note that we assume unittest_blkdev is run from the current directory (or
rather that test/common/test_blkdev_sys_block exists).

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add simple sandboxing function for testing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3155 from athanatos/wip-8011

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Fixes: #8011
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3154 from athanatos/wip-10059

Wip 10059

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3156 from athanatos/wip-10150

ReplicatedPG::scan_range: an object can disappear between the list and t...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3157 from athanatos/wip-10157

PGLog: include rollback_info_trimmed_to in (read|write)_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3158 from athanatos/wip-10259

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o...

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::scan_range: an object can disappear between the list and the attr get

The first item in the range is often last_backfill, upon which writes
can be occuring.  It's trimmed off on the primary side anyway.

Fixes: 10150
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #2579 from yuandong1222/cache-hboject-filestore

Cache hobject filestore_key to avoid massive _reverse_nibbles calls

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3018 from tchaikov/wip-fix-safe_read_file

safe_io: do not set ending \0 in safe_read_file()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3070 from ceph/wip-10080

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

Reviewed-by: Sage Weil <sage@redhat.com>

debian: enable libgoogle-perftools-dev on arm64

These binaries haven't landed in Ubuntu, but they are in
sid and jessie for arm64. On Saucy I've installed them by
hand from ports.ubuntu.com and things seems pretty swell.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3150 from ceph/wip-10288

mon: fix `fs ls` on peons

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common/blkdev: fix block device discard check

- fix base name calculation (do not assume sda)
- reverse sense of check (it was returning false when true before?)
- add a generic helper to get other properties, too

Fixes: #10296
Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: get_block_device_base

And a test.  Ideally the test would get run on a machine with weirdo
devices like /dev/cciss/c0d1.

This mirrors the logic in ceph-disk, which has been tested.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3152 from dachary/wip-10296-journal-discard

os: do not attempt to detect discard by default

Reviewed-by: Sage Weil <sage@redhat.com>

tests: do not ignore ceph-disk.sh::test_activate_dev failure

The status of the test commands must be tested instead of being
silently ignored.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

os: do not attempt to detect discard by default

If the journal_discard option is false, the journal should no try to
guess if the device has discard capabilities.

http://tracker.ceph.com/issues/10296 Refs: #10296

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mon: fix `fs ls` on peons

This was incorrectly using pending_mdsmap instead
of mdsmap.  We didn't notice in test because of
single-mon configurations.

Fixes: #10288
Backport: giant

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3149 from ceph/wip-typo

mds: fix log typo

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: fix log typo

s/trvserse/traverse/

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3137 from thesues/fix-strtol

Fix libstriprados::remove, use strtoll insdead of strtol

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3038 from ceph/wip-mds-readonly

Wip mds readonly

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3145 from XinzeChi/wip-finisher

Finisher: call cond signal if necessary in function queue

Reviewed-by: Sage Weil <sage@redhat.com>

Finisher: call signal if necessary in function queue

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

lttng: add int type definitions

The normal path through #include <lttng/tracepoint.h> doesn't
appear to include int defintions like uint64_t that are used
in Ceph so we add our definitions file.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

cycles: add reader for i386 and aarch64

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3119 from ceph/wip-8797

rados.py: remove Rados.__del__(); it just causes problems

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Call Rados.shutdown() explicitly before exit

This is mostly a demonstration of good behavior, as the resources will
be reclaimed on exit anyway.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rados.py: remove Rados.__del__(); it just causes problems

Recent versions of Python contain a change to thread shutdown that
causes ceph to hang on exit; see http://bugs.python.org/issue21963.
As it turns out, this is relatively easy to avoid by not spawning
threads on exit, as Rados.__del__() will certainly do by calling
shutdown(); I suspect, but haven't proven, that the problem is
that shutdown() tries to start() a threading.Thread() that never
makes it all the way back to signal start().

Also add a PendingReleaseNote and extra doc comments to clarify.

Fixes: #8797
Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #3129 from ceph/wip-gfcomplete-dirty

gf-complete: update submodule to newest upstream

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3071 from ceph/wip-10164

mds: drop dirty dentries in deleted directory

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

check endpoints is not empty before use it.

Signed-off-by: VRan Liu <gliuwr@gmail.com>
(cherry picked from commit 9d5e4edc15678de675d7a77badcbace0b710d553)

Merge pull request #3135 from XinzeChi/wip-3

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher

Reviewed-by: Sage Weil <sage@redhat.com>

os/FileStore.cc: insert not empty list<Context*> to op_finisher/ondisk_finisher.

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>

Merge pull request #3140 from leseb/sample-ceph.conf

ceph.conf: update sample

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.conf: update sample

This commit adds:

* client admin socket
* change some default
* re-indent

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

tests: Minor cleanup to librbd test

The tests can now be repeated to increase the chance of hitting
edge condition failures.  Also added additional logic to immediately
fail IO tests when an error is encountered.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Fix libstriprados::remove, use strtoll insdead of strtol

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #3120 from ceph/wip-10010

ceph-osd: don't close stderr if not daemonizing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: set dirfrag version when fetching dirfrag is skipped

Signed-off-by: Yan, Zheng <zyan@redhat.com>

ceph-osd: remove extra close of stderr

Otherwise, one loses log messages when running with -f or -d.  When
daemonizing, stderr is already closed in global_init_postfork_finish.

Fixes: #10010, #10113, #9810
Signed-off-by: Dan Mick <dan.mick@redhat.com>

gf-complete: update submodule to newest upstream

Keep the tree cleaner!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

osd: Remove dead code related to old pg removal mechanism

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #3073 from cholcombe973/get_pool_values_docs

Updating docs for Get Pool Value

Reviewed-by: Loic Dachary <ldachary@redhat.com>

crush/CrushWrapper: fix create_or_move_item when name exists but item does not

We were using item_exists(), which simply checks if we have a name defined
for the item.  Instead, use _search_item_exists(), which looks for an
instance of the item somewhere in the hierarchy.  This matches what
get_item_weightf() is doing, which ensures we get a non-negative weight
that converts properly to floating point.

Backport: giant, firefly
Fixes: #9998
Reported-by: Pawel Sadowski <ceph@sadziu.pl>
Signed-off-by: Sage Weil <sage@redhat.com>

crush/builder: prevent bucket weight underflow on item removal

It is possible to set a bucket weight that is not the sum of the item
weights if you manually modify/build the CRUSH map.  Protect against any
underflow on the bucket weight when removing items.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: fix _search_item_exists

Reported-by: Pawel Sadowski <ceph@sadziu.pl>
Signed-off-by: Sage Weil <sage@redhat.com>

Modifying the docs to add the Get pool commands to match the CLI.
Signed-off-by: Chris Holcombe <chris.holcombe@nebula.com>

Merge pull request #3106 from danderson/asan-fixes

Asan fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3113 from wonzhq/name-norm

mailmap: Zhiqiang Wang name normalization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Zhiqiang Wang name normalization

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Cleanup:Use get_type()instead of get_header().type

Cleanup the code ,use get_type() instead of get_header().type
for messages.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Fix stack buffer overflow reported by ASan.

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (new[] vs. delete).

Signed-off-by: David Anderson <dave@natulte.net>

Fix alloc-dealloc mismatch reported by ASan (malloc vs. C++ delete).

Signed-off-by: David Anderson <dave@natulte.net>