Merge branch 'wyc-quota-v2' of git://github.com/kylinstorage/ceph into wip-fs-quota

Conflicts:
src/include/ceph_features.h

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'gh/master' into wip-watch-notify

Conflicts:
PendingReleaseNotes

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh During <jdurgin@redhat.com>

Merge pull request #3033 from dachary/wip-10201-osd-scrub-repair

osd scrub repair uses ceph_objecstore_tool + shell toolbox

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3181 from ceph/wip-10299

librbd: complete all pending aio ops prior to closing image

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2300 from ceph/wip-osdmap

add osdmap checksums

    sage-2014-12-16_06:27:10-rados:singleton-wip-sage-testing2-distro-basic-multi
    sage-2014-12-15_13:18:23-rados:thrash-wip-sage-testing2-distro-basic-multi

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3166 from ceph/wip-librados-fadvise

fadvise for librados, librbd + osd guts

    sage-2014-12-16_06:27:10-rados:singleton-wip-sage-testing2-distro-basic-multi
    sage-2014-12-15_13:18:23-rados:thrash-wip-sage-testing2-distro-basic-multi

Merge pull request #3080 from ceph/wip-fadvise

osd: reenable fadvise; use it more often

sage-2014-12-16_06:27:10-rados:singleton-wip-sage-testing2-distro-basic-multi
sage-2014-12-15_13:18:23-rados:thrash-wip-sage-testing2-distro-basic-multi

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3165 from nilamdyuti/wip-doc-ceph

doc: Updates man page for Ceph

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3184 from dachary/wip-10325-cauchy

erasure-code: relax cauchy w restrictions

Reviewed-by: David Zhang <david.z1003@yahoo.com>

erasure-code: relax cauchy w restrictions

A restriction that the w parameter of the cauchy technique is limited to
8, 16 or 32 was added incorrectly while refactoring parameter parsing in
the jerasure plugin and must be relaxed.

http://tracker.ceph.com/issues/10325 Fixes: #10325

Signed-off-by: Loic Dachary <ldachary@redhat.com>

man: Deprecate preferred OSD setting for cephfs

In the source code of cephfs there is no reference to it anymore,
but it's still in the man page. Leave it there for now, but make
it more clear that it is deprecated and unsupported.

Merge pull request #2970 from ceph/wip-swift-storage-policy

rgw: support swift storage policy api

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3163 from ceph/wip-9254

rgw: http headers need to end with \r\n

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

librbd: complete all pending aio ops prior to closing image

It was possible for an image to be closed while aio operations
were still outstanding.  Now all aio operations are tracked and
completed before the image is closed.

Fixes: #10299
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: do not choke on asphyxiate doc generation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #3142 from ceph/wip-10220

mon: suspend proposing paxos values when flushing pending writes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3182 from ceph/wip-9405

librbd: fixed garbage output from test LibRBD.TestIOPP

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: fixed garbage output from test LibRBD.TestIOPP

buffer::list::c_str does not actually provide a C-style, NULL
terminated string.  As a result, its use for console output
resulted in stray garbage characters being printed.

Fixes: #9405
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3161 from anandgbhat/wip-jemalloc

jemalloc-support: Added --with-jemalloc configure option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3180 from ktdreyer/doc-corpus-spelling

doc: fix spelling in corpus page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon: Paxos: reset accept timeout before submiting work to the store

Otherwise we may trigger the timeout while waiting for the work to be
committed to the store -- and it would only take the write to take a bit
longer than 10 seconds (default accept timeout).

We do wait for the work to be properly committed to the store before
extending the lease though.

Fixes: #10220
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3146 from ceph/wip-9323

mon: allow full flag to be manually cleared

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

doc: fix spelling in corpus page

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

tests: convert osd-scrub-repair to ceph_objectstore_tool

http://tracker.ceph.com/issues/10201 Fixes: #10201

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: shell functions toolbox dedicated to testing

The osd/osd-test-helpers.sh mon/mon-test-helpers.sh are useful to create
shell based functional tests. Since they were introduced they did not
grow significantly. As more tests are added, it becomes necessary to
have more sophisticated helpers. The ceph-helpers.sh toolbox is a merge
of the existing helpers intended to be easy to maintain and expand.

  * Each function is documented
  * Each function is tested

When adding to a function or looking for a new function, it is unlikely
the developer will find time to explore the entire file and understand
its structure. The documentation, body and test for each function are
grouped together to limit the scope of exploration required from the
casual developer.

The test are designed to not use internal helpers so that they can be
understood in full just by reading the test function. It means a given
test takes longer than it should (i.e. each getters tests run one mon).
In the context of testing this toolbox, simplicity is preferred over
efficiency.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3075 from theanalyst/doc/v0.89/mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Blaine Gardner affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Lei Dong affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Sandon Van Ness name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

jemalloc-support: Added --with-jemalloc configure option

If --with-jemalloc option is not specified, then it defaults to tcmalloc.

If --with-jemalloc is specified, then --without-tcmalloc is required.
Also if a rocksdb static linking is desired --with-librocksdb-static,
do not use --with-jemalloc, as rocksdb uses tcmalloc for builds.

Support for jemalloc in package build and perf stats/heap profiler is
underway

Signed-off-by: shishir gowda <shishir.gowda@sandisk.com>

mailmap: Chendi Xue name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Conflicts:
.mailmap

mailmap: Ding Dinghua affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Dongmao Zhang affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Kefu Chai affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Michal Jarzabek affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Xinze Chi affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Karl Eichwalder affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Dan Mick name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Adam Spiers affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Nilamdyuti Goswami affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

ceph_test_rados_api_io: fix new test

From 0c802e11d9f0ee993994d391d1e00aeb76965d6a.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3175 from ceph/wip-test-huge-tickets

qa: add script to test how libceph handles huge auth tickets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3174 from dachary/wip-warning

tests: fix signedness compilation warnings

ceph_test_rados_api_watch_notify: use 5 min timeout for notifies

This should make them tolerate thrashing.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_stress_watch: do not unwatch if watch failed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2986 from ceph/wip-9998

crush: fix weight underfloat issue

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa: add script to test how libceph handles huge auth tickets

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #3112 from xiaoxichen/cleanup_getheader

Cleanup:Use get_type()instead of get_header().type

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: fix signedness compilation warnings

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3172 from dachary/wip-10146-retrigger-udev

ceph-disk: test re-using an existing journal partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: LOG.info instead of print

When a non fatal error happens with parted, use LOG.info to display it
instead of print.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: test re-using an existing journal partition

Add a ceph-disk test to first setup an OSD with a separate journal
block device, then tear down the OSD (simulating a failure) and create
a new OSD which re-uses the same journal device.

Add create_dev / destroy_dev helpers that encapsulate the operations
that ensure the partition table is up to date in the kernel and the
symlinks are created as expected. In particular it makes sure the kernel
is aware that the partition table of a newly created device is
empty. If the device previously existed and the kernel was not informed
of the latest partition table updates via partprobe / partx, it may
have cached an old partition table which can create all sorts of
unexpected behaviors such as a failure to create the by-partuuid
symbolic links as described in http://tracker.ceph.com/issues/10146
Refs: #10146

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: fix get_partition_uuid for loop/cciss

Loopback and cciss devices have an extra 'p' before the partition
number. Add exceptional handling of these devices.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

ceph-disk: don't change the journal partition uuid

We observe that the new /dev/disk/by-partuuid/<journal_uuid>
symlink is not always created by udev when reusing a journal
partition. Fix by not changing the uuid of a journal partition
in this case -- instead we can reuse the existing uuid (and
journal_symlink) instead. We also now assert that the symlink
exists before further preparing the OSD.

Fixes: #10146
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Tested-by: Dan van der Ster <daniel.vanderster@cern.ch>

documentation: simplify running make check

Encapsulate the compilation steps (install dependencies, autogen.sh,
configure, make check) in the run-make-check.sh script. Update the
developer documentation to point to this script instead of multiple
steps.

It is intended as a tool to help new developer make sure their patch is
sane, it focuses on efficiency (runs make check in parallel if possible)
and coverage (enables docker based tests if possible).

http://tracker.ceph.com/issues/10265 Fixes: #10265

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install parted in centos container

Because ceph-disk depends on it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install hdparm in containers

The ceph-disk tests use hdparm, install it in containers that run
them.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: update centos docker file to epel 7.5

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: docker tests only need a workdir, not a clone

Instead of cloning the repository, create a work directory that has
symbolic links to the original .git, using

https://github.com/git/git/blob/master/contrib/workdir/git-new-workdir

It resolves the problem of fetching a commit that is not attached to any
ref, which is apparently not implemented in the git protocol (discussed
on irc.freenode.net#git).

http://tracker.ceph.com/issues/10264 Fixes: #10264

Signed-off-by: Loic Dachary <ldachary@redhat.com>

rgw: cannot modify swift bucket policy

When trying to set a policy on the bucket, fail unless policy matches
current placement rule.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3126 from ceph/wip-cleanup-removal

osd: Remove dead code related to old pg removal mechanism

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: dump X-Storage-Policy header on bucket HEAD

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: compare placement target on bucket recreation

and return -EEXIST (409) if requested a mismatching placement target.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librados: remove IoCtxImpl lock member

It is dead!

Signed-off-by: Sage Weil <sage@redhat.com>

librados: stop using IoCtxImpl lock for watch/notify functions

This is a carry-over from olden times when Objecter didn't do its own
locking.

Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: only ping after watch is registered

If we ping before the watch is registered we will get ETIMEDOUT (the ping
is are a read op that will fail to see the watch registration).

Signed-off-by: Sage Weil <sage@redhat.com>

librados: do not overload C++ API methods with new signatures

Bah!  C++ sucks for libraries.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/master' into wip-librados-fadvise

Conflicts:
src/test/librbd/test_librbd.cc

doc: Adds man page for ceph under man/.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds man page for ceph.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

rgw: http headers need to end with \r\n

Fixes: #9254
Backport: firefly, giant

Reported-by: Benedikt Fraunhofer <fraunhofer@traced.net>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon/OSDMonitor: dump inc and full maps we get a crc mismatch

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not reencode OSDMap on 'osd getmap'

Return the bits unmolested.

This is a slight change in that we do not reencode based on the client's
features.  However, this is an admin command and the client is generally
the CLI or REST API.  It makes more sense to provide the unmolested
bits than to reencode them based on features present in a tool that
never inspects the map.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-dencoder: mark OSDMap as featureful, but with stray data

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: allow OSDMap encode

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add osd_inject_bad_map_crc_probability option

Signed-off-by: Sage Weil <sage@redhat.com>

osd: verify our OSDMap encoding based on crc

When we generate a full map and can't replicate teh expected CRC, request
the full map from the mon.

Note that we should do everything we can to avoid doing this:

 - upgrade OSDs before mons
 - do not encode things into OSDMaps that OSDs can't replicate

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: verify full_crc when available

Crash if it is wrong.  The monitors should *always* agree.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3141 from ceph/wip-librbd-test-cleanup

tests: Minor cleanup to librbd test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3160 from xylv/wip-8562-rebase

rgw: Conditional PUT on ETag

Reviewed-by: Yehuda Sadeh <yehuda@rehat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3153 from ceph/wip-10296

osd: fix block device discard detection

Reviewed-by: Loic Dachary <loic@dachary.org>
Tested-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3144 from ceph/wip-aarch64

fixes for aarch64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

rgw: Conditional PUT on ETag

Fixes: #8562
Signed-off-by: Ray Lv <raylv@yahoo-inc.com>

test/librados: Add test case for librados function which can handle fadvise flas.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/librbd: Add test case for librbd function which can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

FileStore: Implement fadvise handle for read-operation.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ECBackend: Make parity data don't cache.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ObjectStore/Transaction: Add fadvise_flags to track write fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PGTransaction: Modify append/write function can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: Make async-read can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

include: Remove unused code: class triple.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

types: Add operator<< for type boost::tuple<A, B, C>.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

encoding: Add type boost::tuple<A,B,C> encoding/decode func.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PGBackend: Make objects_read_sync can handle fadvise flags

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ObjectStore: Modify read function can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: print OSDOp flags

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: increment version for new functions rbd_read2/write2,rbd_aio_read2/write2.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: add new read/write api for librbd C++ interface which handle op flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: add new read/write api for librbd C interface which handle op
flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>