Merge pull request #3166 from ceph/wip-librados-fadvise

fadvise for librados, librbd + osd guts

    sage-2014-12-16_06:27:10-rados:singleton-wip-sage-testing2-distro-basic-multi
    sage-2014-12-15_13:18:23-rados:thrash-wip-sage-testing2-distro-basic-multi

Merge pull request #3080 from ceph/wip-fadvise

osd: reenable fadvise; use it more often

sage-2014-12-16_06:27:10-rados:singleton-wip-sage-testing2-distro-basic-multi
sage-2014-12-15_13:18:23-rados:thrash-wip-sage-testing2-distro-basic-multi

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3165 from nilamdyuti/wip-doc-ceph

doc: Updates man page for Ceph

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3184 from dachary/wip-10325-cauchy

erasure-code: relax cauchy w restrictions

Reviewed-by: David Zhang <david.z1003@yahoo.com>

erasure-code: relax cauchy w restrictions

A restriction that the w parameter of the cauchy technique is limited to
8, 16 or 32 was added incorrectly while refactoring parameter parsing in
the jerasure plugin and must be relaxed.

http://tracker.ceph.com/issues/10325 Fixes: #10325

Signed-off-by: Loic Dachary <ldachary@redhat.com>

man: Deprecate preferred OSD setting for cephfs

In the source code of cephfs there is no reference to it anymore,
but it's still in the man page. Leave it there for now, but make
it more clear that it is deprecated and unsupported.

Merge pull request #2970 from ceph/wip-swift-storage-policy

rgw: support swift storage policy api

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3163 from ceph/wip-9254

rgw: http headers need to end with \r\n

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sweil@redhat.com>

librados: do not choke on asphyxiate doc generation

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #3142 from ceph/wip-10220

mon: suspend proposing paxos values when flushing pending writes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3182 from ceph/wip-9405

librbd: fixed garbage output from test LibRBD.TestIOPP

Reviewed-by: Loic Dachary <ldachary@redhat.com>

librbd: fixed garbage output from test LibRBD.TestIOPP

buffer::list::c_str does not actually provide a C-style, NULL
terminated string.  As a result, its use for console output
resulted in stray garbage characters being printed.

Fixes: #9405
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3161 from anandgbhat/wip-jemalloc

jemalloc-support: Added --with-jemalloc configure option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3180 from ktdreyer/doc-corpus-spelling

doc: fix spelling in corpus page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon: Paxos: reset accept timeout before submiting work to the store

Otherwise we may trigger the timeout while waiting for the work to be
committed to the store -- and it would only take the write to take a bit
longer than 10 seconds (default accept timeout).

We do wait for the work to be properly committed to the store before
extending the lease though.

Fixes: #10220
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3146 from ceph/wip-9323

mon: allow full flag to be manually cleared

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

doc: fix spelling in corpus page

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3075 from theanalyst/doc/v0.89/mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Blaine Gardner affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Lei Dong affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Sandon Van Ness name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

jemalloc-support: Added --with-jemalloc configure option

If --with-jemalloc option is not specified, then it defaults to tcmalloc.

If --with-jemalloc is specified, then --without-tcmalloc is required.
Also if a rocksdb static linking is desired --with-librocksdb-static,
do not use --with-jemalloc, as rocksdb uses tcmalloc for builds.

Support for jemalloc in package build and perf stats/heap profiler is
underway

Signed-off-by: shishir gowda <shishir.gowda@sandisk.com>

mailmap: Chendi Xue name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Conflicts:
.mailmap

mailmap: Ding Dinghua affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Dongmao Zhang affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Kefu Chai affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Michal Jarzabek affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Xinze Chi affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Karl Eichwalder affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Dan Mick name normalization

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Adam Spiers affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

mailmap: Nilamdyuti Goswami affiliation

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

ceph_test_rados_api_io: fix new test

From 0c802e11d9f0ee993994d391d1e00aeb76965d6a.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3175 from ceph/wip-test-huge-tickets

qa: add script to test how libceph handles huge auth tickets

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3174 from dachary/wip-warning

tests: fix signedness compilation warnings

Merge pull request #2986 from ceph/wip-9998

crush: fix weight underfloat issue

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa: add script to test how libceph handles huge auth tickets

Signed-off-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #3112 from xiaoxichen/cleanup_getheader

Cleanup:Use get_type()instead of get_header().type

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: fix signedness compilation warnings

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3172 from dachary/wip-10146-retrigger-udev

ceph-disk: test re-using an existing journal partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: LOG.info instead of print

When a non fatal error happens with parted, use LOG.info to display it
instead of print.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: test re-using an existing journal partition

Add a ceph-disk test to first setup an OSD with a separate journal
block device, then tear down the OSD (simulating a failure) and create
a new OSD which re-uses the same journal device.

Add create_dev / destroy_dev helpers that encapsulate the operations
that ensure the partition table is up to date in the kernel and the
symlinks are created as expected. In particular it makes sure the kernel
is aware that the partition table of a newly created device is
empty. If the device previously existed and the kernel was not informed
of the latest partition table updates via partprobe / partx, it may
have cached an old partition table which can create all sorts of
unexpected behaviors such as a failure to create the by-partuuid
symbolic links as described in http://tracker.ceph.com/issues/10146
Refs: #10146

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-disk: fix get_partition_uuid for loop/cciss

Loopback and cciss devices have an extra 'p' before the partition
number. Add exceptional handling of these devices.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

ceph-disk: don't change the journal partition uuid

We observe that the new /dev/disk/by-partuuid/<journal_uuid>
symlink is not always created by udev when reusing a journal
partition. Fix by not changing the uuid of a journal partition
in this case -- instead we can reuse the existing uuid (and
journal_symlink) instead. We also now assert that the symlink
exists before further preparing the OSD.

Fixes: #10146
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Tested-by: Dan van der Ster <daniel.vanderster@cern.ch>

documentation: simplify running make check

Encapsulate the compilation steps (install dependencies, autogen.sh,
configure, make check) in the run-make-check.sh script. Update the
developer documentation to point to this script instead of multiple
steps.

It is intended as a tool to help new developer make sure their patch is
sane, it focuses on efficiency (runs make check in parallel if possible)
and coverage (enables docker based tests if possible).

http://tracker.ceph.com/issues/10265 Fixes: #10265

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install parted in centos container

Because ceph-disk depends on it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: install hdparm in containers

The ceph-disk tests use hdparm, install it in containers that run
them.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: update centos docker file to epel 7.5

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: docker tests only need a workdir, not a clone

Instead of cloning the repository, create a work directory that has
symbolic links to the original .git, using

https://github.com/git/git/blob/master/contrib/workdir/git-new-workdir

It resolves the problem of fetching a commit that is not attached to any
ref, which is apparently not implemented in the git protocol (discussed
on irc.freenode.net#git).

http://tracker.ceph.com/issues/10264 Fixes: #10264

Signed-off-by: Loic Dachary <ldachary@redhat.com>

rgw: cannot modify swift bucket policy

When trying to set a policy on the bucket, fail unless policy matches
current placement rule.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3126 from ceph/wip-cleanup-removal

osd: Remove dead code related to old pg removal mechanism

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: dump X-Storage-Policy header on bucket HEAD

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: compare placement target on bucket recreation

and return -EEXIST (409) if requested a mismatching placement target.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/master' into wip-librados-fadvise

Conflicts:
src/test/librbd/test_librbd.cc

doc: Adds man page for ceph under man/.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

doc: Adds man page for ceph.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

rgw: http headers need to end with \r\n

Fixes: #9254
Backport: firefly, giant

Reported-by: Benedikt Fraunhofer <fraunhofer@traced.net>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3141 from ceph/wip-librbd-test-cleanup

tests: Minor cleanup to librbd test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3160 from xylv/wip-8562-rebase

rgw: Conditional PUT on ETag

Reviewed-by: Yehuda Sadeh <yehuda@rehat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3153 from ceph/wip-10296

osd: fix block device discard detection

Reviewed-by: Loic Dachary <loic@dachary.org>
Tested-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3144 from ceph/wip-aarch64

fixes for aarch64

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

rgw: Conditional PUT on ETag

Fixes: #8562
Signed-off-by: Ray Lv <raylv@yahoo-inc.com>

test/librados: Add test case for librados function which can handle fadvise flas.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test/librbd: Add test case for librbd function which can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

FileStore: Implement fadvise handle for read-operation.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ECBackend: Make parity data don't cache.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ObjectStore/Transaction: Add fadvise_flags to track write fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PGTransaction: Modify append/write function can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: Make async-read can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

include: Remove unused code: class triple.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

types: Add operator<< for type boost::tuple<A, B, C>.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

encoding: Add type boost::tuple<A,B,C> encoding/decode func.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

PGBackend: Make objects_read_sync can handle fadvise flags

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ObjectStore: Modify read function can handle fadvise flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osd: print OSDOp flags

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: increment version for new functions rbd_read2/write2,rbd_aio_read2/write2.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: add new read/write api for librbd C++ interface which handle op flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: add new read/write api for librbd C interface which handle op
flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Add set_op_flags in AioWrite to set op_flags for rbd write.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Add m_op_flags in AioRead to track op flags for rbd read.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd/internal: Modify read/write api to handle op flags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados/ObjectOperation: add new function set_op_flags2

After adding four read/write fadvise flags OP_FADVISE_*, those flags
can set multi flags at the same time. For compatibility, add new
function. The old func set_op_flags will be deprecated.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados: Add read/write fadvise op flags CEPH_OSD_OP_FLAG_FADVISE_*.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados: Add read/write fadvise op flags in ObjectOperationFlags.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librados.h: Add read/write op fadvise flags  LIBRADOS_OP_FLAG_FADVISE_*.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

unittest_blkdev: test an abbreviated /sys/block dir

Note that we assume unittest_blkdev is run from the current directory (or
rather that test/common/test_blkdev_sys_block exists).

Signed-off-by: Sage Weil <sage@redhat.com>

common/blkdev: add simple sandboxing function for testing

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3155 from athanatos/wip-8011

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG: fail a non-blocking flush if the object is being scrubbed

Fixes: #8011
Backport: firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3154 from athanatos/wip-10059

Wip 10059

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3156 from athanatos/wip-10150

ReplicatedPG::scan_range: an object can disappear between the list and t...

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3157 from athanatos/wip-10157

PGLog: include rollback_info_trimmed_to in (read|write)_log

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3158 from athanatos/wip-10259

osd_types: op_queue_age_hist and fs_perf_stat should be in osd_stat_t::o...

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::scan_range: an object can disappear between the list and the attr get

The first item in the range is often last_backfill, upon which writes
can be occuring.  It's trimmed off on the primary side anyway.

Fixes: 10150
Backport: dumpling, firefly, giant
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #2579 from yuandong1222/cache-hboject-filestore

Cache hobject filestore_key to avoid massive _reverse_nibbles calls

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3018 from tchaikov/wip-fix-safe_read_file

safe_io: do not set ending \0 in safe_read_file()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3070 from ceph/wip-10080

SimpleMessenger: allow RESETSESSION whenever we forget an endpoint

Reviewed-by: Sage Weil <sage@redhat.com>

debian: enable libgoogle-perftools-dev on arm64

These binaries haven't landed in Ubuntu, but they are in
sid and jessie for arm64. On Saucy I've installed them by
hand from ports.ubuntu.com and things seems pretty swell.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #3150 from ceph/wip-10288

mon: fix `fs ls` on peons

Reviewed-by: Greg Farnum <gfarnum@redhat.com>