tests: ceph_objectstore_tool.py test all variants of export/import

Handle change of error message text

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Make --file option consistent by treating "-" as stdout/stdin

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: When exporting to stdout, don't cout messages

Fixes: #10128
Caused by a2bd2aa7

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #2943 from ceph/wip-dumper-coverity

tools: Close the fd before returning in Dumper

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2942 from tchaikov/wip-fix-rongze-affiliation

mailmap: fix Rongze's affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: fix Rongze's affiliation

Merge pull request #2915 from dachary/wip-10063-objectstore-tool-erasure-code

ghobject::dump must display shard_id + make ceph_objectstore_tool run from make check

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2929 from thesues/master

Fix libstriprados::stat, use strtoll insdead of strtol

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix libstriprados::stat, use strtoll insdead of strtol

The return value(long int) of strict_strtol is too small for unstriped
object.

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #2939 from aspiers/fix/architecture-docs

minor tweaks to architecture document

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix typos in diagram for incomplete write

In this example of a write of v2 of the object being interrupted, OSD2
would never have any version of the D1 chunk.  It only has the old v1
version of the D2 chunk.

Signed-off-by: Adam Spiers <aspiers@suse.com>

doc: fix incorrect equalities

The previous punctuation accidentally implied that K == 1 and M == -1.

Signed-off-by: Adam Spiers <aspiers@suse.com>

doc: fix typo (superfluous "no")

Signed-off-by: Adam Spiers <aspiers@suse.com>

tests: ceph_objectstore_tool.py fix list-attr for erasure code

Adding testing of xattr for erasure coded shards
Fix error message when finding an unexpected xattr key

Signed-off-by: David Zafman <dzafman@redhat.com>

tests: ceph_objectstore_tool.py check for malformed JSON for erasure code objs

Signed-off-by: David Zafman <dzafman@redhat.com>

tests: ceph_objectstore_tool.py fix off by 1 ATTR_OBJS handling

Signed-off-by: David Zafman <dzafman@redhat.com>

common: do not omit shard when ghobject NO_GEN is set

Do not silence the display of shard_id when generation is NO_GEN.
Erasure coded objects JSON representation used by ceph_objectstore_tool
need the shard_id to find the file containing the chunk.

Minimal testing is added to ceph_objectstore_tool.py

http://tracker.ceph.com/issues/10063 Fixes: #10063

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: run ceph_objectstore_tool.py from make check

http://tracker.ceph.com/issues/10064 Fixes: #10064

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py skip if /dev/tty fails

Some environments do not have a /dev/tty. When opening /dev/tty fails,
skip the test instead of returning an error.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py encapsulate init-ceph stop

Call init-ceph in kill_daemons and add a call to kill_daemon when main
returns on error so that it never leaves daemons hanging.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py main returns

Instead of calling sys.exit() the main function returns the desired exit
code.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py replace stop.sh with init-ceph

The stop.sh will stop all ceph-* processes. Use the init-ceph script
instead to selectively kill the daemons run by the vstart.sh cluster
used for ceph_objectstore_tool.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py use a dedicated directory

Set CEPH_DIR to a directory that is specific to ceph_objectstore_tool so
that it can run in parallel with other vstart.sh clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py run faster by default

By default use only a small number of objects to speed up the tests. If
the argument "big" is given, use a large number of objects as it may
help find some problems.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py run mon and osd on specific port

By default vstart.sh runs MDS but they are not needed for the tests,
only run mon and osd instead. Instead of using the default vstart.sh
port which may conflict with a already running vstart.sh, set the
CEPH_PORT=7400 which is not used by any other test run with make check.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py can use a WARN cluster

The tests do not need HEALTH_OK exclusively, a HEALTH_WARN cluster can
also run them successfully.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py use env python

Using #/usr/bin/env python instead of a hard coded path is more flexible
and can also be used to run from virtualenv.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

vstart: use hostname -s instead of fqdn

Otherwise init-ceph.in will fail if hostname returns a fqdn. It
validates the host entry of the [osd.x] section does not contain dots.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tools: Close the fd before returning in Dumper

CID 1251577
CID 717084

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2935 from dachary/wip-mailmap

mailmap: Loic Dachary name normalization

mailmap: Loic Dachary name normalization

Do not rewrite the past.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

test_async_driver: fix warning

test/msgr/test_async_driver.cc:233:209: instantiated from here

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2389 from ceph/wip-doc-documenting-ceph

doc: Update documenting ceph to include CentOS/RHEL

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: Removed separate build file for rpm.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2801 from CzBiX/permanent-location

rgw: add location when data in another region.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2932 from nilamdyuti/wip-doc-documenting-ceph

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2930 from dachary/wip-warning

tests: fix cast compilation warning

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #2923 from ktdreyer/doc-install-giant

doc: update packages page for Giant

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: update release descriptions on packages page

Firefly was listed as the "latest release" of Ceph. Add Giant and label
it as the latest release instead.

Releases A-C and E are no longer supported. D will receive updates for
some time longer, and F is a "long term stable" release.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

tests: fix cast compilation warning

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2928 from leseb/improve-exceptions

Improve readability of the exception

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Improve readability of the exception

The error messages were not really clear from a non-programmer
perspective. In the context of OpenStack all the drivers are falling
back to the exceptions provided by the rados library. Having clearer
error messages will help debugging misconfigured environment.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

rgw: add location when data in another region.

When the bucket of requested is in another region, Add location header
in response, so client can follow that to get data.

Signed-off-by: VRan Liu <gliuwr@gmail.com>

Merge pull request #2925 from yuyuyu101/update-mailmap

Mailmap: add Rongze organization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Mailmap: add Rongze organization

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

vstart.sh: complain less about fullish mon disks

Lots of people develop on file systems  > 70% full.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

mds: don't overwrite reply's snapbl

set_trace_dist() updates reply's snapbl, don't overwrite it. For MKSNAP
request, just need to set mdr->tracei, set_trace_dist() will set reply's
snapbl.

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2772 from ceph/wip-init-radosgw

init-radosgw.sysv: set ulimit -n before starting daemon

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2920 from ceph/wip-10095

crush: fix detach_bucket

Reviewed-by: Samuel Just <sjust@redhat.com>

crush/CrushWrapper: fix detach_bucket

In commit 9850227d2f0ca2f692a154de2c14a0a08e751f08 we changed the call that
changed the weight of all instances of item to one that explicitly
changes it in the parent bucket, but parent_id may not be valid at the
call site.  Move this into the conditional block to fix.

Fixes: #10095
Signed-off-by: Sage Weil <sage@redhat.com>

Makefile: include 'ceph' in base target

This makes 'make base' enough to vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2906 from ceph/wip-9835

osd: fix misdirected ec ops stuck in optracker (9835)

Reviewed-by: Samuel Just <sam.just@inktank.com>

msg/Makefile: add new header to tarball

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2909 from yuyuyu101/wip-10079

AsyncMessenger: Add select support and tests for async drivers

Merge pull request #2912 from dachary/wip-warning

tests: fix compilation warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2911 from keichwa/wip-doc-rados

Wip doc rados

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: fix compilation warning

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: remove superfluous white spaces (for testing)

Signed-off-by: Karl Eichwalder <ke@suse.de>

doc: Fix 2 syntax errors.

Signed-off-by: Karl Eichwalder <ke@suse.de>

AsyncMessenger: Fix select driver for add_event

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncMessenger: Add tests for async drivers

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncMessenger: Support select for other OS such as Windows

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

osd/OSD: use OSDMap helper to determine if we are correct op target

Use the new helper.  This fixes our behavior for EC pools where targetting
a different shard is not correct, while for replicated pools it may be. In
the EC case, it leaves the op hanging indefinitely in the OpTracker because
the pgid exists but as a different shard.

Fixes: #9835
Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add osd_is_valid_op_target()

Helper to check whether an osd is a given op target for a pg.  This
assumes that for EC we always send ops to the primary, while for
replicated we may target any replica.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_objectstore_tool: Fixes to make import work again

The is_pg() call is now true even for pgs pending removal, fix broken
    finish_remove_pgs() by removing is_pg() check.
Need to add create_collection() to the initial transaction on import

Fixes: #10090
Signed-off-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2903 from ceph/wip-crush-tree

crush: fix tree bucket functions

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: warn less

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2892 from leseb/doc-juno-client-socket

doc: enable RBD cache and socket on OpenStack deployments

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2902 from dachary/wip-10083-noisy-osd

qa: handle CEPH_CLI_TEST_DUP_COMMAND on ceph osd create

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/next'

qa: handle CEPH_CLI_TEST_DUP_COMMAND on ceph osd create

If CEPH_CLI_TEST_DUP_COMMAND is set when ceph osd create is called, it
will create two osd. They must be cleaned up afterwards instead of
assuming only one is going to be created.

http://tracker.ceph.com/issues/10083 Fixes: #10083

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2124 from ceph/wip-objectstore

simplify ObjectStore interface

sage-2014-11-11_08:26:01-rados-wip-sage-testing-distro-basic-multi

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2722 from ceph/wip-9598

msgr: make mark_down not block on fast dispatch

sage-2014-11-11_08:26:01-rados-wip-sage-testing-distro-basic-multi

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2901 from ktdreyer/doc-spelling-grammar

doc: spelling & grammar fixes

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: correct grammar in RGW SSL cert installation

The SSL certificate generation instruction was missing a preposition.
Add it.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: correct spelling of "fqdn"

The "fqdn" (fully-qualified domain name) was misspelled in a couple
places as fgdn". Use the correct spelling.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #2899 from jdurgin/wip-import-export-next

qa: allow small allocation diffs for exported rbds

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: enable RBD cache and socket on OpenStack deployments

Enabling the RBD cache improves sequential IOs and the socket helps a
lot while troubleshooting. These 2 items are considered as best
practice for OpenStack deployments with Ceph.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

qa: allow small allocation diffs for exported rbds

The local filesytem may behave slightly differently. This isn't
foolproof, but seems to be reliable enough on rhel7 rootfs, where
exact comparison was failing.

Fixes: #10002
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2898 from ceph/wip-rbd-python-tests

librbd: Python unit tests now use unique pools and images

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2820 from ceph/wip-9854

osdc: Constrain max number of in-flight read requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2890 from zhouyuan/fix_mailmap

Fix Yuan's mailmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/next'

Fix Yuan's mailmap

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

doc/release-notes: v0.88

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2897 from ceph/wip-da-revert-10b68b

Revert "osd: detect (some) misordered ondisk tmaps"

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crush: fix tree bucket functions

There are incorrect nodes' weight in tree bucket when construct tree
bucket. The tree bucket don't store item id in items array, so the tree
bucket will not work correctly. The patch fix above bugs and add a
simple test for tree bucket.

Signed-off-by: Rongze Zhu <zrzhit@gmail.com>

crush/builder: replace printf with an empty dprintk macro

This mirrors mapper.c.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "osd: detect (some) misordered ondisk tmaps"

This reverts commit 10b68b2e1a1479c478843fab8232537675053c9b.

The check for 'nextkey < last_disk_key' makes not much sense since
last_disk_key is an empty string and not set before. Comparing a
decoded string to be less than an empty string will be never true.

Since this if() isn't part of a loop last_disk_key is only set
once and there is no other consumer: revert this dead code.

Conflicts:
src/osd/ReplicatedPG.cc

0.88

Merge pull request #2562 from ceph/wip-5595

Wip 5595

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/rados/configuration fix 'ods mkfs options' default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2466 from nereocystis/osd-filesystem-options

doc: Add file system osd config settings

Merge pull request #2883 from ceph/wip-da-SCA-20141028

Fixes for coverity and SCA issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2889 from xuechendi/move_buffers_to_BufferTransaction

KeyValueStore: move buffers from strip_header to BufferTransaction

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2739 from majianpeng/fix2

buffer: Judge bufferlist whether need  rebuild

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2894 from ceph/wip-scrub-coverity

Wip scrub coverity

AuthSessionHandler.h: init protocol in constructor

Init protocol in AuthSessionHandler constructor with
with CEPH_AUTH_UNKNOWN.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>