ceph_objectstore_tool: Add feature called set-allow-sharded-objects

Uses --op set-allow-sharded-objects option
This operation will be rejected if on the target OSD's osdmap there is
    at least one OSD which does not support ERASURE CODES.
Prompt the user that they could import if sharded state allowed
Prompt the user to use new feature if sharded state found inconsistent

Fixes: #10077
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Add utility routine get_osdmap()

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Clear ...INCOMPAT_SHARDS from feature if exporting replicated pg

Don't require importing OSD to have shards feature for replicated pg

http://tracker.ceph.com/issues/10077 Fixes: #10077

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #2951 from sponce/master

Fixed trivial locking issue in the trunc method of libradosstriper

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2910 from swamireddy/wip-organizationmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2907 from xuechendi/update_organizationmap

Add myself to <contact@intel.com>

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fixed locking issue in the trun method of libradosstriper leading to potential race conditions - Fixes: #10129

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #2917 from leseb/rbd-update-glance-config

doc: update the OpenStack glance configuration

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2950 from ceph/wip-10128

Wip 10128

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py test all variants of export/import

Handle change of error message text

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Make --file option consistent by treating "-" as stdout/stdin

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: When exporting to stdout, don't cout messages

Fixes: #10128
Caused by a2bd2aa7

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #2946 from ceph/wip-execstack

Add annotation to all assembly files to turn off stack-execute bit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2933 from ceph/wip-librbd-symbols

librbd: Only public API symbols from the shared library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2947 from HP-Scale-out-Storage/wip-10096

Wip 10096 (ceph-disk umount race condition)

Reviewed-by: Sage Weil <sage@redhat.com>

Fix bug #10096 (ceph-disk umount race condition)

Bug: http://tracker.ceph.com/issues/10096

Brief: Unmounting temporary mount point failed due to file being 'busy'.
Root cause could not be easily determined due to timing variances caused
by debug attempts. Race condition exists.

Solution: Implement a retry with incremental backoff as a viable
workaround. This workaround is okay because (1) Finding the root cause
would take a not insignificant amount of time/effort. (2) The workaround
is a more general fix for any process that might cause the exhibited
behavior.

Signed-off-by: Blaine Gardner <blaine.gardner@hp.com>

Merge pull request #2914 from dachary/wip-10093-monstore-tool

mon: ceph-monstore-tool must close()

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2943 from ceph/wip-dumper-coverity

tools: Close the fd before returning in Dumper

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2942 from tchaikov/wip-fix-rongze-affiliation

mailmap: fix Rongze's affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: fix Rongze's affiliation

Merge pull request #2915 from dachary/wip-10063-objectstore-tool-erasure-code

ghobject::dump must display shard_id + make ceph_objectstore_tool run from make check

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2929 from thesues/master

Fix libstriprados::stat, use strtoll insdead of strtol

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix libstriprados::stat, use strtoll insdead of strtol

The return value(long int) of strict_strtol is too small for unstriped
object.

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #2939 from aspiers/fix/architecture-docs

minor tweaks to architecture document

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix typos in diagram for incomplete write

In this example of a write of v2 of the object being interrupted, OSD2
would never have any version of the D1 chunk.  It only has the old v1
version of the D2 chunk.

Signed-off-by: Adam Spiers <aspiers@suse.com>

doc: fix incorrect equalities

The previous punctuation accidentally implied that K == 1 and M == -1.

Signed-off-by: Adam Spiers <aspiers@suse.com>

doc: fix typo (superfluous "no")

Signed-off-by: Adam Spiers <aspiers@suse.com>

tests: ceph_objectstore_tool.py fix list-attr for erasure code

Adding testing of xattr for erasure coded shards
Fix error message when finding an unexpected xattr key

Signed-off-by: David Zafman <dzafman@redhat.com>

tests: ceph_objectstore_tool.py check for malformed JSON for erasure code objs

Signed-off-by: David Zafman <dzafman@redhat.com>

tests: ceph_objectstore_tool.py fix off by 1 ATTR_OBJS handling

Signed-off-by: David Zafman <dzafman@redhat.com>

common: do not omit shard when ghobject NO_GEN is set

Do not silence the display of shard_id when generation is NO_GEN.
Erasure coded objects JSON representation used by ceph_objectstore_tool
need the shard_id to find the file containing the chunk.

Minimal testing is added to ceph_objectstore_tool.py

http://tracker.ceph.com/issues/10063 Fixes: #10063

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: run ceph_objectstore_tool.py from make check

http://tracker.ceph.com/issues/10064 Fixes: #10064

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py skip if /dev/tty fails

Some environments do not have a /dev/tty. When opening /dev/tty fails,
skip the test instead of returning an error.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py encapsulate init-ceph stop

Call init-ceph in kill_daemons and add a call to kill_daemon when main
returns on error so that it never leaves daemons hanging.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py main returns

Instead of calling sys.exit() the main function returns the desired exit
code.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py replace stop.sh with init-ceph

The stop.sh will stop all ceph-* processes. Use the init-ceph script
instead to selectively kill the daemons run by the vstart.sh cluster
used for ceph_objectstore_tool.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py use a dedicated directory

Set CEPH_DIR to a directory that is specific to ceph_objectstore_tool so
that it can run in parallel with other vstart.sh clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py run faster by default

By default use only a small number of objects to speed up the tests. If
the argument "big" is given, use a large number of objects as it may
help find some problems.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py run mon and osd on specific port

By default vstart.sh runs MDS but they are not needed for the tests,
only run mon and osd instead. Instead of using the default vstart.sh
port which may conflict with a already running vstart.sh, set the
CEPH_PORT=7400 which is not used by any other test run with make check.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py can use a WARN cluster

The tests do not need HEALTH_OK exclusively, a HEALTH_WARN cluster can
also run them successfully.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py use env python

Using #/usr/bin/env python instead of a hard coded path is more flexible
and can also be used to run from virtualenv.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

vstart: use hostname -s instead of fqdn

Otherwise init-ceph.in will fail if hostname returns a fqdn. It
validates the host entry of the [osd.x] section does not contain dots.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Add annotation to all assembly files to turn off stack-execute bit

See discussion in http://tracker.ceph.com/issues/10114

Building with these changes allows output from readelf like this:

 $ readelf -lW src/.libs/librados.so.2 | grep GNU_STACK
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8

(note the absence of 'X' in 'RW')

Fixes: #10114
Signed-off-by: Dan Mick <dan.mick@redhat.com>

tools: Close the fd before returning in Dumper

CID 1251577
CID 717084

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2935 from dachary/wip-mailmap

mailmap: Loic Dachary name normalization

mailmap: Loic Dachary name normalization

Do not rewrite the past.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

test_async_driver: fix warning

test/msgr/test_async_driver.cc:233:209: instantiated from here

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2389 from ceph/wip-doc-documenting-ceph

doc: Update documenting ceph to include CentOS/RHEL

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: Removed separate build file for rpm.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2801 from CzBiX/permanent-location

rgw: add location when data in another region.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2932 from nilamdyuti/wip-doc-documenting-ceph

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2930 from dachary/wip-warning

tests: fix cast compilation warning

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #2923 from ktdreyer/doc-install-giant

doc: update packages page for Giant

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: update release descriptions on packages page

Firefly was listed as the "latest release" of Ceph. Add Giant and label
it as the latest release instead.

Releases A-C and E are no longer supported. D will receive updates for
some time longer, and F is a "long term stable" release.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

tests: fix cast compilation warning

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2928 from leseb/improve-exceptions

Improve readability of the exception

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Improve readability of the exception

The error messages were not really clear from a non-programmer
perspective. In the context of OpenStack all the drivers are falling
back to the exceptions provided by the rados library. Having clearer
error messages will help debugging misconfigured environment.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

rgw: add location when data in another region.

When the bucket of requested is in another region, Add location header
in response, so client can follow that to get data.

Signed-off-by: VRan Liu <gliuwr@gmail.com>

doc: update the OpenStack glance configuration

Juno is now using a section for configuring the stores, so let's use it.
Put spaces between flag and value for clarity and readibility.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

librbd: Only public API symbols from the shared library

The librbd shared library was previously exporting all
symbols.  librbd public API methods are now explicitly
exported and all other symbols are hidden.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2925 from yuyuyu101/update-mailmap

Mailmap: add Rongze organization

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Mailmap: add Rongze organization

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

vstart.sh: complain less about fullish mon disks

Lots of people develop on file systems  > 70% full.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

mds: don't overwrite reply's snapbl

set_trace_dist() updates reply's snapbl, don't overwrite it. For MKSNAP
request, just need to set mdr->tracei, set_trace_dist() will set reply's
snapbl.

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2772 from ceph/wip-init-radosgw

init-radosgw.sysv: set ulimit -n before starting daemon

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2920 from ceph/wip-10095

crush: fix detach_bucket

Reviewed-by: Samuel Just <sjust@redhat.com>

crush/CrushWrapper: fix detach_bucket

In commit 9850227d2f0ca2f692a154de2c14a0a08e751f08 we changed the call that
changed the weight of all instances of item to one that explicitly
changes it in the parent bucket, but parent_id may not be valid at the
call site.  Move this into the conditional block to fix.

Fixes: #10095
Signed-off-by: Sage Weil <sage@redhat.com>

Makefile: include 'ceph' in base target

This makes 'make base' enough to vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2906 from ceph/wip-9835

osd: fix misdirected ec ops stuck in optracker (9835)

Reviewed-by: Samuel Just <sam.just@inktank.com>

msg/Makefile: add new header to tarball

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2909 from yuyuyu101/wip-10079

AsyncMessenger: Add select support and tests for async drivers

mailmap: Dan Mick name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Xan Peng affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: BJ Lougee affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Derrick Schneider affiliation

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mailmap: Federico Gimenez name normalization

Signed-off-by: M Ranga Swami Reddy <swamireddy@gmail.com>

mon: ceph-monstore-tool must close()

Otherwise it asserts because the destructor checks for the is_open flag.

http://tracker.ceph.com/issues/10093 Fixes: #10093

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2912 from dachary/wip-warning

tests: fix compilation warning

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2911 from keichwa/wip-doc-rados

Wip doc rados

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: fix compilation warning

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: remove superfluous white spaces (for testing)

Signed-off-by: Karl Eichwalder <ke@suse.de>

doc: Fix 2 syntax errors.

Signed-off-by: Karl Eichwalder <ke@suse.de>

AsyncMessenger: Fix select driver for add_event

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncMessenger: Add tests for async drivers

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncMessenger: Support select for other OS such as Windows

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

osd/OSD: use OSDMap helper to determine if we are correct op target

Use the new helper.  This fixes our behavior for EC pools where targetting
a different shard is not correct, while for replicated pools it may be. In
the EC case, it leaves the op hanging indefinitely in the OpTracker because
the pgid exists but as a different shard.

Fixes: #9835
Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add osd_is_valid_op_target()

Helper to check whether an osd is a given op target for a pg.  This
assumes that for EC we always send ops to the primary, while for
replicated we may target any replica.

Signed-off-by: Sage Weil <sage@redhat.com>

Add myself to <contact@intel.com>

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

ceph_objectstore_tool: Fixes to make import work again

The is_pg() call is now true even for pgs pending removal, fix broken
    finish_remove_pgs() by removing is_pg() check.
Need to add create_collection() to the initial transaction on import

Fixes: #10090
Signed-off-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2903 from ceph/wip-crush-tree

crush: fix tree bucket functions

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: warn less

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2892 from leseb/doc-juno-client-socket

doc: enable RBD cache and socket on OpenStack deployments

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2902 from dachary/wip-10083-noisy-osd

qa: handle CEPH_CLI_TEST_DUP_COMMAND on ceph osd create

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/next'

qa: handle CEPH_CLI_TEST_DUP_COMMAND on ceph osd create

If CEPH_CLI_TEST_DUP_COMMAND is set when ceph osd create is called, it
will create two osd. They must be cleaned up afterwards instead of
assuming only one is going to be created.

http://tracker.ceph.com/issues/10083 Fixes: #10083

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2124 from ceph/wip-objectstore

simplify ObjectStore interface

sage-2014-11-11_08:26:01-rados-wip-sage-testing-distro-basic-multi

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2722 from ceph/wip-9598

msgr: make mark_down not block on fast dispatch

sage-2014-11-11_08:26:01-rados-wip-sage-testing-distro-basic-multi

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2901 from ktdreyer/doc-spelling-grammar

doc: spelling & grammar fixes

Reviewed-by: John Wilkins <jowilkin@redhat.com>