remove unneeded include file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #2895 from yuyuyu101/wip-10022

AsyncMessenger: Fix several problems

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2980 from ceph/wip-10123

librbd: protect list_children from invalid child pool IoCtxs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2981 from yuyuyu101/quick-fix

GenericObjectMap: Avoid master branch compile error

GenericObjectMap: Fix typo

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2966 from yuyuyu101/wip-10119

KeyValueStore: Fix EC+ KV OSDs crashing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2978 from majianpeng/fix4

blkdev: using strncpy instead of strcpy.

Reviewed-by: Sage Weil <sage@redhat.com>

blkdev: using strncpy instead of strcpy.

Coverity Scan reported this bug:
> New defect(s) Reported-by: Coverity Scan Showing 1 of 1 defect(s)
> *** CID 1255369:  Copy into fixed size buffer  (STRING_OVERFLOW)
> /common/blkdev.cc: 34 in block_device_support_discard(const char *)()
> 28     bool block_device_support_discard(const char *devname)
> 29     {
> 30       bool can_trim = false;
> 31       char *p = strstr((char *)devname, "sd");
> 32       char name[32] = {0};
> 33
> >>>     CID 1255369:  Copy into fixed size buffer  (STRING_OVERFLOW)
> >>>     You might overrun the 32 byte fixed-size string "name" by
> >>>     copying "p" without checking the length.
> 34       strcpy(name, p);
> 35       for (unsigned int i = 0; i < strlen(name); i++) {
> 36         if(isdigit(name[i])) {
> 37           name[i] = 0;
> 38           break;
> 39         }
>

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #2974 from ceph/wip-sepia-update

Updated sepia hardware list.

Updated sepia hardware list.

Added missing details and additional list of hardware.

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

Merge pull request #2752 from dachary/wip-9817-deep-scrub-start

osd: log when scrub,deep scrub, repair starts

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2891 from XinzeChi/master

OSD: batch trim for pg_log

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2885 from ceph/wip-flush-snaps

osd/ReplicatedPG: flush snaps immediately

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2973 from wonzhq/unused-parm

ReplicatedPG: remove unused parameter in function write_update_size_and_usage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG: remove unused parameter in function write_update_size_and_usage

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

KeyValueStore: Fix parse_header_key

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2967 from ceph/wip-10077

ceph_objectstore_tool shard feature handling

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2971 from dachary/wip-mailmap

mailmap: Loic Dachary name normalization

mailmap: Loic Dachary name normalization

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2962 from ceph/wip-9439

osd: Check filter ops for pgls and pgnls

Reviewed-by: Loic Dachary <ldachary@dachary.org>

Merge pull request #2969 from dachary/wip-erasure-code-documentation

doc: erasure-code developer documentation typo

Reviewed-by: Pavan Rallabhandi <Pavan.Rallabhandi@sandisk.com>

doc: erasure-code developer documentation typo

Reported-by: Pavan Rallabhandi <Pavan.Rallabhandi@sandisk.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

StoreTest: Add scrub situation to tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Add KEY_ENDING sign to the end of key

Keys stored in alphabetical order and need to follow ghobject_t comparison
rule. "generation" and "shard_id" are optional fields for object key, but
a default ghobject with UINT64_MAX generation(by default) will larger than
the same ghobject with other generation. GenericObjectMap rejects to store
generation if generation is UINT64_MAX in order to reduce too much words
in key. So we need to add a MAX sign to the end of key to make ordering
is same with ghobject's comparison rule.

For example:

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head!78!0

The first key should larger than the second in ghobject_t ordering because
of generation. But the first key less than the second in GenericObjectMap.

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head!78!0

After we add KEY_ENDING, '!' is (21) in hexadecimal:

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head(21)78!0(FF)
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head(FF)

Fix bug #10119
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2953 from yuyuyu101/fix-keyvaluestore-remove-col-attr

KeyValueStore: Remove assert for collection_getattr method

Reviewed-by: Sage Weil <sage@redhat.com>

StoreTest: Clean up warning

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

MemStore: Return -ENODATA when collection_getattr hit nonexist attr

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Fix collection_getattrs problem when zero-length attr

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Return -ENODATA if collection_getattr is nonexist

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

StoreTest: Add collection_getattr(s) tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Remove assert for collection_getattr method

Merge 22b6c1a5317504e87042ffeff2b2d0d0b71cac5e introduce
5dc990c1ca821620993417ef4acc51a2b0a34606 commit which will lookup "remove"
attr. It shouldn't exist for most of cases but KeyValueStore will crash
if attr isn't exist. It seemed that this assert isn't correct.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ceph_objectstore_tool: Add feature called set-allow-sharded-objects

Uses --op set-allow-sharded-objects option
This operation will be rejected if on the target OSD's osdmap there is
    at least one OSD which does not support ERASURE CODES.
Prompt the user that they could import if sharded state allowed
Prompt the user to use new feature if sharded state found inconsistent

Fixes: #10077
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Add utility routine get_osdmap()

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Clear ...INCOMPAT_SHARDS from feature if exporting replicated pg

Don't require importing OSD to have shards feature for replicated pg

http://tracker.ceph.com/issues/10077 Fixes: #10077

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #2648 from dachary/wip-9665-ceph-disk-partprobe

ceph-disk: run partprobe/partx after zap and data partition creation

Reviewed-by: Sage Weil <sage@redhat.com>

FileJournal: add journal_discard to control ssd whether support discard

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

FileJournal: Add ssd discard for journal which using ssd disk as journal.

Journal is like a ring buffer. After data wrote to media disk, journal
can overwrite. But for those data, ssd dont't know it's nouse and can
remove. So add discard to tell ssd to remove those data.
This maybe not increase the performance. But it can increase the
lifetime of ssd.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #2960 from ceph/wip-10114-fix-warning

erasure-code isa-l: remove duplicated lines (fix warning)

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code isa-l: remove duplicated lines (fix warning)

06a245a added a section def to assembly files; I added it twice to
this file.  There's no damage, but a compiler warning (on machines with
yasm installed)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2951 from sponce/master

Fixed trivial locking issue in the trunc method of libradosstriper

Reviewed-by: Loic Dachary <ldachary@redhat.com>

autotools: add --enable-docker

Docker based tests should be explicit instead of auto-detected. It is
good that they do not run if docker is not available. It would be bad if
they run when the developer does not expect them to create docker
containers.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: test prepare / activate on a device

This indirectly tests that partprobe is called after zap because it
would fail to map the partitions to /dev/disk/by-partuuid otherwise.

It also indirectly test the implementation of init=none when using a
block device because the test would fail to put an object into the rbd
pool using the device otherwise.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: helper to run unit / function tests in docker

For instance

   test/docker-test.sh --os-type ubuntu --os-version 14.04 \
        test/ceph-disk.sh

runs test/ceph-disk.sh in a ubuntu 14.04 docker container. Once the
container is populated and ceph compiled, running a test script roughly
requires entering the container and running make TESTS=tests/foo.sh check

* docker build ceph-ubuntu-14.04 using ubuntu.dockerfile as a Dockerfile
* it will run apt-get install ceph compilation / run dependencies
* git clone the-local-clone ceph-ubuntu-14.04
* docker run ceph-ubuntu-14.04 make -j4 in the ceph-ubuntu-14.04 clone
* docker run test/ceph-disk.sh

test/docker-test.sh is the command line interface for
test/docker-test-helper.sh which can be invoked from shell scripts.
test/ubuntu.dockerfile and test/ubuntu.dockerfile are regular
Dockerfiles which allow substitution of environment variables.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: implement init=none for block devices

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: run partprobe after zap

Not running partprobe after zapping a device can lead to the following:

* ceph-disk prepare /dev/loop2
* links are created in /dev/disk/by-partuuid
* ceph-disk zap /dev/loop2
* links are not removed from /dev/disk/by-partuuid
* ceph-disk prepare /dev/loop2
* some links are not created in /dev/disk/by-partuuid

This is assuming there is a bug in the way udev events are handled by
the operating system.

http://tracker.ceph.com/issues/9665 Fixes: #9665

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: use update_partition in prepare_dev and main_prepare

In the case of prepare_dev the partx alternative was missing and is not
added because update_partition does it.

http://tracker.ceph.com/issues/9721 Fixes: #9721

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: encapsulate partprobe / partx calls

Add the update_partition function to reduce code duplication.
The action is made an argument although it always is -a because it will
be -d when deleting a partition.

Use the update_partition function in prepare_journal_dev

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

doc: update debian compilation dependencies

Using the content of debian/control.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2910 from swamireddy/wip-organizationmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2907 from xuechendi/update_organizationmap

Add myself to <contact@intel.com>

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fixed locking issue in the trun method of libradosstriper leading to potential race conditions - Fixes: #10129

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #2917 from leseb/rbd-update-glance-config

doc: update the OpenStack glance configuration

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2950 from ceph/wip-10128

Wip 10128

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py test all variants of export/import

Handle change of error message text

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Make --file option consistent by treating "-" as stdout/stdin

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: When exporting to stdout, don't cout messages

Fixes: #10128
Caused by a2bd2aa7

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #2946 from ceph/wip-execstack

Add annotation to all assembly files to turn off stack-execute bit

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: protect list_children from invalid child pool IoCtxs

While listing child images, don't ignore error codes returned
from librados when creating an IoCtx. This will prevent seg
faults from occurring when an invalid IoCtx is used.

Fixes: #10123
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2933 from ceph/wip-librbd-symbols

librbd: Only public API symbols from the shared library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2947 from HP-Scale-out-Storage/wip-10096

Wip 10096 (ceph-disk umount race condition)

Reviewed-by: Sage Weil <sage@redhat.com>

Fix bug #10096 (ceph-disk umount race condition)

Bug: http://tracker.ceph.com/issues/10096

Brief: Unmounting temporary mount point failed due to file being 'busy'.
Root cause could not be easily determined due to timing variances caused
by debug attempts. Race condition exists.

Solution: Implement a retry with incremental backoff as a viable
workaround. This workaround is okay because (1) Finding the root cause
would take a not insignificant amount of time/effort. (2) The workaround
is a more general fix for any process that might cause the exhibited
behavior.

Signed-off-by: Blaine Gardner <blaine.gardner@hp.com>

Merge pull request #2914 from dachary/wip-10093-monstore-tool

mon: ceph-monstore-tool must close()

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2943 from ceph/wip-dumper-coverity

tools: Close the fd before returning in Dumper

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2942 from tchaikov/wip-fix-rongze-affiliation

mailmap: fix Rongze's affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: fix Rongze's affiliation

Merge pull request #2915 from dachary/wip-10063-objectstore-tool-erasure-code

ghobject::dump must display shard_id + make ceph_objectstore_tool run from make check

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2929 from thesues/master

Fix libstriprados::stat, use strtoll insdead of strtol

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix libstriprados::stat, use strtoll insdead of strtol

The return value(long int) of strict_strtol is too small for unstriped
object.

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #2939 from aspiers/fix/architecture-docs

minor tweaks to architecture document

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fix typos in diagram for incomplete write

In this example of a write of v2 of the object being interrupted, OSD2
would never have any version of the D1 chunk.  It only has the old v1
version of the D2 chunk.

Signed-off-by: Adam Spiers <aspiers@suse.com>

doc: fix incorrect equalities

The previous punctuation accidentally implied that K == 1 and M == -1.

Signed-off-by: Adam Spiers <aspiers@suse.com>

doc: fix typo (superfluous "no")

Signed-off-by: Adam Spiers <aspiers@suse.com>

tests: ceph_objectstore_tool.py fix list-attr for erasure code

Adding testing of xattr for erasure coded shards
Fix error message when finding an unexpected xattr key

Signed-off-by: David Zafman <dzafman@redhat.com>

tests: ceph_objectstore_tool.py check for malformed JSON for erasure code objs

Signed-off-by: David Zafman <dzafman@redhat.com>

tests: ceph_objectstore_tool.py fix off by 1 ATTR_OBJS handling

Signed-off-by: David Zafman <dzafman@redhat.com>

common: do not omit shard when ghobject NO_GEN is set

Do not silence the display of shard_id when generation is NO_GEN.
Erasure coded objects JSON representation used by ceph_objectstore_tool
need the shard_id to find the file containing the chunk.

Minimal testing is added to ceph_objectstore_tool.py

http://tracker.ceph.com/issues/10063 Fixes: #10063

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: run ceph_objectstore_tool.py from make check

http://tracker.ceph.com/issues/10064 Fixes: #10064

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py skip if /dev/tty fails

Some environments do not have a /dev/tty. When opening /dev/tty fails,
skip the test instead of returning an error.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py encapsulate init-ceph stop

Call init-ceph in kill_daemons and add a call to kill_daemon when main
returns on error so that it never leaves daemons hanging.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py main returns

Instead of calling sys.exit() the main function returns the desired exit
code.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py replace stop.sh with init-ceph

The stop.sh will stop all ceph-* processes. Use the init-ceph script
instead to selectively kill the daemons run by the vstart.sh cluster
used for ceph_objectstore_tool.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py use a dedicated directory

Set CEPH_DIR to a directory that is specific to ceph_objectstore_tool so
that it can run in parallel with other vstart.sh clusters.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py run faster by default

By default use only a small number of objects to speed up the tests. If
the argument "big" is given, use a large number of objects as it may
help find some problems.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py run mon and osd on specific port

By default vstart.sh runs MDS but they are not needed for the tests,
only run mon and osd instead. Instead of using the default vstart.sh
port which may conflict with a already running vstart.sh, set the
CEPH_PORT=7400 which is not used by any other test run with make check.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py can use a WARN cluster

The tests do not need HEALTH_OK exclusively, a HEALTH_WARN cluster can
also run them successfully.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py use env python

Using #/usr/bin/env python instead of a hard coded path is more flexible
and can also be used to run from virtualenv.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

vstart: use hostname -s instead of fqdn

Otherwise init-ceph.in will fail if hostname returns a fqdn. It
validates the host entry of the [osd.x] section does not contain dots.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Add annotation to all assembly files to turn off stack-execute bit

See discussion in http://tracker.ceph.com/issues/10114

Building with these changes allows output from readelf like this:

 $ readelf -lW src/.libs/librados.so.2 | grep GNU_STACK
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000
0x000000 RW  0x8

(note the absence of 'X' in 'RW')

Fixes: #10114
Signed-off-by: Dan Mick <dan.mick@redhat.com>

tools: Close the fd before returning in Dumper

CID 1251577
CID 717084

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2935 from dachary/wip-mailmap

mailmap: Loic Dachary name normalization

mailmap: Loic Dachary name normalization

Do not rewrite the past.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

test_async_driver: fix warning

test/msgr/test_async_driver.cc:233:209: instantiated from here

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2389 from ceph/wip-doc-documenting-ceph

doc: Update documenting ceph to include CentOS/RHEL

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: Removed separate build file for rpm.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge pull request #2801 from CzBiX/permanent-location

rgw: add location when data in another region.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2932 from nilamdyuti/wip-doc-documenting-ceph

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2930 from dachary/wip-warning

tests: fix cast compilation warning

doc: Adds build-doc guidelines for Fedora and CentOS/RHEL.

Signed-off-by: Nilamdyuti Goswami <ngoswami@redhat.com>

Merge pull request #2923 from ktdreyer/doc-install-giant

doc: update packages page for Giant

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: update release descriptions on packages page

Firefly was listed as the "latest release" of Ceph. Add Giant and label
it as the latest release instead.

Releases A-C and E are no longer supported. D will receive updates for
some time longer, and F is a "long term stable" release.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>