Merge pull request #3003 from dachary/wip-10185-neon

arch: fix neon feature detection

Reviewed-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@redhat.com>

arch: fix neon feature detection

The fread function returns the number of elements read, not the number
of bytes.

Introduced by 1cef8339f7a2a503327c20c014e696c91d36281e , after giant.

Add tests for protection against similar regressions.

http://tracker.ceph.com/issues/10185 Fixes: #10185

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3002 from thesues/fix-init

Remove pidfile and asok after stopping ceph

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Remove pidfile and asok after stopping ceph

Signed-off-by: Dongmao Zhang <deanraccoon@gmail.com>

Merge pull request #2872 from dachary/wip-10018-primary-erasure-code-hinfo

osd: erasure code deep scrub must not abort if hinfo is missing

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #2984 from stiopaa1/includeFix

remove unneeded include file

Reviewed-by: Sage Weil <sage@redhat.com>

remove unneeded include file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #2998 from theanalyst/doc/rgw-s3-bucket-loc

doc: rgw document s3 bucket location features.

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: rgw document s3 bucket location features

Fixes: #10142
Reported-by: Neil Levine <nlevine@redhat.com>
Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

Merge pull request #2992 from dachary/wip-10173-autogen

autogen.sh: git submodule sync

Reviewed-by: Sage Weil <sage@redhat.com>

autogen.sh: git submodule sync

It is not enough to git submodule update on a existing git clone: if the
URL of the repository has been changed, the

   git submodule sync

command must be called to synchronizes submodules' remote URL
configuration setting to the value specified in .gitmodules.

http://tracker.ceph.com/issues/10173 Fixes: #10173

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2936 from ceph/wip-mailmap

Update Inktank attribution to point to Red Hat

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mailmap: Update Warren Usui's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Tyler Brekke's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Tamil Muthamizhan's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Sandon Van Ness' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Samuel Just's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Patrick McGarry's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Noah Watkins' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Neil Levine's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Mark Nelson's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Ken Dreyer's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Josh Durgin's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update John Wilkins' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Joao Luis' attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Ilya Dryomov's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Dan Mick's attribution

Point his addresses at Red Hat now, but revert the Inktank->Redhat mappings
so we have more historical accuracy.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Alfredo Deza's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mailmap: Update Greg Farnum's attribution to Red Hat

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #2983 from tchaikov/wip-fix-lock-dep

lockdep: do not use $CEPH_LOCKDEP for g_lockdep

Reviewed-by: Sage Weil <sage@redhat.com>

lockdep: do not use $CEPH_LOCKDEP for g_lockdep

* a non-zero CEPH_LOCKDEP brings ceph down because g_lockdep_ceph_ctx
  is still being constructed when dout_impl() dereferences it.
* fix a typo in comment.
* remove dead code.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

rgw-admin: create subuser if needed when creating user

Fixes: #10103
Backport: firefly, giant
This turned up after fixing #9973. Earlier we also didn't create the
subuser in this case, but we didn't really read the subuser info when it
was authenticating. Now we do that as required, so we end up failing the
authentication. This only applies to cases where a subuser was created
using 'user create', and not the 'subuser create' command.

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2895 from yuyuyu101/wip-10022

AsyncMessenger: Fix several problems

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2980 from ceph/wip-10123

librbd: protect list_children from invalid child pool IoCtxs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2981 from yuyuyu101/quick-fix

GenericObjectMap: Avoid master branch compile error

GenericObjectMap: Fix typo

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2966 from yuyuyu101/wip-10119

KeyValueStore: Fix EC+ KV OSDs crashing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2978 from majianpeng/fix4

blkdev: using strncpy instead of strcpy.

Reviewed-by: Sage Weil <sage@redhat.com>

blkdev: using strncpy instead of strcpy.

Coverity Scan reported this bug:
> New defect(s) Reported-by: Coverity Scan Showing 1 of 1 defect(s)
> *** CID 1255369:  Copy into fixed size buffer  (STRING_OVERFLOW)
> /common/blkdev.cc: 34 in block_device_support_discard(const char *)()
> 28     bool block_device_support_discard(const char *devname)
> 29     {
> 30       bool can_trim = false;
> 31       char *p = strstr((char *)devname, "sd");
> 32       char name[32] = {0};
> 33
> >>>     CID 1255369:  Copy into fixed size buffer  (STRING_OVERFLOW)
> >>>     You might overrun the 32 byte fixed-size string "name" by
> >>>     copying "p" without checking the length.
> 34       strcpy(name, p);
> 35       for (unsigned int i = 0; i < strlen(name); i++) {
> 36         if(isdigit(name[i])) {
> 37           name[i] = 0;
> 38           break;
> 39         }
>

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #2974 from ceph/wip-sepia-update

Updated sepia hardware list.

Updated sepia hardware list.

Added missing details and additional list of hardware.

Signed-off-by: Sandon Van Ness <sandon@redhat.com>

Merge pull request #2752 from dachary/wip-9817-deep-scrub-start

osd: log when scrub,deep scrub, repair starts

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2891 from XinzeChi/master

OSD: batch trim for pg_log

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2885 from ceph/wip-flush-snaps

osd/ReplicatedPG: flush snaps immediately

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2973 from wonzhq/unused-parm

ReplicatedPG: remove unused parameter in function write_update_size_and_usage

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG: remove unused parameter in function write_update_size_and_usage

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

KeyValueStore: Fix parse_header_key

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

osd: deep scrub must not abort if hinfo is missing

Instead it should set read_error.

http://tracker.ceph.com/issues/10018 Fixes: #10018

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2926 from ceph/wip-10052

ceph_test_rados_api_tier: fix whiteout cleanup (10052)

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2967 from ceph/wip-10077

ceph_objectstore_tool shard feature handling

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2971 from dachary/wip-mailmap

mailmap: Loic Dachary name normalization

mailmap: Loic Dachary name normalization

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2962 from ceph/wip-9439

osd: Check filter ops for pgls and pgnls

Reviewed-by: Loic Dachary <ldachary@dachary.org>

Merge pull request #2969 from dachary/wip-erasure-code-documentation

doc: erasure-code developer documentation typo

Reviewed-by: Pavan Rallabhandi <Pavan.Rallabhandi@sandisk.com>

doc: erasure-code developer documentation typo

Reported-by: Pavan Rallabhandi <Pavan.Rallabhandi@sandisk.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

StoreTest: Add scrub situation to tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Add KEY_ENDING sign to the end of key

Keys stored in alphabetical order and need to follow ghobject_t comparison
rule. "generation" and "shard_id" are optional fields for object key, but
a default ghobject with UINT64_MAX generation(by default) will larger than
the same ghobject with other generation. GenericObjectMap rejects to store
generation if generation is UINT64_MAX in order to reduce too much words
in key. So we need to add a MAX sign to the end of key to make ordering
is same with ghobject's comparison rule.

For example:

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head!78!0

The first key should larger than the second in ghobject_t ordering because
of generation. But the first key less than the second in GenericObjectMap.

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head!78!0

After we add KEY_ENDING, '!' is (21) in hexadecimal:

_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head(21)78!0(FF)
_GHOBJTOSEQ_:1%e1ds0_head!D168A7E8!!1!!benchmark_last_metadata!head(FF)

Fix bug #10119
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2953 from yuyuyu101/fix-keyvaluestore-remove-col-attr

KeyValueStore: Remove assert for collection_getattr method

Reviewed-by: Sage Weil <sage@redhat.com>

StoreTest: Clean up warning

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

MemStore: Return -ENODATA when collection_getattr hit nonexist attr

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Fix collection_getattrs problem when zero-length attr

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Return -ENODATA if collection_getattr is nonexist

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

StoreTest: Add collection_getattr(s) tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

KeyValueStore: Remove assert for collection_getattr method

Merge 22b6c1a5317504e87042ffeff2b2d0d0b71cac5e introduce
5dc990c1ca821620993417ef4acc51a2b0a34606 commit which will lookup "remove"
attr. It shouldn't exist for most of cases but KeyValueStore will crash
if attr isn't exist. It seemed that this assert isn't correct.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ceph_objectstore_tool: Add feature called set-allow-sharded-objects

Uses --op set-allow-sharded-objects option
This operation will be rejected if on the target OSD's osdmap there is
    at least one OSD which does not support ERASURE CODES.
Prompt the user that they could import if sharded state allowed
Prompt the user to use new feature if sharded state found inconsistent

Fixes: #10077
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Add utility routine get_osdmap()

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Clear ...INCOMPAT_SHARDS from feature if exporting replicated pg

Don't require importing OSD to have shards feature for replicated pg

http://tracker.ceph.com/issues/10077 Fixes: #10077

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #2648 from dachary/wip-9665-ceph-disk-partprobe

ceph-disk: run partprobe/partx after zap and data partition creation

Reviewed-by: Sage Weil <sage@redhat.com>

FileJournal: add journal_discard to control ssd whether support discard

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

FileJournal: Add ssd discard for journal which using ssd disk as journal.

Journal is like a ring buffer. After data wrote to media disk, journal
can overwrite. But for those data, ssd dont't know it's nouse and can
remove. So add discard to tell ssd to remove those data.
This maybe not increase the performance. But it can increase the
lifetime of ssd.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #2960 from ceph/wip-10114-fix-warning

erasure-code isa-l: remove duplicated lines (fix warning)

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code isa-l: remove duplicated lines (fix warning)

06a245a added a section def to assembly files; I added it twice to
this file.  There's no damage, but a compiler warning (on machines with
yasm installed)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #2951 from sponce/master

Fixed trivial locking issue in the trunc method of libradosstriper

Reviewed-by: Loic Dachary <ldachary@redhat.com>

autotools: add --enable-docker

Docker based tests should be explicit instead of auto-detected. It is
good that they do not run if docker is not available. It would be bad if
they run when the developer does not expect them to create docker
containers.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: test prepare / activate on a device

This indirectly tests that partprobe is called after zap because it
would fail to map the partitions to /dev/disk/by-partuuid otherwise.

It also indirectly test the implementation of init=none when using a
block device because the test would fail to put an object into the rbd
pool using the device otherwise.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: helper to run unit / function tests in docker

For instance

   test/docker-test.sh --os-type ubuntu --os-version 14.04 \
        test/ceph-disk.sh

runs test/ceph-disk.sh in a ubuntu 14.04 docker container. Once the
container is populated and ceph compiled, running a test script roughly
requires entering the container and running make TESTS=tests/foo.sh check

* docker build ceph-ubuntu-14.04 using ubuntu.dockerfile as a Dockerfile
* it will run apt-get install ceph compilation / run dependencies
* git clone the-local-clone ceph-ubuntu-14.04
* docker run ceph-ubuntu-14.04 make -j4 in the ceph-ubuntu-14.04 clone
* docker run test/ceph-disk.sh

test/docker-test.sh is the command line interface for
test/docker-test-helper.sh which can be invoked from shell scripts.
test/ubuntu.dockerfile and test/ubuntu.dockerfile are regular
Dockerfiles which allow substitution of environment variables.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: implement init=none for block devices

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: run partprobe after zap

Not running partprobe after zapping a device can lead to the following:

* ceph-disk prepare /dev/loop2
* links are created in /dev/disk/by-partuuid
* ceph-disk zap /dev/loop2
* links are not removed from /dev/disk/by-partuuid
* ceph-disk prepare /dev/loop2
* some links are not created in /dev/disk/by-partuuid

This is assuming there is a bug in the way udev events are handled by
the operating system.

http://tracker.ceph.com/issues/9665 Fixes: #9665

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: use update_partition in prepare_dev and main_prepare

In the case of prepare_dev the partx alternative was missing and is not
added because update_partition does it.

http://tracker.ceph.com/issues/9721 Fixes: #9721

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

ceph-disk: encapsulate partprobe / partx calls

Add the update_partition function to reduce code duplication.
The action is made an argument although it always is -a because it will
be -d when deleting a partition.

Use the update_partition function in prepare_journal_dev

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

doc: update debian compilation dependencies

Using the content of debian/control.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2910 from swamireddy/wip-organizationmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2907 from xuechendi/update_organizationmap

Add myself to <contact@intel.com>

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fixed locking issue in the trun method of libradosstriper leading to potential race conditions - Fixes: #10129

Signed-off-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #2917 from leseb/rbd-update-glance-config

doc: update the OpenStack glance configuration

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2950 from ceph/wip-10128

Wip 10128

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph_objectstore_tool.py test all variants of export/import

Handle change of error message text

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Make --file option consistent by treating "-" as stdout/stdin

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: When exporting to stdout, don't cout messages

Fixes: #10128
Caused by a2bd2aa7

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #2946 from ceph/wip-execstack

Add annotation to all assembly files to turn off stack-execute bit

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: protect list_children from invalid child pool IoCtxs

While listing child images, don't ignore error codes returned
from librados when creating an IoCtx. This will prevent seg
faults from occurring when an invalid IoCtx is used.

Fixes: #10123
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2933 from ceph/wip-librbd-symbols

librbd: Only public API symbols from the shared library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2947 from HP-Scale-out-Storage/wip-10096

Wip 10096 (ceph-disk umount race condition)

Reviewed-by: Sage Weil <sage@redhat.com>

Fix bug #10096 (ceph-disk umount race condition)

Bug: http://tracker.ceph.com/issues/10096

Brief: Unmounting temporary mount point failed due to file being 'busy'.
Root cause could not be easily determined due to timing variances caused
by debug attempts. Race condition exists.

Solution: Implement a retry with incremental backoff as a viable
workaround. This workaround is okay because (1) Finding the root cause
would take a not insignificant amount of time/effort. (2) The workaround
is a more general fix for any process that might cause the exhibited
behavior.

Signed-off-by: Blaine Gardner <blaine.gardner@hp.com>

Merge pull request #2914 from dachary/wip-10093-monstore-tool

mon: ceph-monstore-tool must close()

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2944 from ceph/wip-9913

mon: auth entity name for forwarded messages

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Monitor: use session's entity name for audit log

Fixes: #9913
Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: Monitor: forward session entity name to leader

And populate mock session (for forwarded request) with forwarded entity
name.

Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: Monitor: stash auth entity name in session

Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>