doc: update the OpenStack glance configuration

Juno is now using a section for configuring the stores, so let's use it.
Put spaces between flag and value for clarity and readibility.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

Merge pull request #2911 from keichwa/wip-doc-rados

Wip doc rados

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: remove superfluous white spaces (for testing)

Signed-off-by: Karl Eichwalder <ke@suse.de>

doc: Fix 2 syntax errors.

Signed-off-by: Karl Eichwalder <ke@suse.de>

ceph_objectstore_tool: Fixes to make import work again

The is_pg() call is now true even for pgs pending removal, fix broken
    finish_remove_pgs() by removing is_pg() check.
Need to add create_collection() to the initial transaction on import

Fixes: #10090
Signed-off-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2903 from ceph/wip-crush-tree

crush: fix tree bucket functions

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: warn less

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2892 from leseb/doc-juno-client-socket

doc: enable RBD cache and socket on OpenStack deployments

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2902 from dachary/wip-10083-noisy-osd

qa: handle CEPH_CLI_TEST_DUP_COMMAND on ceph osd create

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/next'

qa: handle CEPH_CLI_TEST_DUP_COMMAND on ceph osd create

If CEPH_CLI_TEST_DUP_COMMAND is set when ceph osd create is called, it
will create two osd. They must be cleaned up afterwards instead of
assuming only one is going to be created.

http://tracker.ceph.com/issues/10083 Fixes: #10083

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2124 from ceph/wip-objectstore

simplify ObjectStore interface

sage-2014-11-11_08:26:01-rados-wip-sage-testing-distro-basic-multi

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2722 from ceph/wip-9598

msgr: make mark_down not block on fast dispatch

sage-2014-11-11_08:26:01-rados-wip-sage-testing-distro-basic-multi

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2901 from ktdreyer/doc-spelling-grammar

doc: spelling & grammar fixes

Reviewed-by: John Wilkins <jowilkin@redhat.com>

doc: correct grammar in RGW SSL cert installation

The SSL certificate generation instruction was missing a preposition.
Add it.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc: correct spelling of "fqdn"

The "fqdn" (fully-qualified domain name) was misspelled in a couple
places as fgdn". Use the correct spelling.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #2899 from jdurgin/wip-import-export-next

qa: allow small allocation diffs for exported rbds

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: enable RBD cache and socket on OpenStack deployments

Enabling the RBD cache improves sequential IOs and the socket helps a
lot while troubleshooting. These 2 items are considered as best
practice for OpenStack deployments with Ceph.

Signed-off-by: Sébastien Han <sebastien.han@enovance.com>

qa: allow small allocation diffs for exported rbds

The local filesytem may behave slightly differently. This isn't
foolproof, but seems to be reliable enough on rhel7 rootfs, where
exact comparison was failing.

Fixes: #10002
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2898 from ceph/wip-rbd-python-tests

librbd: Python unit tests now use unique pools and images

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2820 from ceph/wip-9854

osdc: Constrain max number of in-flight read requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2890 from zhouyuan/fix_mailmap

Fix Yuan's mailmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/next'

Fix Yuan's mailmap

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

doc/release-notes: v0.88

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2897 from ceph/wip-da-revert-10b68b

Revert "osd: detect (some) misordered ondisk tmaps"

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crush: fix tree bucket functions

There are incorrect nodes' weight in tree bucket when construct tree
bucket. The tree bucket don't store item id in items array, so the tree
bucket will not work correctly. The patch fix above bugs and add a
simple test for tree bucket.

Signed-off-by: Rongze Zhu <zrzhit@gmail.com>

crush/builder: replace printf with an empty dprintk macro

This mirrors mapper.c.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "osd: detect (some) misordered ondisk tmaps"

This reverts commit 10b68b2e1a1479c478843fab8232537675053c9b.

The check for 'nextkey < last_disk_key' makes not much sense since
last_disk_key is an empty string and not set before. Comparing a
decoded string to be less than an empty string will be never true.

Since this if() isn't part of a loop last_disk_key is only set
once and there is no other consumer: revert this dead code.

Conflicts:
src/osd/ReplicatedPG.cc

0.88

Merge pull request #2562 from ceph/wip-5595

Wip 5595

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/rados/configuration fix 'ods mkfs options' default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2466 from nereocystis/osd-filesystem-options

doc: Add file system osd config settings

Merge pull request #2883 from ceph/wip-da-SCA-20141028

Fixes for coverity and SCA issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2889 from xuechendi/move_buffers_to_BufferTransaction

KeyValueStore: move buffers from strip_header to BufferTransaction

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2739 from majianpeng/fix2

buffer: Judge bufferlist whether need  rebuild

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2894 from ceph/wip-scrub-coverity

Wip scrub coverity

AuthSessionHandler.h: init protocol in constructor

Init protocol in AuthSessionHandler constructor with
with CEPH_AUTH_UNKNOWN.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/librbd/librbd.cc: fix potential null pointer deref

Fix potential deref after null check. Move check for max_snaps in
rbd_snap_list up to fail early before call any functions.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedPG.cc: remove redundant check

Fix for:

[src/osd/ReplicatedPG.cc:2171]: (style) Redundant checking of STL
 container element existence before removing it.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/c_read_operations.cc: fix invalid checks

Wrong variable for return of rados_getxattrs_next() and
rados_omap_get_next() used (len vs. val_len).

Fix for:

CID 1219464 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this expression
 key == NULL inside statement if (len == 0UL && key == NU...

CID 1219465 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this expression
 key == NULL inside statement if (len == 0UL && key == NU...

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rados.cc: remove dead code

Fix for:

CID 1128391 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this statement goto done_err;

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados_sync.cc: fix xattr_diff() for the only_in_b checks

In the checks to build only_in_b up the wrong const_iterator x is
build up. it should compare rhs->xattrs with xattrs entries and
not twice rhs->xattrs.

Fix for:

CID 716957 (#1 of 1): Invalid iterator comparison (MISMATCHED_ITERATOR)
 mismatched_comparison: Comparing x from rhs->xattrs to this->xattrs.end()
 from this->xattrs.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

bench_log.cc: catch ceph::FailedAssertion exception

Fix for:

CID 717177 (#2-1 of 3): Uncaught exception (UNCAUGHT_EXCEPT)
 root_function: In function main(int, char const **) an exception of
 type ceph::FailedAssertion is thrown and never caught.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PGLog.cc: prefer empty() over size() for emptiness check

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

librbd: Python unit tests now use unique pools and images

RBD python unit tests no longer utilize the 'rbd' pool for
test cases.  Instead, a new temporary pool is created and
deleted. Additionally, each unit test now uses a unique
image name to reduce the possibility of test case failures
affecting subsequent tests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2873 from ceph/wip-rbd-test-fixtures

librbd: Refactor librbd unit tests to use test fixtures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

bufferlist: Don't call rebuild(), c_strt() call it if bufferlist not contiguous.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: Constrain max number of in-flight read requests

Constrain the number of in-flight RADOS read requests to the
cache size.  This reduces the chance of the cache memory
ballooning during certain scenarios like copy-up which can
invoke many concurrent read requests.

Fixes: #9854
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2893 from ceph/wip-10045

common/Readahead: use correct lock when waiting on the pending ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2878 from ceph/wip-10025

#10025 Fix 'journal import' (aka undump)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common/Readahead: use correct lock when waiting on the pending ops

Readahead was using the incorrect lock with the pending condition,
resulting in a failed assertion.  It now uses the lock associated
with pending ops.

Fixes: #10045
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: remove dead code from inode_t::compare()

CID 1251354

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CInode: catch exception by reference, not value

CID 1251353

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2380 from osynge/wip-suse-systemd-integration

systemd support in ceph-disk activate

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2446 from xylv/wip-9359

rgw: Export user stats in get-user-info Adminops API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2725 from CzBiX/rgw-modified-since

rgw: correct "If-Modified-Since" handle.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2782 from boydc2014/master

rgw: fix can not disable max_size quota

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2624 from ceph/wip-8911

rgw: swift GET / HEAD object returns X-Timestamp field

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2851 from ceph/wip-9986

objecter: fix map skipping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2667 from yuandong1222/avoid-unused-pg_to_acting_osds-call

Avoid unused pg_to_acting_osds call in OSDMap::get_primary_shard

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2474 from dachary/wip-9420-erasure-code-non-regression

erasure-code: store and compare encoded contents

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2456 from ceph/wip-9031-9262

Wip 9031 9262

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2553 from ceph/wip-aio-journal

osd: clean up journal aio+dio header issue

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2745 from ceph/wip-6756

JounralingObjectStore: journal->committed_thru after replay

Merge pull request #2773 from ceph/wip-9852

mon: MDSMonitor: proper error output if pool DNE on 'add_data_pool'

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2822 from ceph/wip-9887

Wip 9887

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2850 from ceph/wip-9987

mon: fix min_last_epoch_started tracking

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2862 from ceph/wip-log-client

osd, mds: fix cluster log

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

qa/workunits/cephtool/test.sh: ceph, not ./ceph

Fixes: #10053
Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/fs/misc: combine sudo and echo effectively

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit abc995b6b4e0117b74c301abcb4faf4ea2ebef25)

Merge pull request #2689 from zhurongze/fix-crush

crush: fix incorrect use of adjust_item_weight method

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2881 from ceph/wip-10030

librbd: don't close an already closed parent image upon failure

Reviewed-by: Sage Weil <sage@redhat.com>

crush: fix incorrect use of adjust_item_weight method

adjust_item_weight method will adjust all buckets which the item
inside. If the osd.0 in host=fake01 and host=fake02, we execute
"ceph osd crush osd.0 10 host=fake01", it not only will adjust fake01's
weight, but also will adjust fake02's weight.

the patch add adjust_item_weightf_in_loc method and fix remove_item,
_remove_item_under, update_item, insert_item, detach_bucket methods.

Signed-off-by: Rongze Zhu <zrzhit@gmail.com>

erasure-code: erasure_code_benchmark exhaustive erasure exploration

Add the --erasure-generation exhaustive flag to try all combinations of
erasures, not just one at random.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add erasure_code_benchmark --verbose

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure_code: implement ceph_erasure_code to assert the existence of a plugin

This is handy when scripting in the context of teuthology and only
conditionally run tests for the isa plugin, for instance.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: ceph_erasure_code does not need to avoid dlclose

The only reason for not dlclosing plugins at exit is for callgrind but
ceph_erasure_code has no workload that would require callgrind.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add corpus verification to make check

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: Makefile.am cosmetics

Cluster benchmark related lines together.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: s/alignement/alignment/ typos in jerasure

The jerasure-per-chunk-alignment prameter was mispelled and while
useable that would lead to confusion.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
      --plugin jerasure \
      --parameter technique=reed_sol_van \
      --parameter k=2 \
      --parameter m=2 \
      --stripe-width 3181 \
      --create \
      --check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2888 from dachary/wip-9970-erasure-code-documentation

erasure-code: document pool operations

Reviewed-by: Laurent Guerby <laurent@guerby.net>

erasure-code: document pool operations

A short introduction to the first time user of an erasure coded pool.
It includes a reminder of how it relates to cache tiering and links to
define new profiles with an example.

There was examples in the developer documentation but the operator
expects to find such a guide in the rados operations chapter.

http://tracker.ceph.com/issues/9970 Fixes: #9970

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2750 from dachary/wip-9815-make-check-parallel

parallelize make check

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

KeyValueStore: move buffers from strip_header to BufferTransaction

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

tests: use kill -0 to check process existence

When killing a daemon, instead of using kill -9 to check the process was
terminated, use kill -0. Should the pid of the process be reused
immediately after, it would be wrong to kill the new process. Worst case
scenario the kill_daemon function returns before the process is
confirmed to be killed but this is not treated as an error and is
unlikely to cause any problem.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: looping to wait for an osd to be up is expected

Remove the reference to a bug that suggests otherwise.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: increase timeout to accommodate slow machines

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: kill_daemon use $name.pid instead of pidfile

So that it can be used instead of stop.sh to stop vstart.sh daemons. The
problem with stop.sh is that it kills any daemon, not just a selection.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: group workunits/cephtool/test.sh tests per daemon

So all tests related to a given daemon (mon, osd, mds) can be run at
once.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: run workunits/cephtool/test.sh

Three scripts are added to run qa/workunits/cephtool/test.sh for each
daemon (mon, mds, osd) so they can be run in parallel.

http://tracker.ceph.com/issues/9815 Fixes: #9815

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: remove vstart_wrapped_tests.sh

Listing tests to be run in a single script does not take advantage of
parallel runs in make.

The vstart_wrapper.sh script is reworked and made less specialized and
let the caller decide which daemons to run via CEPH_START and does not
enforce the number of deamons of each time. It no longer uses stop.sh to
avoid killing the osd/mon/mds that are unrelated to the tests.

http://tracker.ceph.com/issues/9815 Fixes: #9815

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: use different ports for each mon

Run the mon on each test on a different port so they can run in
parallel.

http://tracker.ceph.com/issues/9815 Fixes: #9815

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: tolerate a disk 99% full

The tests do not need much space and will work fine even on a 99% full
disk.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2869 from fgimenez/fix-tests-on-btrfs

Fix tests on btrfs: leftover subvolumes removed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2860 from XinzeChi/master

osd: cache pool: delete dead code in ReplicatedPG::agent_choose_mode

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: use unsigned ints to compare against size()

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2814 from ceph/wip-inode-scrub

Wip inode scrub

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>