FileJournal.cc: fix _open() to return negative value in error case

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

os/FileJournal.cc: fix potential use after close cases

Fix checks after _open() calls to make sure to use the
filehandle only if open returns '0'.

Fix for:

CID 1128395 (#1 of 1): Use after close (USE_AFTER_FREE)
 pass_closed_arg: Passing closed handle this->fd as an argument
 to pwrite.

CID 1128396 (#1 of 2): Use after close (USE_AFTER_FREE)
 pass_closed_arg: Passing closed handle this->fd as an argument
 to read_header.

CID 1128397 (#1 of 1): Use after close (USE_AFTER_FREE)
 pass_closed_arg: Passing closed handle this->fd as an argument
 to read_header.

CID 1128398 (#1 of 1): Use after close (USE_AFTER_FREE)
 pass_closed_arg: Passing closed handle this->fd as an argument
 to read_header.

CID 1128400 (#1 of 1): Use after close (USE_AFTER_FREE)
 pass_closed_arg: Passing closed handle this->fd as an argument
 to read_header.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

c_read_operations.cc: prevent potential null pointer deref

CID 1251575 (#1 of 1): Dereference after null check (FORWARD_NULL)
 var_deref_model: Passing null pointer key to basic_string, which
 dereferences it.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

tools/psim.cc: catch ceph::buffer::end_of_buffer exception

Fix for:

CID 717163 (#1 of 1): Uncaught exception (UNCAUGHT_EXCEPT)
 root_function: In function main(int, char **) an exception of type
 ceph::buffer::end_of_buffer is thrown and never caught.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

librados-config.cc: catch ceph::FailedAssertion exception

CID 717157 (#1-2): Uncaught exception (UNCAUGHT_EXCEPT)
 root_function: In function main(int, char const **) an exception
 of type ceph::FailedAssertion is thrown and never caught.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_authtool.cc: catch ceph::FailedAssertion exception

Fix for:

CID 717141 (#1-19): Uncaught exception (UNCAUGHT_EXCEPT)
 root_function: In function main(int, char const **) an exception
 of type ceph::FailedAssertion is thrown and never caught.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_authtool.cc: catch ceph::buffer::end_of_buffer exception

Fix for:

CID 717142 (#1 of 1): Uncaught exception (UNCAUGHT_EXCEPT)
 root_function: In function main(int, char const **) an exception
 of type ceph::buffer::end_of_buffer is thrown and never caught.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

MDSMonitor.cc: remove cmd_getval for poolid

Remove useless check for invalid parameter for
"mds remove_data_pool", we have already the poolid
from the correct parameter 'poolname'.

Fix for:

CID 1251445 (#1 of 1): Unchecked return value (CHECKED_RETURN)
 check_return: Calling cmd_getval without checking return value
 (is done elsewhere 19 out of 22 times).

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

st_rados_list_objects.cc: free resources in error case

Fix for:

CID 1251453 (#2 of 2): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable io_ctx going out of scope leaks the
 storage it points to.

CID 1251454 (#1 of 1): Resource leak (RESOURCE_LEAK)
 leaked_storage: Variable h going out of scope leaks the storage it
 points to.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

net_handler.cc: close another socket leak

Close socket if ::setsockopt() fails before return.

Fix for:

CID 1249632 (#1 of 1): Resource leak (RESOURCE_LEAK)
9. leaked_handle: Handle variable s going out of scope leaks the handle.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

net_handler.cc: close socket leak

Close socket if set_nonblock() fails before return.

Fix for:

CID 1249633 (#1 of 1): Resource leak (RESOURCE_LEAK)
7. leaked_handle: Handle variable s going out of scope leaks the handle.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_erasure_code_benchmark.cc: don't check unsigned for < 0

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_erasure_code_non_regression.cc: prefer ++op for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

ceph_erasure_code_benchmark.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

crush/CrushWrapper.cc: prefer ++operator for non-primitive iterators

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw/rgw_gc.cc: silence cppcheck

Silence cppcheck warning about std::remove, clarify the
used remove() function from RGWGC.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #2898 from ceph/wip-rbd-python-tests

librbd: Python unit tests now use unique pools and images

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2820 from ceph/wip-9854

osdc: Constrain max number of in-flight read requests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2890 from zhouyuan/fix_mailmap

Fix Yuan's mailmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix Yuan's mailmap

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

doc/release-notes: v0.88

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2897 from ceph/wip-da-revert-10b68b

Revert "osd: detect (some) misordered ondisk tmaps"

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Revert "osd: detect (some) misordered ondisk tmaps"

This reverts commit 10b68b2e1a1479c478843fab8232537675053c9b.

The check for 'nextkey < last_disk_key' makes not much sense since
last_disk_key is an empty string and not set before. Comparing a
decoded string to be less than an empty string will be never true.

Since this if() isn't part of a loop last_disk_key is only set
once and there is no other consumer: revert this dead code.

Conflicts:
src/osd/ReplicatedPG.cc

Merge pull request #2562 from ceph/wip-5595

Wip 5595

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/rados/configuration fix 'ods mkfs options' default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2466 from nereocystis/osd-filesystem-options

doc: Add file system osd config settings

Merge pull request #2883 from ceph/wip-da-SCA-20141028

Fixes for coverity and SCA issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2889 from xuechendi/move_buffers_to_BufferTransaction

KeyValueStore: move buffers from strip_header to BufferTransaction

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2739 from majianpeng/fix2

buffer: Judge bufferlist whether need  rebuild

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2894 from ceph/wip-scrub-coverity

Wip scrub coverity

AuthSessionHandler.h: init protocol in constructor

Init protocol in AuthSessionHandler constructor with
with CEPH_AUTH_UNKNOWN.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/librbd/librbd.cc: fix potential null pointer deref

Fix potential deref after null check. Move check for max_snaps in
rbd_snap_list up to fail early before call any functions.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedPG.cc: remove redundant check

Fix for:

[src/osd/ReplicatedPG.cc:2171]: (style) Redundant checking of STL
 container element existence before removing it.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/c_read_operations.cc: fix invalid checks

Wrong variable for return of rados_getxattrs_next() and
rados_omap_get_next() used (len vs. val_len).

Fix for:

CID 1219464 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this expression
 key == NULL inside statement if (len == 0UL && key == NU...

CID 1219465 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this expression
 key == NULL inside statement if (len == 0UL && key == NU...

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rados.cc: remove dead code

Fix for:

CID 1128391 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this statement goto done_err;

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados_sync.cc: fix xattr_diff() for the only_in_b checks

In the checks to build only_in_b up the wrong const_iterator x is
build up. it should compare rhs->xattrs with xattrs entries and
not twice rhs->xattrs.

Fix for:

CID 716957 (#1 of 1): Invalid iterator comparison (MISMATCHED_ITERATOR)
 mismatched_comparison: Comparing x from rhs->xattrs to this->xattrs.end()
 from this->xattrs.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

bench_log.cc: catch ceph::FailedAssertion exception

Fix for:

CID 717177 (#2-1 of 3): Uncaught exception (UNCAUGHT_EXCEPT)
 root_function: In function main(int, char const **) an exception of
 type ceph::FailedAssertion is thrown and never caught.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PGLog.cc: prefer empty() over size() for emptiness check

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

librbd: Python unit tests now use unique pools and images

RBD python unit tests no longer utilize the 'rbd' pool for
test cases.  Instead, a new temporary pool is created and
deleted. Additionally, each unit test now uses a unique
image name to reduce the possibility of test case failures
affecting subsequent tests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2873 from ceph/wip-rbd-test-fixtures

librbd: Refactor librbd unit tests to use test fixtures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

bufferlist: Don't call rebuild(), c_strt() call it if bufferlist not contiguous.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

osdc: Constrain max number of in-flight read requests

Constrain the number of in-flight RADOS read requests to the
cache size.  This reduces the chance of the cache memory
ballooning during certain scenarios like copy-up which can
invoke many concurrent read requests.

Fixes: #9854
Backport: giant, firefly, dumpling
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #2893 from ceph/wip-10045

common/Readahead: use correct lock when waiting on the pending ops

Reviewed-by: Sage Weil <sage@redhat.com>

common/Readahead: use correct lock when waiting on the pending ops

Readahead was using the incorrect lock with the pending condition,
resulting in a failed assertion.  It now uses the lock associated
with pending ops.

Fixes: #10045
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: remove dead code from inode_t::compare()

CID 1251354

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CInode: catch exception by reference, not value

CID 1251353

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2380 from osynge/wip-suse-systemd-integration

systemd support in ceph-disk activate

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2446 from xylv/wip-9359

rgw: Export user stats in get-user-info Adminops API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2725 from CzBiX/rgw-modified-since

rgw: correct "If-Modified-Since" handle.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2782 from boydc2014/master

rgw: fix can not disable max_size quota

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2624 from ceph/wip-8911

rgw: swift GET / HEAD object returns X-Timestamp field

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2851 from ceph/wip-9986

objecter: fix map skipping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2667 from yuandong1222/avoid-unused-pg_to_acting_osds-call

Avoid unused pg_to_acting_osds call in OSDMap::get_primary_shard

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2474 from dachary/wip-9420-erasure-code-non-regression

erasure-code: store and compare encoded contents

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2456 from ceph/wip-9031-9262

Wip 9031 9262

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2553 from ceph/wip-aio-journal

osd: clean up journal aio+dio header issue

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2745 from ceph/wip-6756

JounralingObjectStore: journal->committed_thru after replay

Merge pull request #2773 from ceph/wip-9852

mon: MDSMonitor: proper error output if pool DNE on 'add_data_pool'

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2822 from ceph/wip-9887

Wip 9887

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2850 from ceph/wip-9987

mon: fix min_last_epoch_started tracking

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2862 from ceph/wip-log-client

osd, mds: fix cluster log

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2689 from zhurongze/fix-crush

crush: fix incorrect use of adjust_item_weight method

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2881 from ceph/wip-10030

librbd: don't close an already closed parent image upon failure

Reviewed-by: Sage Weil <sage@redhat.com>

crush: fix incorrect use of adjust_item_weight method

adjust_item_weight method will adjust all buckets which the item
inside. If the osd.0 in host=fake01 and host=fake02, we execute
"ceph osd crush osd.0 10 host=fake01", it not only will adjust fake01's
weight, but also will adjust fake02's weight.

the patch add adjust_item_weightf_in_loc method and fix remove_item,
_remove_item_under, update_item, insert_item, detach_bucket methods.

Signed-off-by: Rongze Zhu <zrzhit@gmail.com>

erasure-code: erasure_code_benchmark exhaustive erasure exploration

Add the --erasure-generation exhaustive flag to try all combinations of
erasures, not just one at random.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add erasure_code_benchmark --verbose

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure_code: implement ceph_erasure_code to assert the existence of a plugin

This is handy when scripting in the context of teuthology and only
conditionally run tests for the isa plugin, for instance.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: ceph_erasure_code does not need to avoid dlclose

The only reason for not dlclosing plugins at exit is for callgrind but
ceph_erasure_code has no workload that would require callgrind.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add corpus verification to make check

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: Makefile.am cosmetics

Cluster benchmark related lines together.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: s/alignement/alignment/ typos in jerasure

The jerasure-per-chunk-alignment prameter was mispelled and while
useable that would lead to confusion.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
      --plugin jerasure \
      --parameter technique=reed_sol_van \
      --parameter k=2 \
      --parameter m=2 \
      --stripe-width 3181 \
      --create \
      --check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2888 from dachary/wip-9970-erasure-code-documentation

erasure-code: document pool operations

Reviewed-by: Laurent Guerby <laurent@guerby.net>

erasure-code: document pool operations

A short introduction to the first time user of an erasure coded pool.
It includes a reminder of how it relates to cache tiering and links to
define new profiles with an example.

There was examples in the developer documentation but the operator
expects to find such a guide in the rados operations chapter.

http://tracker.ceph.com/issues/9970 Fixes: #9970

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2750 from dachary/wip-9815-make-check-parallel

parallelize make check

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

KeyValueStore: move buffers from strip_header to BufferTransaction

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

tests: use kill -0 to check process existence

When killing a daemon, instead of using kill -9 to check the process was
terminated, use kill -0. Should the pid of the process be reused
immediately after, it would be wrong to kill the new process. Worst case
scenario the kill_daemon function returns before the process is
confirmed to be killed but this is not treated as an error and is
unlikely to cause any problem.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: looping to wait for an osd to be up is expected

Remove the reference to a bug that suggests otherwise.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: increase timeout to accommodate slow machines

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: kill_daemon use $name.pid instead of pidfile

So that it can be used instead of stop.sh to stop vstart.sh daemons. The
problem with stop.sh is that it kills any daemon, not just a selection.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: group workunits/cephtool/test.sh tests per daemon

So all tests related to a given daemon (mon, osd, mds) can be run at
once.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: run workunits/cephtool/test.sh

Three scripts are added to run qa/workunits/cephtool/test.sh for each
daemon (mon, mds, osd) so they can be run in parallel.

http://tracker.ceph.com/issues/9815 Fixes: #9815

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: remove vstart_wrapped_tests.sh

Listing tests to be run in a single script does not take advantage of
parallel runs in make.

The vstart_wrapper.sh script is reworked and made less specialized and
let the caller decide which daemons to run via CEPH_START and does not
enforce the number of deamons of each time. It no longer uses stop.sh to
avoid killing the osd/mon/mds that are unrelated to the tests.

http://tracker.ceph.com/issues/9815 Fixes: #9815

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: use different ports for each mon

Run the mon on each test on a different port so they can run in
parallel.

http://tracker.ceph.com/issues/9815 Fixes: #9815

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: tolerate a disk 99% full

The tests do not need much space and will work fine even on a 99% full
disk.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2869 from fgimenez/fix-tests-on-btrfs

Fix tests on btrfs: leftover subvolumes removed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #2860 from XinzeChi/master

osd: cache pool: delete dead code in ReplicatedPG::agent_choose_mode

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: use unsigned ints to compare against size()

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2814 from ceph/wip-inode-scrub

Wip inode scrub

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #2884 from dachary/wip-mailmap

mailmap: Loic Dachary affiliation

mailmap: Loic Dachary affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2882 from ceph/wip-doc-dumpling-to-firefly

doc: Added Dumpling to Firefly upgrade section.

doc: Added Dumpling to Firefly upgrade section.

Fixes: #7679
Signed-off-by: John Wilkins <john.wilkins@inktank.com>

Merge remote-tracking branch 'origin/master' into wip-inode-scrub

Conflicts:
src/common/Makefile.am
src/mds/Server.cc

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

MDS: clean up internal MDRequests the standard way

All cleanup is now routed through respond_to_request(),
which invokes the internal_op_finish Context*, then does
mdcache->request_finish(). This is easier to reason about,
and indeed fixes a bug (I was not cleaning up locks
following flush). Use the MDSContinuation to facilitate
this in scrub's case.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

MDS: CInode: break out of validation early on symlinks

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

common/ceph_strings: add some MDS internal op names to ceph_mds_op_name()

In addition to my validate and flush, this is also missing exportdir.

Signed-off-by: Greg Farnum <greg@inktank.com>

MDS: add a flush_dentry() function, and wire it up to the admin socket

Signed-off-by: Greg Farnum <greg@inktank.com>

MDS: CInode: create a flush() function

Signed-off-by: Greg Farnum <greg@inktank.com>