0.89

Merge pull request #3065 from ceph/wip-cephtool-exec

qa: fix cephtool tests not getting run

Reviewed-by: Loic Dachary <ldachary@redhat.com>

qa: fix teardown in cephtool's test_mon_mds

This was missing from 17b5fc9a but we didn't notice
because the test wasn't being run by the gitbuilders.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3067 from dachary/wip-cephtool

fix cephtools running only part of the tests + avoid killing wrong daemons

Reviewed-by: Sage Weil <sage@redhat.com>

qa: cephtool/test.sh use regular strings to list tests

Using the array notation to list test is error prone and more
complicated to write.

It also fixes a bug : only the first test of each series (MON, OSD, MDS)
was run and the others were ignored.

Signed-off-by: Loic Dachary <loic@dachary.org>

tests: vstart_wrapper.sh must call vstart.sh with a list of daemons

The list of daemons is provided in the $CEPH_START environment variable
but vstart_wrapper.sh incorrectly uses the $VSTART_ARGS empty variable.

As a consequence vstart.sh calls init-ceph stop which kills daemons that
do not belong to the test.

http://tracker.ceph.com/issues/10215 Fixes: #10215

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #3056 from ceph/wip-9997-1

client: invalidate kernel dentries one by one

Reviewed-by: John Spray <john.spray@redhat.com>

client: invalidate kernel dentries one by one

Our trick to trim the whole kernel dentry tree does not work for 3.18+ kernel.
The fix is trim kernel dentries one by one.

Fixes: #9997
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3008 from ceph/wip-10135

mon: OSDMonitor: allow adding tiers to FS pools

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2945 from ceph/wip-9321

mon: PaxosService: can be readable even if proposing

Reviewed-by: Sage Weil <sage@redhat.com>

mon: OSDMonitor: allow adding tiers to FS pools

This was an overly-strict check.  In fact it is perfectly
fine to set an overlay on a pool that is already in use
as a filesystem data or metadata pool.

Fixes: #10135
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2996 from ceph/wip-10151

mon: fix MDS health status from peons

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon: fix MDS health status from peons

The health data was there, but we were attempting
to enumerate MDS GIDs from pending_mdsmap (empty on
peons) instead of mdsmap (populated from paxos updates)

Fixes: #10151
Backport: giant

Signed-off-by: John Spray <john.spray@redhat.com>

rgw-admin: create subuser if needed when creating user

Fixes: #10103
Backport: firefly, giant
This turned up after fixing #9973. Earlier we also didn't create the
subuser in this case, but we didn't really read the subuser info when it
was authenticating. Now we do that as required, so we end up failing the
authentication. This only applies to cases where a subuser was created
using 'user create', and not the 'subuser create' command.

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2926 from ceph/wip-10052

ceph_test_rados_api_tier: fix whiteout cleanup (10052)

Reviewed-by: Samuel Just <sjust@redhat.com>

mon: PaxosService: can be readable even if proposing

As long as we have a stable version in memory that is lower or equal to
the version we want.

Fixes: #9321
Fixes: #9322
Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2944 from ceph/wip-9913

mon: auth entity name for forwarded messages

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Monitor: use session's entity name for audit log

Fixes: #9913
Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: Monitor: forward session entity name to leader

And populate mock session (for forwarded request) with forwarded entity
name.

Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

mon: Monitor: stash auth entity name in session

Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

messages: MForward: stash auth entity name

This will allow us to assess the entity name, used during auth, for a
forwarded request, which is otherwise unavailable to the receiving
party.

Fixes: #9913 (partially)
Backport: giant

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2916 from ceph/wip-10057

msg/simple: ms_die_on_skipped_message

Reviewed-by: Greg Farnum <greg@inktank.com>

ceph_test_rados_api_tier: fix cleanup of whiteouts

Doing a normal DELETE on a whiteout returns ENOENT.  Use the IGNORE_CACHE
flag to make sure it removes the whiteout, too.

Fixes: #10052
Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: allow whiteout deletion with IGNORE_CACHE flag

If the client specifies IGNORE_CACHE, allow a regular DELETE to zap a
whiteout.  Expand test case to verify this works.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: Fix operator= null ptr references

Fixes: #10082
dzafman-2014-11-13_10:42:58-rgw-wip-10082-testing-basic-multi

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

msg/simple: ms_die_on_skipped_message

Assert if we skip an incoming seq.  The SimpleMessenger never does this,
but the kernel messenger does.  Hopefully useful for tracking down bugs
like #10057.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: don't overwrite reply's snapbl

set_trace_dist() updates reply's snapbl, don't overwrite it. For MKSNAP
request, just need to set mdr->tracei, set_trace_dist() will set reply's
snapbl.

Reviewed-by: Sage Weil <sage@redhat.com>
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #2920 from ceph/wip-10095

crush: fix detach_bucket

Reviewed-by: Samuel Just <sjust@redhat.com>

crush/CrushWrapper: fix detach_bucket

In commit 9850227d2f0ca2f692a154de2c14a0a08e751f08 we changed the call that
changed the weight of all instances of item to one that explicitly
changes it in the parent bucket, but parent_id may not be valid at the
call site.  Move this into the conditional block to fix.

Fixes: #10095
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2906 from ceph/wip-9835

osd: fix misdirected ec ops stuck in optracker (9835)

Reviewed-by: Samuel Just <sam.just@inktank.com>

osd/OSD: use OSDMap helper to determine if we are correct op target

Use the new helper.  This fixes our behavior for EC pools where targetting
a different shard is not correct, while for replicated pools it may be. In
the EC case, it leaves the op hanging indefinitely in the OpTracker because
the pgid exists but as a different shard.

Fixes: #9835
Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add osd_is_valid_op_target()

Helper to check whether an osd is a given op target for a pg.  This
assumes that for EC we always send ops to the primary, while for
replicated we may target any replica.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2899 from jdurgin/wip-import-export-next

qa: allow small allocation diffs for exported rbds

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa: allow small allocation diffs for exported rbds

The local filesytem may behave slightly differently. This isn't
foolproof, but seems to be reliable enough on rhel7 rootfs, where
exact comparison was failing.

Fixes: #10002
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'gh/next'

doc/release-notes: v0.88

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2897 from ceph/wip-da-revert-10b68b

Revert "osd: detect (some) misordered ondisk tmaps"

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Revert "osd: detect (some) misordered ondisk tmaps"

This reverts commit 10b68b2e1a1479c478843fab8232537675053c9b.

The check for 'nextkey < last_disk_key' makes not much sense since
last_disk_key is an empty string and not set before. Comparing a
decoded string to be less than an empty string will be never true.

Since this if() isn't part of a loop last_disk_key is only set
once and there is no other consumer: revert this dead code.

Conflicts:
src/osd/ReplicatedPG.cc

0.88

Merge pull request #2562 from ceph/wip-5595

Wip 5595

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/rados/configuration fix 'ods mkfs options' default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2466 from nereocystis/osd-filesystem-options

doc: Add file system osd config settings

Merge pull request #2883 from ceph/wip-da-SCA-20141028

Fixes for coverity and SCA issues

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2889 from xuechendi/move_buffers_to_BufferTransaction

KeyValueStore: move buffers from strip_header to BufferTransaction

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #2739 from majianpeng/fix2

buffer: Judge bufferlist whether need  rebuild

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2894 from ceph/wip-scrub-coverity

Wip scrub coverity

AuthSessionHandler.h: init protocol in constructor

Init protocol in AuthSessionHandler constructor with
with CEPH_AUTH_UNKNOWN.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

src/librbd/librbd.cc: fix potential null pointer deref

Fix potential deref after null check. Move check for max_snaps in
rbd_snap_list up to fail early before call any functions.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/ReplicatedPG.cc: remove redundant check

Fix for:

[src/osd/ReplicatedPG.cc:2171]: (style) Redundant checking of STL
 container element existence before removing it.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

test/librados/c_read_operations.cc: fix invalid checks

Wrong variable for return of rados_getxattrs_next() and
rados_omap_get_next() used (len vs. val_len).

Fix for:

CID 1219464 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this expression
 key == NULL inside statement if (len == 0UL && key == NU...

CID 1219465 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this expression
 key == NULL inside statement if (len == 0UL && key == NU...

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rgw_rados.cc: remove dead code

Fix for:

CID 1128391 (#1 of 1): Logically dead code (DEADCODE)
 dead_error_line: Execution cannot reach this statement goto done_err;

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

rados_sync.cc: fix xattr_diff() for the only_in_b checks

In the checks to build only_in_b up the wrong const_iterator x is
build up. it should compare rhs->xattrs with xattrs entries and
not twice rhs->xattrs.

Fix for:

CID 716957 (#1 of 1): Invalid iterator comparison (MISMATCHED_ITERATOR)
 mismatched_comparison: Comparing x from rhs->xattrs to this->xattrs.end()
 from this->xattrs.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

bench_log.cc: catch ceph::FailedAssertion exception

Fix for:

CID 717177 (#2-1 of 3): Uncaught exception (UNCAUGHT_EXCEPT)
 root_function: In function main(int, char const **) an exception of
 type ceph::FailedAssertion is thrown and never caught.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

osd/PGLog.cc: prefer empty() over size() for emptiness check

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #2873 from ceph/wip-rbd-test-fixtures

librbd: Refactor librbd unit tests to use test fixtures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

bufferlist: Don't call rebuild(), c_strt() call it if bufferlist not contiguous.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #2893 from ceph/wip-10045

common/Readahead: use correct lock when waiting on the pending ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2878 from ceph/wip-10025

#10025 Fix 'journal import' (aka undump)

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

common/Readahead: use correct lock when waiting on the pending ops

Readahead was using the incorrect lock with the pending condition,
resulting in a failed assertion.  It now uses the lock associated
with pending ops.

Fixes: #10045
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mds: remove dead code from inode_t::compare()

CID 1251354

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CInode: catch exception by reference, not value

CID 1251353

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #2380 from osynge/wip-suse-systemd-integration

systemd support in ceph-disk activate

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2446 from xylv/wip-9359

rgw: Export user stats in get-user-info Adminops API

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2725 from CzBiX/rgw-modified-since

rgw: correct "If-Modified-Since" handle.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2782 from boydc2014/master

rgw: fix can not disable max_size quota

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #2624 from ceph/wip-8911

rgw: swift GET / HEAD object returns X-Timestamp field

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2851 from ceph/wip-9986

objecter: fix map skipping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2667 from yuandong1222/avoid-unused-pg_to_acting_osds-call

Avoid unused pg_to_acting_osds call in OSDMap::get_primary_shard

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2474 from dachary/wip-9420-erasure-code-non-regression

erasure-code: store and compare encoded contents

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2456 from ceph/wip-9031-9262

Wip 9031 9262

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2553 from ceph/wip-aio-journal

osd: clean up journal aio+dio header issue

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2745 from ceph/wip-6756

JounralingObjectStore: journal->committed_thru after replay

Merge pull request #2773 from ceph/wip-9852

mon: MDSMonitor: proper error output if pool DNE on 'add_data_pool'

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2822 from ceph/wip-9887

Wip 9887

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #2850 from ceph/wip-9987

mon: fix min_last_epoch_started tracking

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #2862 from ceph/wip-log-client

osd, mds: fix cluster log

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

qa/workunits/cephtool/test.sh: ceph, not ./ceph

Fixes: #10053
Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/fs/misc: combine sudo and echo effectively

Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit abc995b6b4e0117b74c301abcb4faf4ea2ebef25)

Merge pull request #2689 from zhurongze/fix-crush

crush: fix incorrect use of adjust_item_weight method

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #2881 from ceph/wip-10030

librbd: don't close an already closed parent image upon failure

Reviewed-by: Sage Weil <sage@redhat.com>

crush: fix incorrect use of adjust_item_weight method

adjust_item_weight method will adjust all buckets which the item
inside. If the osd.0 in host=fake01 and host=fake02, we execute
"ceph osd crush osd.0 10 host=fake01", it not only will adjust fake01's
weight, but also will adjust fake02's weight.

the patch add adjust_item_weightf_in_loc method and fix remove_item,
_remove_item_under, update_item, insert_item, detach_bucket methods.

Signed-off-by: Rongze Zhu <zrzhit@gmail.com>

erasure-code: erasure_code_benchmark exhaustive erasure exploration

Add the --erasure-generation exhaustive flag to try all combinations of
erasures, not just one at random.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add erasure_code_benchmark --verbose

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure_code: implement ceph_erasure_code to assert the existence of a plugin

This is handy when scripting in the context of teuthology and only
conditionally run tests for the isa plugin, for instance.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: ceph_erasure_code does not need to avoid dlclose

The only reason for not dlclosing plugins at exit is for callgrind but
ceph_erasure_code has no workload that would require callgrind.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: add corpus verification to make check

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: Makefile.am cosmetics

Cluster benchmark related lines together.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: s/alignement/alignment/ typos in jerasure

The jerasure-per-chunk-alignment prameter was mispelled and while
useable that would lead to confusion.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: workunit to check for encoding regression

Clone the archive of encoded objects and decode all archived objects, up
to and including the current ceph version.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

erasure-code: store and compare encoded contents

Introduce ceph_erasure_code_non_regression to check and compare how an
erasure code plugin encodes and decodes content with a given set of
parameters. For instance:

./ceph_erasure_code_non_regression \
      --plugin jerasure \
      --parameter technique=reed_sol_van \
      --parameter k=2 \
      --parameter m=2 \
      --stripe-width 3181 \
      --create \
      --check

Will create an encoded object (--create) and store it into a directory
along with the chunks, one chunk per file. The directory name is derived
from the parameters. The content of the object is a random pattern of 31
bytes repeated to fill the object size specified with --stripe-width.

The check function (--check) reads the object back from the file,
encodes it and compares the result with the content of the chunks read
from the files. It also attempts recover from one or two erasures.

Chunks encoded by a given version of Ceph are expected to be encoded
exactly in the same way by all Ceph versions going forward.

http://tracker.ceph.com/issues/9420 Refs: #9420

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2888 from dachary/wip-9970-erasure-code-documentation

erasure-code: document pool operations

Reviewed-by: Laurent Guerby <laurent@guerby.net>

erasure-code: document pool operations

A short introduction to the first time user of an erasure coded pool.
It includes a reminder of how it relates to cache tiering and links to
define new profiles with an example.

There was examples in the developer documentation but the operator
expects to find such a guide in the rados operations chapter.

http://tracker.ceph.com/issues/9970 Fixes: #9970

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #2750 from dachary/wip-9815-make-check-parallel

parallelize make check

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

KeyValueStore: move buffers from strip_header to BufferTransaction

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

tests: use kill -0 to check process existence

When killing a daemon, instead of using kill -9 to check the process was
terminated, use kill -0. Should the pid of the process be reused
immediately after, it would be wrong to kill the new process. Worst case
scenario the kill_daemon function returns before the process is
confirmed to be killed but this is not treated as an error and is
unlikely to cause any problem.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: looping to wait for an osd to be up is expected

Remove the reference to a bug that suggests otherwise.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: increase timeout to accommodate slow machines

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: kill_daemon use $name.pid instead of pidfile

So that it can be used instead of stop.sh to stop vstart.sh daemons. The
problem with stop.sh is that it kills any daemon, not just a selection.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: group workunits/cephtool/test.sh tests per daemon

So all tests related to a given daemon (mon, osd, mds) can be run at
once.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

tests: run workunits/cephtool/test.sh

Three scripts are added to run qa/workunits/cephtool/test.sh for each
daemon (mon, mds, osd) so they can be run in parallel.

http://tracker.ceph.com/issues/9815 Fixes: #9815

Signed-off-by: Loic Dachary <loic-201408@dachary.org>