test: test-erasure-eio.sh fix recovery testing and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

test: CLEANUP make test clearer by using "fail" as argument

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12142 from athanatos/wip-17966

osd: Fix read error propogation in ECBackend

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #11994 from ukernel/wip-17193

mds: force client flush snap data before truncating objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11635 from batrick/mds-unique-ptr-20161024

mds: more unique_pointer changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11917 from ajarr/wip-17800

ceph_volume_client: set an existing auth ID's default mon caps

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11983 from liewegas/wip-bluestore-release

os/bluestore: fix alloc release timing on sync submits

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12129 from wjin/fix_backoffthrottle_comment

common/Throttle: fix typo for BackoffThrottle

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9643 from wjin/mon

mon: clean legacy code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

src/test: reenable test-erasure-eio.sh

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12000 from jcsp/wip-doc-exports

src/doc: fix class names in exports.txt

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10474 from songbaisen/b5

mon:  remove the redudant jugement in LogMonitor tick function

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11631 from wolsen/issues/17689

upstart: start ceph-all after static-network-up

Reviewed-by: James Page <james.page@ubuntu.com>

Merge pull request #12128 from tchaikov/wip-cmake

cmake: add -Wno-unknown-pragmas to CMAKE_CXX_FLAGS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12126 from branch-predictor/bp-doc-async-opts

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12135 from kylinstorage/wip-remove-unneeded-forward-declaration

osd/PG.h: remove unneeded forward declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12162 from liewegas/wip-coverity

script/run-coverity: update

script/run-coverity: -x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11848 from javeme/case-CEPH_MSG_POOLOP

mon: move case CEPH_MSG_POOLOP to OSDs group

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #12160 from vshankar/wip-17973

librbd: mark request as finished after failed refresh

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Documentation of a few of Async Messenger configuration options.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #12055 from liewegas/wip-17942

osdc/Objecter: handle race between calc_target and handle_osd_map

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11178 from stiopaa1/osd_pg_changeStructPGRecoveryStatsToClass

osd/PG.h: change PGRecoveryStats struct to class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12009 from 363921219/master

osd: clean up process_peering_events

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12031 from liewegas/wip-ceph-abort

common: assert(0) -> ceph_abort()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: account m_processing when failing request after refresh

AioImageRequestWQ::_void_process() dequeues an item from the
work queue by invoking ThreadPool::_void_process() which also
performs accounting by incrementing ->m_processing which never
gets decremented if an image refresh fails. This is exactly
the flow when running test_lock_fence.sh which blacklists a
client and expects failed I/O operation from the client.

Without this fix, running test_lock_fence.sh in a loop fails
with an assert in PointerWQ dtor within 100 odd runs.

Fixes: http://tracker.ceph.com/issues/17973
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #12034 from wjwithagen/wip-wjw-freebsd-strtoll

common/strtol.cc: Get error testing also to work on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12150 from tchaikov/wip-kill-gf-complete-warnings

erasure-code: fix gf-complete warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cmake: regroup CMAKE_CXX_FLAGS and CMAKE_C_FLAGS

- some options are only meaningful for C++, so move them into
  CMAKE_CXX_FLAGS.
- regroup CMAKE_CXX_FLAGS and CMAKE_C_FLAGS, and remove the duplicated
  options.
- do not reset CMAKE_CXX_FLAGS with ${CMAKE_C_FLAGS}, instead, always
  append ${CMAKE_C_FLAGS} to existing ${CMAKE_CXX_FLAGS}. so we don't
  clobber CMAKE_CXX_FLAGS set by command line.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add -Wno-unknown-pragmas to CMAKE_C_FLAGS

this silences warnings like:

warning: ignoring #pragma clang diagnostic [-Wunknown-pragmas]

and recent clang (3.9 and 4.0) do not complain at seeing
so we only pass this this param to gcc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

erasure-code: fix gf-complete warning

pick up latest gf-complete submodule

$ git log --no-merges --pretty=%s \
  a6847973cba329ae079d3bd26341a4ec2906f012..c431e1ff76bdce6c3f6c0d0cbdf6a01c68f81071
gf_w32.c: silence the -Wmaybe-uninitialized warning

it silences following warning:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c: In
function ‘gf_w32_cfmgk_multiply_region_from_single’:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c:410:5:
warning: ‘a’ may be used uninitialized in this function
[-Wmaybe-uninitialized]
   g = _mm_insert_epi64 (a, g_star, 0);
     ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12146 from dzafman/wip-dz-delays

tests: minor make check cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12136 from dachary/wip-17889-systemd-order

ceph-disk: resolve race conditions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12145 from tchaikov/wip-client-use-after-freed

client: trim_caps() do not dereference cap if it's removed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test: Handle terminal width 0 case for running test in emacs

Signed-off-by: David Zafman <dzafman@redhat.com>

test: CLEANUP: Make wait_for_clean() clearer changing variable name

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Return wait_for_clean() to start sleeping at .1

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Enhance get_timeout_delays()

Do all math using bc so we can have fractions
Allow caller to specify the first step (default 1)
Add testing of fractional first step

Signed-off-by: David Zafman <dzafman@redhat.com>

client: trim_caps() do not dereference cap if it's removed

this silences the warning of "Use of memory after it is freed" reported
by clang static analyzer.

Reported-by: Brad Hubbard <bhubbard@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12127 from renhwztetecs/renhw-wip-doc-mds

doc/mds: fixup "mds bal mode" Description

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12052 from liewegas/wip-osd-fast-dispatch

osd: a few fast dispatch optimizations

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12059 from liewegas/wip-osd-omap-limits

osd: set server-side limits on omap get operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12064 from majianpeng/rocksdb-perf

kv/RocksDBStore: Don't update rocksdb perf_context if rocksdb_perf di…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12068 from yuyuyu101/wip-rdma-cct

msg/async/rdma: ensure CephContext existed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12069 from liupan1111/wip-osd-up-heartbeat-peers

osd: heartbeat peers need to be updated when a new OSD added into an existed cluster

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

mds: use unique_ptr unstable_bits_t in SimpleLock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use unique_ptr in MDCache

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use unique_ptr bufferlist in inline_data_t

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11959 from batrick/mds-header-prot-org

Partial organization of mds/ header sections

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12111 from batrick/client-capsnap-emplace

client: put CapSnap not ptr in cap_snaps map

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12098 from ukernel/wip-17606

multimds: fix state check in Migrator::find_stale_export_freeze()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12063 from batrick/i17858

mds: use projected path construction for access

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

src/doc: move exports.txt into doc/

Signed-off-by: John Spray <john.spray@redhat.com>

src/doc: remove anchortable.txt

The anchor table is long gone.

Signed-off-by: John Spray <john.spray@redhat.com>

src/doc: fix class names in exports.txt

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #12141 from athanatos/wip-17983

osd/ECTransaction: only write out the hinfo if not delete

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12143 from rook/wip-boost-ext

cmake: fix boost build on ubuntu 16.10 yakkety

Merge pull request #12078 from yangdongsheng/rbd_cleanup

librbd: minor cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: fix boost build on ubuntu yakkety

If the host system has boost version 1.61 or higher (as in the case
for ubuntu 16.10 yakkety) the ceph build is currently broken. it
will pickup the system boost libraries but use the headers from the
submodule.

This commit ensure that when WITH_SYSTEM_BOOST is OFF we always
use the boost libraries and headers built from the submodule.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #12110 from batrick/client-failure-msg

client: improve failure messages/debugging

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/PG.h: remove unneeded forward declaration

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

ceph-disk: trigger must ensure device ownership

The udev rules that set the owner/group of the OSD devices are racing
with 50-udev-default.rules and depending on which udev event fires last,
ownership may not be as expected.

Since ceph-disk trigger --sync runs as root, always happens after
dm/lvm/filesystem units are complete and before activation, it is a good
time to set the ownership of the device.

It does not eliminate all races: a script running after systemd
local-fs.target and firing a udev event may create a situation where the
permissions of the device are temporarily reverted while the activation
is running.

Fixes: http://tracker.ceph.com/issues/17813
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: systemd unit must run after local-fs.target

A ceph udev action may be triggered before the local file systems are
mounted because there is no ordering in udev. The ceph udev action
delegates asynchronously to systemd via ceph-disk@.service which will
fail if (for instance) the LVM partition required to mount /var/lib/ceph
is not available yet. The systemd unit will retry a few times but will
eventually fail permanently. The sysadmin can systemctl reset-fail at a
later time and it will succeed.

Add a dependency to ceph-disk@.service so that it waits until the local
file systems are mounted:

After=local-fs.target

Since local-fs.target depends on lvm, it will wait until the lvm
partition (as well as any dm devices) is ready and mounted before
attempting to activate the OSD. It may still fail because the
corresponding journal/data partition is not ready yet (which is
expected) but it will no longer fail because the lvm/filesystems/dm are
not ready.

Fixes: http://tracker.ceph.com/issues/17889
Signed-off-by: Loic Dachary <loic@dachary.org>

common/Throttle: fix typo for BackoffThrottle

Fix formula so that pepople can understand it smoothly.

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

doc/mds: fixup "mds bal mode" Description

mds_load code is:
switch(g_conf->mds_bal_mode) {
  case 0:
    return
      .8 * auth.meta_load() +
      .2 * all.meta_load() +
      req_rate +
      10.0 * queue_len;

  case 1:
    return req_rate + 10.0*queue_len;

  case 2:
    return cpu_load_avg;
  }

from Brett Niver <bniver@redhat.com>

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #12124 from branch-predictor/bp-fast-markdown-doc

doc/rados/configuration/osd-config-ref.rst: document the fast mark down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12123 from branch-predictor/bp-fast-markdown-is-default

test/osd-fast-mark-down.sh: wrong assumption on first subtest

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/osd-config-ref.rst: document the fast mark down

Document the "osd fast fail on connection refused" option.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

test/osd-fast-mark-down.sh: wrong assumption on first subtest

First subtest checks for proper behavior when "osd fast fail on
connection refused" is disabled, yet it doesn't disable it and
fails, as is it enabled by default.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #8948 from cooboos/perf-reset

test: add perf-reset test in test/perf_counters.cc

Reviewed-by: Haomai Wang <haomai@xsky.com>

assert(0) -> ceph_abort()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12072 from dzafman/wip-test-17830

tests: fix osd-scrub-repair.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: cleanup: use a more proper variable name

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

ECTransaction: only write out the hinfo if not delete

Fixes: http://tracker.ceph.com/issues/17983
Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: propogate read errors through callbacks

Signed-off-by: Samuel Just <sjust@redhat.com>

mds: organize CDentry class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: organize Capability class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: organize Beacon class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11993 from ceph/wip-17809

buffer: add advance(unsigned) back

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12049 from liewegas/wip-kv-iterators

kv: remove snapshot iterator

Reviewed-by: Kari, Varada <varada.kari@sandisk.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

ECBackend::ReadOp: fill in complete in constructor, not in do_read_op

Otherwise, send_all_remaining_reads creates duplicate entries.

Fixes: http://tracker.ceph.com/issues/17966
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #11719 from zhangsw/fix-rgw-requestpay-bug

rgw: Anonymous users shouldn't be able to access requester pays buckets.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11989 from idealguo/httpargs-parse-cleanup

rgw: skip empty http args in method parse() to avoid extra effort

Merge pull request #12021 from rzarzynski/wip-rgw-comp-noimplacefact

rgw: compression uses optional::emplace instead of in-place factories

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12108 from adamemerson/wip-compile-compress

compress: Fix compilation failure from missing header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11827 from oritwas/wip-rgw-upgrade-doc

rgw: add recovery procedure for upgrade to older version of jewel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11940 from trociny/wip-17880

librbd: release lock after demote

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11960 from vshankar/wip-librbd-ec-support

librbd: bug fixes for optional data pool support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

compress: Fix compilation failure from missing header

Compress was failing due to the random header not being included.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: compression uses boost::optional::emplace instead of in-place factories.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #11901 from ceph/wip-rgw-compression-type-random

rgw: add rgw_compression_type=random for teuthology testing

Merge pull request #12073 from rzarzynski/wip-rgw-compile-asio-by-default

rgw: compilation of the ASIO front-end is enabled by default.

Merge pull request #12103 from rzarzynski/wip-rgw-auth-static_vector

rgw: avoid dynamic memory allocations in rgw_swift_auth.cc by using static_vector

verified: f23

test: Re-enable osd-scrub-repair.sh again

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix broken ec_size_mismatch setting

Caused by: 1e95f2ce642485e25b74f7d873fda7e4b75a425f

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12101 from tchaikov/wip-mark-down-test-cleanup

test/osd/osd-fast-mark-down.sh: remove unnecessary teardown() calls

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #12100 from tchaikov/wip-rocksdb

rocksdb: update to latest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12093 from tchaikov/wip-yasm-wrapper

yasm-wrapper: translate "-isystem $1" to "-i $1"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12005 from wjwithagen/wip-wjw-freebsd-ceph-helpers

workunits/ceph-helpers.sh: FreeBSD returns a different errorstring.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "rgw: temporarily use std::vector in place of static_vector of Boost."

This reverts commit bc23e0f7fa8491c44fa938eeb954197f6aad2367.
We're doing that because the reverted commit was a makeshift
solution to not fail Ceph compilation on platforms lacking
Boost modern enough to ship the container/static_vector.hpp.
As we got the in-tree Boost the commit isn't necessary anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>