rbd: enabling/disabling rbd feature should report missing dependency

Currently while enabling or disabling rbd feature command does not
give missing dependency for eg: attempting to enable the journaling
feature on an image that doesn't have the exclusive-lock feature
enabled should give missing dependency error message.

Fixes: http://tracker.ceph.com/issues/16985
Reported-by: Jason Dillaman <dillaman@redhat.com>
Signed-off-by: Gaurav Kumar Garg <garg.gaurav52@gmail.com>

Merge pull request #12183 from dachary/wip-17830-eio

tests: facilitate background process debug in ceph-helpers.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12184 from SUSE/wip-18040

doc: we can now run multiple MDS, so qualify warning

Reviewed-by: John Spray <john.spray@redhat.com>

doc: we can now run multiple MDS, so qualify warning

Fixes: http://tracker.ceph.com/issues/18040
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: facilitate background process debug in ceph-helpers.sh

When displaying the output of a background process, do it on stderr so
that it is not bufferized. Otherwise the output of the background
process may be displayed after it completed.

Prefix the output of a background process with the PID of the process
known to the parent instead of the PID of the awk process processing the
output. When wait_background loops, it will print the process on which
it is waiting and it is confusing that they do not match with the PID
prefixing the process output.

Refs: http://tracker.ceph.com/issues/17830

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11733 from tchaikov/wip-17762

common: release g_ceph_context before returns

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12172 from dachary/wip-17830-eio

tests: osd-scrub-repair.sh abort if add_something fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

crushtool: do not release g_ceph_context at exit

it is but a work around of occasionally timeout.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,test: g_ceph_context->put() upon return

prior to this change, global_init() could create a new CephContext
and assign it to g_ceph_context. it's our responsibilty to release
the CephContext explicitly using cct->put() before the application
quits. but sometimes, we fail to do so.

in this change, global_init() will return an intrusive_ptr<CephContext>,
which calls `g_ceph_context->put()` in its dtor. this ensures that
the CephContext is always destroyed before main() returns. so the
log is flushed before _log_exp_length is destroyed.

there are two cases where global_pre_init() is called directly.
- ceph_conf.cc: g_ceph_context->put() will be called by an intrusive_ptr<>
  deleter.
- rgw_main.cc: global_init() is called later on on the success code
  path, so it will be taken care of.

Fixes: http://tracker.ceph.com/issues/17762
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10296 from songbaisen/b4

mon: small change on the HealthMonitor start_epoch  function

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon: small change on the HealthMonitor start_epoch function

No need to call start_epoch function each time in the loop.

Signed-off-by: songbaisen <song.baisen@zte.com.cn>

Merge pull request #9513 from songbaisen/a5

mon: Forbidden copy and assignment function in monoprequest

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12085 from wjwithagen/wip-freebsd-ceph-helpers-2

workunits/ceph-helpers.sh: Fixes for FreeBSD

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: osd-scrub-repair.sh abort if add_something fails

If the script keeps going after this function fails, it will be non
trivial to find the root cause of the problem.

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11943 from wido/docs-pool-remove

doc: Add documentation about mon_allow_pool_delete before pool remove

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

workunits/ceph-helpers.sh: Fixes for FreeBSD

 - stat(1) does not have '%T'

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10402 from ceph/wip-openstack-wusui

qa: Add ceph-ansible installer.

Merge pull request #11673 from zhangsw/fix-rgw-initbucketshardsync-bug

rgw: Class member cookie is not initialized correctly in some coroutine's constructor.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11969 from yehudasa/wip-17897

rgw: don't store empty chains in gc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11815 from ceph/wip-17792

rgw: remove circular reference in RGWAsyncRadosRequest
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #12142 from athanatos/wip-17966

osd: Fix read error propogation in ECBackend

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #11994 from ukernel/wip-17193

mds: force client flush snap data before truncating objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11635 from batrick/mds-unique-ptr-20161024

mds: more unique_pointer changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11917 from ajarr/wip-17800

ceph_volume_client: set an existing auth ID's default mon caps

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11983 from liewegas/wip-bluestore-release

os/bluestore: fix alloc release timing on sync submits

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12129 from wjin/fix_backoffthrottle_comment

common/Throttle: fix typo for BackoffThrottle

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9643 from wjin/mon

mon: clean legacy code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

src/test: reenable test-erasure-eio.sh

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12000 from jcsp/wip-doc-exports

src/doc: fix class names in exports.txt

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10474 from songbaisen/b5

mon:  remove the redudant jugement in LogMonitor tick function

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11631 from wolsen/issues/17689

upstart: start ceph-all after static-network-up

Reviewed-by: James Page <james.page@ubuntu.com>

Merge pull request #12128 from tchaikov/wip-cmake

cmake: add -Wno-unknown-pragmas to CMAKE_CXX_FLAGS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12126 from branch-predictor/bp-doc-async-opts

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12135 from kylinstorage/wip-remove-unneeded-forward-declaration

osd/PG.h: remove unneeded forward declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12162 from liewegas/wip-coverity

script/run-coverity: update

script/run-coverity: -x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11848 from javeme/case-CEPH_MSG_POOLOP

mon: move case CEPH_MSG_POOLOP to OSDs group

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #12160 from vshankar/wip-17973

librbd: mark request as finished after failed refresh

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Documentation of a few of Async Messenger configuration options.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

doc: Add documentation about mon_allow_pool_delete before pool remove

Tell users they need to set this to true before Monitors will allow
pools to be removed.

Also update the Pending Release Notes so that users can find this change
there.

This was changed with commit 5d7f4ea

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #12055 from liewegas/wip-17942

osdc/Objecter: handle race between calc_target and handle_osd_map

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11178 from stiopaa1/osd_pg_changeStructPGRecoveryStatsToClass

osd/PG.h: change PGRecoveryStats struct to class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12009 from 363921219/master

osd: clean up process_peering_events

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12031 from liewegas/wip-ceph-abort

common: assert(0) -> ceph_abort()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: account m_processing when failing request after refresh

AioImageRequestWQ::_void_process() dequeues an item from the
work queue by invoking ThreadPool::_void_process() which also
performs accounting by incrementing ->m_processing which never
gets decremented if an image refresh fails. This is exactly
the flow when running test_lock_fence.sh which blacklists a
client and expects failed I/O operation from the client.

Without this fix, running test_lock_fence.sh in a loop fails
with an assert in PointerWQ dtor within 100 odd runs.

Fixes: http://tracker.ceph.com/issues/17973
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #12034 from wjwithagen/wip-wjw-freebsd-strtoll

common/strtol.cc: Get error testing also to work on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12150 from tchaikov/wip-kill-gf-complete-warnings

erasure-code: fix gf-complete warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cmake: regroup CMAKE_CXX_FLAGS and CMAKE_C_FLAGS

- some options are only meaningful for C++, so move them into
  CMAKE_CXX_FLAGS.
- regroup CMAKE_CXX_FLAGS and CMAKE_C_FLAGS, and remove the duplicated
  options.
- do not reset CMAKE_CXX_FLAGS with ${CMAKE_C_FLAGS}, instead, always
  append ${CMAKE_C_FLAGS} to existing ${CMAKE_CXX_FLAGS}. so we don't
  clobber CMAKE_CXX_FLAGS set by command line.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add -Wno-unknown-pragmas to CMAKE_C_FLAGS

this silences warnings like:

warning: ignoring #pragma clang diagnostic [-Wunknown-pragmas]

and recent clang (3.9 and 4.0) do not complain at seeing
so we only pass this this param to gcc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

erasure-code: fix gf-complete warning

pick up latest gf-complete submodule

$ git log --no-merges --pretty=%s \
  a6847973cba329ae079d3bd26341a4ec2906f012..c431e1ff76bdce6c3f6c0d0cbdf6a01c68f81071
gf_w32.c: silence the -Wmaybe-uninitialized warning

it silences following warning:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c: In
function ‘gf_w32_cfmgk_multiply_region_from_single’:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c:410:5:
warning: ‘a’ may be used uninitialized in this function
[-Wmaybe-uninitialized]
   g = _mm_insert_epi64 (a, g_star, 0);
     ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12146 from dzafman/wip-dz-delays

tests: minor make check cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12136 from dachary/wip-17889-systemd-order

ceph-disk: resolve race conditions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12145 from tchaikov/wip-client-use-after-freed

client: trim_caps() do not dereference cap if it's removed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test: Handle terminal width 0 case for running test in emacs

Signed-off-by: David Zafman <dzafman@redhat.com>

test: CLEANUP: Make wait_for_clean() clearer changing variable name

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Return wait_for_clean() to start sleeping at .1

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Enhance get_timeout_delays()

Do all math using bc so we can have fractions
Allow caller to specify the first step (default 1)
Add testing of fractional first step

Signed-off-by: David Zafman <dzafman@redhat.com>

client: trim_caps() do not dereference cap if it's removed

this silences the warning of "Use of memory after it is freed" reported
by clang static analyzer.

Reported-by: Brad Hubbard <bhubbard@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12127 from renhwztetecs/renhw-wip-doc-mds

doc/mds: fixup "mds bal mode" Description

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12052 from liewegas/wip-osd-fast-dispatch

osd: a few fast dispatch optimizations

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12059 from liewegas/wip-osd-omap-limits

osd: set server-side limits on omap get operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12064 from majianpeng/rocksdb-perf

kv/RocksDBStore: Don't update rocksdb perf_context if rocksdb_perf di…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12068 from yuyuyu101/wip-rdma-cct

msg/async/rdma: ensure CephContext existed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12069 from liupan1111/wip-osd-up-heartbeat-peers

osd: heartbeat peers need to be updated when a new OSD added into an existed cluster

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

mds: use unique_ptr unstable_bits_t in SimpleLock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use unique_ptr in MDCache

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use unique_ptr bufferlist in inline_data_t

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11959 from batrick/mds-header-prot-org

Partial organization of mds/ header sections

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12111 from batrick/client-capsnap-emplace

client: put CapSnap not ptr in cap_snaps map

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12098 from ukernel/wip-17606

multimds: fix state check in Migrator::find_stale_export_freeze()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12063 from batrick/i17858

mds: use projected path construction for access

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

src/doc: move exports.txt into doc/

Signed-off-by: John Spray <john.spray@redhat.com>

src/doc: remove anchortable.txt

The anchor table is long gone.

Signed-off-by: John Spray <john.spray@redhat.com>

src/doc: fix class names in exports.txt

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #12141 from athanatos/wip-17983

osd/ECTransaction: only write out the hinfo if not delete

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12143 from rook/wip-boost-ext

cmake: fix boost build on ubuntu 16.10 yakkety

Merge pull request #12078 from yangdongsheng/rbd_cleanup

librbd: minor cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: fix boost build on ubuntu yakkety

If the host system has boost version 1.61 or higher (as in the case
for ubuntu 16.10 yakkety) the ceph build is currently broken. it
will pickup the system boost libraries but use the headers from the
submodule.

This commit ensure that when WITH_SYSTEM_BOOST is OFF we always
use the boost libraries and headers built from the submodule.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #12110 from batrick/client-failure-msg

client: improve failure messages/debugging

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/PG.h: remove unneeded forward declaration

Signed-off-by: Li Wang <li.wang@kylin-cloud.com>

crushtool: s/exit(EXIT_FAILURE)/return EXIT_FAILURE/

so the destructor(s) can be called.

Signed-off-by: Kefu Chai <kchai@redhat.com>

global/signal_handler: reset injected segv after test

~CephContext() => ~TypedSingletonWrapper() => ~MempoolObs =>
unregister_command() => ldout() << "unregister_command" =>
Log::submit_entry() => *(volatile int *)(0) = 0xdead;

Signed-off-by: Kefu Chai <kchai@redhat.com>

test_cors.cc: fix the mem leak

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-disk: trigger must ensure device ownership

The udev rules that set the owner/group of the OSD devices are racing
with 50-udev-default.rules and depending on which udev event fires last,
ownership may not be as expected.

Since ceph-disk trigger --sync runs as root, always happens after
dm/lvm/filesystem units are complete and before activation, it is a good
time to set the ownership of the device.

It does not eliminate all races: a script running after systemd
local-fs.target and firing a udev event may create a situation where the
permissions of the device are temporarily reverted while the activation
is running.

Fixes: http://tracker.ceph.com/issues/17813
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: systemd unit must run after local-fs.target

A ceph udev action may be triggered before the local file systems are
mounted because there is no ordering in udev. The ceph udev action
delegates asynchronously to systemd via ceph-disk@.service which will
fail if (for instance) the LVM partition required to mount /var/lib/ceph
is not available yet. The systemd unit will retry a few times but will
eventually fail permanently. The sysadmin can systemctl reset-fail at a
later time and it will succeed.

Add a dependency to ceph-disk@.service so that it waits until the local
file systems are mounted:

After=local-fs.target

Since local-fs.target depends on lvm, it will wait until the lvm
partition (as well as any dm devices) is ready and mounted before
attempting to activate the OSD. It may still fail because the
corresponding journal/data partition is not ready yet (which is
expected) but it will no longer fail because the lvm/filesystems/dm are
not ready.

Fixes: http://tracker.ceph.com/issues/17889
Signed-off-by: Loic Dachary <loic@dachary.org>

common/Throttle: fix typo for BackoffThrottle

Fix formula so that pepople can understand it smoothly.

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

doc/mds: fixup "mds bal mode" Description

mds_load code is:
switch(g_conf->mds_bal_mode) {
  case 0:
    return
      .8 * auth.meta_load() +
      .2 * all.meta_load() +
      req_rate +
      10.0 * queue_len;

  case 1:
    return req_rate + 10.0*queue_len;

  case 2:
    return cpu_load_avg;
  }

from Brett Niver <bniver@redhat.com>

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #12124 from branch-predictor/bp-fast-markdown-doc

doc/rados/configuration/osd-config-ref.rst: document the fast mark down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12123 from branch-predictor/bp-fast-markdown-is-default

test/osd-fast-mark-down.sh: wrong assumption on first subtest

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/osd-config-ref.rst: document the fast mark down

Document the "osd fast fail on connection refused" option.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

test/osd-fast-mark-down.sh: wrong assumption on first subtest

First subtest checks for proper behavior when "osd fast fail on
connection refused" is disabled, yet it doesn't disable it and
fails, as is it enabled by default.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #8948 from cooboos/perf-reset

test: add perf-reset test in test/perf_counters.cc

Reviewed-by: Haomai Wang <haomai@xsky.com>

assert(0) -> ceph_abort()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12072 from dzafman/wip-test-17830

tests: fix osd-scrub-repair.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: cleanup: use a more proper variable name

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

ECTransaction: only write out the hinfo if not delete

Fixes: http://tracker.ceph.com/issues/17983
Signed-off-by: Samuel Just <sjust@redhat.com>

ECBackend: propogate read errors through callbacks

Signed-off-by: Samuel Just <sjust@redhat.com>

mds: organize CDentry class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: organize Capability class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: organize Beacon class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>