librbd: account m_processing when failing request after refresh

AioImageRequestWQ::_void_process() dequeues an item from the
work queue by invoking ThreadPool::_void_process() which also
performs accounting by incrementing ->m_processing which never
gets decremented if an image refresh fails. This is exactly
the flow when running test_lock_fence.sh which blacklists a
client and expects failed I/O operation from the client.

Without this fix, running test_lock_fence.sh in a loop fails
with an assert in PointerWQ dtor within 100 odd runs.

Fixes: http://tracker.ceph.com/issues/17973
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #12034 from wjwithagen/wip-wjw-freebsd-strtoll

common/strtol.cc: Get error testing also to work on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12150 from tchaikov/wip-kill-gf-complete-warnings

erasure-code: fix gf-complete warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

erasure-code: fix gf-complete warning

pick up latest gf-complete submodule

$ git log --no-merges --pretty=%s \
  a6847973cba329ae079d3bd26341a4ec2906f012..c431e1ff76bdce6c3f6c0d0cbdf6a01c68f81071
gf_w32.c: silence the -Wmaybe-uninitialized warning

it silences following warning:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c: In
function ‘gf_w32_cfmgk_multiply_region_from_single’:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c:410:5:
warning: ‘a’ may be used uninitialized in this function
[-Wmaybe-uninitialized]
   g = _mm_insert_epi64 (a, g_star, 0);
     ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12146 from dzafman/wip-dz-delays

tests: minor make check cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12136 from dachary/wip-17889-systemd-order

ceph-disk: resolve race conditions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12145 from tchaikov/wip-client-use-after-freed

client: trim_caps() do not dereference cap if it's removed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test: Handle terminal width 0 case for running test in emacs

Signed-off-by: David Zafman <dzafman@redhat.com>

test: CLEANUP: Make wait_for_clean() clearer changing variable name

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Return wait_for_clean() to start sleeping at .1

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Enhance get_timeout_delays()

Do all math using bc so we can have fractions
Allow caller to specify the first step (default 1)
Add testing of fractional first step

Signed-off-by: David Zafman <dzafman@redhat.com>

client: trim_caps() do not dereference cap if it's removed

this silences the warning of "Use of memory after it is freed" reported
by clang static analyzer.

Reported-by: Brad Hubbard <bhubbard@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12127 from renhwztetecs/renhw-wip-doc-mds

doc/mds: fixup "mds bal mode" Description

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12052 from liewegas/wip-osd-fast-dispatch

osd: a few fast dispatch optimizations

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12059 from liewegas/wip-osd-omap-limits

osd: set server-side limits on omap get operations

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12064 from majianpeng/rocksdb-perf

kv/RocksDBStore: Don't update rocksdb perf_context if rocksdb_perf di…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12068 from yuyuyu101/wip-rdma-cct

msg/async/rdma: ensure CephContext existed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12069 from liupan1111/wip-osd-up-heartbeat-peers

osd: heartbeat peers need to be updated when a new OSD added into an existed cluster

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11959 from batrick/mds-header-prot-org

Partial organization of mds/ header sections

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12111 from batrick/client-capsnap-emplace

client: put CapSnap not ptr in cap_snaps map

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #12098 from ukernel/wip-17606

multimds: fix state check in Migrator::find_stale_export_freeze()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12063 from batrick/i17858

mds: use projected path construction for access

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12141 from athanatos/wip-17983

osd/ECTransaction: only write out the hinfo if not delete

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12143 from rook/wip-boost-ext

cmake: fix boost build on ubuntu 16.10 yakkety

Merge pull request #12078 from yangdongsheng/rbd_cleanup

librbd: minor cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: fix boost build on ubuntu yakkety

If the host system has boost version 1.61 or higher (as in the case
for ubuntu 16.10 yakkety) the ceph build is currently broken. it
will pickup the system boost libraries but use the headers from the
submodule.

This commit ensure that when WITH_SYSTEM_BOOST is OFF we always
use the boost libraries and headers built from the submodule.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #12110 from batrick/client-failure-msg

client: improve failure messages/debugging

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-disk: trigger must ensure device ownership

The udev rules that set the owner/group of the OSD devices are racing
with 50-udev-default.rules and depending on which udev event fires last,
ownership may not be as expected.

Since ceph-disk trigger --sync runs as root, always happens after
dm/lvm/filesystem units are complete and before activation, it is a good
time to set the ownership of the device.

It does not eliminate all races: a script running after systemd
local-fs.target and firing a udev event may create a situation where the
permissions of the device are temporarily reverted while the activation
is running.

Fixes: http://tracker.ceph.com/issues/17813
Signed-off-by: Loic Dachary <loic@dachary.org>

ceph-disk: systemd unit must run after local-fs.target

A ceph udev action may be triggered before the local file systems are
mounted because there is no ordering in udev. The ceph udev action
delegates asynchronously to systemd via ceph-disk@.service which will
fail if (for instance) the LVM partition required to mount /var/lib/ceph
is not available yet. The systemd unit will retry a few times but will
eventually fail permanently. The sysadmin can systemctl reset-fail at a
later time and it will succeed.

Add a dependency to ceph-disk@.service so that it waits until the local
file systems are mounted:

After=local-fs.target

Since local-fs.target depends on lvm, it will wait until the lvm
partition (as well as any dm devices) is ready and mounted before
attempting to activate the OSD. It may still fail because the
corresponding journal/data partition is not ready yet (which is
expected) but it will no longer fail because the lvm/filesystems/dm are
not ready.

Fixes: http://tracker.ceph.com/issues/17889
Signed-off-by: Loic Dachary <loic@dachary.org>

doc/mds: fixup "mds bal mode" Description

mds_load code is:
switch(g_conf->mds_bal_mode) {
  case 0:
    return
      .8 * auth.meta_load() +
      .2 * all.meta_load() +
      req_rate +
      10.0 * queue_len;

  case 1:
    return req_rate + 10.0*queue_len;

  case 2:
    return cpu_load_avg;
  }

from Brett Niver <bniver@redhat.com>

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #12124 from branch-predictor/bp-fast-markdown-doc

doc/rados/configuration/osd-config-ref.rst: document the fast mark down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12123 from branch-predictor/bp-fast-markdown-is-default

test/osd-fast-mark-down.sh: wrong assumption on first subtest

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados/configuration/osd-config-ref.rst: document the fast mark down

Document the "osd fast fail on connection refused" option.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

test/osd-fast-mark-down.sh: wrong assumption on first subtest

First subtest checks for proper behavior when "osd fast fail on
connection refused" is disabled, yet it doesn't disable it and
fails, as is it enabled by default.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #8948 from cooboos/perf-reset

test: add perf-reset test in test/perf_counters.cc

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12072 from dzafman/wip-test-17830

tests: fix osd-scrub-repair.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: cleanup: use a more proper variable name

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

ECTransaction: only write out the hinfo if not delete

Fixes: http://tracker.ceph.com/issues/17983
Signed-off-by: Samuel Just <sjust@redhat.com>

mds: organize CDentry class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: organize Capability class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: organize Beacon class members

This amends the code to follow our C++ style guidelines with the goal of
increasing header readability.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11993 from ceph/wip-17809

buffer: add advance(unsigned) back

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12049 from liewegas/wip-kv-iterators

kv: remove snapshot iterator

Reviewed-by: Kari, Varada <varada.kari@sandisk.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11719 from zhangsw/fix-rgw-requestpay-bug

rgw: Anonymous users shouldn't be able to access requester pays buckets.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11989 from idealguo/httpargs-parse-cleanup

rgw: skip empty http args in method parse() to avoid extra effort

Merge pull request #12021 from rzarzynski/wip-rgw-comp-noimplacefact

rgw: compression uses optional::emplace instead of in-place factories

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12108 from adamemerson/wip-compile-compress

compress: Fix compilation failure from missing header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11827 from oritwas/wip-rgw-upgrade-doc

rgw: add recovery procedure for upgrade to older version of jewel

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11940 from trociny/wip-17880

librbd: release lock after demote

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11960 from vshankar/wip-librbd-ec-support

librbd: bug fixes for optional data pool support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

compress: Fix compilation failure from missing header

Compress was failing due to the random header not being included.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: compression uses boost::optional::emplace instead of in-place factories.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #11901 from ceph/wip-rgw-compression-type-random

rgw: add rgw_compression_type=random for teuthology testing

Merge pull request #12073 from rzarzynski/wip-rgw-compile-asio-by-default

rgw: compilation of the ASIO front-end is enabled by default.

Merge pull request #12103 from rzarzynski/wip-rgw-auth-static_vector

rgw: avoid dynamic memory allocations in rgw_swift_auth.cc by using static_vector

verified: f23

test: Re-enable osd-scrub-repair.sh again

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix broken ec_size_mismatch setting

Caused by: 1e95f2ce642485e25b74f7d873fda7e4b75a425f

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12101 from tchaikov/wip-mark-down-test-cleanup

test/osd/osd-fast-mark-down.sh: remove unnecessary teardown() calls

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>

Merge pull request #12100 from tchaikov/wip-rocksdb

rocksdb: update to latest

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12093 from tchaikov/wip-yasm-wrapper

yasm-wrapper: translate "-isystem $1" to "-i $1"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12005 from wjwithagen/wip-wjw-freebsd-ceph-helpers

workunits/ceph-helpers.sh: FreeBSD returns a different errorstring.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "rgw: temporarily use std::vector in place of static_vector of Boost."

This reverts commit bc23e0f7fa8491c44fa938eeb954197f6aad2367.
We're doing that because the reverted commit was a makeshift
solution to not fail Ceph compilation on platforms lacking
Boost modern enough to ship the container/static_vector.hpp.
As we got the in-tree Boost the commit isn't necessary anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

test/osd/osd-fast-mark-down.sh: remove unnecessary teardown() calls

main() does this for us.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12035 from VictorDenisov/fix_snap_remove

rbd: use snap_remove implementation from internal

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12019 from branch-predictor/bp-fix-fastkill

test/osd/osd-fast-mark-down.sh: introduce large timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

rocksdb: update to latest

Latest master includes

- CMakeLists.txt: FreeBSD has jemalloc as default malloc
- cmake: drop "-march=native" from CXX_FLAGS (#1429)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12095 from dachary/wip-17830-eio

tests: fix ceph-helpers.sh wait_for_clean delays

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11813 from ukernel/wip-17801

mds: ignore 'session evict' when mds is replaying log

Reviewed-by: John Spray <john.spray@redhat.com>

mds: fix failed assertion in MDCache::maybe_resolve_finish

It's possible mds's state is RESOLVE and mds's want_state is
RECONNCT or STATE_REJOIN. MDCache::resolve_done is null in that
case.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix state check in Migrator::find_stale_export_freeze()

Iterator p is pointing to next state

Possibly fix: http://tracker.ceph.com/issues/{17606,8405}
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #12065 from xiexingguo/xxg-wip-kill-gc-counters

os/bluestore: kill dead gc-related counters
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

tests: fix ceph-helpers.sh wait_for_clean delays

The TENTH_TIMEOUT was not delcared as an int and failed to be set with
the correct number. The test of the function did not catch this.

Implement computing of the increasingly large sleep delays in a separate
function so that it can be tested more easily. Give up on sub-second
sleep because a the function will not sleep at all if the cluster is
already clean. And if it is not already clean, it is very unlikely to
become clean within less than a second. The downside of having very
short sleep time is that it needlessly stress the machine and also
possibly spam the logs.

Refs: http://tracker.ceph.com/issues/17830

Signed-off-by: Loic Dachary <loic@dachary.org>

msg/async/rdma: avoid use global NetHandler which refer to cephcontext

Signed-off-by: Haomai Wang <haomai@xsky.com>

yasm-wrapper: translate "-isystem $1" to "-i $1"

this silences the warning of:

yasm: warning: can open only one input file, only the last file will be
processed

as yasm does not understand -isystem, but gcc does.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa / workunits: make rbd import/export work with image data pool

Signed-off-by: Venky Shankar <vshankar@redhat.com>

tests / rbd: fix rbd python binding test

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test / rbd-mirror: image data pool test support

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test / librbd: image data pool test support

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test / librbd: use data pool IoCtx while listing objects

Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd / ObjectMap: use data_ctx/md_ctx pool context as required

Signed-off-by: Venky Shankar <vshankar@redhat.com>

librbd / CopyupRequest: use data pool ctx during copyup

Signed-off-by: Venky Shankar <vshankar@redhat.com>

librados: drop io_ctx_impl on ioctx_create/create2

close() was never called for the passed in IoCtx which
could probably result in an IoCtx leak if the original
IoCtx was a valid pool context allocated earlier.

Its kind of better to do it here rather than to leave
the destruction on the caller for better (or cleaner)
common case handling.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #12080 from wjwithagen/wip-wjw-freebsd-install-deps

install-deps.sh: JQ is needed in one script

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12076 from cbodley/wip-rgw-region-root

rgw: region conversion respects pre-existing rgw_region_root_pool
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

install-deps.sh: JQ is needed in one script

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: cleanup: fix a typo in CreateRequest.

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

os/bluestore: refactor ctor of SharedBlob

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: use projected path construction for access

A new CDentry will not have a parent until its projected parent is
flushed to journal. During path construction a given dentry may have no
parent yet which will cause fallbacks to be used (the inode number).
This can cause access checks to fail:

    2016-11-17 19:50:43.830207 7eff9977a700 20 Session check_access path #10000000002/3

compare to an earlier check:

    2016-11-17 19:50:43.824223 7eff9977a700 20 Session check_access path /test/1/2

This commit refactors path construction to optionally use projected
parents for the entire chain of directories. Existing use of real stable
parents is unchanged. For example, this line is the same before and
after the patch:

    2016-11-18 23:17:15.611680 7f153f97a700 12 mds.0.cache.dir(10000000002) add_null_dentry [dentry #10000000002/3 [2,head] auth NULL (dversion lock) pv=0 v=1 inode=0 0x55e0f771f5f0]

Here inode "#10000000002" has no stable parent yet. So the path is
constructed as "#10000000002/3".

One notable change in this commit is the removal of
make_path_string_projected which was only used in debugging code. Here's
an example difference:

    2016-11-17 19:50:43.827915 7eff9977a700 10 mds.0.server traverse_to_auth_dir [dir 10000000003 {#10000000003 #10000000002/3}/ [2,head] auth v=1 cv=0/0 state=1073741826|complete f() n() hs=0+0,ss=0+0 0x55f5d35e2ee0]

to:

    2016-11-18 23:17:15.617757 7f153f97a700 10 mds.0.server traverse_to_auth_dir [dir 10000000003 /test/1/2/3/ [2,head] auth v=1 cv=0/0 state=1073741826|complete f() n() hs=0+0,ss=0+0 0x55e0f7706ee0]

Fixes: http://tracker.ceph.com/issues/17858
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: look for region_map in rgw_region_root_pool

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: region conversion respects pre-existing rgw_region_root_pool

Fixes: http://tracker.ceph.com/issues/17963
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: compilation of the ASIO front-end is enabled by default.

We're changing the default value because the previous one was
a makeshift solution to not fail Ceph compilation due to
the Beast's dependency on Boost >= 1.54 that wasn't available
on CentoOS 7. As we got the in-tree Boost we can compile
the ASIO front-end by default.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #12057 from liewegas/wip-debug-hitset

ceph_test_rados_api_tier: dump hitset that we fail to decode

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com.>

Merge pull request #12040 from liupan1111/wip-bluestore-lat-report

os/bluestore: fix latency calculation

Reviewed-by: Sage Weil <sage@redhat.com>

OSD: remove 'has_inst', which has the same function as 'is_up'

Signed-off-by: Pan Liu <pan.liu@istuary.com>

OSD: heartbeat peers need to be updated when a new OSD added into an already existed cluster

Signed-off-by: Pan Liu <pan.liu@istuary.com>

kv/RocksDBStore: Don't update rocksdb perf_context if rocksdb_perf disable.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

msg/async/rdma: ensure CephContext existed

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11978 from tchaikov/wip-cmake-jemalloc

cmake: should link against ${ALLOC_LIBS}

Reviewed-by: Varada Raja Kumar Kari <kariraja@yahoo.com>

Merge pull request #12048 from dzafman/wip-dz-memstore

tests: Minor clean-ups

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12066 from dachary/wip-16014-cot

tests: save 9 characters for asok paths

Reviewed-by: Kefu Chai <kchai@redhat.com>