test: Turn off test-erasure-eio.sh due to Jenkins failures

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Turn off osd-scrub-repair.sh due to Jenkins failures

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #10672 from ceph/wip-rgw-http-getcmd

RGWHTTPArgs::get_str() - return argument string that was set.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10546 from cbodley/wip-rgw-log-trim-lease

rgw multisite: use a rados lock to coordinate data log trimming

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12114 from liewegas/wip-bluestore-failops

os/bluestore: mark ops that can't tolerate ENOENT

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12022 from jcsp/wip-17853

mds: more deterministic timing on frag split/join

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12094 from 363921219/cleanup_C_CompleteSplits

osd: cleanup C_CompleteSplits::finish()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12148 from tchaikov/wip-scan-build-fixes

scan build fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12099 from wjin/fix_filestore_perfcounter

os/filestore: fix journal logger

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12201 from tchaikov/wip-memstore-leak

os/memstore: fix a mem leak in MemStore::Collection::create_object()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

os/memstore: fix a mem leak in MemStore::Collection::create_object()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12168 from liewegas/doc-perf

doc/dev/perf: a few notes on perf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12176 from ovh/wip-doc-osd-scrub-during-recovery-chunk-min-max-sleep

doc: Add doc about osd scrub {during recovery|chunk {min|max}| sleep}

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10297 from Yan-waller/yj-wip-msgsimple-0714

msg/simple: save the errno in case being changed by subsequent codes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11154 from stiopaa1/osd_pg_useMoveOnSharedPtr

osd/PG.h: move shared ptr instead of copying it

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12186 from chrone81/patch-1

doc: Remove duplicate command for Ubuntu

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12106 from ceph/wip-jlayton-libcephfs

API cleanup for libcephfs interfaces

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: Remove duplicate command for Ubuntu

Remove duplicate "sudo service radosgw restart id=rgw.<short-hostname>" command for Ubuntu in Configure Bucket Sharding.

Signed-off-by: chrone <charlesalva@gmail.com>

Merge pull request #11982 from zhouruisong/master

rgw:bugfix for deleting objects name beginning and ending with underscores of one bucket using POST method of AWS's js sdk.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9859 from x11562/OSDMap

mon: osdmap's epoch should be more than 0

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #9172 from majianpeng/librados-add-fullforce

librados:  For C-API, expose LIBRADOS_OPERATION_FULL_FORCE flag

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #10553 from songbaisen/c3

osd:  remove the redundant clear method in consume_map function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12117 from 363921219/cleanup_PeeringWQ_dequeue

osd: clean up PeeringWQ::_dequeue(), remove unnecessary variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11841 from hjwsm1989/filestore

os/filestore: avoid to get the wrong hardlink number.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11913 from wjwithagen/wip-wjw-freebsd-aio-inline

os/fs/FS.cc: condition on WITH_AIO for FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9062 from majianpeng/aio-pwritev

os/fs/FS: optimize aio::pwritev which make caller provide length.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12183 from dachary/wip-17830-eio

tests: facilitate background process debug in ceph-helpers.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12184 from SUSE/wip-18040

doc: we can now run multiple MDS, so qualify warning

Reviewed-by: John Spray <john.spray@redhat.com>

doc: we can now run multiple MDS, so qualify warning

Fixes: http://tracker.ceph.com/issues/18040
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: facilitate background process debug in ceph-helpers.sh

When displaying the output of a background process, do it on stderr so
that it is not bufferized. Otherwise the output of the background
process may be displayed after it completed.

Prefix the output of a background process with the PID of the process
known to the parent instead of the PID of the awk process processing the
output. When wait_background loops, it will print the process on which
it is waiting and it is confusing that they do not match with the PID
prefixing the process output.

Refs: http://tracker.ceph.com/issues/17830

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11733 from tchaikov/wip-17762

common: release g_ceph_context before returns

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12172 from dachary/wip-17830-eio

tests: osd-scrub-repair.sh abort if add_something fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

crushtool: do not release g_ceph_context at exit

it is but a work around of occasionally timeout.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,test: g_ceph_context->put() upon return

prior to this change, global_init() could create a new CephContext
and assign it to g_ceph_context. it's our responsibilty to release
the CephContext explicitly using cct->put() before the application
quits. but sometimes, we fail to do so.

in this change, global_init() will return an intrusive_ptr<CephContext>,
which calls `g_ceph_context->put()` in its dtor. this ensures that
the CephContext is always destroyed before main() returns. so the
log is flushed before _log_exp_length is destroyed.

there are two cases where global_pre_init() is called directly.
- ceph_conf.cc: g_ceph_context->put() will be called by an intrusive_ptr<>
  deleter.
- rgw_main.cc: global_init() is called later on on the success code
  path, so it will be taken care of.

Fixes: http://tracker.ceph.com/issues/17762
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Add doc about osd scrub {during recovery|chunk {min|max}| sleep}

Signed-off-by: Paweł Sadowski <pawel.sadowski@corp.ovh.com>

Merge pull request #10296 from songbaisen/b4

mon: small change on the HealthMonitor start_epoch  function

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: explain directory fragmenation settings

...and update the config ref.

Includes the new mds_bal_fragment_fast_factor setting.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: try fragmenting new fragments during split

In case insertions have occurred during the split that would
immediately take the new fragments over the split threshold.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: maybe_fragment earlier in openc

Check it during the initial request, not just
on completion, so that when doing lots of creates
we get a chance to split the directory before
it zooms past the size threshold.

Signed-off-by: John Spray <john.spray@redhat.com>

mds/MDBalancer: separate out fragmentation from hit_dir

In _fragment_finish we would like to try splitting
the new frags without applying a spurious hit to their
temperatures.  Same for the start of openc where we would
like to do an early check for splitting without hitting
the dir twice.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: more deterministic timing on frag split/join

...by using timer instead of tick()

Fixes: http://tracker.ceph.com/issues/17853
Signed-off-by: John Spray <john.spray@redhat.com>

mds/MDBalancer: remove stale show_imports fn

This was just dropping its argument and
calling through to show_subtrees.

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove unused mds_bal_merge_[rd|wr]

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix fragment thrasher

The allows_dirfrags() test was in the wrong place, causing
in some cases a root fragment to be passed into queue_merge.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: tweak merge/split counters

Increment these when we have finished splitting
or merging, not partway through the process.

This makes testing more deterministic: once I've seen
the counter increment, I'm sure that the children
no longer have STATE_FRAGMENTING set as a result of
their parent's split-in-progress.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: include fragmenting in state print

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add a performance counter for dirfrag merge

Signed-off-by: John Spray <john.spray@redhat.com>

mon: small change on the HealthMonitor start_epoch function

No need to call start_epoch function each time in the loop.

Signed-off-by: songbaisen <song.baisen@zte.com.cn>

Merge pull request #9513 from songbaisen/a5

mon: Forbidden copy and assignment function in monoprequest

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12085 from wjwithagen/wip-freebsd-ceph-helpers-2

workunits/ceph-helpers.sh: Fixes for FreeBSD

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: osd-scrub-repair.sh abort if add_something fails

If the script keeps going after this function fails, it will be non
trivial to find the root cause of the problem.

Signed-off-by: Loic Dachary <loic@dachary.org>

osd:  remove the redundant clear method in consume_map function

In the above loop before this to_remove to clear,we have remove all

the element.So this clear method is redundant.

Signed-off-by: song baisen <song.baisen@zte.com.cn>

Merge pull request #11943 from wido/docs-pool-remove

doc: Add documentation about mon_allow_pool_delete before pool remove

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw:fix for deleting objects name beginning and ending with underscores of one bucket using POST method of AWS's js sdk.
Fixes: http://tracker.ceph.com/issues/17888
Signed-off-by: zhouruisong 236131368@qq.com

workunits/ceph-helpers.sh: Fixes for FreeBSD

 - stat(1) does not have '%T'

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/dev/perf: a few notes on perf and flame graphs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10402 from ceph/wip-openstack-wusui

qa: Add ceph-ansible installer.

Merge pull request #11673 from zhangsw/fix-rgw-initbucketshardsync-bug

rgw: Class member cookie is not initialized correctly in some coroutine's constructor.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11969 from yehudasa/wip-17897

rgw: don't store empty chains in gc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11815 from ceph/wip-17792

rgw: remove circular reference in RGWAsyncRadosRequest
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #12142 from athanatos/wip-17966

osd: Fix read error propogation in ECBackend

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #11994 from ukernel/wip-17193

mds: force client flush snap data before truncating objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11635 from batrick/mds-unique-ptr-20161024

mds: more unique_pointer changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11917 from ajarr/wip-17800

ceph_volume_client: set an existing auth ID's default mon caps

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11983 from liewegas/wip-bluestore-release

os/bluestore: fix alloc release timing on sync submits

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12129 from wjin/fix_backoffthrottle_comment

common/Throttle: fix typo for BackoffThrottle

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9643 from wjin/mon

mon: clean legacy code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

src/test: reenable test-erasure-eio.sh

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12000 from jcsp/wip-doc-exports

src/doc: fix class names in exports.txt

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10474 from songbaisen/b5

mon:  remove the redudant jugement in LogMonitor tick function

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11631 from wolsen/issues/17689

upstart: start ceph-all after static-network-up

Reviewed-by: James Page <james.page@ubuntu.com>

Merge pull request #12128 from tchaikov/wip-cmake

cmake: add -Wno-unknown-pragmas to CMAKE_CXX_FLAGS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12126 from branch-predictor/bp-doc-async-opts

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12135 from kylinstorage/wip-remove-unneeded-forward-declaration

osd/PG.h: remove unneeded forward declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12162 from liewegas/wip-coverity

script/run-coverity: update

script/run-coverity: -x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11848 from javeme/case-CEPH_MSG_POOLOP

mon: move case CEPH_MSG_POOLOP to OSDs group

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #12160 from vshankar/wip-17973

librbd: mark request as finished after failed refresh

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Documentation of a few of Async Messenger configuration options.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

doc: Add documentation about mon_allow_pool_delete before pool remove

Tell users they need to set this to true before Monitors will allow
pools to be removed.

Also update the Pending Release Notes so that users can find this change
there.

This was changed with commit 5d7f4ea

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #12055 from liewegas/wip-17942

osdc/Objecter: handle race between calc_target and handle_osd_map

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11178 from stiopaa1/osd_pg_changeStructPGRecoveryStatsToClass

osd/PG.h: change PGRecoveryStats struct to class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12009 from 363921219/master

osd: clean up process_peering_events

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12031 from liewegas/wip-ceph-abort

common: assert(0) -> ceph_abort()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: account m_processing when failing request after refresh

AioImageRequestWQ::_void_process() dequeues an item from the
work queue by invoking ThreadPool::_void_process() which also
performs accounting by incrementing ->m_processing which never
gets decremented if an image refresh fails. This is exactly
the flow when running test_lock_fence.sh which blacklists a
client and expects failed I/O operation from the client.

Without this fix, running test_lock_fence.sh in a loop fails
with an assert in PointerWQ dtor within 100 odd runs.

Fixes: http://tracker.ceph.com/issues/17973
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #12034 from wjwithagen/wip-wjw-freebsd-strtoll

common/strtol.cc: Get error testing also to work on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12150 from tchaikov/wip-kill-gf-complete-warnings

erasure-code: fix gf-complete warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

cmake: regroup CMAKE_CXX_FLAGS and CMAKE_C_FLAGS

- some options are only meaningful for C++, so move them into
  CMAKE_CXX_FLAGS.
- regroup CMAKE_CXX_FLAGS and CMAKE_C_FLAGS, and remove the duplicated
  options.
- do not reset CMAKE_CXX_FLAGS with ${CMAKE_C_FLAGS}, instead, always
  append ${CMAKE_C_FLAGS} to existing ${CMAKE_CXX_FLAGS}. so we don't
  clobber CMAKE_CXX_FLAGS set by command line.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add -Wno-unknown-pragmas to CMAKE_C_FLAGS

this silences warnings like:

warning: ignoring #pragma clang diagnostic [-Wunknown-pragmas]

and recent clang (3.9 and 4.0) do not complain at seeing
so we only pass this this param to gcc.

Signed-off-by: Kefu Chai <kchai@redhat.com>

erasure-code: fix gf-complete warning

pick up latest gf-complete submodule

$ git log --no-merges --pretty=%s \
  a6847973cba329ae079d3bd26341a4ec2906f012..c431e1ff76bdce6c3f6c0d0cbdf6a01c68f81071
gf_w32.c: silence the -Wmaybe-uninitialized warning

it silences following warning:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c: In
function ‘gf_w32_cfmgk_multiply_region_from_single’:

ceph/src/erasure-code/jerasure/gf-complete/src/gf_w32.c:410:5:
warning: ‘a’ may be used uninitialized in this function
[-Wmaybe-uninitialized]
   g = _mm_insert_epi64 (a, g_star, 0);
     ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

FileStore::_do_fiemap: do not reference fiemap after it is freed

`last` points to fiemap::fm_extends[n], and if fiemap gets freed, we can
not reference any of its fieldis. so we could remember the check result before
freeing it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

hobject: better left-shift an unsigned

this silences the warning of

The result of the '<<' expression is undefined

quote from n3337, §5.8 expr.shift:

If E1 has an unsigned type, the value ... (is deterministic)...
Otherwise, if E1 has a signed type and non-negative value, and E1 x 2^E2
is representable in the result type, then that is the resulting value;
otherwise, the behavior is undefined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd_types: pg_t::is_split(): use unsigned for bits operations

this silences the warning of

The result of the '<<' expression is undefined

quote from n3337, §5.8 expr.shift:

If E1 has an unsigned type, the value ... (is deterministic)...
Otherwise, if E1 has a signed type and non-negative value, and E1 x 2^E2
is representable in the result type, then that is the resulting value;
otherwise, the behavior is undefined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ReplicatedPG: check retcode before verifying trailing hole

this silences the warning of

Value stored to 'r' is never used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/blkdev: get_device_by_uuid() silence clang warning

early return if blkid_get_cache() fails. this chance is but cosmetic,
it pleases clang static analyzer which thinks that rc will be
overwritten if dev is nullptr.

Signed-off-by: Kefu Chai <kchai@redhat.com>

global/signal_handler: retry if read() returns with EINTR

this silences the warning of

Value stored to 'r' is never used.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/ceph_radosacl: print out return code of "acl set"

this silences the warning of

Value stored to 'r' is never read.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12146 from dzafman/wip-dz-delays

tests: minor make check cleanup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12136 from dachary/wip-17889-systemd-order

ceph-disk: resolve race conditions

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12145 from tchaikov/wip-client-use-after-freed

client: trim_caps() do not dereference cap if it's removed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>