msg/simple/Pipe: handle addr decode error

Fixes: http://tracker.ceph.com/issues/18072
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11764 from bassamtabbara/wip-embedded

cmake: Support for embedding Ceph Daemons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12097 from ceph/wip-17837

src/mds: fix MDSMap upgrade decoding

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12090 from wjwithagen/wip-wjw-do_freebsd.sh

FreeBSD: do_freebsd.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12200 from ddiss/tracker18049_ceph_disk_trigger_flock_timeout

systemd/ceph-disk: reduce ceph-disk flock contention

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12180 from tchaikov/wip-silence-get_timeout_delays

tests: disable the echo when running get_timeout_delays()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

embedded: Add RADOS classes to embedded cephd library

RADOS classes can now be statically compiled and added to
the embedded cephd library.

The RADOS ClassHandler now has an option to skip calling dlclose
just like PluginRegistry.

All RADOS classes where changed to use a CLS_INIT macro that
will either use __cls_init or classname_cls_init. this enables
the static compiling of all RADOS classes in a single library. Also
global method definitions where moved to inside cls_init.

Also added a few aconfig defines including WITH_EMBEDDED, WITH_CEPHFS,
WITH_RBD, and WITH_KVS. Note that WITH_RBD was not defined before
and the ceph-dencoder was broken when it was turned on.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: Add librbd and librados to embedded ceph

This commit add librados, librbd to the embedded ceph static library. This is needed to support daemons built with libcephd that want to run mon commands, create rbd volumes, snapshots etc.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: Support for running MON and OSD from libcephd

Added supported for running MON and OSD daemon code almost
untouched directly from libcephd.

Also added two API functions for generating an FSID and secret
key that can be used to bootstrap a new cluster.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: add compression and EC plugins to libcephd

Compression and erasure coding plugins are now statically compiled
into libcephd. A new method is added to load them into the
respective registry.

The static libraries are only built when WITH_EMBEDDED is enabled
and existing plugins are unaffected.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

embedded: Add a skeleton libcephd library

libcephd is a library that contains ceph daemon code
that can be statically linked in other applications.

Added MergeStaticLibraries.cmake that can merge static libraries
to form a bigger one. This approach avoids the need to mess with
STATIC libraries all over the code base.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #12170 from dzafman/wip-ec-error-tests

test: test-erasure-eio.sh fix recovery testing and enable it

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10672 from ceph/wip-rgw-http-getcmd

RGWHTTPArgs::get_str() - return argument string that was set.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10546 from cbodley/wip-rgw-log-trim-lease

rgw multisite: use a rados lock to coordinate data log trimming

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #12114 from liewegas/wip-bluestore-failops

os/bluestore: mark ops that can't tolerate ENOENT

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12022 from jcsp/wip-17853

mds: more deterministic timing on frag split/join

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12094 from 363921219/cleanup_C_CompleteSplits

osd: cleanup C_CompleteSplits::finish()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12148 from tchaikov/wip-scan-build-fixes

scan build fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12099 from wjin/fix_filestore_perfcounter

os/filestore: fix journal logger

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12201 from tchaikov/wip-memstore-leak

os/memstore: fix a mem leak in MemStore::Collection::create_object()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

systemd/ceph-disk: reduce ceph-disk flock contention

"ceph-disk trigger" invocation is currently performed in a mutually
exclusive fashion, with each call first taking an flock on the path
/var/lock/ceph-disk. On systems with a lot of osds, this leads to a
large amount of lock contention during boot-up, and can cause some
service instances to trip the 120 second timeout.

Take an flock on a device specific path instead of /var/lock/ceph-disk,
so that concurrent "ceph-disk trigger" invocations are permitted for
independent osds. This greatly reduces lock contention and consequently
the chance of service timeout. Per-device concurrency restrictions
required for http://tracker.ceph.com/issues/13160 are maintained.

Fixes: http://tracker.ceph.com/issues/18049
Signed-off-by: David Disseldorp <ddiss@suse.de>

os/memstore: fix a mem leak in MemStore::Collection::create_object()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12168 from liewegas/doc-perf

doc/dev/perf: a few notes on perf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12176 from ovh/wip-doc-osd-scrub-during-recovery-chunk-min-max-sleep

doc: Add doc about osd scrub {during recovery|chunk {min|max}| sleep}

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10297 from Yan-waller/yj-wip-msgsimple-0714

msg/simple: save the errno in case being changed by subsequent codes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11154 from stiopaa1/osd_pg_useMoveOnSharedPtr

osd/PG.h: move shared ptr instead of copying it

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12186 from chrone81/patch-1

doc: Remove duplicate command for Ubuntu

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12106 from ceph/wip-jlayton-libcephfs

API cleanup for libcephfs interfaces

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc: Remove duplicate command for Ubuntu

Remove duplicate "sudo service radosgw restart id=rgw.<short-hostname>" command for Ubuntu in Configure Bucket Sharding.

Signed-off-by: chrone <charlesalva@gmail.com>

Merge pull request #11982 from zhouruisong/master

rgw:bugfix for deleting objects name beginning and ending with underscores of one bucket using POST method of AWS's js sdk.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9859 from x11562/OSDMap

mon: osdmap's epoch should be more than 0

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #9172 from majianpeng/librados-add-fullforce

librados:  For C-API, expose LIBRADOS_OPERATION_FULL_FORCE flag

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #10553 from songbaisen/c3

osd:  remove the redundant clear method in consume_map function

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12117 from 363921219/cleanup_PeeringWQ_dequeue

osd: clean up PeeringWQ::_dequeue(), remove unnecessary variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11841 from hjwsm1989/filestore

os/filestore: avoid to get the wrong hardlink number.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11913 from wjwithagen/wip-wjw-freebsd-aio-inline

os/fs/FS.cc: condition on WITH_AIO for FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9062 from majianpeng/aio-pwritev

os/fs/FS: optimize aio::pwritev which make caller provide length.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12183 from dachary/wip-17830-eio

tests: facilitate background process debug in ceph-helpers.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12184 from SUSE/wip-18040

doc: we can now run multiple MDS, so qualify warning

Reviewed-by: John Spray <john.spray@redhat.com>

doc: we can now run multiple MDS, so qualify warning

Fixes: http://tracker.ceph.com/issues/18040
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: facilitate background process debug in ceph-helpers.sh

When displaying the output of a background process, do it on stderr so
that it is not bufferized. Otherwise the output of the background
process may be displayed after it completed.

Prefix the output of a background process with the PID of the process
known to the parent instead of the PID of the awk process processing the
output. When wait_background loops, it will print the process on which
it is waiting and it is confusing that they do not match with the PID
prefixing the process output.

Refs: http://tracker.ceph.com/issues/17830

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11733 from tchaikov/wip-17762

common: release g_ceph_context before returns

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #12172 from dachary/wip-17830-eio

tests: osd-scrub-repair.sh abort if add_something fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: disable the echo when running get_timeout_delays()

this function is very distracting when one is looking at the log

Signed-off-by: Kefu Chai <kchai@redhat.com>

crushtool: do not release g_ceph_context at exit

it is but a work around of occasionally timeout.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,test: g_ceph_context->put() upon return

prior to this change, global_init() could create a new CephContext
and assign it to g_ceph_context. it's our responsibilty to release
the CephContext explicitly using cct->put() before the application
quits. but sometimes, we fail to do so.

in this change, global_init() will return an intrusive_ptr<CephContext>,
which calls `g_ceph_context->put()` in its dtor. this ensures that
the CephContext is always destroyed before main() returns. so the
log is flushed before _log_exp_length is destroyed.

there are two cases where global_pre_init() is called directly.
- ceph_conf.cc: g_ceph_context->put() will be called by an intrusive_ptr<>
  deleter.
- rgw_main.cc: global_init() is called later on on the success code
  path, so it will be taken care of.

Fixes: http://tracker.ceph.com/issues/17762
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Add doc about osd scrub {during recovery|chunk {min|max}| sleep}

Signed-off-by: Paweł Sadowski <pawel.sadowski@corp.ovh.com>

Merge pull request #10296 from songbaisen/b4

mon: small change on the HealthMonitor start_epoch  function

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: explain directory fragmenation settings

...and update the config ref.

Includes the new mds_bal_fragment_fast_factor setting.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: try fragmenting new fragments during split

In case insertions have occurred during the split that would
immediately take the new fragments over the split threshold.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: maybe_fragment earlier in openc

Check it during the initial request, not just
on completion, so that when doing lots of creates
we get a chance to split the directory before
it zooms past the size threshold.

Signed-off-by: John Spray <john.spray@redhat.com>

mds/MDBalancer: separate out fragmentation from hit_dir

In _fragment_finish we would like to try splitting
the new frags without applying a spurious hit to their
temperatures.  Same for the start of openc where we would
like to do an early check for splitting without hitting
the dir twice.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: more deterministic timing on frag split/join

...by using timer instead of tick()

Fixes: http://tracker.ceph.com/issues/17853
Signed-off-by: John Spray <john.spray@redhat.com>

mds/MDBalancer: remove stale show_imports fn

This was just dropping its argument and
calling through to show_subtrees.

Signed-off-by: John Spray <john.spray@redhat.com>

common: remove unused mds_bal_merge_[rd|wr]

Signed-off-by: John Spray <john.spray@redhat.com>

mds: fix fragment thrasher

The allows_dirfrags() test was in the wrong place, causing
in some cases a root fragment to be passed into queue_merge.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: tweak merge/split counters

Increment these when we have finished splitting
or merging, not partway through the process.

This makes testing more deterministic: once I've seen
the counter increment, I'm sure that the children
no longer have STATE_FRAGMENTING set as a result of
their parent's split-in-progress.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: include fragmenting in state print

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add a performance counter for dirfrag merge

Signed-off-by: John Spray <john.spray@redhat.com>

mon: small change on the HealthMonitor start_epoch function

No need to call start_epoch function each time in the loop.

Signed-off-by: songbaisen <song.baisen@zte.com.cn>

Merge pull request #9513 from songbaisen/a5

mon: Forbidden copy and assignment function in monoprequest

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12085 from wjwithagen/wip-freebsd-ceph-helpers-2

workunits/ceph-helpers.sh: Fixes for FreeBSD

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: osd-scrub-repair.sh abort if add_something fails

If the script keeps going after this function fails, it will be non
trivial to find the root cause of the problem.

Signed-off-by: Loic Dachary <loic@dachary.org>

osd:  remove the redundant clear method in consume_map function

In the above loop before this to_remove to clear,we have remove all

the element.So this clear method is redundant.

Signed-off-by: song baisen <song.baisen@zte.com.cn>

Merge pull request #11943 from wido/docs-pool-remove

doc: Add documentation about mon_allow_pool_delete before pool remove

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw:fix for deleting objects name beginning and ending with underscores of one bucket using POST method of AWS's js sdk.
Fixes: http://tracker.ceph.com/issues/17888
Signed-off-by: zhouruisong 236131368@qq.com

test: test-erasure-eio.sh fix recovery testing and enable it

Signed-off-by: David Zafman <dzafman@redhat.com>

workunits/ceph-helpers.sh: Fixes for FreeBSD

 - stat(1) does not have '%T'

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc/dev/perf: a few notes on perf and flame graphs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10402 from ceph/wip-openstack-wusui

qa: Add ceph-ansible installer.

Merge pull request #11673 from zhangsw/fix-rgw-initbucketshardsync-bug

rgw: Class member cookie is not initialized correctly in some coroutine's constructor.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

test: CLEANUP make test clearer by using "fail" as argument

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11969 from yehudasa/wip-17897

rgw: don't store empty chains in gc
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11815 from ceph/wip-17792

rgw: remove circular reference in RGWAsyncRadosRequest
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #12142 from athanatos/wip-17966

osd: Fix read error propogation in ECBackend

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #11994 from ukernel/wip-17193

mds: force client flush snap data before truncating objects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11635 from batrick/mds-unique-ptr-20161024

mds: more unique_pointer changes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11917 from ajarr/wip-17800

ceph_volume_client: set an existing auth ID's default mon caps

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11983 from liewegas/wip-bluestore-release

os/bluestore: fix alloc release timing on sync submits

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12129 from wjin/fix_backoffthrottle_comment

common/Throttle: fix typo for BackoffThrottle

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9643 from wjin/mon

mon: clean legacy code

Reviewed-by: Loic Dachary <ldachary@redhat.com>

src/test: reenable test-erasure-eio.sh

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #12000 from jcsp/wip-doc-exports

src/doc: fix class names in exports.txt

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10474 from songbaisen/b5

mon:  remove the redudant jugement in LogMonitor tick function

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #11631 from wolsen/issues/17689

upstart: start ceph-all after static-network-up

Reviewed-by: James Page <james.page@ubuntu.com>

Merge pull request #12128 from tchaikov/wip-cmake

cmake: add -Wno-unknown-pragmas to CMAKE_CXX_FLAGS

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12126 from branch-predictor/bp-doc-async-opts

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12135 from kylinstorage/wip-remove-unneeded-forward-declaration

osd/PG.h: remove unneeded forward declaration

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12162 from liewegas/wip-coverity

script/run-coverity: update

script/run-coverity: -x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11848 from javeme/case-CEPH_MSG_POOLOP

mon: move case CEPH_MSG_POOLOP to OSDs group

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #12160 from vshankar/wip-17973

librbd: mark request as finished after failed refresh

doc/rados/configuration/ms-ref.rst: document a few async msgr options

Documentation of a few of Async Messenger configuration options.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

doc: Add documentation about mon_allow_pool_delete before pool remove

Tell users they need to set this to true before Monitors will allow
pools to be removed.

Also update the Pending Release Notes so that users can find this change
there.

This was changed with commit 5d7f4ea

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #12055 from liewegas/wip-17942

osdc/Objecter: handle race between calc_target and handle_osd_map

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11178 from stiopaa1/osd_pg_changeStructPGRecoveryStatsToClass

osd/PG.h: change PGRecoveryStats struct to class

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12009 from 363921219/master

osd: clean up process_peering_events

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12031 from liewegas/wip-ceph-abort

common: assert(0) -> ceph_abort()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: account m_processing when failing request after refresh

AioImageRequestWQ::_void_process() dequeues an item from the
work queue by invoking ThreadPool::_void_process() which also
performs accounting by incrementing ->m_processing which never
gets decremented if an image refresh fails. This is exactly
the flow when running test_lock_fence.sh which blacklists a
client and expects failed I/O operation from the client.

Without this fix, running test_lock_fence.sh in a loop fails
with an assert in PointerWQ dtor within 100 odd runs.

Fixes: http://tracker.ceph.com/issues/17973
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #12034 from wjwithagen/wip-wjw-freebsd-strtoll

common/strtol.cc: Get error testing also to work on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>