os/bluestore: fix bufferspace stats leak due to blob splitting

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11926 from dzafman/wip-test-17830

test: fix test-erasure-eio and osd-scrub-repair races (17830)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12030 from branch-predictor/bp-rmxattr-bl

librados: remove unused bufferlist from rados_write_op_rmxattr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

test: Add sleep 1 so that we look for 60 seconds

This failed because test blew through 60 requests in 8 seceons before
the scrub repair even started on Jenkins.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix duplicate port in test-erasure-code-plugins.sh

Add comments about uniqueness of port number required

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12032 from liewegas/wip-bluestore-warning

os/bluestore: fix warning and uninit variable

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon: Fix compiler warnings, missed _dump() change

Caused by: af720cc87b4d631695c1549c31d57bfbc82bc6ba

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Re-enable osd-scrub-repair.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test: In osd-scrub-repair.sh fix use of wait_for_clean()

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11996 from dachary/wip-17849-bootstrap-osd

mon,ceph-disk: add lockbox permissions to bootstrap-osd

Reviewed-by: Sage Weil <sage@redhat.com>

include/mempool: silence warnings

/home/sage/src/ceph4/src/include/mempool.h:421:68: warning: no return statement in function returning non-void [-Wreturn-type]
   void *operator new[](size_t size) { assert(0 == "no array new"); } \

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: initialize oldest

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12029 from batrick/mon-_dump-bugfix

mon: remove utime_t param in _dump

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librados: remove unused bufferlist from rados_write_op_rmxattr

rados_write_op_rmxattr declares the bl bufferlist variable, then doesn't make
any use of it. Get rid of it, as it's probably an effect of copy+paste.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #12028 from batrick/client-redundant-init

client: remove redundant initialization

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test: Re-enable test-erasure-eio.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test: activate_osd() doesn't need to set crush

Tests use objectstore_tool() which stops and starts OSDs,
but may assume consistency of object locations.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: wait_for_clean() add sleep backoff

Reduce size of log on timeout by doing a backoff so that
we don't log 3000 loops at 1/10 second sleeps.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: CLEANUP to later enable TEST_rados_get_with_subreadall_eio_shard_*

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: remove utime_t param in _dump

Missing in af720cc87b4d631695c1549c31d57bfbc82bc6ba.

Found in warning:

In file included from /home/pdonnell/ceph/src/mds/MDSRank.h:18:0,
                 from /home/pdonnell/ceph/src/mds/MDBalancer.cc:18:
/home/pdonnell/ceph/src/common/TrackedOp.h:153:16: warning: ‘virtual void TrackedOp::_dump(ceph::Formatter*) const’ was hidden [-Woverloaded-virtual]
   virtual void _dump(Formatter *f) const {}
                ^
In file included from /home/pdonnell/ceph/src/mon/mon_types.h:23:0,
                 from /home/pdonnell/ceph/src/mon/MonMap.h:22,
                 from /home/pdonnell/ceph/src/mon/MonClient.h:20,
                 from /home/pdonnell/ceph/src/mds/MDBalancer.cc:19:
/home/pdonnell/ceph/src/mon/MonOpRequest.h:106:8: warning:   by ‘void MonOpRequest::_dump(utime_t, ceph::Formatter*) const’ [-Woverloaded-virtual]
   void _dump(utime_t now, Formatter *f) const {
        ^

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon,ceph-disk: add lockbox permissions to bootstrap-osd

ceph-disk --dmcrypt needs to put a config-key and authorize
the OSD to get it back. The corresponding permissions are
added to the bootstrap-osd profile in the monitor.

When preparing the OSD lockbox, use the bootstrap-osd profile instead of
implicitly requiring admin permissions to perform the initial config-key
and auth get-or-create operations.

Fixes: http://tracker.ceph.com/issues/17849
Signed-off-by: Loic Dachary <loic@dachary.org>

client: remove redundant initialization

messenger is already initialized in the constructor member
initialization list.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12025 from dachary/wip-17923-tox

tests: no python3 tests for ceph-disk

Reviewed-by: Samuel Just <sjust@redhat.com>

tests: no python3 tests for ceph-disk

2.5.0 published today made it so an environment with no action fails
where it previously did not.

Fixes: http://tracker.ceph.com/issues/17923
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #12024 from tchaikov/wip-ceph-erasure-code-corpus-arch

tests: sync ceph-erasure-code-corpus for using 'arch' not 'uname -p'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: sync ceph-erasure-code-corpus for using 'arch' not 'uname -p'

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12011 from ceph/wip-git-archive-all

git-archive-all.sh: use an actually unique tmp dir

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12007 from jcsp/wip-trackedop-now

common/TrackedOp: remove unused 'now' in _dump()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12016 from dachary/wip-17919-osd-crush

tests: osd-crush.sh must retry crush dump

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12012 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: a script to prepare for debugging on teuthology@sepia

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11167 from jcsp/wip-17308

mon: make MDSMonitor tolerant of slow mon elections

Reviewed-by: John Spray <john.spray@redhat.com>

tests: osd-crush.sh must retry crush dump

After ceph osd setcrushmap, the script must retry because ceph osd crush
dump races against the update of the crushmap.

Fixes: http://tracker.ceph.com/issues/17919
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11979 from ceph/wip-disable-osd-scrub-repair

test: disable osd-scrub-repair and test-erasure-eio

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

script/sepia_bt.sh: a script to prepare for debugging on teuthology@sepia

Signed-off-by: Kefu Chai <kchai@redhat.com>

git-archive-all.sh: use an actually unique tmp dir

git archive into $TMPDIR/$(basename "$(pwd)").$FORMAT is not unique;
if two runs are running simultaneously, this will collide.  Make
TMPDIR actually unique, and then the cleanup can just remove the whole
directory as well.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #12003 from athanatos/wip-17899

OSDMonitor: only reject MOSDBoot based on up_from if inst matches

Reviewed-by: Sage Weil <sage@redhat.com>

mon: make MDSMonitor tolerant of slow mon elections

Previously MDS daemons would get failed incorrectly
when they appeared to have timed out due to
delays in calling into MDSMonitor that were
actually caused by e.g. slow leveldb writes leading
to slow mon elections.

Fixes: http://tracker.ceph.com/issues/17308
Signed-off-by: John Spray <john.spray@redhat.com>

mon: clean up MDSMonitor interface

This was almost all public.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11836 from batrick/client-new-20161108

 client: remove unnecessary bufferptr[] for writev

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11837 from batrick/client-unique-20161108

client: use unique_ptr

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11904 from stiopaa1/mds_scrubheader_passStringByConstRef

mds/ScrubHeader.h: pass string by const reference

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11651 from stiopaa1/mds_mdsrank_protectedDestructor

mds/MDSRank.h: make destructor protected

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11338 from stiopaa1/mds_sessionmap_unneededCount

mds/SessionMap.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11957 from batrick/mds-deactivate-doc

doc: clarify mds deactivate purpose

Reviewed-by: John Spray <john.spray@redhat.com>

common/TrackedOp: remove unused 'now' in _dump()

Greg pointed this out in PR #11985

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11985 from batrick/op-tracker-mds-event

mds: revert "mds/Mutation: remove redundant _dump method"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11958 from batrick/cls-lua-lua_next

lua: use simpler lua_next traversal structure

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11833 from stiopaa1/mds_migrator_removeUnneededUseOfCount

mds/Migrator.h: remove unneeded use of count

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10940 from stiopaa1/mds_beacon_moveCMDSBeaconSenderClass

mds/Beacon: move C_MDS_BeaconSender class to .cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11887 from jcsp/wip-efragment-cleanup

mds: remove unused EFragment::OP_ONESHOT

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12004 from bassamtabbara/wip-fix-civetweb-submodule

rgw: revert unintentional change to civetweb

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revert unintentional change to civetweb

https://github.com/ceph/ceph/pull/11800 uninternionally updated
the civetweb submodule.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #11908 from stiopaa1/mds_mdcache_removeUnusedFunctions

mds/MDCache.h: remove unused functions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11964 from xiexingguo/xxg-wip-bluestore-1112

os/bluestore: misc. fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11986 from xiexingguo/xxg-wip-bluestore-1115

os/bluestore: dedup if space overlap truly exists

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

test: disable osd-scrub-repair and test-erasure-eio

After Sage's suggestion of earlier today, in now-closed
PR #11970, which can't be resurrected just now

Kefu is trying to reproduce osd-scrub-repair failures

David is working test-erasure-eio in
https://github.com/ceph/ceph/pull/11926

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #11803 from XinzeChi/wip-openpgs

osd: do not open pgs when the pg is not in pg_map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11962 from tchaikov/wip-doc-rebuilddb

doc: fix the script for rebuild monitor db

Reviewed-by: Shylesh Kumar <shmohan@redhat.com>

Merge pull request #11984 from batrick/sphinx-py3-error

doc/man: avoid file builtin to solve build error

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man: avoid file builtin to solve build error

Sphinx running with Python 3.X fails:

    # Sphinx version: 1.4.8
    # Python version: 3.5.2 (CPython)
    # Docutils version: 0.12 release
    # Jinja2 version: 2.8
    # Last messages:

    # Loaded extensions:
    Traceback (most recent call last):
      File "/usr/lib/python3.5/site-packages/sphinx/cmdline.py", line 243, in main
        opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
      File "/usr/lib/python3.5/site-packages/sphinx/application.py", line 137, in __init__
        confoverrides or {}, self.tags)
      File "/usr/lib/python3.5/site-packages/sphinx/config.py", line 287, in __init__
        execfile_(filename, config)
      File "/usr/lib/python3.5/site-packages/sphinx/util/pycompat.py", line 130, in execfile_
        exec_(code, _globals)
      File "conf.py", line 56, in <module>
      File "conf.py", line 47, in _get_manpages
      File "conf.py", line 12, in _get_description
    NameError: name 'file' is not defined

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11808 from trociny/wip-17010

librbd: default features should be negotiated with the OSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11821 from trociny/wip-test_notify

test/librbd: in test_notify set object-map and fast-diff features by default

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11840 from yangdongsheng/rbd_mirror_options

rbd-mirror: configuration overrides for hard coded timers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: note 1st existed coll instead of 1st coll

We store the coll mainly for cache trim and we shall try
our best to do it whenever it is possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: move comments to proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop wal_op_onodes

Which is currently unused.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: dedup if space overlap truly exists

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: revert "mds/Mutation: remove redundant _dump method"

This reverts commit 93d0fc9d937186861b41b054235955475c65b5c1.

This commit had the unintentional side-effect of preventing TrackedOp
dumps from including MDS specific events.

Fixes: http://tracker.ceph.com/issues/17906
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: simplify OmapIteratorImpl::valid()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting nits

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make fsck() can catch uninitialized sbid too

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11973 from ceph/wip-17898-ceph-disk-flake8

ceph-disk: fix flake8 errors

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #11971 from dachary/wip-17898-flake8

tests: pin flake8 to avoid behavior changes

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

ceph-disk: fix flake8 errors

flake8 3.1.1 surfaces the following issues:

ceph_disk/main.py:173:1: E305 expected 2 blank lines after class or
function definition, found 1
ceph_disk/main.py:5011:1: E305 expected 2 blank lines after class or
function definition, found 1

Fixes: http://tracker.ceph.com/issues/17898
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11954 from stiopaa1/mds_MDCache_removeUnneededClear

mds/MDCache.h: remove unneeded call to clear func

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11912 from SUSE/wip-17747-followup

doc: drop --journal-check from ceph-mds man page

Reviewed-by: John Spray <john.spray@redhat.com>

OSDMonitor: only reject MOSDBoot based on up_from if inst matches

If the osd actually restarts, there is no guarrantee that the epoch will
advance past up_from.  If the inst is different, it can't really be a
dup.  At worst, it might be a queued MOSDBoot from a previous inst, but
in that case, the real inst would see itself marked up, and then back
down causing it to try booting again.

Fixes: http://tracker.ceph.com/issues/17899
Signed-off-by: Samuel Just <sjust@redhat.com>

tests: pin flake8 to avoid behavior changes

Fixes: http://tracker.ceph.com/issues/17898
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11967 from wjwithagen/wip-wjw-buffer-splice

common/buffer.cc: raw_pipe depends on splice(2)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11914 from tchaikov/wip-17857

ReplicatedPG::failed_push: release read lock on failure

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

common/buffer.cc: raw_pipe depends on splice(2)

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11919 from liewegas/wip-bluestore-cache-meta

os/bluestore: bluestore_cache_meta_ratio = .5

Reviewed-by: Mark Nelson <mnelson@redhat.com>

os/bluestore: precondition enough to write two 256mb buffers

We now match the new bluestore_rocksdb_options.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_cache_meta_ratio = .5

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: configuration overrides for hard coded timers

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #11842 from ceph/wip-cmake

cmake: librbd cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ReplicatedPG::failed_push: release read lock on failure

and requeue the blocked ops.

Fixes: http://tracker.ceph.com/issues/17857
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake:librbd: move common to the end of linked libraries

so the linked libraries are able to find the symbols in libcommon.
and do not `list(APPEND librbd ...)` anymore, which is wrong and not
necessary. librbd does not depends on krbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move ContextCompletion.cc into rbd_internal

- ContextCompletion.cc is used by TrimRequest.cc which is included by
  rbd_internal, it's more natural to put ContextCompletion.cc into
  rbd_internal also. as rbd_internal is the only consumer of this
  translation unit.
- librbd/internal.cc is not referencing any symbols from util.cc, so
  remove this include. and also, do not add
  $<TARGET_OBJECTS:common_util_obj> to librbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: let rbd_internal depend on librbd-tp

as librbd/AioCompletion.cc includes "tracing/librbd.h", which is offered
by librbd-tp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11906 from tchaikov/wip-11829

ceph-detect-init: FreeBSD introduction of bsdrc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix the script for rebuild monitor db

- escape the "$" in "$osd", so it won't be expanded by local shell
- fix the permission bits of /var/lib/ceph/mon/mon.0/store.db
- add the doc to explain the step to prepare the keyring

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-detect-init: FreeBSD introduction of bsdrc

ceph-detect-init/ceph_detect_init/freebsd/__init__.py
    New file

ceph_detect_init/__init__.py: Only test FreeBSD after it is not Linux

    If we do it the other way around it will not work during testing on FreeBSD
    because it will always have platform.system() == FreeBSD
    So first test for Linux, and only then check to see if it is FreeBSD.

ceph-detect-init/tests/test_all.py: add tests for FreeBSD
ceph-detect-init/run-tox.sh: ReEnable to run test for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: static apply() method for bluestore internal-use only

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

lua: use simpler lua_next traversal structure

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: clarify mds deactivate purpose

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: remove broken sentence

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for fsmap_user

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for FSMap

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for MDSMap

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>