Merge pull request #12046 from dachary/wip-16014-cot

tests: use shorter directories for tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12036 from tchaikov/wip-osd-scrub-repair-cleanup

test/osd-scrub-repair.sh: use repair() instead of "ceph pg repair"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12039 from xiexingguo/xxg-wip-bluestreo-1117

os/bluestore: fix bufferspace stats leak due to blob splitting

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #12042 from ifed01/wip-bluestore-drop-blob_depth

os/bluestore: remove garbage collector staff

Reviewed-by: Sage Weil <sage@redhat.com>

tests: use shorter directories for tests

So that jenkins can use longer directories. We can't have both otherwise
the limit UNIX domain socket path length limit triggers errors such as:

... client.admin.12750.asok is too long! The maximum length on this system is 107

Fixes: http://tracker.ceph.com/issues/16014
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11961 from tianshan/fix-RGWSimpleRadosLockCR

rgw: fix RGWSimpleRadosLockCR set_description()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11965 from ceph/wip-rgw-fix-period-commit

rgw: fix missing master zone for a single zone zonegroup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12020 from leseb/glance-mitaka

doc: openstack glance mitaka uses show_multiple_locations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix missing master zone for a single zone zonegroup

Fixes: http://tracker.ceph.com/issues/17364
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

os/bluestore: remove garbage collector staff

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11424 from renhwztetecs/renhw-wip-mgr

mgr: init() return when connection daemons failed && add some err info

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12015 from xiexingguo/xxg-wip-bluestore-init-csum

os/bluestore: avoid unnecessary call to init_csum()

Reviewed-by: Sage Weil <sage@redhat.com>

test/osd-scrub-repair.sh: use repair() instead of "ceph pg repair"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12038 from tchaikov/wip-ceph-objectstore-tool

test: ceph-objectstore-tool: should import platform before using it

Reviewed-by: Owen Synge <osynge@suse.com>

doc: openstack glance mitaka uses show_multiple_locations

As of the Mitaka release show_image_direct_url is not needed, but
instead show_multiple_locations should be used.
Adding the necessary guidance for Mitaka release.

Signed-off-by: Sébastien Han <seb@redhat.com>

test: ceph-objectstore-tool: should import platform before using it

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9613 from dzafman/wip-16064

common osd: Improve scrub analysis, list-inconsistent-obj output and osd-scrub-repair test

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10507 from wjwithagen/wip-wjw-tests-ceph_objectstore_tool

ceph_objectstore_tool.py: Don't use btrfs on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11890 from liupan1111/wip-code-clean-ups

os/filestore: handle EINTR returned by io_getevents()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11765 from bassamtabbara/wip-crushtool-fix

mon: if crushtool config is empty use internal crush test

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: fix bufferspace stats leak due to blob splitting

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11926 from dzafman/wip-test-17830

test: fix test-erasure-eio and osd-scrub-repair races (17830)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12030 from branch-predictor/bp-rmxattr-bl

librados: remove unused bufferlist from rados_write_op_rmxattr

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

test: Add sleep 1 so that we look for 60 seconds

This failed because test blew through 60 requests in 8 seceons before
the scrub repair even started on Jenkins.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix duplicate port in test-erasure-code-plugins.sh

Add comments about uniqueness of port number required

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12032 from liewegas/wip-bluestore-warning

os/bluestore: fix warning and uninit variable

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mon: Fix compiler warnings, missed _dump() change

Caused by: af720cc87b4d631695c1549c31d57bfbc82bc6ba

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Re-enable osd-scrub-repair.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test: In osd-scrub-repair.sh fix use of wait_for_clean()

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #11996 from dachary/wip-17849-bootstrap-osd

mon,ceph-disk: add lockbox permissions to bootstrap-osd

Reviewed-by: Sage Weil <sage@redhat.com>

include/mempool: silence warnings

/home/sage/src/ceph4/src/include/mempool.h:421:68: warning: no return statement in function returning non-void [-Wreturn-type]
   void *operator new[](size_t size) { assert(0 == "no array new"); } \

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/KernelDevice: initialize oldest

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12029 from batrick/mon-_dump-bugfix

mon: remove utime_t param in _dump

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librados: remove unused bufferlist from rados_write_op_rmxattr

rados_write_op_rmxattr declares the bl bufferlist variable, then doesn't make
any use of it. Get rid of it, as it's probably an effect of copy+paste.

Signed-off-by: Piotr Dałek <git@predictor.org.pl>

Merge pull request #12028 from batrick/client-redundant-init

client: remove redundant initialization

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test: Re-enable test-erasure-eio.sh

Signed-off-by: David Zafman <dzafman@redhat.com>

test: activate_osd() doesn't need to set crush

Tests use objectstore_tool() which stops and starts OSDs,
but may assume consistency of object locations.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: wait_for_clean() add sleep backoff

Reduce size of log on timeout by doing a backoff so that
we don't log 3000 loops at 1/10 second sleeps.

Signed-off-by: David Zafman <dzafman@redhat.com>

test: CLEANUP to later enable TEST_rados_get_with_subreadall_eio_shard_*

Signed-off-by: David Zafman <dzafman@redhat.com>

mon: remove utime_t param in _dump

Missing in af720cc87b4d631695c1549c31d57bfbc82bc6ba.

Found in warning:

In file included from /home/pdonnell/ceph/src/mds/MDSRank.h:18:0,
                 from /home/pdonnell/ceph/src/mds/MDBalancer.cc:18:
/home/pdonnell/ceph/src/common/TrackedOp.h:153:16: warning: ‘virtual void TrackedOp::_dump(ceph::Formatter*) const’ was hidden [-Woverloaded-virtual]
   virtual void _dump(Formatter *f) const {}
                ^
In file included from /home/pdonnell/ceph/src/mon/mon_types.h:23:0,
                 from /home/pdonnell/ceph/src/mon/MonMap.h:22,
                 from /home/pdonnell/ceph/src/mon/MonClient.h:20,
                 from /home/pdonnell/ceph/src/mds/MDBalancer.cc:19:
/home/pdonnell/ceph/src/mon/MonOpRequest.h:106:8: warning:   by ‘void MonOpRequest::_dump(utime_t, ceph::Formatter*) const’ [-Woverloaded-virtual]
   void _dump(utime_t now, Formatter *f) const {
        ^

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon,ceph-disk: add lockbox permissions to bootstrap-osd

ceph-disk --dmcrypt needs to put a config-key and authorize
the OSD to get it back. The corresponding permissions are
added to the bootstrap-osd profile in the monitor.

When preparing the OSD lockbox, use the bootstrap-osd profile instead of
implicitly requiring admin permissions to perform the initial config-key
and auth get-or-create operations.

Fixes: http://tracker.ceph.com/issues/17849
Signed-off-by: Loic Dachary <loic@dachary.org>

client: remove redundant initialization

messenger is already initialized in the constructor member
initialization list.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12025 from dachary/wip-17923-tox

tests: no python3 tests for ceph-disk

Reviewed-by: Samuel Just <sjust@redhat.com>

tests: no python3 tests for ceph-disk

2.5.0 published today made it so an environment with no action fails
where it previously did not.

Fixes: http://tracker.ceph.com/issues/17923
Signed-off-by: Loic Dachary <loic@dachary.org>

doc, test: Add schemas for list-inconsistent-* rados command output

If jsonschema cmd is available use it to test output against schema

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Update testing for new list-inconsistent-obj output

Signed-off-by: David Zafman <dzafman@redhat.com>

rados, osd: Improve attrs output of list-inconsistent-obj

Persist the user_version and shard id of scrubbed obj
Rados command dump inconsistent obj's version and shard-id
    so they can be passed to repair command
Rados list-inconsistent-obj output of attrs
    Make attrs an array since there are more than one
    Use base64 encode for values with non-printable chars
    Add indication if base64 encoding used
Add checking for ss_attr_missing and ss_attr_corrupted
    Rename attr errors to attr_key_mismatch and attr_value_mismatch
Add missing size_mismatch_oi scrub checking
For erasure coded pools add ec_size_error and ec_hash_error not just read_error
Use oi_attr_missing and oi_attr_corrupted just like list-inconsistent-snap does
Pick an object info based on version and use that to find specific shards in error
    Check for object info inconsistency which should be rare
Make all errors based on comparing shards to each other object errors
    We don't want give the impression that we've picked the correct one

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #12024 from tchaikov/wip-ceph-erasure-code-corpus-arch

tests: sync ceph-erasure-code-corpus for using 'arch' not 'uname -p'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: sync ceph-erasure-code-corpus for using 'arch' not 'uname -p'

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12011 from ceph/wip-git-archive-all

git-archive-all.sh: use an actually unique tmp dir

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #12007 from jcsp/wip-trackedop-now

common/TrackedOp: remove unused 'now' in _dump()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12016 from dachary/wip-17919-osd-crush

tests: osd-crush.sh must retry crush dump

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12012 from tchaikov/wip-sepia-bt

script/sepia_bt.sh: a script to prepare for debugging on teuthology@sepia

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #11167 from jcsp/wip-17308

mon: make MDSMonitor tolerant of slow mon elections

Reviewed-by: John Spray <john.spray@redhat.com>

tests: osd-crush.sh must retry crush dump

After ceph osd setcrushmap, the script must retry because ceph osd crush
dump races against the update of the crushmap.

Fixes: http://tracker.ceph.com/issues/17919
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11979 from ceph/wip-disable-osd-scrub-repair

test: disable osd-scrub-repair and test-erasure-eio

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: more readability improvements

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: avoid unnecessary call to init_csum()

We have to initiate CSumType from 1, which represents CSUM_NONE,
to be aligned with OSDMnitor's pool_opts_t handling.

So we have to explicitly check against CSUM_NONE to skip init_csum(),
which will set FLAG_CSUM and alloc memory for csum_data and thus
shall be avoided whenever it is possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

script/sepia_bt.sh: a script to prepare for debugging on teuthology@sepia

Signed-off-by: Kefu Chai <kchai@redhat.com>

git-archive-all.sh: use an actually unique tmp dir

git archive into $TMPDIR/$(basename "$(pwd)").$FORMAT is not unique;
if two runs are running simultaneously, this will collide.  Make
TMPDIR actually unique, and then the cleanup can just remove the whole
directory as well.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #12003 from athanatos/wip-17899

OSDMonitor: only reject MOSDBoot based on up_from if inst matches

Reviewed-by: Sage Weil <sage@redhat.com>

mon: make MDSMonitor tolerant of slow mon elections

Previously MDS daemons would get failed incorrectly
when they appeared to have timed out due to
delays in calling into MDSMonitor that were
actually caused by e.g. slow leveldb writes leading
to slow mon elections.

Fixes: http://tracker.ceph.com/issues/17308
Signed-off-by: John Spray <john.spray@redhat.com>

mon: clean up MDSMonitor interface

This was almost all public.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11836 from batrick/client-new-20161108

 client: remove unnecessary bufferptr[] for writev

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11837 from batrick/client-unique-20161108

client: use unique_ptr

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11904 from stiopaa1/mds_scrubheader_passStringByConstRef

mds/ScrubHeader.h: pass string by const reference

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11651 from stiopaa1/mds_mdsrank_protectedDestructor

mds/MDSRank.h: make destructor protected

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11338 from stiopaa1/mds_sessionmap_unneededCount

mds/SessionMap.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11957 from batrick/mds-deactivate-doc

doc: clarify mds deactivate purpose

Reviewed-by: John Spray <john.spray@redhat.com>

common/TrackedOp: remove unused 'now' in _dump()

Greg pointed this out in PR #11985

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11985 from batrick/op-tracker-mds-event

mds: revert "mds/Mutation: remove redundant _dump method"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11958 from batrick/cls-lua-lua_next

lua: use simpler lua_next traversal structure

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11833 from stiopaa1/mds_migrator_removeUnneededUseOfCount

mds/Migrator.h: remove unneeded use of count

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10940 from stiopaa1/mds_beacon_moveCMDSBeaconSenderClass

mds/Beacon: move C_MDS_BeaconSender class to .cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11887 from jcsp/wip-efragment-cleanup

mds: remove unused EFragment::OP_ONESHOT

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12004 from bassamtabbara/wip-fix-civetweb-submodule

rgw: revert unintentional change to civetweb

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revert unintentional change to civetweb

https://github.com/ceph/ceph/pull/11800 uninternionally updated
the civetweb submodule.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #11908 from stiopaa1/mds_mdcache_removeUnusedFunctions

mds/MDCache.h: remove unused functions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11964 from xiexingguo/xxg-wip-bluestore-1112

os/bluestore: misc. fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11986 from xiexingguo/xxg-wip-bluestore-1115

os/bluestore: dedup if space overlap truly exists

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

test: disable osd-scrub-repair and test-erasure-eio

After Sage's suggestion of earlier today, in now-closed
PR #11970, which can't be resurrected just now

Kefu is trying to reproduce osd-scrub-repair failures

David is working test-erasure-eio in
https://github.com/ceph/ceph/pull/11926

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #11803 from XinzeChi/wip-openpgs

osd: do not open pgs when the pg is not in pg_map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11962 from tchaikov/wip-doc-rebuilddb

doc: fix the script for rebuild monitor db

Reviewed-by: Shylesh Kumar <shmohan@redhat.com>

Merge pull request #11984 from batrick/sphinx-py3-error

doc/man: avoid file builtin to solve build error

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man: avoid file builtin to solve build error

Sphinx running with Python 3.X fails:

    # Sphinx version: 1.4.8
    # Python version: 3.5.2 (CPython)
    # Docutils version: 0.12 release
    # Jinja2 version: 2.8
    # Last messages:

    # Loaded extensions:
    Traceback (most recent call last):
      File "/usr/lib/python3.5/site-packages/sphinx/cmdline.py", line 243, in main
        opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
      File "/usr/lib/python3.5/site-packages/sphinx/application.py", line 137, in __init__
        confoverrides or {}, self.tags)
      File "/usr/lib/python3.5/site-packages/sphinx/config.py", line 287, in __init__
        execfile_(filename, config)
      File "/usr/lib/python3.5/site-packages/sphinx/util/pycompat.py", line 130, in execfile_
        exec_(code, _globals)
      File "conf.py", line 56, in <module>
      File "conf.py", line 47, in _get_manpages
      File "conf.py", line 12, in _get_description
    NameError: name 'file' is not defined

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11808 from trociny/wip-17010

librbd: default features should be negotiated with the OSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11821 from trociny/wip-test_notify

test/librbd: in test_notify set object-map and fast-diff features by default

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11840 from yangdongsheng/rbd_mirror_options

rbd-mirror: configuration overrides for hard coded timers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: note 1st existed coll instead of 1st coll

We store the coll mainly for cache trim and we shall try
our best to do it whenever it is possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: move comments to proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop wal_op_onodes

Which is currently unused.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: dedup if space overlap truly exists

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: revert "mds/Mutation: remove redundant _dump method"

This reverts commit 93d0fc9d937186861b41b054235955475c65b5c1.

This commit had the unintentional side-effect of preventing TrackedOp
dumps from including MDS specific events.

Fixes: http://tracker.ceph.com/issues/17906
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: simplify OmapIteratorImpl::valid()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting nits

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make fsck() can catch uninitialized sbid too

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11973 from ceph/wip-17898-ceph-disk-flake8

ceph-disk: fix flake8 errors

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #11971 from dachary/wip-17898-flake8

tests: pin flake8 to avoid behavior changes

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

ceph-disk: fix flake8 errors

flake8 3.1.1 surfaces the following issues:

ceph_disk/main.py:173:1: E305 expected 2 blank lines after class or
function definition, found 1
ceph_disk/main.py:5011:1: E305 expected 2 blank lines after class or
function definition, found 1

Fixes: http://tracker.ceph.com/issues/17898
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11954 from stiopaa1/mds_MDCache_removeUnneededClear

mds/MDCache.h: remove unneeded call to clear func

Reviewed-by: Greg Farnum <gfarnum@redhat.com>