doc: instructions for provisioning OpenStack VMs ad hoc

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge remote-tracking branch 'gh/luminous'

Merge pull request #13218 from guits/ceph_detect_init

ceph-detect-init: Add docker detection

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

12.0.0

ceph-detect-init: Add docker detection

if running `ceph-detect-init` from a container, the given output is the
reference from the hosting system. This patch add the capability of
detecting if the tool is being run inside a container.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #12586 from ktdreyer/wip-18313-rgw-upstart

upstart: start radosgw-all according to runlevel

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #12086 from wjwithagen/wip-wjw-freebsd-ceph-disk

ceph-disk: FreeBSD changes to get it working and passing tests

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13015 from chuanhong-wang/dev_change_osd_df_tree_output

mon: in output of "ceph osd df tree", display "-", not "0",  for pg amount of a bucket

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13159 from bassam/pr-fix-embedded-zstd

compressor/zstd: add zstd to embedded ceph

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13236 from adamemerson/wip-alarum

cleanup: resolve compiler warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13265 from tchaikov/wip-librados-hide-private-symbols

librados: do not expose non-public symbols

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13268 from tchaikov/wip-better-error-ping-mon

ceph: do not throw TypeError on connection failure

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13286 from Liuchang0812/remove-useless-close

msg/async: remove useless close function

Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #13288 from dmick/master

ceph.in: add help for locally-handled commands

Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #9976 from songbaisen/a13

mon: PGMonitor add check only concern our self cluster command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10150 from songbaisen/b

mon: avoid start election twice when quorum enter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10309 from songbaisen/b8

mon: OSDMonitor add check only concern our self cluster command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12273 from songbaisen/s4

mon: add warn info for osds were removed from osdmap but still kept in crushmap

Reviewed-by: Piotr Dałek <piotr.dalek@ts.fujitsu.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13121 from LiumxNL/wip-170125

osd: we know the definite epoch of marking down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13297 from dachary/wip-buildpackages

buildpackages: remove because it does not belong

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #13162 from liewegas/wip-bluestore-reshard

os/bluestore: partial reshard support

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

buildpackages: remove because it does not belong

It should live in teuthology, not in Ceph. And it is currently broken:
there is no need to keep it around.

Fixes: http://tracker.ceph.com/issues/18846
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #13035 from yehudasa/wip-18621

rgw: change default chunk size to 4MB

Merge pull request #13259 from jcsp/wip-jcsp-relnotes

PendingReleaseNotes: note the fuse fstab format change

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.in: add help for locally-handled commands

Signed-off-by: Dan Mick <dan.mick@redhat.com>

msg/async: remove useless close function

fix coverity cid 1395314: it's invalid to pass a negative value to close function, and we have closed socket in net.connect/nonblock_connect in error cases indeed

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #13279 from smithfarm/wip-fix-regression

tests: fix regression in qa/tasks/ceph_master.py

FreeBSD: changes to ceph-disk to get it working and passing tests

 - It currently only supports running with filestore
 - Testing is executed while running on a ZFS partition
 - All disktypes and naming is different on FreeBSD
 - Partitioning and tools are not workable on FreeBSD
 - add diagnostic timeout

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: don't update bucket index multiple times in overwrite

Instead of this for overwrites:
 prepare (index),
   write (head) [-EEXIST]
 cancel (index)
 read (head)
 prepare (index)
   write (head)
 complete (index)

We now do:
 prepare (index),
   write (head) [-EEXIST]
   read (head)
   write (head)
 complete (index)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: assume obj write is a first write

if fails and object already exists then retry. This improves first obj
write performance on the expense of overwrites.

Fixes: http://tracker.ceph.com/issues/18622
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: configurable write obj window size

Fixes: http://tracker.ceph.com/issues/18623
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

tests: fix regression in qa/tasks/ceph_master.py

https://github.com/ceph/ceph/pull/13194 introduced a regression:

2017-02-06T16:14:23.162 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 722, in wrapper
    return func(self)
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 839, in do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/ceph_manager.py", line 305, in kill_osd
    output = proc.stderr.getvalue()
AttributeError: 'NoneType' object has no attribute 'getvalue'

This is because the original patch failed to pass "stderr=StringIO()" to run().

Fixes: http://tracker.ceph.com/issues/16263
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12979 from theanalyst/scripts/release-notes-tracker

script: ceph-release-notes check orig. issue only for backports

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13277 from jan--f/janf-teuth-cloud-doc

doc: minor change to a cloud testing paragraph

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13275 from idryomov/wip-require-osd-features-fix

osd/OSDMap: require OSD features only of OSDs

Reviewed-by: Sage Weil <sage@redhat.com>

doc: minor change to a cloud testing paragraph

Use terminate for the instances to match the terminology used by the
horizon interface.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #13256 from dongbula/correct-cached_crc-comment

osd: correct comment of perfcounter cached_crc in code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13220 from rzarzynski/wip-rgw-18364-2

rgw: be aware abount tenants on cls_user_bucket -> rgw_bucket conversion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #12993 from jcsp/wip-cephfs-doc-authcaps

doc: clarify the path restriction mds cap example

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #13242 from Jing-Scott/dev/fix-swift-anon-getobj-error-code

rgw: swift: fix anonymous user's error code of getting object

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

osd/OSDMap: require OSD features only of OSDs

Fix up commit 1a5cc32f0a3b ("osd/OSDMap: reflect REQUIRE_*_OSDS flag in
required features") -- require_*_osds shouldn't affect older clients.

Fixes: http://tracker.ceph.com/issues/18831
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #13271 from Liuchang0812/fix-file-description-leak

msg/async: fix file description leak in NetHandler

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: fix file description leak in NetHandler

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

rgw: swift: fix anonymous user's error code of getting object

The openstack swift will return 401 rather than 403 when
the anon user has no permission to get objects.

Fixes: http://tracker.ceph.com/issues/18806
Signed-off-by: Jing Wenjun <jingwenjun@cmss.chinamobile.com>

ceph: do not throw TypeError on connection failure

otherwise we could concat None with a string on connection problem.
which will result in TypeError. and the caller will print misleading
error like

Error connecting to cluster: TypeError

see also #12934

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13237 from smithfarm/wip-18799

tests: Thrasher: eliminate a race between kill_osd and __init__

Reviewed-by: Sage Weil <sage@redhat.com>

librados: do not expose non-public symbols

librados could be referencing the wrong symbol when calling the
destructor. for example, a variable of type AioGetxattrsData in
IoCtxImpl.cc) could be destructed using the destructor defined in
librados.cc. this is reproduciable using

ceph_test_rados_api_aio --gtest_filter=LibRadosAio.XattrIter

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13263 from Adirl/fix_log_spaces

msg/async/rdma: fix log line spacing

Reviewed-by: Haomai Wang <haomai@xsky.com>

PendingReleaseNotes: note the fuse fstab format change

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #13257 from tchaikov/wip-less-verbose-ceph-helper

tests: ceph-helpers.sh reduce get_timeout_delays() verbosity

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh reduce get_timeout_delays() verbosity

`set +o` prints out the full command line which is echoed if "xtrace" is
enabled. this increases the verbosity of get_timeout_delays().
in this change, we follow the way of kill_daemons() to kill the extra
output. see aefcf6d.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: correct comment of perfcounter cached_crc in code

Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

Merge pull request #13080 from yuyuyu101/wip-async-fd

msg/async/AsyncConnection: socket's fd can be zero, avoid false assert

Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #13251 from scuttlemonkey/master

script: adding contributor credits script

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9396 from songbaisen/doc2

doc: fix the librados c api can not compile problem

Reviewed-by: Sage Weil <sage@redhat.com>

[librados,libcephfs] Avoid redefinition of rados_t

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Classes with virtual functions get virtual destructors

Including template specializations.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

crush: Do not defeat RVO with unnecessary std::move

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

bluestore: Fix redeclaration of struct

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Use non-deprecated get_omap function

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix rgw_format construction order

Since we have only one constructor, just use default initialization for
every member that doesn't depend on an argument.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Fix sign comparison warning

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

key_value_store: Stop using deprecated omap_get_* functions

This gets rid of deprecation warnings.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #13252 from linuxbox2/wip-rgw-setattr

rgw_file:  fix RGWLibFS::setattr for directory objects

rgw_file:  fix RGWLibFS::setattr for directory objects

Fixes: http://tracker.ceph.com/issues/18808
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #13147 from yehudasa/wip-18331

rgw: fix use of marker in List::list_objects()

Merge pull request #12023 from yuyuyu101/wip-msgr-type

msg: allow different ms type for cluster network and public network

Reviewed-by: Adir Lev <adirl@mellanox.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13114 from liewegas/wip-osdmap-required-features

osd: condition object_info_t encoding on required (not up) features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13206 from liewegas/wip-osd-enxio

osd: do not send ENXIO on misdirected op by default

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13213 from ovh/bp-osd-updateable-throttles

osd: allow client throttler to be adjusted on-fly, without restart

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13063 from ovh/wip-mailmap-ovh-contributors

mailmap: Update OVH contributors

Reviewed-by: Loic Dachary <ldachary@redhat.com>

adding contributor credits script

Signed-off-by: Patrick McGarry <pmcgarry@redhat.com>

Merge pull request #13223 from wjwithagen/wip-wjw-freebsd-sphynx

build/install-deps.sh: Add sphinx package for building docs on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11448 from ledgr/change/mount_fuse_ceph_options_in_fs_mntops_and_python

Rewrite mount.fuse.ceph (to python) and move ceph-fuse options to fs_mntops

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13194 from smithfarm/wip-16263

tests: ignore bogus ceph-objectstore-tool error in ceph_manager

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

os/bluestore: drop ExtentMap::Shard offset field

This is mirrored in shard_info, which we have a pointer to.

Note that shard_info is also a bit redundant as we can also look it up in
the onode vector in the same position, but it makes it more awkward to use
iterators.  Something to consider later to save memory.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: simplify dirty shard handling in ExtentMap::update

We can do this with a single allocation with a simple vector<>.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ExtentMap: reshard only requested range

Only reshard extents in the requested range.

Note that the strategy for unspanning blobs changed; we now span or
unspan specific blobs at the end based on our reshard result instead
of unspanning all at the start.  This keeps the spanning id stable,
which is important because it may be referenced from another that
we aren't even looking at.

Also note that this requires a bit of restructuring: an encode_some may
hit a spanning shard, *requiring* us to reshard, which means we should
always conduct the initial pass through update to discover other reshard
requirements, even if we already know some resharding will be needed.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: dump num spanning blobs from dump_onode

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: trivial cleanups in csum error code

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: dump_onode after shards are loaded

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_objectstore: error out on fsck failure

It's much more helpful this way.

Signed-off-by: Sage Weil <sage@redhat.com>

os/memstore: return 0 from fsck

Signed-off-by: Sage Weil <sage@redhat.com>

tests: Thrasher: eliminate a race between kill_osd and __init__

If Thrasher.__init__() spawns the do_thrash thread before initializing the
ceph_objectstore_tool property, do_thrash races with the rest
of Thrasher.__init__() and in some cases do_thrash can call kill_osd() before
Trasher.__init__() progresses much further. This can lead to an exception
("AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool'")
being thrown in kill_osd().

This commit eliminates the race by making sure the ceph_objectstore_tool
attribute is initialized before the do_thrash thread is spawned.

Fixes: http://tracker.ceph.com/issues/18799
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13234 from ifed01/wip-bluestore-performance-sandbox2017

os/bluestore: fix potential assert in cache _trim method.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #12965 from linuxbox2/wip-librgw-expand-argv

rgw_file:  split last argv on ws, if provided

rgw_file:  split last argv on ws, if provided

This is intended to allow an "extra" unparsed argument string
containing various cmdline options to be passed as the last argument
in the argv array of librgw_create(), which nfs-ganesha is
expecting to happen.

While at it, hook env_args() too.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: be aware abount tenants on cls_user_bucket -> rgw_bucket conversion.

Fixes: http://tracker.ceph.com/issues/18364
Fixes: http://tracker.ceph.com/issues/16355
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

mailmap: Update OVH contributors

Signed-off-by: Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>

Merge pull request #12335 from theanalyst/mailmap/v11.1.0

mailmap updates for v11.1.0

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os/bluestore: fsck: check for shards past EOF

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: reshard on truncate if shards past EOF

This ensure we mop up shards past EOF instead of encoding them
empty and confusing future code (that, say, assumes no shards past
EOF).

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/bluestore_types: beautify use_tracker_t operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ExtentMap: request reshard range from update() intelligently

Split ourselves, or merge with our immediate predecessor or
successor.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ExtentMap: use normal iterator for reshard loop

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ExtentMap: put un-spanning in normal loop

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ExtentMap: remove reshard 0 extent special case

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ExtentMap: factor old shard key removal into reshard()

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: cosmetic cleanup in ExtentMap::update

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: ExtentMap: count extents per Shard

Signed-off-by: Sage Weil <sage@redhat.com>