mon: make MDSMonitor tolerant of slow mon elections

Previously MDS daemons would get failed incorrectly
when they appeared to have timed out due to
delays in calling into MDSMonitor that were
actually caused by e.g. slow leveldb writes leading
to slow mon elections.

Fixes: http://tracker.ceph.com/issues/17308
Signed-off-by: John Spray <john.spray@redhat.com>

mon: clean up MDSMonitor interface

This was almost all public.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #11836 from batrick/client-new-20161108

 client: remove unnecessary bufferptr[] for writev

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11837 from batrick/client-unique-20161108

client: use unique_ptr

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11904 from stiopaa1/mds_scrubheader_passStringByConstRef

mds/ScrubHeader.h: pass string by const reference

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11651 from stiopaa1/mds_mdsrank_protectedDestructor

mds/MDSRank.h: make destructor protected

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11338 from stiopaa1/mds_sessionmap_unneededCount

mds/SessionMap.cc: remove unneeded use of count

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11957 from batrick/mds-deactivate-doc

doc: clarify mds deactivate purpose

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11985 from batrick/op-tracker-mds-event

mds: revert "mds/Mutation: remove redundant _dump method"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11958 from batrick/cls-lua-lua_next

lua: use simpler lua_next traversal structure

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11833 from stiopaa1/mds_migrator_removeUnneededUseOfCount

mds/Migrator.h: remove unneeded use of count

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10940 from stiopaa1/mds_beacon_moveCMDSBeaconSenderClass

mds/Beacon: move C_MDS_BeaconSender class to .cc

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11887 from jcsp/wip-efragment-cleanup

mds: remove unused EFragment::OP_ONESHOT

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #12004 from bassamtabbara/wip-fix-civetweb-submodule

rgw: revert unintentional change to civetweb

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: revert unintentional change to civetweb

https://github.com/ceph/ceph/pull/11800 uninternionally updated
the civetweb submodule.

Signed-off-by: Bassam Tabbara <bassam.tabbara@quantum.com>

Merge pull request #11908 from stiopaa1/mds_mdcache_removeUnusedFunctions

mds/MDCache.h: remove unused functions

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11964 from xiexingguo/xxg-wip-bluestore-1112

os/bluestore: misc. fixes and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11986 from xiexingguo/xxg-wip-bluestore-1115

os/bluestore: dedup if space overlap truly exists

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11803 from XinzeChi/wip-openpgs

osd: do not open pgs when the pg is not in pg_map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11962 from tchaikov/wip-doc-rebuilddb

doc: fix the script for rebuild monitor db

Reviewed-by: Shylesh Kumar <shmohan@redhat.com>

Merge pull request #11984 from batrick/sphinx-py3-error

doc/man: avoid file builtin to solve build error

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man: avoid file builtin to solve build error

Sphinx running with Python 3.X fails:

    # Sphinx version: 1.4.8
    # Python version: 3.5.2 (CPython)
    # Docutils version: 0.12 release
    # Jinja2 version: 2.8
    # Last messages:

    # Loaded extensions:
    Traceback (most recent call last):
      File "/usr/lib/python3.5/site-packages/sphinx/cmdline.py", line 243, in main
        opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
      File "/usr/lib/python3.5/site-packages/sphinx/application.py", line 137, in __init__
        confoverrides or {}, self.tags)
      File "/usr/lib/python3.5/site-packages/sphinx/config.py", line 287, in __init__
        execfile_(filename, config)
      File "/usr/lib/python3.5/site-packages/sphinx/util/pycompat.py", line 130, in execfile_
        exec_(code, _globals)
      File "conf.py", line 56, in <module>
      File "conf.py", line 47, in _get_manpages
      File "conf.py", line 12, in _get_description
    NameError: name 'file' is not defined

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11808 from trociny/wip-17010

librbd: default features should be negotiated with the OSD

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11821 from trociny/wip-test_notify

test/librbd: in test_notify set object-map and fast-diff features by default

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11840 from yangdongsheng/rbd_mirror_options

rbd-mirror: configuration overrides for hard coded timers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: note 1st existed coll instead of 1st coll

We store the coll mainly for cache trim and we shall try
our best to do it whenever it is possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: move comments to proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: drop wal_op_onodes

Which is currently unused.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: dedup if space overlap truly exists

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mds: revert "mds/Mutation: remove redundant _dump method"

This reverts commit 93d0fc9d937186861b41b054235955475c65b5c1.

This commit had the unintentional side-effect of preventing TrackedOp
dumps from including MDS specific events.

Fixes: http://tracker.ceph.com/issues/17906
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: simplify OmapIteratorImpl::valid()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: formatting nits

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make fsck() can catch uninitialized sbid too

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11973 from ceph/wip-17898-ceph-disk-flake8

ceph-disk: fix flake8 errors

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #11971 from dachary/wip-17898-flake8

tests: pin flake8 to avoid behavior changes

Reviewed-by: Nathan Cutler <ncutler@suse.cz>

ceph-disk: fix flake8 errors

flake8 3.1.1 surfaces the following issues:

ceph_disk/main.py:173:1: E305 expected 2 blank lines after class or
function definition, found 1
ceph_disk/main.py:5011:1: E305 expected 2 blank lines after class or
function definition, found 1

Fixes: http://tracker.ceph.com/issues/17898
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11954 from stiopaa1/mds_MDCache_removeUnneededClear

mds/MDCache.h: remove unneeded call to clear func

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11912 from SUSE/wip-17747-followup

doc: drop --journal-check from ceph-mds man page

Reviewed-by: John Spray <john.spray@redhat.com>

tests: pin flake8 to avoid behavior changes

Fixes: http://tracker.ceph.com/issues/17898
Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #11967 from wjwithagen/wip-wjw-buffer-splice

common/buffer.cc: raw_pipe depends on splice(2)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11914 from tchaikov/wip-17857

ReplicatedPG::failed_push: release read lock on failure

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

common/buffer.cc: raw_pipe depends on splice(2)

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #11919 from liewegas/wip-bluestore-cache-meta

os/bluestore: bluestore_cache_meta_ratio = .5

Reviewed-by: Mark Nelson <mnelson@redhat.com>

os/bluestore: precondition enough to write two 256mb buffers

We now match the new bluestore_rocksdb_options.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: bluestore_cache_meta_ratio = .5

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: configuration overrides for hard coded timers

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #11842 from ceph/wip-cmake

cmake: librbd cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ReplicatedPG::failed_push: release read lock on failure

and requeue the blocked ops.

Fixes: http://tracker.ceph.com/issues/17857
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake:librbd: move common to the end of linked libraries

so the linked libraries are able to find the symbols in libcommon.
and do not `list(APPEND librbd ...)` anymore, which is wrong and not
necessary. librbd does not depends on krbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move ContextCompletion.cc into rbd_internal

- ContextCompletion.cc is used by TrimRequest.cc which is included by
  rbd_internal, it's more natural to put ContextCompletion.cc into
  rbd_internal also. as rbd_internal is the only consumer of this
  translation unit.
- librbd/internal.cc is not referencing any symbols from util.cc, so
  remove this include. and also, do not add
  $<TARGET_OBJECTS:common_util_obj> to librbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: let rbd_internal depend on librbd-tp

as librbd/AioCompletion.cc includes "tracing/librbd.h", which is offered
by librbd-tp.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11906 from tchaikov/wip-11829

ceph-detect-init: FreeBSD introduction of bsdrc

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix the script for rebuild monitor db

- escape the "$" in "$osd", so it won't be expanded by local shell
- fix the permission bits of /var/lib/ceph/mon/mon.0/store.db
- add the doc to explain the step to prepare the keyring

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-detect-init: FreeBSD introduction of bsdrc

ceph-detect-init/ceph_detect_init/freebsd/__init__.py
    New file

ceph_detect_init/__init__.py: Only test FreeBSD after it is not Linux

    If we do it the other way around it will not work during testing on FreeBSD
    because it will always have platform.system() == FreeBSD
    So first test for Linux, and only then check to see if it is FreeBSD.

ceph-detect-init/tests/test_all.py: add tests for FreeBSD
ceph-detect-init/run-tox.sh: ReEnable to run test for FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore: static apply() method for bluestore internal-use only

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

lua: use simpler lua_next traversal structure

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: clarify mds deactivate purpose

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: remove broken sentence

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for fsmap_user

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for FSMap

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: use unique_ptr for MDSMap

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: remove unnecessary bufferptr[] for writev

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #11844 from liupan1111/master

os/bluestore: speedup the performance of multi-replication flow by switc…

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11799 from KerLiu/fixdupassignment

osd/OSDMap.cc: fix duplicated assignment for new_blacklist_entries

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11378 from ryneli/watch_notify_add_timeout

librados: add timeout to watch/notify

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds/MDCache.h: remove unneeded call to clear func

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

librados: add timeout parameter to rados_watch

Signed-off-by: Ryne Li <lizhenqiangsnake@gmail.com>

Merge pull request #10221 from Yan-waller/yj-wip-testrados-0709

test/rados: remove unused bufferlist variable

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11941 from liewegas/wip-envlibrados

qa/workunits/rados/test_envlibrados_for_rocksdb: force librados-dev install

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9357 from Aran85/wip-assert-build-push

osd/PGBackend: build_push_op segment fault

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #11800 from bassamtabbara/wip-make-leveldb-optional

mon: support for building without leveldb + mon mkfs bug fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11664 from sysnote/wip-17719

mon: fix "OSDs marked OUT wrongly after monitor failover"

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10196 from xiaoxichen/fix_16609

log/LogClient: fill seq & who for syslog and graylog

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

BlueStore: speedup the performance of multi-replication flow by
switching the callback order in bluestore.

In Bluestore ack callback and commit callback are queued one by one in
the function "BlueStore;:_txc_finish_kv". Therefore, just only one callback
is needed to call, in order to improve performance. We do this by switching
the callback order in bluestore, and the callback work can be done in
sub_op_modify_commit and doesn't need to do it again in sub_op_modify_applied.

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #11805 from tchaikov/wip-deprecate-readdir-r

deprecate readdir_r() with readdir()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11824 from xiexingguo/xxg-wip-kill-pg-splitting

osd: kill PG_STATE_SPLITTING

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #11874 from wido/osd-no-scrub-recovery

osd: Add config option to disable new scrubs during recovery

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11880 from cbodley/wip-ms-type-random

msg: seed random engine used for ms_type="random"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11898 from liewegas/wip-filestore-hashindex

os/filestore/HashIndex: fix list_by_hash_* termination on reaching end

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/workunits/rados/test_envlibrados_for_rocksdb: force librados-dev install

On trusty we see

 WARNING: The following packages cannot be authenticated!
   librados-dev
 E: There are problems and -y was used without --force-yes

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: assert from access violation

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: use built-in methods for ExtentMap resharding and encoding

Hope this can improve the code readability a little...

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #11907 from badone/wip-pipe-socket-typos

msg: Fix typos in socket creation error message

Reviewed-by: xiexingguo <xie.xingguo@zte.com.cn>

Merge pull request #11392 from Oloremo/keystone_docs

Update keystone doc about v3 options

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #11925 from ktdreyer/doc-infernalis-eol

doc: add infernalis EOL date

doc: add infernalis EOL date

Remove the estimated Infernalis EOL date, and add the actual EOL date.

As discussed on the ceph-devel mailing list, we decided to say that this
infernalis EOL date is the same date that we made the first Jewel
release available.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11924 from liewegas/wip-balloc

unittest_bit_alloc: fix type

Merging before make check because it clearly breaks the build and the build part is done
Reviewed-by: Dan Mick <dmick@redhat.com>

unittest_bitalloc: fix type

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #11922 from liewegas/wip-mempool-bluestore

mempool: changes for bitmap allocator

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #11920 from ifed01/wip-bluestore-tmp

os/bluestore: remove build warning in a better way

Reviewed-by: Sage Weil <sage@redhat.com>

mempool changes for bitmap allocator

[with bstore_balloc -> bluestore_alloc rename by sage]

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #11839 from liewegas/wip-mempool-buffer

buffer: put buffers in buffer_{data,meta} mempools

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

Merge pull request #11834 from liewegas/wip-bluestore-mempool

os/bluestore: move bluefs into its own mempool

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>

buffer: put data and metadata in a mempool

Note that for raw_combined we leak some metadata into the data pool.

Also, we do not account for non-raw metadata or anything else in buffer.h,
as that would pollute the ABI and public interface.

Drop the namespace ceph in buffer.cc (which serves no real purpose) as it
confuses the MEMPOOL_DEFINE_* macros (they cannot be used inside a
namespace).

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: add [de]allocate_aligned

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move pg logs into osd mempool

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: move peering evt into pool

Signed-off-by: Sage Weil <sage@redhat.com>

osd: move ReplicatedPG into mempool

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: create an osd mempool

Signed-off-by: Sage Weil <sage@redhat.com>

mempool: attach mempool asok commands to all ccts

Signed-off-by: Sage Weil <sage@redhat.com>