doc: peering.rst, fix typo

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #9933 from chhabaramesh/master

kv: In memory keyvalue db implementation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10286 from ceph/wip_leftover_with_mds

Removes remaining reference to WITH_MDS

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Not all references to WITH_MDS were removed (see PR #10186). This
removes the final one (dependency of vstart on ceph-mds).

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #10120 from oritwas/bz16412

rgw: remove bucket index objects when deleting the bucket

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10161 from ceph/wip-dang-cmake

CMake - stop pip checking for updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10170 from ceph/wip-pg-log-errors-11

include write error codes in the pg log

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge branch 'googletest'

Update Ceph to use Google Test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10282 from ceph/wip-cmake-cxx-flags

cmake: fix the FTBFS introduced by dc8b3ba

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: fix the FTBFS introduced by dc8b3ba

the flags are not list

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10279 from ceph/wip-cmake

cmake changes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9784 from ceph/wip-ms-async-fixes

msg/async: connect authorizer fix + recv_buf size

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10276 from tchaikov/wip-doc-drop-i386

doc: remove i386 from minimal hardware recommendations

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #10254 from dillaman/wip-16536

rbd-mirror: include local pool id in resync throttle unique key

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10245 from jcsp/wip-mds-cache-warn

mds: add health warning for oversized cache

Reviewed-by: Yan, Zheng <zyan@redhat.com>

cmake: use CMAKE_CXX_STANDARD for enabling C++11

hope we can drop the crufts when moving to cmake 3.1+

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10191 from batrick/mds-minor

mds: use reference to avoid copy

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10268 from batrick/i16664

doc: fix standby replay config

Reviewed-by: John Spray <john.spray@redhat.com>

cmake: do not check for availability of static_cast<> anymore

it's implied by C++11 compliance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use CHECK_STRUCT_HAS_MEMBER for checking member variables in struct

we were using CHECK_CXX_SOURCE_COMPILES() for this purpose, which was a
little bit clumsy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

kv: In memory keyvalue db implementation

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

doc: remove i386 from minimal hardware recommendations

as we don't build or test on i386 machines now. also, we don't build
i386 releases anymore. and regular users are on amd64 or arm.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10102 from liewegas/wip-mon-nosnap

mon/MonitorDBStore: do not use snapshot iterator; close on close

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10275 from ceph/wip-cmake

cmake: disable -fvar-tracking-assignments for ceph_dencoder.cc

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: disable -fvar-tracking-assignments for ceph_dencoder.cc

there are too many variables involved in main() of ceph_dencoder.cc, but
-fvar-tracking-assignments is enabled by default, when the compiler
finds that the number of variables exceeds the limit
"max-vartrack-size", it gives up, and tries again disabling this option.

this change avoids the recompilation of ceph_dencoder.cc and also silences
the warning of
```
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/encoding/ceph_dencoder.cc:
In function ‘int main(int, const char**)’:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/encoding/ceph_dencoder.cc:289:5:
note: variable tracking size limit exceeded with
-fvar-tracking-assignments, retrying without
 int main(int argc, const char **argv)
     ^
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10113 from stiopaa1/msg_asyncmess_moveC_handle_reap

msg/AsyncMessenger: move C_handle_reap class to cc

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10183 from stiopaa1/msg_asyncmess_addConstToSetAddrUnknowns

msg/Messenger: add const and override to function

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10141 from stiopaa1/msg_asyncmess_removeUnneededCast

msg/AsyncMessenger.cc: remove unneeded cast

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #10266 from tchaikov/wip-ceph-detect-init-py3

ceph-detect-init: fix the py3 test

Reviewed-by: Loic Dachary <ldachary@redhat.com>

googletest: use ceph branch

Added the same functionality as the previous gtest/gmock branches -
.gitignore and not installing shared libs that we only want to
statically link to test programs.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Replace gmock/gtest with googletest

Google Mock has been absorbed into the Google Test project, and
is now available under the googletest repository.

Signed-off-by: Anirudha Bose <ani07nov@gmail.com>

doc: fix standby replay config

I tried using these settings in tests without success. The correct config names
are prefixed with "mds".

Fixes: http://tracker.ceph.com/issues/16664
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-detect-init: fix the py3 test

the mock_open() does not support "for i in open()", and readline() and
readlines() support was added in py3.3 and py3.4 so for better backwards
compatibility, we should change the code being tested to use the plain
read() call.
and also use open(path) instead of open(path, 'rb') for simplicity,
otherwise we need to use bytestring for comparison and pass the same
parameters to mock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph-detect-init: fix the indent

this fixes the pep8 failure:
```
pep8 runtests: commands[0] | flake8 ceph_detect_init tests
tests/test_all.py:121:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:125:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:129:17: E128 continuation line under-indented for
visual indent
tests/test_all.py:133:17: E128 continuation line under-indented for
visual indent
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10090 from yuyuyu101/wip-remove-async-lock

msg/async: remove file event lock

Reviewed-by: Kefu Chai <kefu@redhat.com>

ceph_test_msgr: clean previous sent queue infos

1. A -> B
2. goto standby
3. B mark down
4. A reconnect to B
5. got reset session and dispatch remote reset
6. because remote reset is executed in DispatchQueue, it will be delayed
7. A -> B successfully and begin to send message
8. assert because we found the first message is missing but it's reasonble

if policy.resetcheck is true

Signed-off-by: Haomai Wang <haomai@xsky.com>

ceph_test_msgr: use ceph log infrastructure to output

because we want to get the right log sequence which mixes ceph logginer and
cerr. Otherwise, cerr output make the logs a little disordered.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: make sure no STATE_WAIT -> STATE_STANDBY

Otherwise if message in queue, we will continue to reconnect right now,
it won't meet our expectation that we want our connect request delay

Signed-off-by: Haomai Wang <haomai@xsky.com>

test_msgr: add server_msgr cleanup

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: add REPLACING write status to indicate replacing

When replacing, we don't expect any AsyncConnection dispatch new event which
will cause thing chaos

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: hold lock avoid state changing

Fixes: http://tracker.ceph.com/issues/16554
Signed-off-by: Haomai Wang <haomai@xsky.com>

test_async_driver: accept new api changes

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: dont delete time event when exchange eventcenter

When replacing and someone called mark_down, it will delete_time_event which
isn't allowed. Because we're exchaning EventCenter now!

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: simplify shutdown socket flow

Let cleanup resources things all in shutdown_socket

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: add assert to ensure create/delete file event within thread

Signed-off-by: Haomai Wang <haomai@xsky.com>

Event: remove file_lock

Now all EventCenter will exists within one thread, it will let all file events
api changes without locks.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncMessenger: make sure all connection cleanup handler finished

Fixes: http://tracker.ceph.com/issues/16552
Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: remove close socket line

because if we are in STATE_CLOSED, fd must be -1

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: make sure worker started before let msgr ready

When we create event thread, it need a little time to enter event loop(like
calling set_owner), if caller is going to call create_file_event before event
thread enter event loop, it will trigger assert.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: make EventCenter notify file event creating when set_owner

EventCenter::init is called by other thread instead of event thread, so we
need to move create_file_event to set_owner which is called by event thread.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: swap eventcenter when replacing

Previously we only exchange fd when replacing, now we will introduce dpdk
plugin in the near future. It needs all fd used locally which not like
kernel socket shared by all cores.

So we need to add EventCenter swapping to let each socket is associated to
EventCenter.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10153 from tchaikov/wip-no-copy-plugins

test: test_{compression_plugin,async_compressor}: do not copy plugins

Reviewed-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: fix delete_time_event leak

Signed-off-by: Haomai Wang <haomai@xsky.com>

AsyncMessenger: make create/delete_file_event within event thread

We are make each AsyncConnection/AsyncMessenger only modify its file event
in event thread. So make sure create/delete_file_event aren't directly called.

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10174 from theanalyst/test/rgw/multisite-cleanup

rgw: test_multi.py avoid creating mds

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10211 from oritwas/wip-rgw-fix-zone-modify

rgw: remove unused realm from radosgw-admin zone modify

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10256 from ceph/wip-kill-ceph_test_rados_api_tmap_migrate

test: remove ceph_test_rados_api_tmap_migrate

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10229 from badone/wip-libgmock_main.la-build-failures

makefile: change librgw_file_* as check_PROGRAMS

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: remove ceph_test_rados_api_tmap_migrate

see also: http://tracker.ceph.com/issues/16144

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: include local pool id in resync throttle unique key

Fixes: http://tracker.ceph.com/issues/16536
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10252 from jdurgin/wip-rgw-build

rgw: fix compilation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix compilation

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10205 from oritwas/wip-rgw-fix-upgrade

rgw: add missing master_zone when running with old default region config

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10040 from xiexingguo/xxg-wip-bit-alloc-tree-configurable

os/bluestore: make zone/span size of bitmap-allocator configurable

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10075 from cbodley/wip-rgw-tenant-sync

rgw: add tenant support to multisite sync

Merge pull request #8964 from songbaisen/song24

global: log which process/command sent a signal

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10195 from stiopaa1/msg_asyncmess_removeIncFile

msg/AsyncMessenger: remove unneeded include file

Merge pull request #10224 from stiopaa1/msg_asyncmess_event_addConst

msg/async/Event.h: add const to member function

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10230 from stiopaa1/msg_asyncmess_Processor_accept_changeToVoid

msg/AsyncMessenger: change return type to void

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10152 from tchaikov/wip-16277

osd: increment stats on recovery pull also

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10222 from ceph/wip-cmake

cmake changes for pypi

verfied f23

Event: no need to delete_file_Event when deconstruct

Since we are going to close all epoll and cleanup resources, no need to delete
notify fd resource. And another reason is "delete_file_event" doesn't expect
other threads to call

Signed-off-by: Haomai Wang <haomai@xsky.com>

async: change set_socket_option caller

Signed-off-by: Haomai Wang <haomai@xsky.com>

net_handler: adjust set_socket_options to avoid read from conf

We don't want net_handler rely on config value, caller may expect to pass
different value to set_socket_options according to different socket type
like heartbeat, client socket or server socket.

Signed-off-by: Haomai Wang <haomai@xsky.com>

cmake: fix the detection of curl_multi_wait()

it was broken by b5bb77d5

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: add health warning for oversized cache

Various issues can lead to an oversized cache, which
is a precursor to an OOM condition: let's give users
a clearer message with some useful counts that might
hint at what is wrong.

Fixes: http://tracker.ceph.com/issues/16570
Signed-off-by: John Spray <john.spray@redhat.com>

pybind/rados: remove rados.c from MANIFEST.in

this fixes the warning of

```
running egg_info
creating
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info
writing
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/PKG-INFO
writing top-level names to
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/top_level.txt
writing dependency_links to
/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/dependency_links.txt
writing manifest file
'/srv/autobuild-ceph/gitbuilder.git/build/build/src/pybind/rados/rados.egg-info/SOURCES.txt'
warning: no files found matching 'rados.c'
```

in the out-of-source tree build, rados.c is not located in the same
directory of setup.py, rados.pyx, and rados.pyd. it is in the build_dir,
for example, it could be:

  ceph/build/src/pybind/rados/pyrex/rados.c

where ceph/build is the build directory of cmake.
and sdist will include it when running egg_info, because cython's
build_ext module lists `rados.c` as one of the "source_files".

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass necessary cflags to build_ext

this fixes:
```
creating
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados
creating
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados/pyrex
/usr/bin/gcc-6 -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
-fno-strict-aliasing -iquote /var/ceph/ceph/src/include -fPIC
-I/usr/include/python2.7 -I/usr/include/x86_64-linux-
gnu/python2.7 -I/usr/include/python2.7 -c
/home/kefu/ceph/build/src/pybind/rados/pyrex/rados.c -o
/home/kefu/ceph/build/src/pybind/rados/var/ceph/ceph/build/src/pybind/rados/pyrex/ra
dos.o -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g
-fstack-protector-strong -Wformat -Werror=format-security -DNDEBUG -g
-fwrapv -O2 -Wall -Wstrict-prototypes -lpython2.
7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1
-Wl,-Bsymbolic-functions
/home/kefu/ceph/build/src/pybind/rados/pyrex/rados.c:239:28: fatal error:
rados/librados.h: No such file or directory
 #include "rados/librados.h"
                            ^
compilation terminated.
```

seems we can not pass the CFLAGS with space in it to setup.py using env variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: put cythonized .c files into ${CMAKE_CURRENT_BINARY_DIR}

it was put into the directory where .pyx is located. it's not expected
in an out-of-tree build.
* {rados,rbd,cephfs}/setup.py: use build_ext from cython if possible, fallback to
  the one from setuptools or distutils
* Distutils.cmake: pass --cython-c-in-temp --build-temp and
  --cython-include-dirs to "build_ext" command

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9138 from cxwshawn/wip-cl-fix

osd: replace ceph:atomic_t with std::atomic in osd module.

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9871 from odivlad/fix-init-el7-master

master: remove SYSTEMD_RUN from initscript

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8317 from dreamhost/gentoo-python

Gentoo support for ceph-disk / ceph-detect-init; pip speedup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9085 from renhwsky/renhw-wip-osdmonitor-updatepool

osd/osdmonitor: pool of objects and bytes beyond quota should all be warn

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10234 from tchaikov/wip-drop-tmap_upgrade

ceph_test_rados_api_tmap_migrate: remove test for tmap_upgrade

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9117 from wjin/clean

osd: remove duplicated function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9252 from vumrao/wip-vumrao-15983

mon: change osdmap flags set and unset messages

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9452 from mslovy/wip-preserve-alloc-hint

osd:preserve allocation hint attribute during recovery

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9698 from LiumxNL/fix-clones-flush-evict

rados: add option to include clones when doing flush or evict

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9703 from badone/wip-cryptoKey-encrypt-segfault

cephx: Fix multiple segfaults due to attempts to encrypt or decrypt

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9717 from songbaisen/a7

mon: do not recalculate 'to_remove' when it's known

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9763 from dx9/wip-pthread-np

compat: add abstractions for non portable pthread name funcs

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph_test_rados_api_tmap_migrate: remove test for tmap_upgrade

the tested feature was removed in ad6cc41

> tmap_upgrade was only for Jewel, so remove it for Kraken.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/AsyncMessenger: change return type to void

Change the return type of function Processor::start to void. It
doesn't make sense to return constant value(zero).

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

os/bluestore: add support for allocating 0-sized extent

For testing, we may choose randomized length of extent to allocate,
which means we could trim down the length to 0.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: fix allocation of discontiguous blocks

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: add sanity check for get_free() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: simplify is_exhausted() method

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore: make zone/span size of bitmap-allocator configurable

Perviously as zone_size_block and span_size are identical, so we could use
span_size only to simplify the calcuation.

Now that the zone_size_block and span_size are both configurable, so
it is necessary to treat them respectively.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10190 from ktdreyer/doc-delete-pool

doc: more details for pool deletion

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

makefile: change librgw_file_* as check_PROGRAMS

These targets depend on libgmock_main.la which is not built during
a default build.

Fixes: http://tracker.ceph.com/issues/16646
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>