msg/async/rdma: Device::last_poll_dev must be positive

Make Device::last_poll_dev `unsigned` - it could overlap and should not
be negative.
Also, add missing initialization.

Change-Id: I558f02292ae4f5d1ce48d9bcda5047d382604237
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #14245 from tchaikov/wip-revert-14201

Revert "msg/async/rdma: Debug prints for ibv_*"

Revert "msg/async/rdma: Debug prints for ibv_*"

This reverts commit 7d1475f6274531988854cf62f6056f9cfd13ff29.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14201 from Adirl/rdma-cm-4

msg/async/rdma: Introduce RDMAConnMgr + Debug prints

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14214 from dmick/wip-workunit-shallow-clone

tasks/workunit.py: when cloning, use --depth=1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14202 from tchaikov/wip-cast-size_t-arm32

rbd: use min<uint64_t>() explicitly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14122 from hrchu/rgwadmin-doc-quota

doc: correct the quota section

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

msg/async/rdma: Debug prints for ibv_*

Setting RDMA_DEBUG to 1, will enable debug prints for every ibv_*
function called.
This makes it easier to debug RDMA issues.

Issue: 995322
Change-Id: I49d327d5d4386b44264f5619d50f2dbc7d76dcae
Signed-off-by: Amir Vadai <amir@vadai.me>

msg/async/rdma: Introduce RDMAConnMgr

Encapsulate all connection establishment stuff in a new class -
RDMAConnMgr and make it a friend class of RDMAConnectedSocketImpl.
This class will be inherited for every type of connection establishment
- Currently only TCP is supported, very soon CM will be added too.

RDMAServerConnImpl which only handle connection establishment became an
abstract class and RDMAServerConnTCP is inherting it for connections of
type TCP.

Some of the code was left in its original file and place, and therefore
it looks misplaced. This was done to make it easier to review and rebase.
Once it is accepted a cleanup patch will be sent to move the code into
the right place.

Issue: 995322
Change-Id: I8b0e163525ec80c2452f4b6481bf696968cc1e51
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #14078 from tchaikov/wip-19258

mgr: shutdown py_modules in Mgr::shutdown()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13337 from ifed01/wip-bluestore-minimiza-blobs2

os/bluestore: optimize blob usage when doing appends/overwrites

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14223 from tchaikov/wip-rbd-nbd-ftbfs

rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

mgr: shutdown py_modules in Mgr::shutdown()

Fixes: http://tracker.ceph.com/issues/19258
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS

the build failure was introduced by ff4dcf0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14079 from liupan1111/wip-support-signal-handler

rbd-nbd: support signal handle for SIGHUP, SIGINT and SIGTERM.

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

tasks/workunit.py: when cloning, use --depth=1

Help avoid killing git.ceph.com.  A depth 1 clone takes about
7 seconds, whereas a full one takes about 3:40 (much of it
waiting for the server to create a huge compressed pack)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #14184 from SUSE/wip-mgr-rest-timezone

pybind/mgr/rest: don't set timezone to Chicago

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14199 from andihit/doc-update-default-osd-op-thread-timeout

doc: change osd_op_thread_timeout default value to 15

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rgw: correct the quota section

Add the missing option and fix typo.

Fixes: http://tracker.ceph.com/issues/19397
Signed-off-by: Chu, Hua-Rong <hrchu@cht.com.tw>

Merge pull request #14203 from cbodley/wip-rgw-slo-auth-identity

rgw: fix build of conflict after auth rework

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix build of conflict after auth rework

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13764 from guihecheng/rgwfs-errcode

rgw_file: fix non-posix errcode EINVAL to ENAMETOOLONG

rbd: use min<uint64_t>() explicitly

on arm32, size_t is actually int, which cannot be compared with uint64_t
using std::min().

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14108 from sven15/doc-fix-pools

doc: typo in hit_set_search_last_n

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #14128 from yehudasa/wip-19367

cls/user: cls_user_bucket backward compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13987 from shinobu-x/wp-sk-clean-up-timeindexlistctx-20170316

cls/timeindex: clean up cls_timeindex_client.h|cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14009 from yuyuyu101/wip-objecter-ping

msg/async/AsyncConnection: keepalive objecter ping connection to avoid timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14133 from runsisi/wip-pg-state-cleanup

osd/PG: some minor cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14089 from ceph/wip-rgw-doc-changing-master

doc/rgw: instructions for changing multisite master zone

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14090 from cbodley/wip-rgw-test-delete-notempty

test/rgw: test_bucket_delete_notempty in test_multi.py

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13561 from cbodley/wip-19027

rgw: allow system users to read SLO parts

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: change osd_op_thread_timeout default value to 15

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@catalysts.cc>

Merge pull request #13984 from liewegas/wip-osdmap-remap

mon,osd: explicitly remap some pgs

Reviewed-by: Josh Durgin <jduring@redhat.com>

Merge pull request #13687 from Liuchang0812/wip-19106-eliminate-unsafe-option

common/config: eliminate config_t::set_val unsafe option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13855 from yuyuyu101/wip-bluestore-rm-range

kv/RocksDBStore: implement rm_range_keys operator interface and test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13879 from liewegas/wip-19076

osd/ReplicatedBackend: clear pull source once we are done with it

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13961 from wangzhengyong/network

msg/SimpleMessenger: error out misplace in set_socket_options

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14093 from hjwsm1989/sparse-read

osd/PrimaryLogPG: set return value if sparse read failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14099 from tchaikov/wip-denc-arm32

include/denc: add {encode,decode}_nohead for denc_traits<basic_string>

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14129 from SUSE/wip-18764

mgr: fix crash on missing 'ceph_version' in daemon metadata (fixes #18764)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14003 from dillaman/wip-19287

librbd: is_exclusive_lock_owner API should ping OSD

Reviewed-by: Mike Christie <mchristi@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14179 from Adirl/rdma-cm-3

msg/async/rdma: RDMA-CM Initialize device on first connect

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14166 from dillaman/wip-19388

librbd: random unit test failures due to shut down race

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/tasks/thrashosds,ceph_manager: thrash pg_remap[_items]

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool/remap.t: add --remap test

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: add --remap to run the remap_pgs() method

Output ceph cli commands to realize the new remaps entries.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: clean_pg_remaps

Helper to clean out no-op pg_remap cruft:

- pg_remap entries that match the raw result
- pg_remap_items entries where the 'from' part of the (from,to)
  set isn't present.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: implement remap_pgs

Run a specified number of iterations trying to install new
pg_remap_items entries that improve the PG distribution.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: implement try_remap_rule

Simulate a CRUSH mapping but try to identify alternative OSD
choices (based on an underfull list and overfull set) that still
respect the CRUSH rule constraints.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow 'single step' blob reuse when overwriting

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/store_test: adjust OnodeSizeTracking test case to measure full object metadata using empty Onode as a basis

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: remove excessive 0x in logging

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add test cases for BlueStore blob reuse.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: try to reuse blob rather than create new one on overwrite.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14021 from batrick/i19291

mds: set ceph-mds name uncond for external tools

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14017 from jcsp/wip-19282

mds: don't assert on read errors in RecoveryQueue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14015 from jcsp/wip-19288

mds: remove legacy "mds tell" command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #13904 from jcsp/wip-16709

mon: fix hiding mdsmonitor informative strings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12074 from batrick/i17604

mds: warn if insufficient standbys exist

Reviewed-by: John Spray <john.spray@redhat.com>

pybind/mgr/rest: don't set timezone to Chicago

Setting TIME_ZONE in the Django app causes the timestamps printed in
the mgr log to suddenly be wrong after the rest module starts up
(unless, I imagine, if the host happens to be in Chicago).

Signed-off-by: Tim Serong <tserong@suse.com>

rbd-nbd: polish the output info before and after ioctl NBD_DISCONNECT.

Signed-off-by: Pan Liu <liupan1111@gmail.com>

rbd-nbd: support signal handle for SIGHUP, SIGINT, and SIGTERM.

Fixes: http://tracker.ceph.com/issues/19349
Signed-off-by: Pan Liu <liupan1111@gmail.com>

msg/async/rdma: Extract sockets stuff from RDMAStack.h

This is a preparation commit, in order to make review easier. In this
commit I move code from RDMAStack.h into the new file
RDMAConnectedSocketImpl.h - without changing the code.

In the next commit, the actual logic changes will be done and socket
classes will be split into a base RDMAConnected classes and child
classes with TCP connection establishment specific code.

Issue: 995322
Change-Id: I639fda490a6fbd02addb95d3158c5ac1e7390ef0
Signed-off-by: Amir Vadai <amir@vadai.me>

msg/async/rdma: Initialize device on first connect

Allocate Device's IB resources only when first connect for a device is
called.

Also, removed Infiniband::MemoryManager::Chunk::post_srq() which is not
used anywhere in the code.

Issue: 995322
Change-Id: I2ac1e36aff645ad7d8bbc06c87530bf33c4b3ecf
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #13423 from dzafman/wip-18533-tool

Fixes: http://tracker.ceph.com/issues/18533
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14132 from dmick/wip-arm-crc

arch: fix cmake's ARM CRC intrinsics test to handle duplicitous gcc 4.8.5

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake/modules/SIMDExt.cmake: armv8 crypto intrinsics

Test not only for -march support, but also the actual
presence of the intrinsic routines.  Not sure why, but gcc
4.8.5 passes the first but not the second.

Fixes: http://tracker.ceph.com/issues/19386
Signed-off-by: Dan Mick <dan.mick@redhat.com>

librbd: random unit test failures due to shut down race

Fixes: http://tracker.ceph.com/issues/19389
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

filestore, tools: Fix logging of DBObjectMap check() repairs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Simplify DBObjectMap by no longer creating complete tables

Bump the version for new maps to 3
Make clone less efficient but simpler
Add rename operation (use instead of clone/unlink)
For now keep code that understands version 2 maps

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix seg fault with large amount of check error output

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add automatic repair for DBObjectMap bug

Add repair command to ceph-osdomap-tool too

Under some situations the previous rm_keys() code would
generated a corrupt complete table.  There is no way to
figure out what the table should look like now.  By removing
the entries we fix the corruption and aren't much worse off
because the corruption caused some deleted keys to re-appear.

This doesn't breaking the parent/child relationship during
repair because some of the keys may still be contained
in the parent.

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix tool exit status

Signed-off-by: David Zafman <dzafman@redhat.com>

DBObjectMap: rewrite rm_keys and merge_new_complete

Leverage the updated in_complete_region and needs_parent to simplify
these methods.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: strengthen in_complete_region post condition

Previously, in_complete_region didn't guarantee anything about
where it left complete_iter pointing.  It will be handy for
complete_iter to be pointing at the lowest interval which ends
after to_test.  Make it so.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: fix next_parent()

The previous implementation assumed that
lower_bound(parent_iter->key()) always leaves the iterator
on_parent().  There isn't any guarantee, however, that that
key isn't present on the child as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

test_object_map: add tests to trigger some bugs related to 18533

Signed-off-by: Samuel Just <sjust@redhat.com>

test: Add ceph_test_object_map to make check tests

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Add --debug and only show internal logging if enabled

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: DBOjectMap::check: Dump complete mapping when inconsistency found

Signed-off-by: David Zafman <dzafman@redhat.com>

test_object_map: Use ASSERT_EQ() for check() so failure doesn't stop testing

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check for overlaps in internal "complete" table for DBObjectMap

Changed check to return an error count and fix tool error message

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add dump-headers command to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add --oid option to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Remove unnecessary assert and assignment in DBObjectMap

Fix and add comment(s)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #13754 from liewegas/wip-objectstore-random

os: allow 'osd objectstore = random' to pick either filestore or bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: remove unuse perfcounter:l_bluestore_read_wait_flush_lat.

Now read don't need flush(). See commit 83e33a32fde7f83.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14088 from Adirl/rdma-cm-2

msg/async/rdma: Move resource handling to Device

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13915 from jcsp/wip-19253

qa: whitelist new fullness messages in fs tests

Merge pull request #14159 from tchaikov/wip-rbd-size_t-is-not-uint64_t-on-arm32

librbd: pass an uint64_t to clip_io() as the third param

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: do garbage collection after main write path to be able to call set_lextent from _do_alloc_write (will be required later for blob alignment).

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: make punch_hole call from set_lextent optional.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: refactor bluestore_blob_t to encapsulate extents/*length fields.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: store blob logical length rather than recalculate it on pextent vector

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

librbd: pass an uint64_t to clip_io() as the 3rd param

because, on arm32, size_t is "unsigned", while uint64_t is "long long
unsigned int", compiler is not able to promote the former to the latter
automatically. and ImageRequest::Extents::value_type::second_type is
indeed uint64_t, which is expected by clip_io(). so let's just use
"auto" here.

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10412 from stiopaa1/mon_monitorh_addConstToGetLeader

mon/Monitor.h: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14146 from liupan1111/wip-fix-comment

rbd-nbd: clean up the doc and help information

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

kv/rocksdb: add option to control whether enable DeleteRange API

Signed-off-by: Haomai Wang <haomai@xsky.com>

kv/RocksDBStore: implement rm_range_keys operator interface and test

Signed-off-by: Haomai Wang <haomai@xsky.com>