mgr/DaemonServer: set messenger policy; set throttles

Put clients, osds, mons, and mdss in different buckets.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: move most PGMonitor commands into PGMap.{cc} helper

We don't need private access to PGMap members, so a simple
function will do.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: move dump_stuck_pg_stats from PGMonitor -> PGMap

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: constify a few things

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/ceph_argparse: send MGR-flagged commands to ceph-mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: add generic Formatter for handle_command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: include MgrCommands.h in mgr_commands

Signed-off-by: Sage Weil <sage@redhat.com>

mon,mgr: move "pg ..." commands to MgrCommands, but keep #including on mon

These will become mgr commands.  Move them untouched to
MgrCommands.h, but for now include them in the mon_commands with
the new MGR flag on them.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: hide the MGR flag on mon commands until require_luminous is set

Signed-off-by: Sage Weil <sage@redhat.com>

mon: add a MGR flag to mon commands

The flag will indicate that a command should be sent to ceph-mgr
instead of ceph-mon (rados_mgr_command instead of
rados_mon_command).

For legacy clients, we'll have to make the mon forward the message
on behalf of the client.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14122 from hrchu/rgwadmin-doc-quota

doc: correct the quota section

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #14078 from tchaikov/wip-19258

mgr: shutdown py_modules in Mgr::shutdown()

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13337 from ifed01/wip-bluestore-minimiza-blobs2

os/bluestore: optimize blob usage when doing appends/overwrites

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14223 from tchaikov/wip-rbd-nbd-ftbfs

rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

mgr: shutdown py_modules in Mgr::shutdown()

Fixes: http://tracker.ceph.com/issues/19258
Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-nbd: s/cpp_error/cpp_strerror/ to fix FTBFS

the build failure was introduced by ff4dcf0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14079 from liupan1111/wip-support-signal-handler

rbd-nbd: support signal handle for SIGHUP, SIGINT and SIGTERM.

Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14184 from SUSE/wip-mgr-rest-timezone

pybind/mgr/rest: don't set timezone to Chicago

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14199 from andihit/doc-update-default-osd-op-thread-timeout

doc: change osd_op_thread_timeout default value to 15

Reviewed-by: Sage Weil <sage@redhat.com>

doc: rgw: correct the quota section

Add the missing option and fix typo.

Fixes: http://tracker.ceph.com/issues/19397
Signed-off-by: Chu, Hua-Rong <hrchu@cht.com.tw>

Merge pull request #14203 from cbodley/wip-rgw-slo-auth-identity

rgw: fix build of conflict after auth rework

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix build of conflict after auth rework

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13764 from guihecheng/rgwfs-errcode

rgw_file: fix non-posix errcode EINVAL to ENAMETOOLONG

Merge pull request #14108 from sven15/doc-fix-pools

doc: typo in hit_set_search_last_n

Reviewed-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #14128 from yehudasa/wip-19367

cls/user: cls_user_bucket backward compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13987 from shinobu-x/wp-sk-clean-up-timeindexlistctx-20170316

cls/timeindex: clean up cls_timeindex_client.h|cc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14009 from yuyuyu101/wip-objecter-ping

msg/async/AsyncConnection: keepalive objecter ping connection to avoid timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14133 from runsisi/wip-pg-state-cleanup

osd/PG: some minor cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14089 from ceph/wip-rgw-doc-changing-master

doc/rgw: instructions for changing multisite master zone

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14090 from cbodley/wip-rgw-test-delete-notempty

test/rgw: test_bucket_delete_notempty in test_multi.py

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13561 from cbodley/wip-19027

rgw: allow system users to read SLO parts

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: change osd_op_thread_timeout default value to 15

Signed-off-by: Andreas Gerstmayr <andreas.gerstmayr@catalysts.cc>

Merge pull request #13984 from liewegas/wip-osdmap-remap

mon,osd: explicitly remap some pgs

Reviewed-by: Josh Durgin <jduring@redhat.com>

Merge pull request #13687 from Liuchang0812/wip-19106-eliminate-unsafe-option

common/config: eliminate config_t::set_val unsafe option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13855 from yuyuyu101/wip-bluestore-rm-range

kv/RocksDBStore: implement rm_range_keys operator interface and test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13879 from liewegas/wip-19076

osd/ReplicatedBackend: clear pull source once we are done with it

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13961 from wangzhengyong/network

msg/SimpleMessenger: error out misplace in set_socket_options

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14093 from hjwsm1989/sparse-read

osd/PrimaryLogPG: set return value if sparse read failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14099 from tchaikov/wip-denc-arm32

include/denc: add {encode,decode}_nohead for denc_traits<basic_string>

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14129 from SUSE/wip-18764

mgr: fix crash on missing 'ceph_version' in daemon metadata (fixes #18764)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14003 from dillaman/wip-19287

librbd: is_exclusive_lock_owner API should ping OSD

Reviewed-by: Mike Christie <mchristi@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14179 from Adirl/rdma-cm-3

msg/async/rdma: RDMA-CM Initialize device on first connect

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14166 from dillaman/wip-19388

librbd: random unit test failures due to shut down race

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/tasks/thrashosds,ceph_manager: thrash pg_remap[_items]

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool/remap.t: add --remap test

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: add --remap to run the remap_pgs() method

Output ceph cli commands to realize the new remaps entries.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: clean_pg_remaps

Helper to clean out no-op pg_remap cruft:

- pg_remap entries that match the raw result
- pg_remap_items entries where the 'from' part of the (from,to)
  set isn't present.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: implement remap_pgs

Run a specified number of iterations trying to install new
pg_remap_items entries that improve the PG distribution.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: implement try_remap_rule

Simulate a CRUSH mapping but try to identify alternative OSD
choices (based on an underfull list and overfull set) that still
respect the CRUSH rule constraints.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: allow 'single step' blob reuse when overwriting

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

test/store_test: adjust OnodeSizeTracking test case to measure full object metadata using empty Onode as a basis

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: remove excessive 0x in logging

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: add test cases for BlueStore blob reuse.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: try to reuse blob rather than create new one on overwrite.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #14021 from batrick/i19291

mds: set ceph-mds name uncond for external tools

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14017 from jcsp/wip-19282

mds: don't assert on read errors in RecoveryQueue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14015 from jcsp/wip-19288

mds: remove legacy "mds tell" command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #13904 from jcsp/wip-16709

mon: fix hiding mdsmonitor informative strings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12074 from batrick/i17604

mds: warn if insufficient standbys exist

Reviewed-by: John Spray <john.spray@redhat.com>

pybind/mgr/rest: don't set timezone to Chicago

Setting TIME_ZONE in the Django app causes the timestamps printed in
the mgr log to suddenly be wrong after the rest module starts up
(unless, I imagine, if the host happens to be in Chicago).

Signed-off-by: Tim Serong <tserong@suse.com>

rbd-nbd: polish the output info before and after ioctl NBD_DISCONNECT.

Signed-off-by: Pan Liu <liupan1111@gmail.com>

rbd-nbd: support signal handle for SIGHUP, SIGINT, and SIGTERM.

Fixes: http://tracker.ceph.com/issues/19349
Signed-off-by: Pan Liu <liupan1111@gmail.com>

msg/async/rdma: Extract sockets stuff from RDMAStack.h

This is a preparation commit, in order to make review easier. In this
commit I move code from RDMAStack.h into the new file
RDMAConnectedSocketImpl.h - without changing the code.

In the next commit, the actual logic changes will be done and socket
classes will be split into a base RDMAConnected classes and child
classes with TCP connection establishment specific code.

Issue: 995322
Change-Id: I639fda490a6fbd02addb95d3158c5ac1e7390ef0
Signed-off-by: Amir Vadai <amir@vadai.me>

msg/async/rdma: Initialize device on first connect

Allocate Device's IB resources only when first connect for a device is
called.

Also, removed Infiniband::MemoryManager::Chunk::post_srq() which is not
used anywhere in the code.

Issue: 995322
Change-Id: I2ac1e36aff645ad7d8bbc06c87530bf33c4b3ecf
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #13423 from dzafman/wip-18533-tool

Fixes: http://tracker.ceph.com/issues/18533
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14132 from dmick/wip-arm-crc

arch: fix cmake's ARM CRC intrinsics test to handle duplicitous gcc 4.8.5

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake/modules/SIMDExt.cmake: armv8 crypto intrinsics

Test not only for -march support, but also the actual
presence of the intrinsic routines.  Not sure why, but gcc
4.8.5 passes the first but not the second.

Fixes: http://tracker.ceph.com/issues/19386
Signed-off-by: Dan Mick <dan.mick@redhat.com>

librbd: random unit test failures due to shut down race

Fixes: http://tracker.ceph.com/issues/19389
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

filestore, tools: Fix logging of DBObjectMap check() repairs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Simplify DBObjectMap by no longer creating complete tables

Bump the version for new maps to 3
Make clone less efficient but simpler
Add rename operation (use instead of clone/unlink)
For now keep code that understands version 2 maps

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix seg fault with large amount of check error output

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add automatic repair for DBObjectMap bug

Add repair command to ceph-osdomap-tool too

Under some situations the previous rm_keys() code would
generated a corrupt complete table.  There is no way to
figure out what the table should look like now.  By removing
the entries we fix the corruption and aren't much worse off
because the corruption caused some deleted keys to re-appear.

This doesn't breaking the parent/child relationship during
repair because some of the keys may still be contained
in the parent.

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix tool exit status

Signed-off-by: David Zafman <dzafman@redhat.com>

DBObjectMap: rewrite rm_keys and merge_new_complete

Leverage the updated in_complete_region and needs_parent to simplify
these methods.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: strengthen in_complete_region post condition

Previously, in_complete_region didn't guarantee anything about
where it left complete_iter pointing.  It will be handy for
complete_iter to be pointing at the lowest interval which ends
after to_test.  Make it so.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: fix next_parent()

The previous implementation assumed that
lower_bound(parent_iter->key()) always leaves the iterator
on_parent().  There isn't any guarantee, however, that that
key isn't present on the child as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

test_object_map: add tests to trigger some bugs related to 18533

Signed-off-by: Samuel Just <sjust@redhat.com>

test: Add ceph_test_object_map to make check tests

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Add --debug and only show internal logging if enabled

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: DBOjectMap::check: Dump complete mapping when inconsistency found

Signed-off-by: David Zafman <dzafman@redhat.com>

test_object_map: Use ASSERT_EQ() for check() so failure doesn't stop testing

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check for overlaps in internal "complete" table for DBObjectMap

Changed check to return an error count and fix tool error message

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add dump-headers command to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add --oid option to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Remove unnecessary assert and assignment in DBObjectMap

Fix and add comment(s)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #13754 from liewegas/wip-objectstore-random

os: allow 'osd objectstore = random' to pick either filestore or bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: remove unuse perfcounter:l_bluestore_read_wait_flush_lat.

Now read don't need flush(). See commit 83e33a32fde7f83.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14088 from Adirl/rdma-cm-2

msg/async/rdma: Move resource handling to Device

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13915 from jcsp/wip-19253

qa: whitelist new fullness messages in fs tests

Merge pull request #14159 from tchaikov/wip-rbd-size_t-is-not-uint64_t-on-arm32

librbd: pass an uint64_t to clip_io() as the third param

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: do garbage collection after main write path to be able to call set_lextent from _do_alloc_write (will be required later for blob alignment).

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: make punch_hole call from set_lextent optional.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: refactor bluestore_blob_t to encapsulate extents/*length fields.

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

os/bluestore: store blob logical length rather than recalculate it on pextent vector

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>

librbd: pass an uint64_t to clip_io() as the 3rd param

because, on arm32, size_t is "unsigned", while uint64_t is "long long
unsigned int", compiler is not able to promote the former to the latter
automatically. and ImageRequest::Extents::value_type::second_type is
indeed uint64_t, which is expected by clip_io(). so let's just use
"auto" here.

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10412 from stiopaa1/mon_monitorh_addConstToGetLeader

mon/Monitor.h: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14146 from liupan1111/wip-fix-comment

rbd-nbd: clean up the doc and help information

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

kv/rocksdb: add option to control whether enable DeleteRange API

Signed-off-by: Haomai Wang <haomai@xsky.com>

kv/RocksDBStore: implement rm_range_keys operator interface and test

Signed-off-by: Haomai Wang <haomai@xsky.com>