Merge pull request #14133 from runsisi/wip-pg-state-cleanup

osd/PG: some minor cleanups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #14089 from ceph/wip-rgw-doc-changing-master

doc/rgw: instructions for changing multisite master zone

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #14090 from cbodley/wip-rgw-test-delete-notempty

test/rgw: test_bucket_delete_notempty in test_multi.py

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13561 from cbodley/wip-19027

rgw: allow system users to read SLO parts

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13984 from liewegas/wip-osdmap-remap

mon,osd: explicitly remap some pgs

Reviewed-by: Josh Durgin <jduring@redhat.com>

Merge pull request #13687 from Liuchang0812/wip-19106-eliminate-unsafe-option

common/config: eliminate config_t::set_val unsafe option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13855 from yuyuyu101/wip-bluestore-rm-range

kv/RocksDBStore: implement rm_range_keys operator interface and test

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13879 from liewegas/wip-19076

osd/ReplicatedBackend: clear pull source once we are done with it

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13961 from wangzhengyong/network

msg/SimpleMessenger: error out misplace in set_socket_options

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #14093 from hjwsm1989/sparse-read

osd/PrimaryLogPG: set return value if sparse read failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14099 from tchaikov/wip-denc-arm32

include/denc: add {encode,decode}_nohead for denc_traits<basic_string>

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14129 from SUSE/wip-18764

mgr: fix crash on missing 'ceph_version' in daemon metadata (fixes #18764)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14003 from dillaman/wip-19287

librbd: is_exclusive_lock_owner API should ping OSD

Reviewed-by: Mike Christie <mchristi@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14179 from Adirl/rdma-cm-3

msg/async/rdma: RDMA-CM Initialize device on first connect

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14166 from dillaman/wip-19388

librbd: random unit test failures due to shut down race

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/tasks/thrashosds,ceph_manager: thrash pg_remap[_items]

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool/remap.t: add --remap test

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli/osdmaptool: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

osdmaptool: add --remap to run the remap_pgs() method

Output ceph cli commands to realize the new remaps entries.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: clean_pg_remaps

Helper to clean out no-op pg_remap cruft:

- pg_remap entries that match the raw result
- pg_remap_items entries where the 'from' part of the (from,to)
  set isn't present.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: implement remap_pgs

Run a specified number of iterations trying to install new
pg_remap_items entries that improve the PG distribution.

Signed-off-by: Sage Weil <sage@redhat.com>

crush: implement try_remap_rule

Simulate a CRUSH mapping but try to identify alternative OSD
choices (based on an underfull list and overfull set) that still
respect the CRUSH rule constraints.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #14021 from batrick/i19291

mds: set ceph-mds name uncond for external tools

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14017 from jcsp/wip-19282

mds: don't assert on read errors in RecoveryQueue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14015 from jcsp/wip-19288

mds: remove legacy "mds tell" command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #13904 from jcsp/wip-16709

mon: fix hiding mdsmonitor informative strings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12074 from batrick/i17604

mds: warn if insufficient standbys exist

Reviewed-by: John Spray <john.spray@redhat.com>

msg/async/rdma: Extract sockets stuff from RDMAStack.h

This is a preparation commit, in order to make review easier. In this
commit I move code from RDMAStack.h into the new file
RDMAConnectedSocketImpl.h - without changing the code.

In the next commit, the actual logic changes will be done and socket
classes will be split into a base RDMAConnected classes and child
classes with TCP connection establishment specific code.

Issue: 995322
Change-Id: I639fda490a6fbd02addb95d3158c5ac1e7390ef0
Signed-off-by: Amir Vadai <amir@vadai.me>

msg/async/rdma: Initialize device on first connect

Allocate Device's IB resources only when first connect for a device is
called.

Also, removed Infiniband::MemoryManager::Chunk::post_srq() which is not
used anywhere in the code.

Issue: 995322
Change-Id: I2ac1e36aff645ad7d8bbc06c87530bf33c4b3ecf
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #13423 from dzafman/wip-18533-tool

Fixes: http://tracker.ceph.com/issues/18533
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14132 from dmick/wip-arm-crc

arch: fix cmake's ARM CRC intrinsics test to handle duplicitous gcc 4.8.5

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake/modules/SIMDExt.cmake: armv8 crypto intrinsics

Test not only for -march support, but also the actual
presence of the intrinsic routines.  Not sure why, but gcc
4.8.5 passes the first but not the second.

Fixes: http://tracker.ceph.com/issues/19386
Signed-off-by: Dan Mick <dan.mick@redhat.com>

librbd: random unit test failures due to shut down race

Fixes: http://tracker.ceph.com/issues/19389
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

filestore, tools: Fix logging of DBObjectMap check() repairs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Simplify DBObjectMap by no longer creating complete tables

Bump the version for new maps to 3
Make clone less efficient but simpler
Add rename operation (use instead of clone/unlink)
For now keep code that understands version 2 maps

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix seg fault with large amount of check error output

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add automatic repair for DBObjectMap bug

Add repair command to ceph-osdomap-tool too

Under some situations the previous rm_keys() code would
generated a corrupt complete table.  There is no way to
figure out what the table should look like now.  By removing
the entries we fix the corruption and aren't much worse off
because the corruption caused some deleted keys to re-appear.

This doesn't breaking the parent/child relationship during
repair because some of the keys may still be contained
in the parent.

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix tool exit status

Signed-off-by: David Zafman <dzafman@redhat.com>

DBObjectMap: rewrite rm_keys and merge_new_complete

Leverage the updated in_complete_region and needs_parent to simplify
these methods.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: strengthen in_complete_region post condition

Previously, in_complete_region didn't guarantee anything about
where it left complete_iter pointing.  It will be handy for
complete_iter to be pointing at the lowest interval which ends
after to_test.  Make it so.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: fix next_parent()

The previous implementation assumed that
lower_bound(parent_iter->key()) always leaves the iterator
on_parent().  There isn't any guarantee, however, that that
key isn't present on the child as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

test_object_map: add tests to trigger some bugs related to 18533

Signed-off-by: Samuel Just <sjust@redhat.com>

test: Add ceph_test_object_map to make check tests

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Add --debug and only show internal logging if enabled

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: DBOjectMap::check: Dump complete mapping when inconsistency found

Signed-off-by: David Zafman <dzafman@redhat.com>

test_object_map: Use ASSERT_EQ() for check() so failure doesn't stop testing

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check for overlaps in internal "complete" table for DBObjectMap

Changed check to return an error count and fix tool error message

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add dump-headers command to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add --oid option to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Remove unnecessary assert and assignment in DBObjectMap

Fix and add comment(s)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #13754 from liewegas/wip-objectstore-random

os: allow 'osd objectstore = random' to pick either filestore or bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: remove unuse perfcounter:l_bluestore_read_wait_flush_lat.

Now read don't need flush(). See commit 83e33a32fde7f83.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14088 from Adirl/rdma-cm-2

msg/async/rdma: Move resource handling to Device

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13915 from jcsp/wip-19253

qa: whitelist new fullness messages in fs tests

Merge pull request #14159 from tchaikov/wip-rbd-size_t-is-not-uint64_t-on-arm32

librbd: pass an uint64_t to clip_io() as the third param

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: pass an uint64_t to clip_io() as the 3rd param

because, on arm32, size_t is "unsigned", while uint64_t is "long long
unsigned int", compiler is not able to promote the former to the latter
automatically. and ImageRequest::Extents::value_type::second_type is
indeed uint64_t, which is expected by clip_io(). so let's just use
"auto" here.

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10412 from stiopaa1/mon_monitorh_addConstToGetLeader

mon/Monitor.h: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14146 from liupan1111/wip-fix-comment

rbd-nbd: clean up the doc and help information

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

kv/rocksdb: add option to control whether enable DeleteRange API

Signed-off-by: Haomai Wang <haomai@xsky.com>

kv/RocksDBStore: implement rm_range_keys operator interface and test

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #12893 from rzarzynski/wip-rgw-auth-rework-cont-2

rgw: continuation of the auth rework

mon/Monitor.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10121 from theanalyst/wip-16357

rgw: cls_user don't clobber existing bucket stats when creating bucket
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rbd-nbd: polish the doc and help information.

Signed-off-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #14134 from wangzhengyong/doc

doc: add some undocumented options to rbd-nbd

Reviewed-by: Pan Liu <liupan1111@gmail.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14091 from trociny/wip-prepare_async_request

librbd: potential use of uninitialised value in ImageWatcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13740 from Adirl/forksafe

msg/async/rdma: Add fork safe on RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

crush: make get_immediate_parent const

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add 'osd [rm-]pg-remap[-items] ...' commands

Add commands to add and remove pg_remap mappings.

Require that a mon config option is set before this is allowed so that
users don't inadvertantly prevent older clients from interacting with the
cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: ignore pg_remap[_items] mappings when target osd isout

If the remap target is fully out, ignore the mapping.

Do not do a probabilistic 'osd_weight' rejection; pg-remap is intended to
avoid probabilistic mappings.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: add pg_remap and pg_remap_items

The first simply forces a pg to map to a particular value.  The second
replaces specific OSDs with specific other OSDs in the CRUSH mapping.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMap: clean up primary selection

_pg_to_raw_osds was selecting a primary, but it was generally
getting thrown out since _raw_to_up_osds was picking a *new*
primary after filtering out down and nonexistent osds.  Add a
_pick_primary helper to do it and call that explicitly in the
callers where it can be used by all.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #13965 from liewegas/wip-bluestore-pc

os/bluestore: fix perf counters

Merge pull request #13962 from Liuchang0812/wip-add-override-in-osd-headers

osd: add override in headers files

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add some undocumented options to rbd-nbd

Signed-off-by: wangzhengyong@cmss.chinamobile.com

osd: add override in headers files

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #14114 from dmick/wip-boost-j

debian/rules, ceph.spec.in: invoke cmake with -DBOOST_J

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

include/denc: add missing {encode,decode}_nohead for denc_traits

implement {encode,decode}_nohead() for dec_traits<basic_string> and
bufferlist.

so compiler can find an appropriate version of decode_nohead() when
performing overload resolution. in some architectures, compiler resolves
the __u32 to size_t, which leads to a compilation failure, because the
compiler fails to instantiate the ::decode_nohead() template exposing
traits::decode_nohead().

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PrimaryLogPG: remove unused custom reaction method for WaitReservation

the KickTrim event can be handled by the outer state Trimming properly

Signed-off-by: runsisi <runsisi@zte.com.cn>

osd: remove dispatch_op_fast method declaration

Signed-off-by: runsisi <runsisi@zte.com.cn>

osd/PG: remove unused Recovering event

Signed-off-by: runsisi <runsisi@zte.com.cn>

cmake/modules/SIMDExt.cmake: add whitespace

readability, man, readability

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rgw: implement the dynamic reconfiguration of auth strategies.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: the S3's local v2 auth engine becomes a fallback conditionally.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement SwiftAnonymousEngine.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: parametrize the implicit tenancy of rgw::auth::RemoteApplier.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: RGWPostObj_ObjStore_S3 doesn't instantiate auth strategy for each request anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix appending '\0' in the rgw::auth::LocalApplier::to_str().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13889 from liewegas/wip-denc-nullptr

include/denc: remove nullptr runtime magic boundedness check

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14096 from baiyanchun/remove_useless_parameter

common: remove useless parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #14131 from liewegas/wip-crush-encode

crush: only encode class info if SERVER_LUMINOUS

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13960 from wangzhengyong/kstore

os/kstore: some error handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13973 from shinobu-x/wp-sk-primarylogpg-null-nullptr

osd/PrimaryLogPG: nullptr not NULL

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13995 from liuhongtong/wip-config

common/config: set rocksdb_cache_size to OPT_U64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14013 from ShiqiCooperation/newshiqi

test/unittest_bluefs: check whether add_block_device success

Reviewed-by: Sage Weil <sage@redhat.com>

crush: only encode class info if SERVER_LUMINOUS

This fixes OSDMap reencode crc mismatches on jewel to
luminous upgrades.

Fixes: http://tracker.ceph.com/issues/19361
Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: derive _smp_ncpus and use it for -DBOOST_J

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph.spec.in: move lowmem_build setting of _smp_mflags

Signed-off-by: Dan Mick <dan.mick@redhat.com>

debian/rules: invoke cmake with -DBOOST_J

Allow boost build during toplevel cmake from Debian package build
to benefit from multiple processors.  Should speed build a lot
on many-proc machines (say, arm64).  Use argument passed to
debhelper.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rgw: improve logs in the RGWAccessControlPolicy class.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>