Merge pull request #14093 from hjwsm1989/sparse-read

osd/PrimaryLogPG: set return value if sparse read failed

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14099 from tchaikov/wip-denc-arm32

include/denc: add {encode,decode}_nohead for denc_traits<basic_string>

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14129 from SUSE/wip-18764

mgr: fix crash on missing 'ceph_version' in daemon metadata (fixes #18764)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14003 from dillaman/wip-19287

librbd: is_exclusive_lock_owner API should ping OSD

Reviewed-by: Mike Christie <mchristi@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14179 from Adirl/rdma-cm-3

msg/async/rdma: RDMA-CM Initialize device on first connect

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #14166 from dillaman/wip-19388

librbd: random unit test failures due to shut down race

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14021 from batrick/i19291

mds: set ceph-mds name uncond for external tools

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #14017 from jcsp/wip-19282

mds: don't assert on read errors in RecoveryQueue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #14015 from jcsp/wip-19288

mds: remove legacy "mds tell" command

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #13904 from jcsp/wip-16709

mon: fix hiding mdsmonitor informative strings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #12074 from batrick/i17604

mds: warn if insufficient standbys exist

Reviewed-by: John Spray <john.spray@redhat.com>

msg/async/rdma: Extract sockets stuff from RDMAStack.h

This is a preparation commit, in order to make review easier. In this
commit I move code from RDMAStack.h into the new file
RDMAConnectedSocketImpl.h - without changing the code.

In the next commit, the actual logic changes will be done and socket
classes will be split into a base RDMAConnected classes and child
classes with TCP connection establishment specific code.

Issue: 995322
Change-Id: I639fda490a6fbd02addb95d3158c5ac1e7390ef0
Signed-off-by: Amir Vadai <amir@vadai.me>

msg/async/rdma: Initialize device on first connect

Allocate Device's IB resources only when first connect for a device is
called.

Also, removed Infiniband::MemoryManager::Chunk::post_srq() which is not
used anywhere in the code.

Issue: 995322
Change-Id: I2ac1e36aff645ad7d8bbc06c87530bf33c4b3ecf
Signed-off-by: Amir Vadai <amir@vadai.me>

Merge pull request #13423 from dzafman/wip-18533-tool

Fixes: http://tracker.ceph.com/issues/18533
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #14132 from dmick/wip-arm-crc

arch: fix cmake's ARM CRC intrinsics test to handle duplicitous gcc 4.8.5

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake/modules/SIMDExt.cmake: armv8 crypto intrinsics

Test not only for -march support, but also the actual
presence of the intrinsic routines.  Not sure why, but gcc
4.8.5 passes the first but not the second.

Fixes: http://tracker.ceph.com/issues/19386
Signed-off-by: Dan Mick <dan.mick@redhat.com>

librbd: random unit test failures due to shut down race

Fixes: http://tracker.ceph.com/issues/19389
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

filestore, tools: Fix logging of DBObjectMap check() repairs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Simplify DBObjectMap by no longer creating complete tables

Bump the version for new maps to 3
Make clone less efficient but simpler
Add rename operation (use instead of clone/unlink)
For now keep code that understands version 2 maps

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix seg fault with large amount of check error output

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add automatic repair for DBObjectMap bug

Add repair command to ceph-osdomap-tool too

Under some situations the previous rm_keys() code would
generated a corrupt complete table.  There is no way to
figure out what the table should look like now.  By removing
the entries we fix the corruption and aren't much worse off
because the corruption caused some deleted keys to re-appear.

This doesn't breaking the parent/child relationship during
repair because some of the keys may still be contained
in the parent.

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Fix tool exit status

Signed-off-by: David Zafman <dzafman@redhat.com>

DBObjectMap: rewrite rm_keys and merge_new_complete

Leverage the updated in_complete_region and needs_parent to simplify
these methods.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: strengthen in_complete_region post condition

Previously, in_complete_region didn't guarantee anything about
where it left complete_iter pointing.  It will be handy for
complete_iter to be pointing at the lowest interval which ends
after to_test.  Make it so.

Signed-off-by: Samuel Just <sjust@redhat.com>

DBObjectMap: fix next_parent()

The previous implementation assumed that
lower_bound(parent_iter->key()) always leaves the iterator
on_parent().  There isn't any guarantee, however, that that
key isn't present on the child as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

test_object_map: add tests to trigger some bugs related to 18533

Signed-off-by: Samuel Just <sjust@redhat.com>

test: Add ceph_test_object_map to make check tests

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-osdomap-tool: Add --debug and only show internal logging if enabled

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: DBOjectMap::check: Dump complete mapping when inconsistency found

Signed-off-by: David Zafman <dzafman@redhat.com>

test_object_map: Use ASSERT_EQ() for check() so failure doesn't stop testing

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Check for overlaps in internal "complete" table for DBObjectMap

Changed check to return an error count and fix tool error message

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add dump-headers command to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add --oid option to ceph-osdomap-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Remove unnecessary assert and assignment in DBObjectMap

Fix and add comment(s)

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #13754 from liewegas/wip-objectstore-random

os: allow 'osd objectstore = random' to pick either filestore or bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: remove unuse perfcounter:l_bluestore_read_wait_flush_lat.

Now read don't need flush(). See commit 83e33a32fde7f83.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14088 from Adirl/rdma-cm-2

msg/async/rdma: Move resource handling to Device

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13915 from jcsp/wip-19253

qa: whitelist new fullness messages in fs tests

Merge pull request #14159 from tchaikov/wip-rbd-size_t-is-not-uint64_t-on-arm32

librbd: pass an uint64_t to clip_io() as the third param

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: pass an uint64_t to clip_io() as the 3rd param

because, on arm32, size_t is "unsigned", while uint64_t is "long long
unsigned int", compiler is not able to promote the former to the latter
automatically. and ImageRequest::Extents::value_type::second_type is
indeed uint64_t, which is expected by clip_io(). so let's just use
"auto" here.

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10412 from stiopaa1/mon_monitorh_addConstToGetLeader

mon/Monitor.h: add const to member function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14146 from liupan1111/wip-fix-comment

rbd-nbd: clean up the doc and help information

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #12893 from rzarzynski/wip-rgw-auth-rework-cont-2

rgw: continuation of the auth rework

mon/Monitor.h: add const to member functions

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10121 from theanalyst/wip-16357

rgw: cls_user don't clobber existing bucket stats when creating bucket
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rbd-nbd: polish the doc and help information.

Signed-off-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #14134 from wangzhengyong/doc

doc: add some undocumented options to rbd-nbd

Reviewed-by: Pan Liu <liupan1111@gmail.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #14091 from trociny/wip-prepare_async_request

librbd: potential use of uninitialised value in ImageWatcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13740 from Adirl/forksafe

msg/async/rdma: Add fork safe on RDMA

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #13965 from liewegas/wip-bluestore-pc

os/bluestore: fix perf counters

Merge pull request #13962 from Liuchang0812/wip-add-override-in-osd-headers

osd: add override in headers files

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add some undocumented options to rbd-nbd

Signed-off-by: wangzhengyong@cmss.chinamobile.com

osd: add override in headers files

Signed-off-by: liuchang0812 <liuchang0812@gmail.com>

Merge pull request #14114 from dmick/wip-boost-j

debian/rules, ceph.spec.in: invoke cmake with -DBOOST_J

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

include/denc: add missing {encode,decode}_nohead for denc_traits

implement {encode,decode}_nohead() for dec_traits<basic_string> and
bufferlist.

so compiler can find an appropriate version of decode_nohead() when
performing overload resolution. in some architectures, compiler resolves
the __u32 to size_t, which leads to a compilation failure, because the
compiler fails to instantiate the ::decode_nohead() template exposing
traits::decode_nohead().

Fixes: http://tracker.ceph.com/issues/18938
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/modules/SIMDExt.cmake: add whitespace

readability, man, readability

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rgw: implement the dynamic reconfiguration of auth strategies.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: the S3's local v2 auth engine becomes a fallback conditionally.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement SwiftAnonymousEngine.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: parametrize the implicit tenancy of rgw::auth::RemoteApplier.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: RGWPostObj_ObjStore_S3 doesn't instantiate auth strategy for each request anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix appending '\0' in the rgw::auth::LocalApplier::to_str().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #13889 from liewegas/wip-denc-nullptr

include/denc: remove nullptr runtime magic boundedness check

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14096 from baiyanchun/remove_useless_parameter

common: remove useless parameter

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #14131 from liewegas/wip-crush-encode

crush: only encode class info if SERVER_LUMINOUS

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13960 from wangzhengyong/kstore

os/kstore: some error handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13973 from shinobu-x/wp-sk-primarylogpg-null-nullptr

osd/PrimaryLogPG: nullptr not NULL

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13995 from liuhongtong/wip-config

common/config: set rocksdb_cache_size to OPT_U64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #14013 from ShiqiCooperation/newshiqi

test/unittest_bluefs: check whether add_block_device success

Reviewed-by: Sage Weil <sage@redhat.com>

crush: only encode class info if SERVER_LUMINOUS

This fixes OSDMap reencode crc mismatches on jewel to
luminous upgrades.

Fixes: http://tracker.ceph.com/issues/19361
Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: derive _smp_ncpus and use it for -DBOOST_J

Signed-off-by: Dan Mick <dan.mick@redhat.com>

ceph.spec.in: move lowmem_build setting of _smp_mflags

Signed-off-by: Dan Mick <dan.mick@redhat.com>

debian/rules: invoke cmake with -DBOOST_J

Allow boost build during toplevel cmake from Debian package build
to benefit from multiple processors.  Should speed build a lot
on many-proc machines (say, arm64).  Use argument passed to
debhelper.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rgw: improve logs in the RGWAccessControlPolicy class.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve logs in rgw::auth::Strategy.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: RGWOp is responsible now for the authentication process.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: refactor error handling in rgw::auth::keystone.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement descriptive authentication failure reasons.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: unify handling S3's ::authorize_v2 and ::get_policy methods.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: remove the parts of auth framework that aren't necessary anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: move ACL Strategies to the newer auth framework.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: switch from req_state::auth_identity to req_state::auth::identity.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: remove rgw/rgw_auth_decoimpl.h as it isn't necessary anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: add rgw::auth::s3::AWSv2AuthStrategy and switch the S3 auth to it.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: replace RGWDummyIdentityApplier with rgw::auth::DummyIdentityApplier.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: apply the new filters over appliers in the Swift's auth strategy.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the rgw::auth::SysReqApplier over IdentityApplier.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement rgw::auth::ThirdPartyAccountApplier.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add Control::FALLBACK mode to rgw::auth::AuthStrategy.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: aggregate S3's external auth engine as the ExternalAuthStrategy.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: make rgw::keystone::TokenEnvelope::expired() to be const-correct.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: switch S3-specific auth engines to the new infra.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: integrate the new EC2Engine with current S3 auth code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: introduce basement for all S3 auth engines.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: fix indentation in RGWPostObj_ObjStore_S3::get_policy().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: improve const-correctness and refactor S3 canonized string crafting.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add rgw::auth::keystone::EC2Engine.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: the authorize() method of Swift's handler operates on auth strategy now.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: the Swift's auth strategy is instantiated and injected from rgw_main.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: settle rgw::auth::{IdentityApplier,Completer} in req_state.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>