rgw: handle the Boto2 compatibility of AWSv4 in an abstract way.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: introduce rgw::auth::Strategy::apply() to deduplicate code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: introduce string_to_sign_t abstraction to the AWS auth.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: only rename AWSv2AuthStrategy -> AWSAuthStrategy.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: optimize AWSv4 parsing with Boost's small_vector.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: switch to Ceph's sstring in AWS signature generation process.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

common/backport14: add the constexpr-capable variant of std::max().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

common/sstring: switch to boost::string_view as string_ref is deprecated.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: replace magic strings in the AWSv4 code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: use preallocated std::strings when concatenating in AWSv4.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rework the implementation of rgw::auth::s3::get_v4_canonical_headers().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw; rework interface and implementation of url_decode.

This commit alters the url_decode() to remove its dependency
on Variable Length Array and unnecessary memory allocations.
It also adjust its signature to the sole usage pattern which
is spread across the code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rework and optimise crafting of AWSv4's canonical query string.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: drop 'using ceph::crypto::SHA256' from rgw_common.h.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: remove the duplicative trim_whitespace from rgw_common.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: switch from boost::string_ref to string_view in AWSv4-related code (part 2).

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: switch from boost::string_ref to string_view in AWSv4-related code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: introduce rgw::auth::s3::AWS4_HMAC_SHA256_STR to kill magics.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: optimize and clean-up the AWSv4 signature processing.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: clean-up AWSv4's Canonical QS crafting.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: use std::make_shared for AWSv4 completers creation.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: clean-up rgw::auth::s3::get_v4_signature().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: handle AWSv4 in Browser Upload using the new auth infra.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add support for HTTP_X_AMZ_DECODED_CONTENT_LENGTH.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: split the AWSv4Completer and clean-up the code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: remove the old AWS v4 streaming mode's implementation.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: the AWSv4 completer verifies chunks' signatures now.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement calc_hash_sha256_restart_stream().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: AWSv4Completer dechunks data in the streaming mode.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: drop aws4_auth_needs_complete from req_state.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: turn AWSv4Completer into a filter over rgw::io::RestfulClient.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: extend RGWRestfulIO to cover dynamic filter injection.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY move AWSv4Completer from rgw_rest_s3.cc to rgw_auth_s3.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement rgw::auth::s3::is_v4_payload_empty.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: drop the old AWSv4 code.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: implement the rgw::auth::Completer for AWSv4.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: extend rgw::auth::Completer to handle commiting modifications to req_state.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: switch to the Completer interface for the AWSv4 payload check.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: make AWS_AUTHv4_IO const-friendly and const-correct.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: integrate AWSv4 auth schema with the new auth infra.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: AWSv4's completer_factory takes the secret_key optionally.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: dissect AWSv4's expected payload extraction into a dedicated function.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: make the get_v4_canonical_request_hash identified shorter.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rgw::auth::s3::parse_credentials() extracts access_key_id now.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: all S3 auth engines are able to use AWSv4 now.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: ONLY change names due to AWSv4 adoption in the auth infra.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: extend Version2ndEngine::Extractor to handle AWSv4 as well.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: decompose rgw::auth::s3::parse_credentials regarding the query string handling.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: AWSv4 completion verifies the payload's fingerprint only.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: clean up the AWSv4 completion calls across RGWOps.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: further minimise the req_state::aws4_auth shared state.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: decouple AWSv4 signature verification and payload fingerprint check.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: eradicate req_state::aws4_auth::credential.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: eradicate req_state::aws4_auth::signed_hdrs.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: clean up rgw::auth::s3::get_v4_signing_key.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: refactor buf_to_hex and improve its const-correctness.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: split generation AWSv4's SigningKey into a separate func.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add std::array-aware variants of calc_hmac_sha256() and buf_to_hex().

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: minimise the number of parameters of rgw::auth::s3::get_v4_signature.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: drop req_state::aws4_auth::payload_hash as it doesn't need to be global.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rgw::auth::s3::get_v4_signature doesn't depend on req_state anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: dissect basic AWSv4's credentials parsing into separated function.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: eradicate req_state::http_auth.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: dissect AWSv4's Canonical Headers crafting into a separated function.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: dissect AWSv4's Canonical QS crafting into a separated function.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: dissect AWSv4's Canonical URI crafting into a separated function.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: get_v4_canonical_request_hash doesn't depend on req_state anymore.

In AWSv4 the hash of real, transfered payload IS NOT necessary to form
a Canonical Request, and thus verify a Signature. x-amz-content-sha256
header lets get the information very early -- before seeing first byte
of HTTP body. As a consequence, we can decouple Signature verification
from payload's fingerprint check. Although RadosGW doesn't do that for
now, the situation will definitely change in the future.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: rework interfaces of AWSv4 helper primitives.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: aws4: add AWS4 auth support for S3 Post Object API

RGW S3 supports HTTP POST requests so users can upload content directly.

This patch adds AWS v4 to handle form data properly.

Fixes: http://tracker.ceph.com/issues/18800
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #15530 from tchaikov/wip-jq-without-first

qa/workunits/ceph-helpers.sh: use syntax understood by jq 1.3

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #15423 from liewegas/wip-fix-indent

mon/FSCommand: fix indentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15497 from liewegas/wip-log-last

mon/LogMonitor: 'log last' command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15522 from liewegas/wip-19901

ceph_test_rados_api_misc: fix LibRadosMiscConnectFailure.ConnectFailure retry

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15422 from liewegas/wip-osd-rotational

osd: hdd vs ssd defaults for osd op thread pool

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #15517 from liewegas/wip-20197

qa/suites/rados/upgrade: restart mds

Merge pull request #15523 from ceph/wip-16674

selinux: Allow read on var_run_t

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/ceph-helpers.sh: use syntax understood by jq 1.3

trusty still ships jq 1.3 which does not offer "first". see
https://stedolan.github.io/jq/manual/v1.3/ .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #14745 from trociny/wip-18789

rbd-mirror: coordinate image syncs with leader

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #15243 from yuyuyu101/wip-20049

msg: don't set msgr addr when disabing client bind

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #14117 from zhangsw/fix-rgw-getobjectmetadata-bug

rgw: return the version id in get object and object metadata request.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #15386 from ivancich/wip-add-readme-that-dmclock-is-a-subtree

doc: add README to dmclock subdir to inform developers it's a git subtree

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15392 from dillaman/wip-20049

msg: do not enable client-side binding by default

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #15513 from Fabian-Gruenbichler/fixmgrdesc

debian: ceph-mgr: fix package description

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #15375 from ivancich/wip-fix-dmclock-testing

osd/dmclock/testing: reorganize testing, building now optional

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #15378 from Liuchang0812/print_ret_code_of_json_command

ceph.in: print return code when json_command failed

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15496 from wjwithagen/wip-wjw-host-to-ceph-errno

core: include/types.h, introduce host_to_ceph_errno

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15500 from liewegas/wip-20135

pybind/ceph_argparse: fix empty string check

Merge pull request #15502 from xiexingguo/wip-fix-overload-extents

os/bluestore: avoid overloading extents during reshard; atomic deferred_batch_ops

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/upgrade: restart mds

Signed-off-by: Sage Weil <sage@redhat.com>

selinux: Allow read on var_run_t

Fixes: http://tracker.ceph.com/issues/16674
Signed-off-by: Boris Ranto <branto@redhat.com>

ceph_test_rados_api_misc: fix LibRadosMiscConnectFailure.ConnectFailure retry

Fixes: http://tracker.ceph.com/issues/19901
Signed-off-by: Sage Weil <sage@redhat.com>

osd/dmclock/testing: reorganize testing, building now optional

Modify cmake files to take advantage or re-organization of dmclock's
cmake set-up, specifically not having dmclock's cmake files call
add_test. Remove dmclock tests from being dependencies on ceph's
"test" target. Make change so dmclock tests are not built unless
WITH_TESTS and WITH_DMCLOCK_TESTS are both set. This is so openSUSE
Leap will build correctly.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge commit '5c612d7e32a07a01b3b97f9cf58f2a3cef0e2e73'

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Squashed 'src/dmclock/' changes from d6586d7..64bcc6e

64bcc6e Remove FindGTest and FindBoost, which already exist in cmake required version 2.8.11 and later. Also clean up.
ed6dc60 Do some clean-up and remove memory leaks from tests and simulations.
360172a Allow RunEvery object to be ended before destruction via "join" member function.
135b5d0 Fix two bugs discovered by valgrind. Do some other code clean-up as well.
a26d29e Modify cmake files to match naming of cmake's upstream gtest and ceph's boost.
e23e63f Reorganize CMakeLists.txt files, so that projects that incorporate dmclock can control whether add_test is called. Now, add_test is called only at the top-level. Projects that incorporate dmclock can use add_subdirectory on subdirectories beneath the top-level to selective exactly what is included.

git-subtree-dir: src/dmclock
git-subtree-split: 64bcc6ea697c7e790ec4a4229d1bb1d492f9e803

qa/workunits/cephtool/test.sh: test 'log last [n]'

Signed-off-by: Sage Weil <sage@redhat.com>

common/LogEntry: make LogEntryKey opaque and cache its hash value

This avoids recalculating.  We eliminate the default ctor at the same
time and make the type totally opaque (so that its fields cannot be
modified except by the ctor and decode).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #15390 from liewegas/wip-osdmap-state

osd/OSDMap: make osd_state 32 bits wide

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #15515 from tchaikov/wip-crc32-aarch64

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

Reviewed-by: Dan Mick <dmick@redhat.com>

crc32c: include acconfig.h to fix ceph_crc32c_aarch64()

HAVE_ARMV8_CRC is defined by "acconfig.h". so we need to include it for
the definition.

it's a regresssion introduced by 2a3382f. in that change,
crc32c_intel_baseline.h was removed from crc32.cc, and that header
includes accconfig.h indirectly. that's why we will have a dummy
ceph_crc32c_aarch64() which always returns 0. the dummy implementation
was intended to please the compiler. but we can probably just drop it in
future as a cleanup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #15511 from tchaikov/wip-doc-cartesian-product

doc: update sample explaning "%" operator in test suites

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Loic Dachary <loic@dachary.org>