rgw: catch up aws4 with the new features

Fix pending conflicts after massive merging. It catches up with SLO, bucket
website, bulk deletes and payment request features.

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: fix host field in canonical headers under qs

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: fix bug encoding percentage char in X-Amz-Credential

The X-Amz-Credential value in the URL shows the "/" character only for
readability. In practice, it should be encoded as %2F

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: add aws4 auth grace period

Protect against reuse of the signed portions of the request. In AWS, the signed
portions (using AWS Signatures) of requests are valid within 15 minutes of the
timestamp in the request.

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: add x-amz-expires support

Provides the time period, in seconds, for which the generated presigned URL is
valid. For example, 86400 (24 hours). This value is an integer. The minimum
value you can set is 1, and the maximum is 604800 (seven days).

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: escape aws4 query string

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement missing handlers for aws4 signatures

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement RGWOp_Metadata_Put::get_type()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: raise error on not implemented AWS4 completions

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: add AWS4 completion support for RGW_OP_PUT_ACLS

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: add AWS4 completion support for RGW_OP_PUT_OBJ

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: return -EINVAL on unknow authorization

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: verify X-Amz-Date

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: do not encode key/val when computing canonical qs

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Revert "rgw: avoid re-encoding already encoded query strings in AWS4 auth"

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: multiple fixes and adjustments following aws4 code review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: achieve same error behaviour in S3 and RGW when processing signedheaders

This patch gets the same error response in S3 and RGW when the error is related
to the signedheaders processing.

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: validate x-amz-sha256

Verify content's sha256 sum matches the expected value.

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: check if x-amz-date has a value

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: verify Content-MD5 is a valid base64 value

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: initialize aws4_auth_complete

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: AWS4 auth support for positive content-length

Handle AWS4 auth on requests with positive content-length. It splits the
auth process along several steps to process the body content on the fly
instead of reading the whole body in memory. After that, it completes
the delayed AWS4 auth properly.

Requests with content-length <= 0 are validated as usual. They don't
require any kind of completion.

Requests with content-length > 0 use a streaming approach together with
a completion step.

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: UNSIGNED-PAYLOAD support in AWS4 auth

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: avoid re-encoding already encoded query strings in AWS4 auth

When computing V4 signature, we need to encode the query string. But it
could come already encoded, at least partially.

So do not encode the entities that are already encoded.

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: AWS4 auth support when using request params

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: add proper AWSv4 and AWSv2 auth detection

Adds a new authorize function to identify/handle AWSv4 and AWSv2 auth
properly, handling common code, etc.

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: handle AWS4 auth case when query string exists

Crafts the canonical query string. URI-encode each parameter name and
value properly.

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

rgw: AWS4 authentication minimal support

Amazon S3 supports Signature Version 4. This patch contains the minimal
implementation supporting AWS4 in RGW. It implements AWS4 authentication
on http methods without body content and vars not shipping in the
request query string.

Fixes: #10333
Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

Merge pull request #7605 from dachary/wip-python-wheelhouse

python: use pip instead of python setup.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-disk,ceph-detect-init: clean the build directory

b030d8fc113736b479e84246b6b36d5f698a2da3 should have kept the build
directory in .gitignore and the clean target. Re-adding.

Signed-off-by: Loic Dachary <loic@dachary.org>

python: use pip instead of python setup.py

python setup.py develop may try to pull dependencies from the net and
has no way to collect them from the wheelhouse that was populated by
install-deps.sh. Use pip install -e instead

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #7528 from objoo/YD-jewel-mailmap-updates-take2

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7609 from dillaman/wip-librbd-flatten-test

test: new librbd flatten test case

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7583 from dillaman/wip-librbd-journal-fsx

librbd: integrate journal replay with fsx testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7613 from wjwithagen/patch-1

osd: add missing newline to usage message

Reviewed-by: Sage Weil <sage@redhat.com>

src/ceph_osd.cc Add missing newline to usage message

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

common: fix build error with Graylog

Graylog includes boost stuff, which clobbers our assert macros.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6920 from oritwas/wip-mstart-cmake

ajdust mstart and mstop script to run with cmake build

Merge pull request #7523 from tchaikov/wip-fix-cmake

cmake: fix the build of tests

Merge pull request #7607 from FlorentCoppint/master

mount.fuse.ceph: better parsing of arguments passed to mount.fuse.ceph by mount command

Reviewed-by: Sage Weil <sage@redhat.com>

test: new librbd flatten test case

AIO operations after a flatten operation were previously
hanging during the close of the parent image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Better parsing of arguments passed to mount.fuse.ceph by mount command.

Signed-off-by: Florent Bautista <florent@coppint.com>

Merge pull request #5961 from majianpeng/small-fix

osd: clean up CMPXATTR checks

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7533 from Cdiscount/master

debian: include cpio in build-requiers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7537 from ifed01/wip-no-promote-for-delete-fix

osd: fix unnecessary object promotion when deleting from cache pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7567 from branch-predictor/bp-slimdown-conns

msg: significantly reduce minimal memory usage of connections

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7579 from javacruft/wip-systemd-escaping

systemd: correctly escape block device paths

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7581 from jcsp/wip-asok-lockdep

librados: mix lock cycle (un)registering asok commands

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7604 from ceph/wip-selinux-update-radosgw-log

selinux: allow log files to be located in /var/log/radosgw

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7606 from tchaikov/wip-fix-ceph-disk

tests: ceph-disk.sh: use "readlink -f" instead for fullpath

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh: use "readlink -f" instead for fullpath

if $PATH has ".." in it, and the program happen to be located in
"..", `which program` will print `../program` instead of its fullpath,
so we should always use `readlink -f` for the fullpath.

Signed-off-by: Kefu Chai <kchai@redhat.com>

selinux: Allow log files to be located in /var/log/radosgw

We do suggest users to put their logs in /var/log/radosgw in the
documentation at times. We should also label that directory with
ceph_var_log_t so that ceph daemons can also write there.

The commit also updates the man page for this policy. This man page is
automatically generated by

* sepolicy manpage -p . -d ceph_t

and have not been reloaded in a while. Hence, it contains few more
changes than the new radosgw directory.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #7601 from jjhuo/master

os/bluestore: fix a typo in SPDK path parsing

Reviewed-by: Haomai Wang <haomai@xsky.com>

os/bluestore: fix a typo in SPDK path parsing

SPDK block path with bluestore is something like spdk:55cd2e404bd73932.
The size of the prefix to compare should be 5.
sizeof(SPDK_PREFIX)-1 returns 5, while sizeof(SPDK_PREFIX-1) returns 8.

Signed-off-by: Jianjian Huo <samuel.huo@gmail.com>

librbd: protect journal replay against overlapping writes

If fsx issues a back-to-back synchronous write, these will be
replayed as AIO writes.  If object map is enabled, it's possible
for the two writes to commit to disk out-of-order if the first
write required an object map update.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track in-flight AIO flush requests during journal replay

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: possible librbd journal replay flush race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7588 from dmick/master

common/page.cc: _page_mask has too many bits

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7413 from YankunLi/dev

rgw: remove duplicated code in RGWRados::get_bucket_info()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge branch 'wip-11642-0203'

rgw: make function parameter 'errordoc_key' passed by reference

Signed-off-by: Bo Cai <cai.bo@h3c.com>

Merge branch 'master' of github.com:ceph/ceph

Merge pull request #7544 from ceph/wip-rh-covscan

rgw/rgw_orphan: check the return value of save_state

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7586 from bsedyang/master

rgw: user quota may not adjust on bucket removal

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc/release-notes: v10.0.3

Finalized release notes.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7592 from dachary/wip-release-notes

release-notes: draft v10.0.3 release notes

Merge pull request #7594 from tchaikov/fix-ceph-disk-test

tests: ceph-disk.sh: should use "readlink -f" instead

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk.sh: should use "readlink -f" instead

turns out trusty does not have `realpath` unless the package named
`realpath` is installed.

Reported-by: Joao Eduardo Luis <joao@suse.de>
Signed-off-by: Kefu Chai <kchai@redhat.com>

release-notes: draft v10.0.3 release notes

Signed-off-by: Loic Dachary <loic@dachary.org>

scripts: ceph-release-notes fix merge messages handling

Strip the title line if it is the same as the pr title instead of
ignoring the message entirely.

Signed-off-by: Loic Dachary <loic@dachary.org>

cmake: remove Boost libraries from EXTRALIBS

* and remove ${Boost_SYSTEM_LIBRARY} from test_rados_api_tier

Signed-off-by: Kefu Chai <kchai@redhat.com>

log: remove "boost/asio.hpp" from Log.h

graylog uses boost/asio.hpp which introduces the link-time dependency on
libboost_system to the compilation units which includes Log.h and
LogClient.h. so it appears that perfglue/heap_profiler.cc is referencing
libboost_system.so, and fails the cmake build of all tests which links
against tcmalloc.

in this change, we:

* remove unnecessary #includes from Graylog.h
* forward declare Graylog class, so that "Graylog.h" is not included in
  any header files to avoid the link-time dependency pollution

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: disable warnings introduced by 457f023

Signed-off-by: Kefu Chai <kchai@redhat.com>

Revert "cmake: add libboost_system to EXTRALIBS"

This reverts commit 21438a6773a3153556de07096919ab46c4540eee.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/page.cc: _page_mask has too many bits

It's not used anywhere, but in case it is one day...

Signed-off-by: Dan Mick <dan.mick@redhat.com>

rgw: user quota may not adjust on bucket removal

Description:
If the user/admin removes a bucket using --force/--purge-objects options with s3cmd/radosgw-admin respectively, the user stats will continue to reflect the deleted objects for quota purposes, and there seems to be no way to reset them. User stats need to be sync'ed prior to bucket removal.

Solution:
Sync user stats before removing a bucket.

Fixes: #14507
Signed-off-by: Edward Yang eyang@us.fujitsu.com

test: fsx now exercises librbd journal replay

write, discard, snap create, snap protect, flatten, and resize journal
events are tests when using the new '-j' option.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: missing lock on maint operation replay callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not fire flush completions before associated write

Prevents a race condition in journal replay when committing events
to the journal.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7496 from xiexingguo/xxg-wip-fixnvme

osd: BlueStore/NVMEDevice: fix compiling and fd leak

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7349 from yuyuyu101/add-pwritev

osd: FileStore: use pwritev instead of lseek+writev

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7504 from renhwsky/renhw-wip-mon-shutdown

mon: unregister command on shutdown

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7550 from tchaikov/wip-osd-cleanup

osd: a fix for HeartbeatDispatcher and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7553 from XinzeChi/wip-osdmap-affinity

osd: OSDMap: reset osd_primary_affinity shared_ptr when deepish_copy_from

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7560 from tchaikov/wip-fix-icl

crush/CrushTester: workaround a bug in boost::icl

Merge pull request #7568 from liewegas/wip-bluestore-mutex

osd: bluestore/blockdevice: use std::mutex et al

debian: fix build-depends

Signed-off-by: Rémi BUISSON <remi.buisson@cdiscount.com>

Merge pull request #7578 from ceph/fix-stripper

Stripper: Fix incorrect push_front -> append_zero change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osdc/Objecter: drop lock to unregister asok hooks

unregister blocks until any running hook has
completed, and that running hook might
be one that takes the objecter lock, so you
can't hold it while unregistering.

Signed-off-by: John Spray <john.spray@redhat.com>

common: asok should drop lock to call hook

Otherwise, hooks cannot take any locks which
were held during registration of hooks, because
it creates a cycle.

Signed-off-by: John Spray <john.spray@redhat.com>

Correctly escape block device paths for systemd

Ensure that the leading / is stripped from block device
paths before escaping for using in systemd unit names.

Fixes: #14706
Signed-off-by: James Page <james.page@ubuntu.com>

mailmap: Matt Benjamin affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Emile Snyder affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Jonas Keidel affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap:  Alan Grosskurth affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Alexey Sheplyakov affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Benoît Knecht affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Chuanhong Wang affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Gu Zhongyan affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>

mailmap: Ivan Grcic affiliation

Signed-off-by: Yann Dupont <yann@objoo.org>