print foo -> print(foo)

Signed-off-by: Sage Weil <sage@redhat.com>

python 3

Signed-off-by: Sage Weil <sage@redhat.com>

bootstrap: switch to bash

Use of declare that is bash specific was introduced in:
d44c923405e22ea8b6cc1ab4316be3cc07f29b5

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit e7fbcee0d5d418675ddbb2f21102508d6b36067a)

Merge pull request #306 from hairesis/wip-sse-kms-tests-refactor

Passing sse-kms keys from configuration instead of hard coding in tests

sse-kms test keys parametrization

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>

sse-kms: adding missing kms test to parametrization

boto2: tag more tests with fails_with_subdomain

Signed-off-by: Casey Bodley <cbodley@redhat.com>

add test cases for object lock.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

startafter or continuation token shouldn't  be returned if they are not specified in the request

Signed-off-by: Albin Antony <aantony@redhat.com>

add test case for list with delimiter not skip special keys

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #285 from soumyakoduri/bucket-name-tests

Fix create_bucket tests as per new bucket naming conventions

Merge pull request #294 from linuxbox2/wip-lc-exphdr

lifecycle: expiration header checks do not require @lifecycle_expirat…

lc: remove date delta check from lc header checks

This was prone to false positives.  We still check:
1. the expiration header has valid form
2. the rule-id matches

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

lc: add rule to test disjoint object tag filters

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

lc: fix lifecycle expiration header tests

Fixes various issues with existing HEAD check and helper methods.
Add an explicit test for GET.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

lifecycle: expiration header checks do not require @lifecycle_expiration tag

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

fix syntax error in test_bucket_get_location

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit eaa68f4f7700c88b9c46c3db3abb0b2464fd46e5)

skip bucket location test if no api_name is configured

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 250122e17d7254227bb1a27d698f28f12f39f7dc)

set 'api_name = default' in sample config

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 75182b7e03b088926d415f6b68c93d0ccd5babfd)

Fix create_bucket tests as per new bucket naming conventions

As per amazon s3 spec -
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-s3-bucket-naming-requirements.html

The s3 bucket names should not contain upper case letters or underscore.
Name cannot end with dash or have consecutive periods, or dashes adjacent to periods
Name length shouldn't exceed 63 characters.

These rules are being enforced via
- https://github.com/ceph/ceph/pull/26787

This patch is to update the respective testcases as well.

Note: check_invalid_bucket_name() seems to have been broken. It should
try to create bucket using invalid name. Have addressed it in this
patch. Because of this few testcases (which were incorrectly passing
earlier) are failing in validate_bucket_name. Have marked them
'fails_on_rgw' as done for test_bucket_create_naming_bad_punctuation

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Added testcases for listobjectsv2

Signed-off-by: Albin Antony <aantony@redhat.com>

test versioned copy with urlencoded keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

boto{2,3}: rm functional.test_s3.test_lifecycle_rules_conflict

The assumption that there may be only one rule per prefix has
been removed.  The rules specifically tested here--without tag
filters--do overlap but are not in conflict.

I am proposing to remove altogether rather than writing new
deconfliction logic.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

remove boto2 tests already passing in boto3

Signed-off-by: Ali Maredia <amaredia@redhat.com>

boto3: tag boto3 tests that are failing on ssl as 'fails_on_rgw'

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 5db9eb059dbab52100ee3a30e77b8ef3a9e3faec)

boto3: verify certificates for ssl connections

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0e04dcd6aa62d75a7ed29c6f86e54327d2a34ee0)

boto3: _get_post_url() uses config endpoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 7f49adda30c2b68c7bb94c823f0522c0c91d8e96)

boto3: use https:// for secure endpoints

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ac18365f7527e2fcf76ad763502130ef4e535ca5)

boto3: use getboolean() for is_secure

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 774d40d1147802efdea407f07576993582e4b179)

boto3_s3tests: add tests for lifecycle expiration header

The response to a successful PUT or HEAD (or GET) of/on
an object matching any enabled lifecycle expiration rule should
include an x-amz-expiration header for the object.  The
x-amz-expiration header consists of an expiry-date, rule-id tuple
indicating the earliest matching rule and the corresponding
expiration date for the object.

Also, while at it, add test for lifecycle expiration rule with
'Days' : 0, which should both apply and...work.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 2d12e98ce0f3b975e2c861762e68045c22780553)

boto3: Foundation laid for boto3 tests

Added sample config file for boto3 and vstart.sh

Modified setup.py, requirements.txt, and README

Added an rgw_interactive.py to use interactively
with vstart.sh and python -i

Ported 400+ tests over to boto3 from functional/test_s3.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 67f4f5d3569f511a3034c9976c25809f73856a96)

bootstrap: add libffi-dev for gevent

because gevent 1.3.2 requires cffi>=1.11.5, and cffi requires
libffi-dev to build.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a2b3e4ef771e840f124847380ed39e9ce641d7b7)

Merge pull request #192 from joke-lee/cors_option_header_check

rgw: test cors option request with Access-Control-Request-Headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #220 from joke-lee/max-keys-test

enable test for https://github.com/ceph/ceph/pull/21285

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #182 from gaul/versioning

Add missing versioning tag

enable test for https://github.com/ceph/ceph/pull/21285

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #217 from cbodley/wip-unpin-requests

unpin the requests library version

requirements: unpin 'requests' from old v0.14

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cors: allow None to match missing headers

this is apparently needed to run against newer versions of the requests
library

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #209 from fangyuxiangGL/compression

compression: add case to download range from big object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #205 from malc0lm/multipartcopy-no-range

Add test for muiltpart copy without 'x-amz-copy-source-range' header

compression: add case to download range from big object

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #191 from theanalyst/policy-conditionals

test bucket policy with conditionals

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Add test for muiltpart copy without 'x-amz-copy-source-range' header

policy: refactor make_json_policy to use the new Policy classes

since make_json_policy is redundantly doing most of the same work,
refactor to use the new policy classes instead

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test existingobject tag on get acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object with reqeust object tag

policy: test policy with sse-c encryption

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

typo fix for pub_obj_acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: add S3 tests for grants conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

move ListBucket to new policy class

policy: add a new policy class to make creation of complex policies

Since policies can have allow/deny rules etc

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object with canned acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test metadta copy key

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put obj using copy conditionals

Test s3:x-amz-copy-source conditional on put obj with x-amz-copy-source

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test for acl grants conditionals on put bucket acls

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: add tests for put bucket acl with canned acl conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

s3: policy tests for ListBucket with prefix, delimiter & max-keys

Allow conditionals on ListBucket similar to s3 docs which allow for
these clauses

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object tagging with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test get object tagging with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add policy tests for get object with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add bucket-policy attr to all the policy tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

test_s3: modify make_json_policy to support conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

bucket policy: improve the helper functions used for tagging upwards

- Improve `make_json_policy` to support conditionals in policy
- Move the helper functions for creating policies up so that bucket
  policy tests can use these
- add bucket-policy attribute to the tagging tests using policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #204 from linuxbox2/wip-post-ctype

Add test case for POST with no Content-Type header

Add test case for POST with no Content-Type header

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #199 from cfanz/cfanz-version-id-wip

Add test case for getting version-id when upload objects  bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Add test case for getting version-id when upload objects to bucket

For default-bucket and versioning-suspended-bucket, no version-id should return when upload.
For versioning-enabled-bucket, an non-empty version-id should return and equal to what
bucket list shows.

Signed-off-by: Xinying Song <songxinying@cloudin.cn>

Merge pull request #196 from ZVampirEM77/wip-lc-typos

Fix a typos for lifecycle test case

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #195 from ceph/wip-fix-416

Check for either 400 or 416 response code for invalid range test

cleanup unneeded comments

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Fix a typos for lifecycle test case

According to the commit 9204283def8e0bcfe14a57f0c87ed20735214f53,
lc debug interval should be set to 10s.

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

check for either 400 or 416 response code for invalid range test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #189 from joke-lee/list_bucket_versiong_with_marker

test for https://github.com/ceph/ceph/pull/17934

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test cors option request with Access-Control-Request-Headers
test for https://github.com/ceph/ceph/pull/18556

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #190 from cbodley/wip-pr-17882

expect 400 for missing sse headers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

expect 400 for missing sse headers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #171 from cbodley/wip-amz-date-precedence

check precedence of Date and X-Amz-Date in signatures

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test for https://github.com/ceph/ceph/pull/17934

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #186 from tipabu/check-grant-ordering

Care less about ordering when checking grants

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #185 from theanalyst/lc-time-fixes

lc: Give more flexibility for LC expiration times

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Care less about ordering when checking grants

Some tests, like test_object_header_acl_grants and
test_bucket_header_acl_grants, use the same id for all permissions.
Sort() is stable, so those tests end up testing the order of acl grants
returned by Boto. Not sure, but I think this may in turn depend on the
order of HTTP headers, where the order is not significant.

Signed-off-by: Tim Burke <tim.burke@gmail.com>

lc: Give more flexibility for LC expiration times

Earlier values expected a lc debug interval of 2s, which is a pretty
short time window and can often lead to failures if the processing
didn't complete within the next day. This commit assumes the currently
configured LC debug interval of 10s, and gives time intervals using the
following logic:

Worst case:
LC start-time : 00:00
obj upload    : 00:01
LC run1       : 00:10 -> object not expired as it is only 9s old
LC run2       : 00:20 -> object will expire in this run, however we
can't exactly guess when this run will complete, for a moderate amount
of objects this can take anywhere between 1 to a max of 10s, let us give
it a wiggle room to complete around 8s, given the amount of objects in a
teuthology run, it should be mostly probable that the object is already
deleted within this time, so at 28s, we should have seen day1 objects being
expired.

Best case:
LC start-time: 00:00
obj upload : 00:09
LC run1 : 00:10
LC run2 : 00:20 -> obj expires, so elapsed time is around 11->19s (of
course it would almost close to 10s too), We should probably configure
the LC lock time to 10s as well just so as to ensure that the lock isn't
held for the default 60s in which case it is possible that the object
might expire in a time greater than the lock interval.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #173 from chuang-he/fix_decrypt

Data encryption is not follow the AWS agreement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

modify test_sse_kms_method_head() for aws standard

Signed-off-by: hechuang <hechuang@xsky.com>

add test_sse_kms_read_declare() for aws standard

Signed-off-by: hechuang <hechuang@xsky.com>

rgw: Data encryption is not follow the AWS agreement

Encryption request headers should not be sent for GET requests and HEAD
requests if your object uses SSE-KMS/SSE-S3 or you’ll get an HTTP 400
BadRequest error.

Signed-off-by: hechuang <hechuang@xsky.com>

Add missing versioning tag

Signed-off-by: Andrew Gaul <andrew@gaul.org>

Merge pull request #179 from andrewgaul/copy-part-invalid-range

Add test for invalid copy part range

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #181 from theanalyst/lc-expir-disable

lc: set attribute as lifecycle_time for lc tests using timing ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

lc: add attr lifecycle_expiration for lc tests using expiration

Allows us to additionally filter out these tests which currently fail in
the teuthology runs

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #180 from theanalyst/lc-filter

lifecycle: add test cases using filter & no id

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

lifecycle: add test cases using filter & no id

Added test cases for LC policies with no ID in xml, no prefix and an
empty and non empty Filter.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Add test for invalid copy part range

References kahing/goofys#212.

Signed-off-by: Andrew Gaul <andrew@gaul.org>

Merge pull request #159 from theanalyst/wip-tagging

add tests for tagging functionality

add tests for get/put/delete/post object tagging

- adds test for put obj with tagging query
- adds test for HEAD obj with x-amz-tagging-count verification
- adds test for put object tagging
- adds test for get object tagging
- adds test for post object tagging
- adds test for delete object tagging
- also verify& exceed the max tags
- add tagging condition to bucket acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #167 from joke-lee/PolicyConditionOpEndWithifExist

add test case: test set policy with condition operator end with IfExists

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

add test case: test set policy with condition operator end with IfExists

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #174 from ceph/wip-underscore-prefix

Add test for bucket list with prefix with underscore

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #170 from zhangsw/lifecycle-date

add test case for lifecycle expiration date.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>