boto3: Foundation laid for boto3 tests

Added sample config file for boto3 and vstart.sh

Modified setup.py, requirements.txt, and README

Added an rgw_interactive.py to use interactively
with vstart.sh and python -i

Ported 400+ tests over to boto3 from functional/test_s3.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #228 from tchaikov/wip-cffi

bootstrap: add libffi-dev for gevent

Reviewed-by: Casey Bodley <cbodley@redhat.com>

bootstrap: add libffi-dev for gevent

because gevent 1.3.2 requires cffi>=1.11.5, and cffi requires
libffi-dev to build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #192 from joke-lee/cors_option_header_check

rgw: test cors option request with Access-Control-Request-Headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #220 from joke-lee/max-keys-test

enable test for https://github.com/ceph/ceph/pull/21285

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #182 from gaul/versioning

Add missing versioning tag

enable test for https://github.com/ceph/ceph/pull/21285

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #217 from cbodley/wip-unpin-requests

unpin the requests library version

requirements: unpin 'requests' from old v0.14

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cors: allow None to match missing headers

this is apparently needed to run against newer versions of the requests
library

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #209 from fangyuxiangGL/compression

compression: add case to download range from big object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #205 from malc0lm/multipartcopy-no-range

Add test for muiltpart copy without 'x-amz-copy-source-range' header

compression: add case to download range from big object

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #191 from theanalyst/policy-conditionals

test bucket policy with conditionals

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Add test for muiltpart copy without 'x-amz-copy-source-range' header

policy: refactor make_json_policy to use the new Policy classes

since make_json_policy is redundantly doing most of the same work,
refactor to use the new policy classes instead

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test existingobject tag on get acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object with reqeust object tag

policy: test policy with sse-c encryption

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

typo fix for pub_obj_acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: add S3 tests for grants conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

move ListBucket to new policy class

policy: add a new policy class to make creation of complex policies

Since policies can have allow/deny rules etc

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object with canned acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test metadta copy key

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put obj using copy conditionals

Test s3:x-amz-copy-source conditional on put obj with x-amz-copy-source

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test for acl grants conditionals on put bucket acls

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: add tests for put bucket acl with canned acl conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

s3: policy tests for ListBucket with prefix, delimiter & max-keys

Allow conditionals on ListBucket similar to s3 docs which allow for
these clauses

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object tagging with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test get object tagging with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add policy tests for get object with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add bucket-policy attr to all the policy tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

test_s3: modify make_json_policy to support conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

bucket policy: improve the helper functions used for tagging upwards

- Improve `make_json_policy` to support conditionals in policy
- Move the helper functions for creating policies up so that bucket
  policy tests can use these
- add bucket-policy attribute to the tagging tests using policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #204 from linuxbox2/wip-post-ctype

Add test case for POST with no Content-Type header

Add test case for POST with no Content-Type header

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #199 from cfanz/cfanz-version-id-wip

Add test case for getting version-id when upload objects  bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Add test case for getting version-id when upload objects to bucket

For default-bucket and versioning-suspended-bucket, no version-id should return when upload.
For versioning-enabled-bucket, an non-empty version-id should return and equal to what
bucket list shows.

Signed-off-by: Xinying Song <songxinying@cloudin.cn>

Merge pull request #196 from ZVampirEM77/wip-lc-typos

Fix a typos for lifecycle test case

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #195 from ceph/wip-fix-416

Check for either 400 or 416 response code for invalid range test

cleanup unneeded comments

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Fix a typos for lifecycle test case

According to the commit 9204283def8e0bcfe14a57f0c87ed20735214f53,
lc debug interval should be set to 10s.

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

check for either 400 or 416 response code for invalid range test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #189 from joke-lee/list_bucket_versiong_with_marker

test for https://github.com/ceph/ceph/pull/17934

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test cors option request with Access-Control-Request-Headers
test for https://github.com/ceph/ceph/pull/18556

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #190 from cbodley/wip-pr-17882

expect 400 for missing sse headers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

expect 400 for missing sse headers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #171 from cbodley/wip-amz-date-precedence

check precedence of Date and X-Amz-Date in signatures

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test for https://github.com/ceph/ceph/pull/17934

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #186 from tipabu/check-grant-ordering

Care less about ordering when checking grants

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #185 from theanalyst/lc-time-fixes

lc: Give more flexibility for LC expiration times

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Care less about ordering when checking grants

Some tests, like test_object_header_acl_grants and
test_bucket_header_acl_grants, use the same id for all permissions.
Sort() is stable, so those tests end up testing the order of acl grants
returned by Boto. Not sure, but I think this may in turn depend on the
order of HTTP headers, where the order is not significant.

Signed-off-by: Tim Burke <tim.burke@gmail.com>

lc: Give more flexibility for LC expiration times

Earlier values expected a lc debug interval of 2s, which is a pretty
short time window and can often lead to failures if the processing
didn't complete within the next day. This commit assumes the currently
configured LC debug interval of 10s, and gives time intervals using the
following logic:

Worst case:
LC start-time : 00:00
obj upload    : 00:01
LC run1       : 00:10 -> object not expired as it is only 9s old
LC run2       : 00:20 -> object will expire in this run, however we
can't exactly guess when this run will complete, for a moderate amount
of objects this can take anywhere between 1 to a max of 10s, let us give
it a wiggle room to complete around 8s, given the amount of objects in a
teuthology run, it should be mostly probable that the object is already
deleted within this time, so at 28s, we should have seen day1 objects being
expired.

Best case:
LC start-time: 00:00
obj upload : 00:09
LC run1 : 00:10
LC run2 : 00:20 -> obj expires, so elapsed time is around 11->19s (of
course it would almost close to 10s too), We should probably configure
the LC lock time to 10s as well just so as to ensure that the lock isn't
held for the default 60s in which case it is possible that the object
might expire in a time greater than the lock interval.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #173 from chuang-he/fix_decrypt

Data encryption is not follow the AWS agreement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

modify test_sse_kms_method_head() for aws standard

Signed-off-by: hechuang <hechuang@xsky.com>

add test_sse_kms_read_declare() for aws standard

Signed-off-by: hechuang <hechuang@xsky.com>

rgw: Data encryption is not follow the AWS agreement

Encryption request headers should not be sent for GET requests and HEAD
requests if your object uses SSE-KMS/SSE-S3 or you’ll get an HTTP 400
BadRequest error.

Signed-off-by: hechuang <hechuang@xsky.com>

Add missing versioning tag

Signed-off-by: Andrew Gaul <andrew@gaul.org>

Merge pull request #179 from andrewgaul/copy-part-invalid-range

Add test for invalid copy part range

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #181 from theanalyst/lc-expir-disable

lc: set attribute as lifecycle_time for lc tests using timing ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

lc: add attr lifecycle_expiration for lc tests using expiration

Allows us to additionally filter out these tests which currently fail in
the teuthology runs

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #180 from theanalyst/lc-filter

lifecycle: add test cases using filter & no id

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

lifecycle: add test cases using filter & no id

Added test cases for LC policies with no ID in xml, no prefix and an
empty and non empty Filter.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Add test for invalid copy part range

References kahing/goofys#212.

Signed-off-by: Andrew Gaul <andrew@gaul.org>

Merge pull request #159 from theanalyst/wip-tagging

add tests for tagging functionality

add tests for get/put/delete/post object tagging

- adds test for put obj with tagging query
- adds test for HEAD obj with x-amz-tagging-count verification
- adds test for put object tagging
- adds test for get object tagging
- adds test for post object tagging
- adds test for delete object tagging
- also verify& exceed the max tags
- add tagging condition to bucket acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #167 from joke-lee/PolicyConditionOpEndWithifExist

add test case: test set policy with condition operator end with IfExists

Reviewed-By: Adam C. Emerson <aemerson@redhat.com>

add test case: test set policy with condition operator end with IfExists

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #174 from ceph/wip-underscore-prefix

Add test for bucket list with prefix with underscore

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #170 from zhangsw/lifecycle-date

add test case for lifecycle expiration date.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #168 from zhangsw/lifecycle-expiration

modify the thread sleep time in lifecycle expiration test so that it can pass on ceph master branch.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #163 from pritha-srivastava/wip-rgw-bucket-policy-tests

rgw: Adding tests for Bucket Policy.

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Add test for bucket list with prefix with underscore

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

check precedence of Date and X-Amz-Date in signatures

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #166 from cbodley/wip-get-version

clear key.version_id now that GET requests return x-amz-version-id header

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

modify the thread sleep time in lifecycle expiration test so that it can
pass on ceph master branch.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

add test case for lifecycle expiration date.

Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

clear key.version_id now that GET requests return x-amz-version-id header

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: Adding tests for Bucket Policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #157 from zhangsw/verionid-getobjectmeta

Add test case for getting version id when getting object and object's metadata

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #158 from zhangsw/lifecycle-noncurrent

Test case for non-current expiration，multipart upload expiration and delete marker expiration

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Test case for non-current expiration, multipart upload expiration and
delete marker expiration.

Fixes: http://tracker.ceph.com/issues/19519
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Add test case for getting version id when getting object and object's metadata

Fixes: http://tracker.ceph.com/issues/19565
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #154 from ceph/wip-pr-118

functional tests for server side encryption

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

crypto: add key headers to valid HEAD requests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Added tests for HEAD method for encrypted objects.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Added micro instuction how to configure tests for KMS (barbican) integration

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Modified aws kms tests to use config variable kms_keyid

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

A bit more test for kms

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Additional tests for server side encryption, S3 SSE-KMS mode.
All tests belong to group 'encryption'.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Additional tests for server side encryption, S3 SSE-C mode.
All tests belong to group 'encryption'.

Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>

Merge pull request #136 from jkorvin/fix-ifmodifiedsince-failed

Fix 'If-Modified-Since' format string

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #135 from jkorvin/fix

Fix If-Match test

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Fix 'If-Modified-Since' format string

According to https://tools.ietf.org/html/rfc7232
header 'If-Modified-Since' should be a correct HTTP-date.
For example: `Tue, 15 Nov 1994 12:45:26 GMT`;
not `1994-11-15T12:45:26.000Z`.

Signed-off-by: Evgenii Gorinov <korvin.lucky@gmail.com>

Fix If-Match test

According to https://tools.ietf.org/html/rfc7232, If-Match field should looks like QUOTES ETag QUOTES.

Signed-off-by: Evgenii Gorinov <korvin.lucky@gmail.com>

Merge pull request #152 from zhangsw/set-current-acl

Add a test case for setting acl on object with no version id specified.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #138 from zhangsw/lifecycle-validate

add lifecycle validate test cases.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #128 from ceph/wip-multipart-copy

Wip multipart copy

Add a test case for setting acl on object with no version id specified.

Fixes: http://tracker.ceph.com/issues/18923
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>