update the case for lifecycle expiration on versioning enabled bucket

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Add a case for lifecycle expiration on versining enabled bucket.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #279 from zhangsw/object-lock

add test cases for object lock.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #249 from cbodley/wip-copy-version-urlencode

test versioned copy with urlencoded keys

Merge pull request #277 from liuchang0812/bucket-tagging

rgw: bucket tagging test

test versioned copy with urlencoded keys

Signed-off-by: Casey Bodley <cbodley@redhat.com>

add test cases for object lock.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #245 from gaul/attrs

Add missing attributes for versioning

rgw: bucket tagging test

Signed-off-by: chang liu <liuchang0812@gmail.com>

Add missing attributes for versioning

Also add cors attributes.

Signed-off-by: Andrew Gaul <andrew@gaul.org>

Merge pull request #272 from cbodley/wip-delete-retry

nuke_prefixed_buckets() ignores 404 from bucket delete

nuke_prefixed_buckets() ignores 404 from bucket delete

if a bucket delete request times out, the retry will likely get a 404
NoSuchBucket response. ignore that error and treat this as a success

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #261 from linuxbox2/wip-lifecycle-prefix

boto{2,3}: rm functional.test_s3.test_lifecycle_rules_conflict

Reviewed-by: Casey Bodley <cbodley@redhat.com>

boto{2,3}: rm functional.test_s3.test_lifecycle_rules_conflict

The assumption that there may be only one rule per prefix has
been removed.  The rules specifically tested here--without tag
filters--do overlap but are not in conflict.

I am proposing to remove altogether rather than writing new
deconfliction logic.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #269 from ceph/wip-boto2-remove

Wip boto2 remove

Reviewed-by: Casey Bodley <cbodley@redhat.com>

remove boto2 tests already passing in boto3

Signed-off-by: Ali Maredia <amaredia@redhat.com>

boto3: tag boto3 tests that are failing on ssl as 'fails_on_rgw'

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #262 from tianshan/test_for_38811

add case test for read not exist null version

Reviewed-by: Casey Bodley <cbodley@redhat.com>

add case test for read not exist null version

http://tracker.ceph.com/issues/38811

Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #260 from cbodley/wip-boto3-ssl

Wip boto3 ssl

boto3: verify certificates for ssl connections

Signed-off-by: Casey Bodley <cbodley@redhat.com>

boto3: _get_post_url() uses config endpoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>

boto3: use https:// for secure endpoints

Signed-off-by: Casey Bodley <cbodley@redhat.com>

boto3: use getboolean() for is_secure

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #247 from zhangsw/append-object

Add test case for appending object.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

add https support in apppend object test cases

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #256 from linuxbox2/wip-expiration-header

boto3_s3tests: add tests for lifecycle expiration header

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

boto3_s3tests: add tests for lifecycle expiration header

The response to a successful PUT or HEAD (or GET) of/on
an object matching any enabled lifecycle expiration rule should
include an x-amz-expiration header for the object.  The
x-amz-expiration header consists of an expiry-date, rule-id tuple
indicating the earliest matching rule and the corresponding
expiration date for the object.

Also, while at it, add test for lifecycle expiration rule with
'Days' : 0, which should both apply and...work.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #250 from yehudasa/wip-rgw-lifecycle-trans

lifecycle transitions, and storage class related tests

Merge pull request #210 from ceph/wip-boto-3

boto3: Foundation laid for boto3 tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

boto3: Foundation laid for boto3 tests

Added sample config file for boto3 and vstart.sh

Modified setup.py, requirements.txt, and README

Added an rgw_interactive.py to use interactively
with vstart.sh and python -i

Ported 400+ tests over to boto3 from functional/test_s3.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>

s3tests: test copy objects (storage class)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

s3tests: more object class related tests

modify object's storage_class, multipart

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

s3tests: test create object with storage class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

s3tests: lifecycle transitions: configurable storage classes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

s3tests: tests for lifecycle transitions

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Add test case for appending object.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #228 from tchaikov/wip-cffi

bootstrap: add libffi-dev for gevent

Reviewed-by: Casey Bodley <cbodley@redhat.com>

bootstrap: add libffi-dev for gevent

because gevent 1.3.2 requires cffi>=1.11.5, and cffi requires
libffi-dev to build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #192 from joke-lee/cors_option_header_check

rgw: test cors option request with Access-Control-Request-Headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #220 from joke-lee/max-keys-test

enable test for https://github.com/ceph/ceph/pull/21285

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #182 from gaul/versioning

Add missing versioning tag

enable test for https://github.com/ceph/ceph/pull/21285

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #217 from cbodley/wip-unpin-requests

unpin the requests library version

requirements: unpin 'requests' from old v0.14

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cors: allow None to match missing headers

this is apparently needed to run against newer versions of the requests
library

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #209 from fangyuxiangGL/compression

compression: add case to download range from big object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #205 from malc0lm/multipartcopy-no-range

Add test for muiltpart copy without 'x-amz-copy-source-range' header

compression: add case to download range from big object

Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>

Merge pull request #191 from theanalyst/policy-conditionals

test bucket policy with conditionals

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Add test for muiltpart copy without 'x-amz-copy-source-range' header

policy: refactor make_json_policy to use the new Policy classes

since make_json_policy is redundantly doing most of the same work,
refactor to use the new policy classes instead

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test existingobject tag on get acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object with reqeust object tag

policy: test policy with sse-c encryption

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

typo fix for pub_obj_acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: add S3 tests for grants conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

move ListBucket to new policy class

policy: add a new policy class to make creation of complex policies

Since policies can have allow/deny rules etc

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object with canned acl

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test metadta copy key

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put obj using copy conditionals

Test s3:x-amz-copy-source conditional on put obj with x-amz-copy-source

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test for acl grants conditionals on put bucket acls

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: add tests for put bucket acl with canned acl conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

s3: policy tests for ListBucket with prefix, delimiter & max-keys

Allow conditionals on ListBucket similar to s3 docs which allow for
these clauses

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test put object tagging with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

policy: test get object tagging with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add policy tests for get object with conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

add bucket-policy attr to all the policy tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

test_s3: modify make_json_policy to support conditionals

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

bucket policy: improve the helper functions used for tagging upwards

- Improve `make_json_policy` to support conditionals in policy
- Move the helper functions for creating policies up so that bucket
  policy tests can use these
- add bucket-policy attribute to the tagging tests using policy

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #204 from linuxbox2/wip-post-ctype

Add test case for POST with no Content-Type header

Add test case for POST with no Content-Type header

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #199 from cfanz/cfanz-version-id-wip

Add test case for getting version-id when upload objects  bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Add test case for getting version-id when upload objects to bucket

For default-bucket and versioning-suspended-bucket, no version-id should return when upload.
For versioning-enabled-bucket, an non-empty version-id should return and equal to what
bucket list shows.

Signed-off-by: Xinying Song <songxinying@cloudin.cn>

Merge pull request #196 from ZVampirEM77/wip-lc-typos

Fix a typos for lifecycle test case

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #195 from ceph/wip-fix-416

Check for either 400 or 416 response code for invalid range test

cleanup unneeded comments

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

Fix a typos for lifecycle test case

According to the commit 9204283def8e0bcfe14a57f0c87ed20735214f53,
lc debug interval should be set to 10s.

Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>

check for either 400 or 416 response code for invalid range test

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #189 from joke-lee/list_bucket_versiong_with_marker

test for https://github.com/ceph/ceph/pull/17934

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test cors option request with Access-Control-Request-Headers
test for https://github.com/ceph/ceph/pull/18556

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #190 from cbodley/wip-pr-17882

expect 400 for missing sse headers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

expect 400 for missing sse headers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #171 from cbodley/wip-amz-date-precedence

check precedence of Date and X-Amz-Date in signatures

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test for https://github.com/ceph/ceph/pull/17934

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #186 from tipabu/check-grant-ordering

Care less about ordering when checking grants

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #185 from theanalyst/lc-time-fixes

lc: Give more flexibility for LC expiration times

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Care less about ordering when checking grants

Some tests, like test_object_header_acl_grants and
test_bucket_header_acl_grants, use the same id for all permissions.
Sort() is stable, so those tests end up testing the order of acl grants
returned by Boto. Not sure, but I think this may in turn depend on the
order of HTTP headers, where the order is not significant.

Signed-off-by: Tim Burke <tim.burke@gmail.com>

lc: Give more flexibility for LC expiration times

Earlier values expected a lc debug interval of 2s, which is a pretty
short time window and can often lead to failures if the processing
didn't complete within the next day. This commit assumes the currently
configured LC debug interval of 10s, and gives time intervals using the
following logic:

Worst case:
LC start-time : 00:00
obj upload    : 00:01
LC run1       : 00:10 -> object not expired as it is only 9s old
LC run2       : 00:20 -> object will expire in this run, however we
can't exactly guess when this run will complete, for a moderate amount
of objects this can take anywhere between 1 to a max of 10s, let us give
it a wiggle room to complete around 8s, given the amount of objects in a
teuthology run, it should be mostly probable that the object is already
deleted within this time, so at 28s, we should have seen day1 objects being
expired.

Best case:
LC start-time: 00:00
obj upload : 00:09
LC run1 : 00:10
LC run2 : 00:20 -> obj expires, so elapsed time is around 11->19s (of
course it would almost close to 10s too), We should probably configure
the LC lock time to 10s as well just so as to ensure that the lock isn't
held for the default 60s in which case it is possible that the object
might expire in a time greater than the lock interval.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #173 from chuang-he/fix_decrypt

Data encryption is not follow the AWS agreement

Reviewed-by: Casey Bodley <cbodley@redhat.com>

modify test_sse_kms_method_head() for aws standard

Signed-off-by: hechuang <hechuang@xsky.com>

add test_sse_kms_read_declare() for aws standard

Signed-off-by: hechuang <hechuang@xsky.com>

rgw: Data encryption is not follow the AWS agreement

Encryption request headers should not be sent for GET requests and HEAD
requests if your object uses SSE-KMS/SSE-S3 or you’ll get an HTTP 400
BadRequest error.

Signed-off-by: hechuang <hechuang@xsky.com>

Add missing versioning tag

Signed-off-by: Andrew Gaul <andrew@gaul.org>

Merge pull request #179 from andrewgaul/copy-part-invalid-range

Add test for invalid copy part range

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #181 from theanalyst/lc-expir-disable

lc: set attribute as lifecycle_time for lc tests using timing ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

lc: add attr lifecycle_expiration for lc tests using expiration

Allows us to additionally filter out these tests which currently fail in
the teuthology runs

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #180 from theanalyst/lc-filter

lifecycle: add test cases using filter & no id

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

lifecycle: add test cases using filter & no id

Added test cases for LC policies with no ID in xml, no prefix and an
empty and non empty Filter.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>