test Get/HeadObject with partNumber for single-multipart upload

test_multipart_get_part() tests 'normal' multipart uploads. add a new
test case for a multipart upload with a single part to tests the fix
for https://tracker.ceph.com/issues/66705

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #569 from cbodley/wip-66655

requirements: unpin pytz version

requirements: unpin pytz version

Fixes: https://tracker.ceph.com/issues/66655
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #565 from bobham-bloomberg/FixListBucketsCtimeTest

Fix wrong assertion of the test: `test_buckets_list_ctime`

Fix wrong assertion of the test: `test_buckets_list_ctime`

TestName:
s3tests_boto3.functional.test_s3:test_buckets_list_ctime

Problem:
The test creates 5 buckets for a user but in an assertion check,
it asserts false if any bucket of the user has CreationTime less
than a day prior to current time.
Due to this reason the test fails if the user has pre-existing
buckets older than a day.

Solution:
Assert only on the CreationTime of buckets that were created with
test execution.

Signed-off-by: Sumedh A. Kulkarni <sumedh.a.kulkarni@seagate.com>
Co-developed-by: Bob Ham <bham12@bloomberg.net>
Signed-off-by: Bob Ham <bham12@bloomberg.net>

Merge pull request #537 from cbodley/wip-iam-user-apis

iam: add tests for account-based IAM apis

Merge pull request #557 from yuvalif/wip-yuval-mpu-etag

test etag on mpu complete replies

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

test etag on mpu complete replies

this is to cover the fix of: https://tracker.ceph.com/issues/58879

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

add tests for ObjectSizeGreater(Less)Than

Add tests for the new ObjectSizeGreaterThan and
ObjectSizeLessThan lifecycle operators.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

add test test_lifecycle_expiration_newer_noncurrent()

This verifies the new NewerNoncurrentVersions lifecycle filter
operator.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

sns: add test_sns.py for simple topic testing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: move iam_root, iam_alt_root fixtures to iam.py

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: remove test_bucket_acl_no_grants()

aws doesn't consult acls for same-account access. rgw doesn't for
account users either

Fixes: https://github.com/ceph/s3-tests/issues/184
Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: test cross-account policy with assumed role

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add account tests for GroupPolicy apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add account tests for Group apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add account test for OpenIDConnectProvider apis

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: test cross-account permissions

test the [iam alt root] user's access to buckets owned by [iam root]
using various policy principals and acl grantees

Signed-off-by: Casey Bodley <cbodley@redhat.com>

config: add [iam alt root] for an alt account's root user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: test managed role policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add account test for RolePolicy apis

adds test cases for the following iam actions:
* PutRolePolicy
* GetRolePolicy
* DeleteRolePolicy
* ListRolePolicies

verified to pass against aws when an account root user's credentials are
provided in the [iam] section of s3tests.conf

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add account tests for Role apis

adds test cases for the following iam actions:
* CreateRole
* GetRole
* ListRoles
* DeleteRole
* UpdateRole

verified to pass against aws when an account root user's credentials are
provided in the [iam] section of s3tests.conf

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: test managed user policy

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add account tests for UserPolicy apis

adds test cases for the following iam actions:
* PutUserPolicy
* GetUserPolicy
* DeleteUserPolicy
* ListUserPolicies

verified to pass against aws when an account root user's credentials are
provided in the [iam] section of s3tests.conf

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: test bucket policy principal for iam user with path

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add tests for AccessKey apis

adds test cases for the following iam actions:
* CreateAccessKey
* UpdateAccessKey
* DeleteAccessKey
* ListAccessKeys

verified to pass against aws when an account root user's credentials are
provided in the [iam] section of s3tests.conf

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: add tests for User apis

adds test cases for the following iam actions:
* CreateUser
* GetUser
* UpdateUser
* DeleteUser
* ListUsers

verified to pass against aws when an account root user's credentials are
provided in the [iam] section of s3tests.conf

Signed-off-by: Casey Bodley <cbodley@redhat.com>

config: add [iam root] for an account root user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

config: parse iam config during setup()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: rename test_of_iam mark to iam_tenant

differentiate the test cases that expect a tenant-wide IAM api from new
ones that expect an account-wide api

Signed-off-by: Casey Bodley <cbodley@redhat.com>

config: add fixtures for iam name/path prefixes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

fixtures: split setup() and configure()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #555 from cbodley/wip-64822

test_headers: use fixture to hook request headers

test_headers: use fixture to hook request headers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #547 from cbodley/wip-63724

s3: object lock tests for deletion of multipart objects

Merge pull request #551 from clwluvw/bucket-public-block

BucketPublicAccessBlock: compatibility issues

Merge pull request #552 from alimaredia/lc-utcnow-fix

replace datetime.now with datetime.utcnow()

replace datetime.now with datetime.utcnow()

when the local timezone is not UTC and if it is a day behind,
lifecycle_header tests fails with 2 days not equal to 1
so replacing datetime.now() with datetime.utcnow()

Signed-off-by: Ali Maredia <amaredia@redhat.com>

s3: object lock tests for deletion of multipart objects

Signed-off-by: Casey Bodley <cbodley@redhat.com>

PublicAccessBlock: test access deny via bucket policy

Make sure 403 is returned when access is denied via s3:GetBucketPublicAccessBlock action on GetBucketPublicAccessBlock

Refs: https://github.com/ceph/ceph/pull/55652
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

PublicAccessBlock: test 404 on no block configuration

Make sure NoSuchPublicAccessBlockConfiguration is returned when no public block is configured on bucket:

Refs: https://github.com/ceph/ceph/pull/55652
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #546 from jzhu116-bloomberg/wip-64340

add test case for object copy in versioning suspended bucket

Merge pull request #544 from tobias-urdin/improve-presigned

Add more testing for presigned URLs

Add more testing for presigned URLs

This improves the testing for presigned URLs for
both get_object and put_object when using
generate_presigned_url().

It covers the case where you pass for example
a x-amz-acl (ACL in params for generated_presigned_url)
header that should be signed.

Tests the regression in [1].

[1] https://tracker.ceph.com/issues/64308

Signed-off-by: Tobias Urdin <tobias.urdin@binero.se>

tag test_versioning_obj_suspended_copy with fails_on_dbstore

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

add test case for object copy in versioning suspended bucket

Signed-off-by: Juan Zhu <jzhu4@dev-10-34-20-139.pw1.bcc.bloomberg.com>

Merge pull request #533 from cbodley/wip-45736

304 Not Modified tests still expect etag header

Merge pull request #532 from cbodley/wip-63537

object lock: test very large RetainUntilDate

Merge pull request #491 from cbodley/wip-get-part

test Head/GetObject on completed multipart parts

test Head/GetObject on completed multipart parts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

304 Not Modified tests still expect etag header

test case for https://tracker.ceph.com/issues/45736

Signed-off-by: Casey Bodley <cbodley@redhat.com>

object lock: test very large RetainUntilDate

https://tracker.ceph.com/issues/63537 reported that large dates (with
year after 2107) got truncated when written. test with a later date, and
check that get_object_retention() gives back the date we put

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #527 from cbodley/wip-63004

add test_post_object_wrong_bucket() for CVE-2023-43040

add test_post_object_wrong_bucket()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #524 from cbodley/wip-62013

lc: test transition of plain null version

Merge pull request #525 from cbodley/wip-62919

boto2: fix byte vs. string comparison in verify_object

boto2: copy configured_storage_classes() fix from boto3

some boto2 storage class tests are failing because the list returned by
configured_storage_classes() included an empty string

the boto3 version had an extra line that removes empty values; copy that
for boto2

Signed-off-by: Casey Bodley <cbodley@redhat.com>

boto2: fix byte vs. string comparison in verify_object

the storage class tests were failing on comparisons between the input
data and output data:

AssertionError: assert 'oFbdZvtRj' == b'oFbdZvtRj'

convert the byte representation back to string for comparison

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #523 from tobias-urdin/http-options-presigned

Add test to verify HTTP OPTIONS on presigned URL

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Add test to verify HTTP OPTIONS on presigned URL

Related: https://tracker.ceph.com/issues/62033

Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

lc: test transition of plain null version

for validating the fix of https://tracker.ceph.com/issues/62013

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #522 from nadavMiz/boto3-tests-readme-clerify

Remove filtering from boto3 test example for simplicity

remove filtering from boto3 test example for simplicity

Merge pull request #517 from cbodley/wip-rm-rgw-stat-bucket-headers

remove reliance on x-rgw-object-count and x-rgw-bytes-used headers

Merge pull request #521 from soumyakoduri/wip-skoduri-setattr

Add testcase to verify obj mtime post setattrs

Add testcase to verify obj mtime post setattrs

Object mtime should not change for any attr changes unless
its a copy operation. Verify the same using PutObjectACL op.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

remove reliance on x-rgw-object-count and x-rgw-bytes-used headers

stop using head_bucket() to fetch these response headers, and use
list_objects_v2() instead to count objects and sizes

Fixes: #315
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #513 from galsalomon66/using_get_bucket_name

Using get bucket name

Merge pull request #266 from theanalyst/wip-38700-range

add multipart ranges to encryption tests

Merge pull request #514 from cbodley/wip-tox-readme

readme: change fails_on_aws back to fails_on_rgw

readme: change fails_on_aws back to fails_on_rgw

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge branch 'master' into using_get_bucket_name

instead of hard-coded bucket-name using get_bucket_name.merging PR #488 alphaprinz

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

boto3/test_s3: add ranges around part boundary for encryption tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

boto3/test_s3: range encoding helper function takes a size argument

This is to avoid a get_object call for every range check as the object size will
not change during this duration and we'd most likely already know the object
sizes beforehand

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

sse: add sse-c test with unaligned multipart parts

Fixes: http://tracker.ceph.com/issues/38700
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #463 from cfsnyder/wip-56646

Add test_versioning_concurrent_multi_object_delete

Add test_versioning_concurrent_multi_object_delete

Tests that concurrent multi-object delete requests which specify
the same versioned object instances return successful object
responses within response body.

relates to: https://tracker.ceph.com/issues/56646

Signed-off-by: Cory Snyder <csnyder@iland.com>

Merge pull request #506 from galsalomon66/json_s3tests

Json s3tests

Merge pull request #505 from cbodley/wip-59218

test_sse_s3_default_multipart_upload verifies encryption header

progress nessage is sent back upon processing the object, the change make sure it stay with the max result

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

modifying of the run_s3select routine; to handle the different statuses (progress,stats,end)

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #501 from mkogan1/wip-chunked-trans-enc

Test object PUT with chunked transfer encoding

upon removing the payload tag, the response index should be changed also

Signed-off-by: galsalomon66 <gal.salomon@gmail.com>

Merge pull request #503 from cbodley/wip-multi-delete

boto3: multi-object-delete tests use client.delete_objects()

test_sse_s3_default_multipart_upload verifies encryption header

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #502 from cbodley/wip-58960

s3website: collections.Container removed from python3.10

Merge pull request #495 from cbodley/wip-get-object-torrent

add test_get_object_torrent

boto3: multi-object-delete tests use client.delete_objects()

when the tests were converted from boto2, they were rewritten as loops
over client.delete_object(). switch back to multi-delete

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3website: collections.Container removed from python3.10

Fixes: https://tracker.ceph.com/issues/58960
Signed-off-by: Casey Bodley <cbodley@redhat.com>

add test_get_object_torrent

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #500 from soumyakoduri/wip-skoduri-lifecycle

lifecycle: Fix test_lifecycle_expiration_header_* testcases

test object PUT with chunked transfer enconding

Before the RGW fix PR was responding with 411 instead of 200

RGW fix PR: https://github.com/ceph/ceph/pull/50235

Signed-off-by: Mark Kogan <mkogan@redhat.com>

lifecycle: Fix test_lifecycle_expiration_header_* testcases

Few checks were incorrectly mapped when switched to 'assert'. This
commit fixes the same.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #499 from soumyakoduri/wip-bz58916

dbstore: add back missing 'fails_on_dbstore' tag

Merge pull request #490 from cbodley/wip-489

boto3: list_versions() omits empty KeyMarker/VersionIdMarker

dbstore: add back missing 'fails_on_dbstore' tag

Mark testcase "test_lifecycle_expiration_header_and_tags_head" as
fails_on_dbstore

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #498 from m-ildefons/wip/tox-ini-fixes

QoL: Fix tox.ini syntax and other minor things

QoL: Fix tox.ini syntax and other minor things

- Fix tox.ini syntax

Modern tox versions require the expected environment variables to be
listed one by one on separate lines in tox.ini

- Add `venv` to list of ignored names for git

This is a common name for a local Python virtual environment. Less
typing than `virtualenv`

- Add `tox` to requirements.txt

Installing `tox` via `pip` has the advantage of including it in the
virtual environment, thus avoiding trouble on operating systems shipping
by default with python3.6 or older. It's also nice that `pip install -r
requirements.txt` is now sufficient to set up the testing environment,
after initializing the virtual environment with a moder-enough python
version.

Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

Merge pull request #496 from cbodley/wip-58762

iam: add back missing fails_on_dbstore tags