test_s3: verify list_object_versions after multi-object delete with versioning

Extend test_versioning_concurrent_multi_object_delete to also verify that
list_object_versions returns no Versions or DeleteMarkers after all objects
and their versions have been deleted. This ensures both list_objects and
list_object_versions reflect an empty bucket.

(cherry picked from commit 1062efc8c43a3c8ee539ac98f9cd5d9c96457c6c)

test_s3 versioning multi object delete - add more versions for each object

Signed-off-by: Oguzhan Ozmen <oozmen@bloomberg.net>
(cherry picked from commit 8c43ceb5b8dd2a3aa3e257038c98ca87fd67304f)

extend test_versioning_obj_suspended_copy to test copy to another non-versioned bucket

Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>
(cherry picked from commit 6b715be9fe992ff79552bd46a60864fddc1f115e)

s3: Mark tests known to fail on dbstore

Fixes: https://tracker.ceph.com/issues/72094
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 85a9b0662b0b4f7d705a4987e7161c69e7e811ad)
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

s3: test list_buckets with MaxBuckets/ContinuationToken

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b2c4c07a21533fd15a5fbbf5701fe43625953d68)

s3: test_block_public_object_canned_acls() tests that 'private' works

Fixes: https://tracker.ceph.com/issues/49135
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1f6e6f2d2c13c35851f197a4cebf232fc0071e60)

s3: test for aws-chunked in Content-Encoding responses

aws-chunked should not be persisted on upload for return on download

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit f549c9055f4db62743ec69f97a91b5f97a9eb157)

rgw/cloud-restore: verify if the expiry-date is updated

This is the test-case for https://github.com/ceph/ceph/pull/64804.
If the `restoe-object` request is re-issued on a cloud-restored object,
it should update restoration period accordingly from the time the request
is re-issued.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>
(cherry picked from commit ccdb9d15ab928c0e106a22ca86c88c1b969f2b54)

Merge pull request #675 from yuvalif/wip-ceph-master--bucket-logging

rgw/logging: ceph master backports 2

bucket_logging: remove duplicate function

Removes the duplicate _verify_record_field function definition.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit 9c2181144b87140b633493a4706afc1abf8c3083)

bucket-logging: test the AuthType log field

Adds a test to check the value of the AuthType field.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit 8f865c2b4a0c325f04692b521b5b2b618e3a12aa)

bucket-logging: fix typo

Fixes typo in function name.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit 089d96df7041cefd31a46a728d61d650374dbe5c)

bucket-logging: tests for the ACLRequired field

Tests that the AclRequired field is set correctly in the bucket logging record.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit 0616754ea56fbe70f1ad3e703d9905162933c77e)

readme: fix typo

Fixed typos in the example commands.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit 67cbe2587be6d4a3f9a96cf20b71f3a5ea7d807d)

s3: more testing for conditional put

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>
(cherry picked from commit 8be24ac631850e7da8e52fad6041c36684acb606)

s3: fixed tests

we expected an error if we delete non-existent objects but that is not
true

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>
(cherry picked from commit 68334301db8e2c24d4d6f2ed2f0d54ae4f453e0f)

s3: test delete_objects() with Size

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 474a7e25b109e8e5394b8c11a509b51e40dbef2f)

s3: test delete_objects() with LastModifiedTime

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e6c5ad3efbc73e4396eb5bdddb54c1b3ef1b1d91)

s3: test delete_objects() with ETag

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d9c90411b92417a7067138a127bdc0ddbc6e9ba6)

s3: test delete_object() with IfMatchSize

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit f7647bae0f40a3b5cbc9a9bbe01500c97f2aad64)

s3: test delete_object() with IfMatchLastModifiedTime

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d67d70d0f11aacfdf01a1b80d9bc23d21b041367)

s3: test put_object() with IfMatch and IfNoneMatch

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5f6d5b12450df7919d175435b5ab05e8fea77e56)

delete, rgwlc: delete marker tests

0. non-creation in non-versioned buckets
1. creation in versioned buckets
2. creation in versioning-suspended buckets
3. delete-marker expiration (lifecycle)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 47712fff0a1a2ea08ca8a866c88a1473e13df708)

rgw/sts: GetAccountSummary test

Signed-off-by: Raja Sharma <raja@ibm.com>
(cherry picked from commit 1a6dd6ea9bebd8cd054c5b44fdfb0683b0dd07c8)

Merge pull request #665 from yuvalif/wip-ceph-master-logging

rgw/logging: ceph-master backports

rgw/logging: add tests for logging object key format

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 617437bb1787173acbf08c7fcea21ecba120fae9)

rgw/logging: test conf last modified time

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 7c217886514b00cb9e9eb807b9f04c9f619dba97)

rgw/logging: test part upload in mpu in standard mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit ea7930fd6dda0539457a4b737ab26dffc6bcae75)

rgw/logging: test log object name order

verify that the names of the log objects in the log bucket
are lexicographically ordered

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 6febed20488c21f233ef62b9effb466efcbd5f50)

rgw/logging: test mpu complete in standard mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit fcd58a8a49e07503af4db958f1c6fc85e41fb738)

rgw/logging: test object metadata changes in journal mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit 60be14c4adb295b0cc2a533f2af7cbcb7f761539)

rgw/logging: add tests to verify wildcrad policy

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit bddaa945e1b7090a97824b7d86ca94e1a1ba01b9)

rgw/logging: add tests for flushed object name

when using API call to flush the pending logging object of
the bucket, the flshed object name is returned.
tests were added to verify the right name is returned.

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
(cherry picked from commit c9de24cfdc025d5b52c068df4c93b0cf0c32789e)

fix: Fix typo

Signed-off-by: Emin <emin.sunacoglu@clyso.com>
(cherry picked from commit 5090f49508d52723753621093ae53bc59efdb7b2)

s3: fix: Changed test so that it accepts the character that has % sign in it as a key

(cherry picked from commit cca670f505fac6d448edf721cb83ecc4eb58eae8)

s3: Add tests for test_upload_part_copy_percent_encoded_key

(cherry picked from commit f33d068f6df36251fc3b1e3996836e03279d4109)

increase time out for restore op in test_restore_object_permanent
and in test_lifecycle_expiration_size_gt() for lc.

Fixes: https://tracker.ceph.com/issues/71348
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
(cherry picked from commit b58d24c95003aa4defb7a5424a7458910bece5df)

website: routing_setup() as pytest fixture

c80e9d2118ca5bcdec5285898a532f8e18faa6b6 had removed the use of
routing_setup() without adding a fixture to replace it. this had broken
routing_check(), but we hadn't noticed it because pytest hadn't
supported the 'yield' pattern in test_routing_generator() to call it

test_routing_generator() now uses this routing_setup fixture to provide
the kwargs that had previously been injected into routing_check() via
@common.with_setup_kwargs(setup=routing_setup, teardown=routing_teardown)

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9839a407c2353125bb867dec67a8f9960b1a2065)

website: use pytest parametrize for test_routing_generator()

resolves error with recent pytest:

==================================== ERRORS ====================================
____________ ERROR collecting s3tests/functional/test_s3_website.py ____________
'yield' keyword is allowed in fixtures, but not in tests (test_routing_generator)

use https://docs.pytest.org/en/7.1.x/example/parametrize.html instead to
generate a test_routing_generator() for each entry in ROUTING_RULES_TESTS

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d587a4a49983651258f91a8a47b7faf0e7138ec3)

add missing 'assert' to test_bucket_logging_roll_time

saw a failure after this:
```
        response = client.list_objects_v2(Bucket=log_bucket_name)
        keys = _get_keys(response)
        len(keys) == 1

>       key = keys[0]
E       IndexError: list index out of range

s3tests_boto3/functional/test_s3.py:16637: IndexError
```

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6e26d4433591f0e054df96b19ef103c92f0e3f34)

s3: avoid 100-continue bug for test_encryption_sse_c_deny_algo_with_bucket_policy

was failing for a reason unrelated to sse-c or bucket policy

Fixes: https://tracker.ceph.com/issues/70607
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 562f0f77c07efb1c070d7aef7ecaf1a6bf2756cd)

add test coverage for DeleteBucketLifecycle

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 886a769174bc3ca01243ffd0868037fc51a14892)

Merge pull request #641 from yuvalif/wip-yuval-bucket-logging-rollover

rgw/logging: mark test that require rollover time set

rgw/logging: mark test that require rollover time set

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #628 from thotz/handle-allow-readthrough-in-testcases

rgw/s3tests: handle allow read through values in cloud transitions cases

Merge pull request #624 from pritha-srivastava/wip-rgw-remove-fails-on-rgw

rgw/s3tests: removing fails_on_rgw for

Merge pull request #563 from clwluvw/restrict-public-buckets

BucketPublicAccessBlock: add tests for RestrictPublicBuckets

Merge pull request #639 from thotz/increase-time-out-for-transition-test-cases

rgw/cloud-transition: increase time out for test cases

rgw/cloud-transition: increase time out for test cases

Fixes: https://tracker.ceph.com/issues/69411
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #634 from yuvalif/wip-bucket-logging-policy

rgw/logging: bucket logging policy

Merge pull request #626 from linuxbox2/wip-crc64nvme

Wip crc64nvme

rgw/logging: test bucket logging with accounts

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: fix tenant test

tests were failing in teuthology since the tenanted user name
already have the tenant in its name

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

enhance additional checksum tests

includes tests for CRC64NVME, tests for selecting
COMPOSITE and FULL_OBJECT checksums

a decomposed matrix of tests for all checksum types

also removes the mixed checksum upload case that no longer
works in recent boto3

cleanups, add sha1

checksum validation failure (mismatch) returns BadDigest

multipart checksum matrix helper now validates checksum and checksum type
for all operations which can return them (complete-multipart, head-object,
and get-object-attributes)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

add minimal put-object for CRC64NVME

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #635 from cbodley/wip-69527

test put_bucket_acl with group uri

rgw/logging: add bucket logging permission tests

this is to cover new functionality added in:
https://github.com/ceph/ceph/pull/62284

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: add more bucket logging error cases

this is to cover these checks:
* source bucket ownership
* "requester pays" on log bucket

that were added in: https://github.com/ceph/ceph/pull/62284

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: add basic policy to target bucket

this commit needed to be able to run bucket logging regression
against: https://github.com/ceph/ceph/pull/62284
since target bucket requires policy for bucket logging to work
this only covers the positive cases from bucket logging policy
perspective

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test put_bucket_acl with group uri

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/logging: add missing marker to remove warnings

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #614 from jzhu116-bloomberg/wip-63799

add test case for stacked delete-markers

Merge pull request #625 from cbodley/wip-70191

s3: test bucket policy evaluation for CreateMultipartUpload

Merge pull request #567 from clwluvw/policy-remove-self-access

BucketPolicy: add tests for ConfirmRemoveSelfBucketAccess header

BucketPolicy: add tests for ConfirmRemoveSelfBucketAccess header

Refrence: https://github.com/ceph/ceph/pull/57629
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #627 from soumyakoduri/wip-skoduri-restore

cloud-restore:  Add testcase for restoring non-current versioned objects

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

rgw/s3tests: handle allow read through values in cloud transitions cases

Check the allow_read_through values for cases test_lifecycle_cloud_transition
and test_read_through

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

cloud-restore: testcase for non-current versioned object

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #599 from shreyanshjain7174/restore-s3tests

rgw/restore: s3tests to test restore object functionality

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #622 from cbodley/wip-70013

s3: test that object lock can be enabled on an existing bucket

Merge pull request #621 from cbodley/wip-69936

s3: reenable tests with range requests and checksums

mark test_versioning_stack_delete_merkers fails_on_dbstore

Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>

s3: test bucket policy evaluation for CreateMultipartUpload

test case for https://tracker.ceph.com/issues/70191

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/s3tests: removing fails_on_rgw for
test_versioned_concurrent_object_create_concurrent_remove
as the associated tracker is marked as resolved and
the test also passes against main.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

running cloud restore tests seperately

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

s3: test that object lock can be enabled on an existing bucket

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/restore: s3tests to test restore object functionality.

This tests are added to tests temporary restore, permanent restore and
read through resotre object functionality, this includes zonegroup
parameters and checks.

Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com>
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

s3: reenable tests with range requests and checksums

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #619 from cbodley/wip-unpin-botocore

unpin botocore, disable failing sigv2/checksum tests

Merge pull request #620 from ArbitCode/wip-rgw-raja-remove-clientid-oidc-provider

rgw/iam: add RemoveClientIDFromOpenIDConnectProvider

rgw/iam: add RemoveClientIDFromOpenIDConnectProvider

Signed-off-by: Raja Sharma <raja@ibm.com>

sns: clear region_name for s3 clients

work around sns failures due to default region_name='':

> ERROR s3tests_boto3/functional/test_sns.py::test_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name.
> ERROR s3tests_boto3/functional/test_sns.py::test_cross_account_topic_publish - botocore.exceptions.EndpointResolutionError: Invalid region: region was not a valid DNS name.

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: disable checksum calculation for test_object_create_bad_contentlength_negative

if tls is enabled, boto will switch to STREAMING-UNSIGNED-PAYLOAD-TRAILER
and omit the provided content-length header. this leads to test failure:

> ________________ test_object_create_bad_contentlength_negative _________________
>       e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, ContentLength=-1)
> AssertionError: ClientError not raised

Signed-off-by: Casey Bodley <cbodley@redhat.com>

iam: remove invalid Bucket param from list_buckets()

> botocore.exceptions.ParamValidationError: Parameter validation failed:
> Unknown parameter in input: "Bucket", must be one of: MaxBuckets, ContinuationToken, Prefix, BucketRegion

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: disable multipart/ranged tests with checksum failures

boto update exposed some bugs in our checksum feature around multipart
copy and ranged requests. disabling them with fails_on_rgw until
https://tracker.ceph.com/issues/69936 is resolved

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: disable request_checksum_calculation for test_multipart_checksum_upload_fallback

the test sends create_multipart_upload() with a requested checksum
algorithm, and upload_part() requests without. newer boto automatically
adds `x-amz-sdk-checksum-algorithm: CRC32` when nothing is specified,
so we have to explicitly disable that behavior via botocore config

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: clean up use of tenanted bucket names

the botocore.handlers.validate_bucket_name validation is enabled by
default, but we can disable it with unregister(). this avoids having to
mess with the signature or url

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: sse-kms policy tests use v4 client

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: mark sigv2 tests with fails_on_rgw

Signed-off-by: Casey Bodley <cbodley@redhat.com>

requirements: unpin botocore version

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #616 from ceph/wip-alimasa-sns-cross-tenant

RGW | BucketNotification: fix test to expect authorization failure

Reviewed-by: yuvalif<ylifshit@redhat.com>

rgw|BN: fix test to expect authorization failure

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #611 from ArbitCode/wip-raja-fix-multipart-upload-cant-get-obj-tag

RGW:test-case to fix obj by multipart upload cant get tag

Merge branch 'ceph:master' into wip-raja-fix-multipart-upload-cant-get-obj-tag

Merge pull request #610 from ArbitCode/wip-raja-rgw-test-support-64526

rgw test-case to support x-expected-bucket-owner

add test case for stacked delete-markers

Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>

test-case to fix obj by multipart upload cant get tag

Signed-off-by: Raja Sharma <raja@ibm.com>

Merge pull request #607 from yuvalif/wip-yuval-bucket-logging-tenant

rgw/logging: increase test coverage

Reviewed-By: Ali Masarwe <ali.masarwa@ibm.com>

rgw test-case to support x-expected-bucket-owner
If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).
PR: ceph/ceph#61215
Fixes: https://tracker.ceph.com/issues/64526
Signed-off-by: Raja Sharma <raja@ibm.com>

rgw/logging: fix typo in readme

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>