Merge pull request #680 from soumyakoduri/wip-skoduri-exp-date

rgw/cloud-restore: verify if the expiry-date is updated

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

rgw/cloud-restore: verify if the expiry-date is updated

This is the test-case for https://github.com/ceph/ceph/pull/64804.
If the `restoe-object` request is re-issued on a cloud-restored object,
it should update restoration period accordingly from the time the request
is re-issued.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #678 from galsalomon66/fix_assert_upon_empty_result

fix upon an empty results.

a fix for empty results.for some of the queries
an empty result that may rarely happen, could cause an assertion.
the fix handle that cases.

Signed-off-by: gal salomon <gal.salomon@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge pull request #663 from cbodley/wip-s3-conditional-write

s3: test conditional put_object() and delete_object()

Reviewed-by: Ali Masarwa <amasarwa@redhat.com>

Merge pull request #664 from ArbitCode/wip-raja-get-account-summary-tests

rgw/sts: GetAccountSummary test

s3: more testing for conditional put

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

Merge pull request #661 from ArbitCode/wip-raja-get-caller-identity-tests

rgw/sts: GetCallerIndentity test

s3: fixed tests

we expected an error if we delete non-existent objects but that is not
true

Signed-off-by: Ali Masarwa <ali.saed.masarwa@gmail.com>

s3: test delete_objects() with Size

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: test delete_objects() with LastModifiedTime

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: test delete_objects() with ETag

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: test delete_object() with IfMatchSize

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: test delete_object() with IfMatchLastModifiedTime

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: test delete_object() with IfMatch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: test put_object() with IfMatch and IfNoneMatch

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #672 from ceph/wip-dm-70853

delete, rgwlc: delete marker tests

Merge branch 'master' into wip-dm-70853

Merge pull request #674 from nbalacha/wip-nbalacha-cleanup

bucket_logging: remove duplicate function

bucket_logging: remove duplicate function

Removes the duplicate _verify_record_field function definition.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

Merge pull request #667 from nbalacha/wip-nbalacha-71730

bucket-logging: test for log record fields

Merge pull request #670 from nbalacha/wip-nbalacha-71731

bucket-logging: test the AuthType log field

delete, rgwlc: delete marker tests

0. non-creation in non-versioned buckets
1. creation in versioned buckets
2. creation in versioning-suspended buckets
3. delete-marker expiration (lifecycle)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw/sts: GetCallerIndentity test

Signed-off-by: Raja Sharma <raja@ibm.com>

bucket-logging: test the AuthType log field

Adds a test to check the value of the AuthType field.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

bucket-logging: fix typo

Fixes typo in function name.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

bucket-logging: tests for the ACLRequired field

Tests that the AclRequired field is set correctly in the bucket logging record.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

Merge pull request #666 from nbalacha/wip-nbalacha-typo

readme: fix typo

readme: fix typo

Fixed typos in the example commands.

Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>

rgw/sts: GetAccountSummary test

Signed-off-by: Raja Sharma <raja@ibm.com>

Merge pull request #658 from yuvalif/wip-yuval-bucket-logging-key

rgw/logging: add tests for logging object key format

Merge pull request #653 from yuvalif/test-logging-conf-mtime

rgw/logging: test conf last modified time

Merge pull request #651 from yuvalif/test-log-obj-order

rgw/logging: test log object name order

Merge pull request #650 from yuvalif/logging-mpu-standard-mode

rgw/logging: test mpu complete in standard mode

Merge pull request #649 from yuvalif/test-logging-object-meta

rgw/logging: test object metadata changes in journal mode

Merge pull request #648 from yuvalif/logging_wildcrd_policy_test

rgw/logging: add tests to verify wildcrad policy

Merge pull request #647 from yuvalif/flushed-logging-object-name

rgw/logging: add tests for flushed object name

rgw/logging: add tests for logging object key format

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: test conf last modified time

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #655 from mertsunacoglu/wip-copy-source

Add tests for % included characters for copy source

Merge pull request #657 from thotz/fix-test_restore_object_permanent-failure

increase time out for restore op in test_restore_object_permanent

fix: Fix typo

Signed-off-by: Emin <emin.sunacoglu@clyso.com>

increase time out for restore op in test_restore_object_permanent
and in test_lifecycle_expiration_size_gt() for lc.

Fixes: https://tracker.ceph.com/issues/71348
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #659 from cbodley/wip-website-generate-routing

pytest: use parametrize for s3website test_routing_generator()

website: routing_setup() as pytest fixture

c80e9d2118ca5bcdec5285898a532f8e18faa6b6 had removed the use of
routing_setup() without adding a fixture to replace it. this had broken
routing_check(), but we hadn't noticed it because pytest hadn't
supported the 'yield' pattern in test_routing_generator() to call it

test_routing_generator() now uses this routing_setup fixture to provide
the kwargs that had previously been injected into routing_check() via
@common.with_setup_kwargs(setup=routing_setup, teardown=routing_teardown)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

website: use pytest parametrize for test_routing_generator()

resolves error with recent pytest:

==================================== ERRORS ====================================
____________ ERROR collecting s3tests/functional/test_s3_website.py ____________
'yield' keyword is allowed in fixtures, but not in tests (test_routing_generator)

use https://docs.pytest.org/en/7.1.x/example/parametrize.html instead to
generate a test_routing_generator() for each entry in ROUTING_RULES_TESTS

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #656 from cbodley/wip-bucket-logging-roll-time-assert-len

add missing 'assert' to test_bucket_logging_roll_time

Merge pull request #654 from cbodley/wip-70607

s3: avoid 100-continue bug for test_encryption_sse_c_deny_algo_with_bucket_policy

add missing 'assert' to test_bucket_logging_roll_time

saw a failure after this:
```
        response = client.list_objects_v2(Bucket=log_bucket_name)
        keys = _get_keys(response)
        len(keys) == 1

>       key = keys[0]
E       IndexError: list index out of range

s3tests_boto3/functional/test_s3.py:16637: IndexError
```

Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: fix: Changed test so that it accepts the character that has % sign in it as a key

s3: Add tests for test_upload_part_copy_percent_encoded_key

s3: avoid 100-continue bug for test_encryption_sse_c_deny_algo_with_bucket_policy

was failing for a reason unrelated to sse-c or bucket policy

Fixes: https://tracker.ceph.com/issues/70607
Signed-off-by: Casey Bodley <cbodley@redhat.com>

s3: test retry after error instead of 100-continue

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/logging: test part upload in mpu in standard mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: test log object name order

verify that the names of the log objects in the log bucket
are lexicographically ordered

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: test mpu complete in standard mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: test object metadata changes in journal mode

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #600 from gaul/copy

Tag copy tests

Tag copy tests

rgw/logging: add tests to verify wildcrad policy

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: add tests for flushed object name

when using API call to flush the pending logging object of
the bucket, the flshed object name is returned.
tests were added to verify the right name is returned.

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #645 from cbodley/wip-71083

add test coverage for DeleteBucketLifecycle

add test coverage for DeleteBucketLifecycle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #641 from yuvalif/wip-yuval-bucket-logging-rollover

rgw/logging: mark test that require rollover time set

rgw/logging: mark test that require rollover time set

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #628 from thotz/handle-allow-readthrough-in-testcases

rgw/s3tests: handle allow read through values in cloud transitions cases

Merge pull request #624 from pritha-srivastava/wip-rgw-remove-fails-on-rgw

rgw/s3tests: removing fails_on_rgw for

Merge pull request #563 from clwluvw/restrict-public-buckets

BucketPublicAccessBlock: add tests for RestrictPublicBuckets

Merge pull request #639 from thotz/increase-time-out-for-transition-test-cases

rgw/cloud-transition: increase time out for test cases

rgw/cloud-transition: increase time out for test cases

Fixes: https://tracker.ceph.com/issues/69411
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

Merge pull request #634 from yuvalif/wip-bucket-logging-policy

rgw/logging: bucket logging policy

Merge pull request #626 from linuxbox2/wip-crc64nvme

Wip crc64nvme

rgw/logging: test bucket logging with accounts

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: fix tenant test

tests were failing in teuthology since the tenanted user name
already have the tenant in its name

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

enhance additional checksum tests

includes tests for CRC64NVME, tests for selecting
COMPOSITE and FULL_OBJECT checksums

a decomposed matrix of tests for all checksum types

also removes the mixed checksum upload case that no longer
works in recent boto3

cleanups, add sha1

checksum validation failure (mismatch) returns BadDigest

multipart checksum matrix helper now validates checksum and checksum type
for all operations which can return them (complete-multipart, head-object,
and get-object-attributes)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

add minimal put-object for CRC64NVME

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #635 from cbodley/wip-69527

test put_bucket_acl with group uri

rgw/logging: add bucket logging permission tests

this is to cover new functionality added in:
https://github.com/ceph/ceph/pull/62284

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: add more bucket logging error cases

this is to cover these checks:
* source bucket ownership
* "requester pays" on log bucket

that were added in: https://github.com/ceph/ceph/pull/62284

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: add basic policy to target bucket

this commit needed to be able to run bucket logging regression
against: https://github.com/ceph/ceph/pull/62284
since target bucket requires policy for bucket logging to work
this only covers the positive cases from bucket logging policy
perspective

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

test put_bucket_acl with group uri

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/logging: add missing marker to remove warnings

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #614 from jzhu116-bloomberg/wip-63799

add test case for stacked delete-markers

Merge pull request #625 from cbodley/wip-70191

s3: test bucket policy evaluation for CreateMultipartUpload

Merge pull request #567 from clwluvw/policy-remove-self-access

BucketPolicy: add tests for ConfirmRemoveSelfBucketAccess header

BucketPolicy: add tests for ConfirmRemoveSelfBucketAccess header

Refrence: https://github.com/ceph/ceph/pull/57629
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #627 from soumyakoduri/wip-skoduri-restore

cloud-restore:  Add testcase for restoring non-current versioned objects

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

rgw/s3tests: handle allow read through values in cloud transitions cases

Check the allow_read_through values for cases test_lifecycle_cloud_transition
and test_read_through

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

cloud-restore: testcase for non-current versioned object

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #599 from shreyanshjain7174/restore-s3tests

rgw/restore: s3tests to test restore object functionality

Reviewed-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
Reviewed-by: Adam Emerson <aemerson@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #622 from cbodley/wip-70013

s3: test that object lock can be enabled on an existing bucket

Merge pull request #621 from cbodley/wip-69936

s3: reenable tests with range requests and checksums

mark test_versioning_stack_delete_merkers fails_on_dbstore

Signed-off-by: Jane Zhu <jzhu116@bloomberg.net>

s3: test bucket policy evaluation for CreateMultipartUpload

test case for https://tracker.ceph.com/issues/70191

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/s3tests: removing fails_on_rgw for
test_versioned_concurrent_object_create_concurrent_remove
as the associated tracker is marked as resolved and
the test also passes against main.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

running cloud restore tests seperately

Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

s3: test that object lock can be enabled on an existing bucket

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw/restore: s3tests to test restore object functionality.

This tests are added to tests temporary restore, permanent restore and
read through resotre object functionality, this includes zonegroup
parameters and checks.

Signed-off-by: shreyanshjain7174 <ssanchet@redhat.com>
Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>

s3: reenable tests with range requests and checksums

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #619 from cbodley/wip-unpin-botocore

unpin botocore, disable failing sigv2/checksum tests