test_s3: fix test_bucket_logging_request_id for newer botocore

The test was using a boto3 event handler to capture the x-amz-request-id
header, incorrectly accessing kwargs["response"] in an after-call event.
Botocore's after-call event does not pass a "response" kwarg; it passes
"parsed", "http_response", "model", and "context". Fix by reading the
request ID directly from the put_object() response, which is simpler and
works across all botocore versions.

Fixes: https://tracker.ceph.com/issues/76587
Signed-off-by: cheese-cakee <farzanaman99@gmail.com>

Merge pull request #746 from cheese-cakee/wip-75416-ceph-master

test: add test for standard access log request ID

test: add test for standard access log request ID

Add test_bucket_logging_request_id to verify that the RequestID field
in the standard access log matches the x-amz-request-id returned in
the S3 PUT object response.

Related: ceph/ceph#67984, tracker.ceph.com/issues/75416
Signed-off-by: cheese-cakee <farzanaman99@gmail.com>
(cherry picked from commit 3b99093049296d761a77baa234508688767c6090)

rgw/multipart: test idempotent complete-multipart cksum and etag

Verify that correct etag and checksum (if any) are returned from
a complete-multipart operation that is retried.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 9ca96ba0fc3d3537d947c4157c40abbe7f0fca1e)

s3tests: add tests for per-bucket cloud transition targeting

Add tests to validate the target_by_bucket feature which allows each
source bucket to transition objects to a dedicated destination bucket
rather than sharing a common target.

New tests:
- test_lifecycle_cloud_transition_target_by_bucket: validates objects
  land in bucket-specific targets without source bucket name prefix
- test_lifecycle_cloud_transition_target_by_bucket_multiple_buckets:
  validates isolation between different source buckets

Each test will perform a restore of the object transitioned.

Signed-off-by: Matthew N. Heler <matthew.heler@hotmail.com>
(cherry picked from commit 2fc33fc980b1e5d474e463e4e48db20c2a244c65)

s3: add fails_on_dbstore mark

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4895b3af073d2c31436d9a4159c8aceaff1b31aa)

Remove fails_on_rgw to facilitate testing a fix

Signed-off-by: Jacques Heunis <jheunis@bloomberg.net>
(cherry picked from commit f1453774f7bbc36f722681bdb872cfc1af1bf9b5)

s3: Add a test for attempting to complete an unstarted multipart upload

The AWS documentation for CompleteMultipartUpload specifies that the call will
return a 404 with the text "NoSuchUpload" to indicate that the specified
multipart upload does not exist. This could be the result of an invalid upload
ID or a multipart upload that has been aborted or completed.

Of particular significance is the requirement that this API call return a
400-series error (indicating a client-side problem) and not a 500-series error
(indicating a server-size problem).
At the time of writing, Ceph returns a 500 in this scenario which may be
handled incorrectly by client SDKs and is likely to be misinterpreted by Ceph
monitoring as an increase in errors that require operator intervention to fix.

CompleteMultipartUpload docs available at:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html

Signed-off-by: Jacques Heunis <jheunis@bloomberg.net>
(cherry picked from commit 5506af825b6dde397da7926cdcab889ff537c506)

s3: add test_head_object_404_with_policy_prefix()

Fixes: https://tracker.ceph.com/issues/74398
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ee1ca235d94f9ed272878943ae6991ddaf60763b)

Merge pull request #743 from nbalacha/wip-nbalacha-backports

rgw-bucket-logging: handle presigned SigV2 requests

rgw-bucket-logging: handle presigned SigV2 requests

Adds a test to check that the log record AuthenticationType is
correctly set to QueryString for SigV2 presigned URLs.

Fixes: https://tracker.ceph.com/issues/76206
Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit 3ff252aa1950023a05390acb0cfbb9f4a7d26bd1)

sns: test sns:ListTopics with non-root account user

test that identity policy can grant access to ListTopics

Fixes: https://tracker.ceph.com/issues/74595
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 77e550338255e6251eca6241f989812e5ee47951)

s3: replace deprecated ssl.wrap_socket()

this was deprecated in python 3.7 and removed in 3.12

Fixes: https://tracker.ceph.com/issues/75943
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 54796a4965436bf85ffe635411d856f2f4935afe)

rgw/sts: modifying test_assume_role_creds_expiry
to check for ExpiredToken (400) error code
instead of AccessDenied (403).

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>
(cherry picked from commit c7961972036bb0cb73a136e718e106609f3de0a0)

Merge pull request #724 from nbalacha/ceph-master

rgw/bucket-logging: set retain date to the future

rgw/bucket-logging: set retain date to the future

Sets the date for PutObjectRetention to 60 days in the
future.

Fixes: https://tracker.ceph.com/issues/74322
Signed-off-by: N Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit 03ee2030aaecc7e191488a7dba0f792634174045)

Merge pull request #713 from yuvalif/wip-yuval-74102

rgw/bucket-logging: handle async log commits

rgw/bucket-logging: handle async log commits

Modified to work with the new asynchronous buckt logging commit
mechanism.

Fixes: https://tracker.ceph.com/issues/71365
Signed-off-by: Nithya Balachandran <nithya.balachandran@ibm.com>
(cherry picked from commit ec113ccbca1dcb3c34fce4f47e9ec26b58d30259)

s3: test checksum for get_object with part number

(cherry picked from commit 01075ff90fff4b6e7161aa850cf2d9ed9b58140e)

s3: update etag for get_part tests

(cherry picked from commit c7a41f8e75e694703e77fc06502bed7595a49aef)

test_multipart_sse_c_get_part: pass enc attrs on mp complete

Based on the new fixes around copy enc, those headers are required
when completing mp upload.

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 5982e64bc60362cded5e9fabdd52fb302575e034)

refactor: merge test_copy_enc_ with test_copy_enc_storage_class

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 1edf7e5b214b137d252a279134ba921f2b7e74f5)

UploadPart: test to verify sse-c headers

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 9ca6d04e698ab64f8aec2ffc005fdd8b5bf80bbe)

CompleteMultipartUpload: test to verify sse-c headers

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 0086db16fc57a84e0ff5d8dba4a096636088bb29)

UploadPart: add copy part with encryption

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit 9732871b964e797cce74de0e1af5ed6a3d058290)

lc-transition: add test with encrypted objects

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit db3d28bbcf1997850ca70ebf83416b36a2a78fda)

CopyObject: add storage class copy tests

Signed-off-by: Seena Fallah <seenafallah@gmail.com>
(cherry picked from commit eed0551f3308778ab561146e3d6e1c8226d8f9dc)

s3: test object ownership

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6e67add3832e46be91ff29817ca2417ccbabcdf9)

Conflicts:
s3tests_boto3/functional/test_s3.py moved to s3tests/

functional: fix invalid escape sequence deprication warning

This change address multiple messages in the log:

    DeprecationWarning: invalid escape sequence '\$'

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@clyso.com>
(cherry picked from commit f3fc14f42ee37e21afc890a3a3b4b15c5c39170a)

Merge pull request #707 from yuvalif/ceph-master-73675

rgw/logging: change test to match the new source cleanup process