Matt Benjamin [Sat, 22 Jun 2024 20:12:54 +0000 (16:12 -0400)]
rgw_cksum: address review comments
* remove rgw_cksum_pipe state enum, not needed [Casey review]
* remove a format that just took a single string substitution
and passed it to an iostream [Casey review]
* use boost::to_upper* [Casey review]
* remove unused RGW_ATTR_CKSUM_ALGORITHM decl [Casey review]
* negate error code values in two places [Casey review]
* split cksum digests from base type decls
* resolve comment when checksum requested but not available
* remove redundant memset
* remove junk from rgw_blake3_digest.h
* s/ldpp_dout + fmt::format/ldpp_dout_fmt/g;
* fix conditional return of parts_count
from RGWRados::Object::prepare(). A value for parts_count should
be returned iff a *multipart* object manifest exists.
* remove /tmp output test
* finish moving ceph_crypto headers out of rgw_cksum.h
* consume the optional in multipart_parts_count
* target_attrs can be a reference (but not const)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Mon, 15 Apr 2024 20:44:58 +0000 (16:44 -0400)]
rgw_cksum: implement POST upload checksums
* properly transform pseudo headers in PostObj
* enable cksum verify in PostObj
* match checksum headers in match_policy_vars
* fixup add POST headers to environment
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Sun, 5 May 2024 18:27:06 +0000 (14:27 -0400)]
rgw_cksum: fix ReadOp comment
parts_count now returns a value for parts_count whenever the
target object is a multipart upload. this has no additional
overhead, since this can be read off the manifest
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Wed, 10 Apr 2024 21:58:15 +0000 (17:58 -0400)]
rgw_cksum: multipart upload checksums
includes commits:
* fixes init-multipart header return
* introduce checksum to SAL MultipartPart interface
* thread optional checksum through DataProcessor
* code complete multipart checksum verify
* fix formatter
* fix ckecksum format for multipart objects in GET/HEAD ops
* always return parts_count from ReadOp::prepare() if applicable
This behavior is used when returning the checksum of a multipart
upload object.
* tweak conditional multipart_parts_count
* add checksum output to ListMultipart
* fix nil-return from GetHeaderCksumResult
* re-arm truncated if re-entering list-parts
* complete-multipart w/list-parts
* validate supplied checksum in CompleteMultipart
* verify checksum type against initial checksum algorithm
* rgw_op: suppress more x-amz headers
* final fixes and cleanups
* remove unused t0
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Tue, 24 Sep 2019 21:12:02 +0000 (17:12 -0400)]
rgw: add checksum and digest machinery
Adds new Blake3 digest format (native), a concrete type to
represent digests, and static_visitor machinery to unify varying
checksum computations.
This framework, together with new trailing checksum header support,
is used to implement S3 additional checksum verification. Parts of
the AWS content checksum API work build on a prior contribution from
imtzw <tongzhiwei_yewu@cmss.chinamobile.com>.
Thank you!
Fixes: https://tracker.ceph.com/issues/42080 Fixes: https://tracker.ceph.com/issues/63951
squashed commits:
* rgw_cksum: add trival test vectors for sha-format digests
Computed digests match those produced by sha1sum, sha256sum,
and sha512sum utilities.
* rgw_cksum: add test vectors for blake3
Tests the same input strings with digests validated by
b3sum (https://crates.io/crates/b3sum).
* rgw_ckum: switch to accel crc32c
The internal Ceph convention appears to be to omit a final
xor where ceph_crc32c is used, but it's required for compatibility
with AWS implementations.
* rgw_cksum: add XXH3 digest
* rgw_cksum: write class encoder for rgw::digest::Cksum
* rgw_cksum: also reverse crc32c (REBASEME)
Mark noticed that the crc32c output was being tested against a
byteswapped value (crc32c also needs byteswap on LE).
* rgw_cksum: add digest::Cksum serde tests
* rgw_cksum: fix main(...) linkage
(so we run our main unit and not the one in gmock
* rgw_cksum: convenience extensions for integration with RGW/S3
* introduce rgw_cksum unique_ptr factory
* rgw_cksum: mark string transform accessors const
* rgw_cksum: fixup unittest_rgw_chksum compilation--all existing tests pass
* rgw_cksum: hook up put-object checksum workflow
* tweaks to report on content checksum mismatch
* rgw_cksum: match SDK as well as general checksum header
* make it more efficient
* initialize RGWPutObj_Cksum::digest
* rgw_cksum: write parse_cksum_type w/const char* arg
* initialize _type correctly; doing armored wrong
* fix expected checksum header name, clean up verify
* fix output on checksum verify fail, cleanup
* introduce Cksum::to_armor(); all AWS cases pass
* oops, extra 0-byte at end of to_armor() result
* use to_armor() with decoded checksums (i.e., for all S3 presentation)
* remove unnecessary finalize() in RGWPutObj_Cksum dtor
* RGWPutObj_Cksum::Factory fixes
* fixes test_object_checksum_sha256
* choose preferred checksum algorithm header if both are present
* log verified checksums in RGWPutObj::execute at 16
* checksum not needed in policy condition
* fix checksum trailing header format
* move Blake3 to rgw_digest_blake3.h
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
rgw: fix bucket sync thread waiting inordinate amount of time
A signed value was read in as an unsigned value, so -1 was interpreted
as a very large value. This made the thread wait period in the bucket
sync thread inordinately long, preventing bucket sync and dynamic
resharding (unless values appropriate for debugging were set).
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
cls/rgw: bump cls_rgw_reshard_entry decode version to match encode
9302fbb3f5416871c1978af5d45f3bf568c2c190 bumped the version in
ENCODE_START() but missed DECODE_START(). i don't think that would cause
any decode failures, unless we later raise ENCODE_START's compat_v above
2
qa/tasks: Initialize 'monitoring_profiles' spec to an empty dict
Initialize 'monitoring_profiles' to an empty python dictionary instead of
'None' to prevent the cbt task from failing due to the TypeError exception
when attempting to iterate a 'NoneType'.
Merge pull request #58092 from zdover23/wip-doc-2024-06-18-start-os-recs
doc/start: remove mention of Centos 8 support
Reviewed-by: Adam King <adking@redhat.com> Reviewed-by: Dan Mick <dan.mick@redhat.com> Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com> Reviewed-by: Ken Dreyer <kdreyer@redhat.com> Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Repair the link to cephfs-shell.rst in doc/cephfs/cephfs-shell.rst that
was broken in https://github.com/ceph/ceph/pull/41165/ when
doc/cephfs/cephfs-shell.rst was moved to doc/man/8/cephfs-shell.rst.
This commit is made in response to a request by Lander Duncan that was
made on the [ceph-users] mailing list, and can be seen here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/F7V4CWLIYCAJ4JXI2JLNY6QPCFPR4SLA/
Seena Fallah [Fri, 28 Jun 2024 10:30:54 +0000 (12:30 +0200)]
rgw: use s3website REST when higher priority and no host header
If the Host header is missing, but the s3website API is enabled and
has a higher priority than the regular S3 API, then we should still
treat the request as a website request.
Xiubo Li [Mon, 4 Mar 2024 03:38:28 +0000 (11:38 +0800)]
mds/client: return -ENODATA when xattr doesn't exist for removexattr
The POSIX says we should return -ENODATA when the corresponding
attribute doesn't exist when removing it. While there is one
exception for the acl ones in the local filesystems, for exmaple
for xfs, which will treat it as success.
While in the MDS side there have two ways to remove the xattr:
sending a CEPH_MDS_OP_SETXATTR request by setting the 'flags' with
CEPH_XATTR_REMOVE and just issued a CEPH_MDS_OP_RMXATTR request
directly.
For the first one it will always return 0 when the corresponding
xattr doesn't exist, while for the later one it will return
-ENODATA instead, this should be fixed in MDS to make them to be
consistent.
Added a CEPH_XATTR_REMOVE2 new flags and will return -ENODATA errno
directly when the crresponding xattr doesn't exist. Just keeps the
old CEPH_XATTR_REMOVE flags to make it to be compatible with old
clients.
Fixes: https://tracker.ceph.com/issues/64679 Signed-off-by: Xiubo Li <xiubli@redhat.com>
Venky Shankar [Thu, 27 Jun 2024 16:34:37 +0000 (22:04 +0530)]
Merge PR #57619 into main
* refs/pull/57619/head:
qa/cephfs: use wait_for_daemon() instead of sleep()-ing
qa/cephfs: mark file system joinable for fs rename tests before unmounting clients
Venky Shankar [Thu, 27 Jun 2024 14:28:50 +0000 (19:58 +0530)]
Merge PR #53755 into main
* refs/pull/53755/head:
PendingReleaseNotes: add note about CephFS set_vxattrs
doc/cephfs: Update docs to match remove functionality and respective vxattrs
qa: Add test coverage for vxattr behavior
qa: Add removexattr to support setfattr removal.
mds: Implement remove for ceph vxattrs
After rollback started being tested in commit b3977c53c930
("test/librbd: make rollback in TestGroup.add_snapshot{,PP}
meaningful"), these tests can fail on comparing post-rollback
data to expected data if run with exclusive lock disabled.
This doesn't occur with exclusive lock enabled because the RBD
cache gets invalidated implicitly before releasing the lock.
While at it, pass LIBRADOS_OP_FLAG_FADVISE_FUA to avoid relying
on any cache settings that happen to be in effect.
projects / ceph.git / log