John Mulligan [Tue, 9 Jul 2024 18:28:49 +0000 (14:28 -0400)]
qa/suites/orch: enable cephadm single_host_defaults for smb tests
Set the cephadm single_host_defaults option for the smb tests that use a
single node ceph cluster. Without this option the mgr_pool and the
sqlite3 db support needed by the smb mgr module will not be available.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 6 Jul 2024 17:37:31 +0000 (13:37 -0400)]
mgr/smb: use the sqlite mirroring store by default
We added the sqlite db for performance reasons and now have enhanced it
with the mirroring store for a consistent policy of only storing
possibly sensitive information in the mon config key store, like cephadm
does for certs and stuff. Enable the sqlite mirroring store by default.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 6 Jul 2024 18:00:29 +0000 (14:00 -0400)]
mgr/smb: filter out password fields in sqlite store
Currently, all of ceph orchestration stores sensitive data in the mon
config-key store. Keep doing that by eliding passwords in the sqlite
store but retaining them in the mon based store. Perhaps, in the future
we can even use a 'vault' type store for even better sensitive info
retention.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 6 Jul 2024 17:37:06 +0000 (13:37 -0400)]
mgr/smb: add SqliteMirroringStore class
Add a SqliteMirroringStore class that can mirror objects from a
particular namespace into the same namespace in another store.
The Mirror configuration classes can be used to modify and
merge objects from each store when fetched.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 30 May 2024 19:00:57 +0000 (15:00 -0400)]
mgr/smb: add transactions to handler funcs underlying show command
Add transaction context manager to the handler functions that underlie
the smb module's show command. This makes a pretty good difference in
the speed of showing the resources.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 10 May 2024 19:38:25 +0000 (15:38 -0400)]
mgr/smb: make the new sqlite store our default internal store
Add the new sqlite store to the smb mgr module, making it the default.
Add some configuration params, mainly for devs only, to also allow going
back to the previous store for debugging and testing purposes.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 10 May 2024 19:37:57 +0000 (15:37 -0400)]
mgr/smb: add a new sqlite db backed store
Add a new store type that uses a sqlite3 database, stored in rados, for
managing internal resources. The main motivation for this store is to
improve performance, specifically in the case where we need to look
up shares by fields other than the store entry key.
In this store each namespace maps to a table. Each table is managed by a
class that can store a JSON object. The shares table is specialized so
that it can index fields within the JSON and find (cluster_id, name)
pairs efficiently. This allows us to much more quickly determine if
a share's name is already in use by a different share in the same
cluster (the name may be reused in a different cluster w/o issue).
One other important factor when using this store is to use transactions,
via the transaction method (see TransactingConfigStore). If transactions
are not used this store can perform worse than the ModuleConfigStore.
When they are used (correctly) there is a good performance improvement.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 15 May 2024 22:25:40 +0000 (18:25 -0400)]
mgr/smb: add store transactions to parts of the handler
Add a wrapper contextmethod that initiates a transaction on a store if said
store supports transactions (otherwise the ctx manager is a no-op).
Use the wrapper over certain functions in the handler where many store
operations may be needed. This makes a big performance difference when
using a sqlite db backed store.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 29 May 2024 23:27:57 +0000 (19:27 -0400)]
mgr/smb: add TransactingConfigStore protocol
Add a new TransactingConfigStore protocol. This TransactingConfigStore is a
ConfigStore that provides a method for establishing a transaction.
What a transaction means can depend on the store but generally it is
intended to provide a way to use database transactions with the store
abstraction.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Tue, 14 May 2024 14:19:10 +0000 (10:19 -0400)]
mgr/smb: validate that a share name is unique per-cluster
We already require that the share_id be unique as it's part of the entry
key of the resource. Add a validation function to the handler that
checks if a (cluster_id, name) tuple is not duplicated. This function
uses the newly added config_store.find_in_store function to find the
potential dupes.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 13 May 2024 21:10:36 +0000 (17:10 -0400)]
mgr/smb: add find_in_store function to config_store.py
Add a new `find_in_store` function to the config store module. This
function can be called on any store to search for a particular object
using the FindParams, with an exact match of the subset of keys & values
in the dict. If the store is a FindingConfigStore this operation may be
able to be done more efficiently but we always can fall back to the
simple approach of reading in every object from the store & namespace.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 13 May 2024 20:29:22 +0000 (16:29 -0400)]
mgr/smb: add ObjectCachingEntry class to config_store.py
Add a general store entry class that caches objects so that if an
object needs to be "gotten" from the store multiple times we may
avoid actually calling the store entry get method. This will be
used by a future function for finding particular items in a store.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 13 May 2024 21:10:26 +0000 (17:10 -0400)]
mgr/smb: add a new protocol for a FindingConfigStore
Add a new FindingConfigStore protocol. This FindingConfigStore is a
ConfigStore that provides a method for more quickly/efficiently finding
specific entries in the store. This is opposed to walking all keys and
getting all objects and comparing them. This protocol is optional and
will only be implemented by a few specialized stores.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 29 May 2024 23:06:49 +0000 (19:06 -0400)]
pybind/mgr: add convenient context mgrs for accessing module db
Add two new functions returning context managers for accessing the
sqlite db from the module base class. exclusive_db_access takes
the db lock and yields the sqlite db connection. exclusive_db_cursor
does the same as the previous function but returns a cursor instead
of the connection.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
authentication.rst described the steps to generate a v2 signature,
without reference to aws docs. replace that with sections that reference
aws docs for v2 and v4 signatures. list which values of the request
header x-amz-content-sha256 are supported for v4
Matt Benjamin [Sat, 22 Jun 2024 20:12:54 +0000 (16:12 -0400)]
rgw_cksum: address review comments
* remove rgw_cksum_pipe state enum, not needed [Casey review]
* remove a format that just took a single string substitution
and passed it to an iostream [Casey review]
* use boost::to_upper* [Casey review]
* remove unused RGW_ATTR_CKSUM_ALGORITHM decl [Casey review]
* negate error code values in two places [Casey review]
* split cksum digests from base type decls
* resolve comment when checksum requested but not available
* remove redundant memset
* remove junk from rgw_blake3_digest.h
* s/ldpp_dout + fmt::format/ldpp_dout_fmt/g;
* fix conditional return of parts_count
from RGWRados::Object::prepare(). A value for parts_count should
be returned iff a *multipart* object manifest exists.
* remove /tmp output test
* finish moving ceph_crypto headers out of rgw_cksum.h
* consume the optional in multipart_parts_count
* target_attrs can be a reference (but not const)
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Mon, 15 Apr 2024 20:44:58 +0000 (16:44 -0400)]
rgw_cksum: implement POST upload checksums
* properly transform pseudo headers in PostObj
* enable cksum verify in PostObj
* match checksum headers in match_policy_vars
* fixup add POST headers to environment
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Sun, 5 May 2024 18:27:06 +0000 (14:27 -0400)]
rgw_cksum: fix ReadOp comment
parts_count now returns a value for parts_count whenever the
target object is a multipart upload. this has no additional
overhead, since this can be read off the manifest
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Wed, 10 Apr 2024 21:58:15 +0000 (17:58 -0400)]
rgw_cksum: multipart upload checksums
includes commits:
* fixes init-multipart header return
* introduce checksum to SAL MultipartPart interface
* thread optional checksum through DataProcessor
* code complete multipart checksum verify
* fix formatter
* fix ckecksum format for multipart objects in GET/HEAD ops
* always return parts_count from ReadOp::prepare() if applicable
This behavior is used when returning the checksum of a multipart
upload object.
* tweak conditional multipart_parts_count
* add checksum output to ListMultipart
* fix nil-return from GetHeaderCksumResult
* re-arm truncated if re-entering list-parts
* complete-multipart w/list-parts
* validate supplied checksum in CompleteMultipart
* verify checksum type against initial checksum algorithm
* rgw_op: suppress more x-amz headers
* final fixes and cleanups
* remove unused t0
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Matt Benjamin [Tue, 24 Sep 2019 21:12:02 +0000 (17:12 -0400)]
rgw: add checksum and digest machinery
Adds new Blake3 digest format (native), a concrete type to
represent digests, and static_visitor machinery to unify varying
checksum computations.
This framework, together with new trailing checksum header support,
is used to implement S3 additional checksum verification. Parts of
the AWS content checksum API work build on a prior contribution from
imtzw <tongzhiwei_yewu@cmss.chinamobile.com>.
Thank you!
Fixes: https://tracker.ceph.com/issues/42080 Fixes: https://tracker.ceph.com/issues/63951
squashed commits:
* rgw_cksum: add trival test vectors for sha-format digests
Computed digests match those produced by sha1sum, sha256sum,
and sha512sum utilities.
* rgw_cksum: add test vectors for blake3
Tests the same input strings with digests validated by
b3sum (https://crates.io/crates/b3sum).
* rgw_ckum: switch to accel crc32c
The internal Ceph convention appears to be to omit a final
xor where ceph_crc32c is used, but it's required for compatibility
with AWS implementations.
* rgw_cksum: add XXH3 digest
* rgw_cksum: write class encoder for rgw::digest::Cksum
* rgw_cksum: also reverse crc32c (REBASEME)
Mark noticed that the crc32c output was being tested against a
byteswapped value (crc32c also needs byteswap on LE).
* rgw_cksum: add digest::Cksum serde tests
* rgw_cksum: fix main(...) linkage
(so we run our main unit and not the one in gmock
* rgw_cksum: convenience extensions for integration with RGW/S3
* introduce rgw_cksum unique_ptr factory
* rgw_cksum: mark string transform accessors const
* rgw_cksum: fixup unittest_rgw_chksum compilation--all existing tests pass
* rgw_cksum: hook up put-object checksum workflow
* tweaks to report on content checksum mismatch
* rgw_cksum: match SDK as well as general checksum header
* make it more efficient
* initialize RGWPutObj_Cksum::digest
* rgw_cksum: write parse_cksum_type w/const char* arg
* initialize _type correctly; doing armored wrong
* fix expected checksum header name, clean up verify
* fix output on checksum verify fail, cleanup
* introduce Cksum::to_armor(); all AWS cases pass
* oops, extra 0-byte at end of to_armor() result
* use to_armor() with decoded checksums (i.e., for all S3 presentation)
* remove unnecessary finalize() in RGWPutObj_Cksum dtor
* RGWPutObj_Cksum::Factory fixes
* fixes test_object_checksum_sha256
* choose preferred checksum algorithm header if both are present
* log verified checksums in RGWPutObj::execute at 16
* checksum not needed in policy condition
* fix checksum trailing header format
* move Blake3 to rgw_digest_blake3.h
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Document how to manually pass the search domain to "mon_dns_srv_name" in
doc/rados/configuration/mon-lookup-dns.rst.
This commit is made in response to a request by Lander Duncan that was made on the [ceph-users] mailing list, and can be seen here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/F7V4CWLIYCAJ4JXI2JLNY6QPCFPR4SLA/
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
rgw: fix bucket sync thread waiting inordinate amount of time
A signed value was read in as an unsigned value, so -1 was interpreted
as a very large value. This made the thread wait period in the bucket
sync thread inordinately long, preventing bucket sync and dynamic
resharding (unless values appropriate for debugging were set).
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
cls/rgw: bump cls_rgw_reshard_entry decode version to match encode
9302fbb3f5416871c1978af5d45f3bf568c2c190 bumped the version in
ENCODE_START() but missed DECODE_START(). i don't think that would cause
any decode failures, unless we later raise ENCODE_START's compat_v above
2
qa/tasks: Initialize 'monitoring_profiles' spec to an empty dict
Initialize 'monitoring_profiles' to an empty python dictionary instead of
'None' to prevent the cbt task from failing due to the TypeError exception
when attempting to iterate a 'NoneType'.
Merge pull request #58092 from zdover23/wip-doc-2024-06-18-start-os-recs
doc/start: remove mention of Centos 8 support
Reviewed-by: Adam King <adking@redhat.com> Reviewed-by: Dan Mick <dan.mick@redhat.com> Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com> Reviewed-by: Ken Dreyer <kdreyer@redhat.com> Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
projects / ceph-ci.git / log