rgw/d4n: removing code that writes object entry to the
object directory in redis
Objects were being written to the directory for bucket listing
but another data structure needs to be used that allows for
ordered listing indexed by a bucket name/ id, hence removing
the entries that do not serve this purpose (they also add to the
number of calls to the redis server).
rgw/d4n: squashing the following commits that fix s3 test
failures.
1. rgw/d4n: fixing logical errors for ranged requests in iterate(), aligning
start chunk/block to beginning of the chunk/block boundary.
2. cache head object in write path in case write-cache
is disabled - for maintaining cache consistensy
3. delete dirty blocks of older versions of a non-versioned
bucket as dirty blocks are not evicted, and older versions won't
be written to the disk also.
4. remove dirty objects from the dirty objects data structure,
in case the bucket it belongs to has been deleted.
5. rgw/d4n: fixing compilation issue after rebasing with main.
6. Handle multipart objects correctly in copy object
7. Fix more ranged request errors.
8. handling part_num correctly for multipart and non-multipart objects
9. handling obect name starting with '_'.
10. bug in range request for sending back data to the client, after reading from
backend store.
11. rgw/d4n: code to populate RGW_ATTR_CRYPT_PARTS attr
in d4n filter.
12. rgw/d4n: fix for 'if-match' and 'if-nomatch' for
put object and copy object.
13. rgw/cache: Account for slashes in object name
14. d4n/filter: Handle request headers during read if `next->prepare` isn't called
15. d4n/filter: Fix overwriting of object size and instance
16. rgw/d4n: Supply attrs to `calculate_version`
17. rgw/d4n: Handle ACL and etag in copy_object
18. rgw/filter: Ensure cksum values are maintained
19. rgw/d4n: Set parts_count correctly and check for bufferlist length in calculate_version
20. rgw/d4n: adding a case for iterate method, where last
part to read is smaller than chunk size.
Samarah [Fri, 17 May 2024 19:19:12 +0000 (19:19 +0000)]
rgw/d4n: Squash the following commits related to directory, eviction and filter driver logs:
1. d4n/directory: Remove `blockHosts` and change `objHosts` to store the hosts for the block
2. d4n/directory: Remove `dirtyBlock` metadata and use the cacheObj's `dirty` flag in block logic
3. rgw/d4n: This commit introduces the following changes:
a. Fix `get_victim_block` method to correctly handle version
b. Return `0` instead of error for non-error circumstances in `eviction`
c. Switch `increase` to `decrease` for heap after localWeight update in
`eviction`
d. Update filter writer logs
4. d4n/filter: Make minor adjustments
5. test/d4n: Update `update_field` test calls
Samarah [Wed, 27 Mar 2024 17:42:41 +0000 (17:42 +0000)]
rgw/d4n: squashing the following commits for miscellaneous fixes
to policy driver, redis driver, directory and filter driver.
1. d4n/directory: Use boost::split for simpler code
2. rgw: Lower log levels for failures in D4N and redis cache files
3. rgw: Add dpp and logs to directory, cache, and policy
4. rgw: Reduce Redis calls and fix workflow
5. qa/d4n: Remove D4N task and add S3 user creation to workunit driver script
6. d4n: Use Redis transactions to serialize consecutive requests for safe
data handling and faster completion
7. d4n/directory: Remove boost lexical_cast calls
8. rgw/d4n: Add return values to error logs
9. rgw/d4n: Change directory hostsList to use `unordered_set`
10. d4n/filter: Simplify logic for storing block in `handle_data`
11. rgw/policy: Properly delete `LFUDAEntry` instances
12. rgw/d4n: Add support for `dirty` block metadata, `check_bool` for consistent values, and fix directory updates in `cleanup` method
rgw/d4n: miscellaneous corrections/additions:
1. storing objects in directory using their oid, so that the version
is included.
2. making sure that the head block corresponds to latest
version in the block directory.
3. add a directory entry for head block for every version
in case of a versioned bucket.
4. Populating hostsList correctly for blocks and objects.
rgw/d4n: squashing all commits related to caching head in the
write-back workflow, modifying set_obj_attrs(), get_obj_attrs() and
delete_obj_attrs based on the cached head and modifying the cleaning
method to use a min-heap data structure for storing dirty objects only.
1. rgw/d4n: implementation for caching head object in write-back workflow.
2. rgw/d4n: modifications to get write back cache working after cleaning
process.
3. rgw/d4n: modifications for eviction of dirty blocks.
4. rgw/d4n: modifications include adding a heap of dirty objects
which has objects ordered by their creation time and the top
element of which is fetched in the cleaning method, processed
and deleted in a loop.
5. rgw/d4n: changing the format of cached blocks to
bucket_name_version_object_name_ofs_len, to avoid checks
for versioned and non-versioned objects.
6. rgw/d4n: modifications to set_obj_attrs(), modify_obj_attrs()
and delete_obj_attrs() to check if the head object exists in a cache,
else direct the calls to backend store.
7. rgw/d4n: handling version in case of bucket versioning being suspended
while writing the object.
Co-authored by: Samarah <samarah.uriarte@ibm.com>
Changed dynamic_cast to static_cast for D4NFilterObject
in D4NFilterWriter class constructors.
mosayyebzadeh [Mon, 26 Feb 2024 15:43:01 +0000 (15:43 +0000)]
rgw/d4n: squashing all commits related to initial implementation
of write-back cache for non-multipart objects (writing objects to
cache and then to backend store in the cleaning process):
1. combining write cache with latest D4N code
2. cleaning the code
3. removing some bugs on bigger objects
4. updating iterate function to check the dirty flag
5. Updating write cache based on the pull request comments.
Read process needs to be updated based on write process. It needs to check where is the data and if it is dirty or clean.
If it is in the cache and dirty, we need to put D_ in the oid of the object before reading it from cache.
If it is clean, there is nothing to do.
6. updating flush functions and comments.
pybind: switch from pkgutil.find_loader() to importlib.util.find_spec()
Replace pkgutil.find_loader() with importlib.util.find_spec() throughout
Python bindings. This addresses the deprecation warning in Python 3.10
(scheduled for removal in 3.14) that appeared when generating librbd
Python bindings.
The importlib.util.find_spec() API has been available since Python 3.4
and is compatible with our minimum required Python version (3.9, since
commit 51f71fc1).
The warning resolved:
```
/home/kefu/dev/ceph/src/pybind/rbd/setup.py:8: DeprecationWarning: 'pkgutil.find_loader' is deprecated and slated for removal in Python 3.14; use importlib.util.find_spec() instead
if not pkgutil.find_loader('setuptools'):
```
J. Eric Ivancich [Wed, 16 Apr 2025 16:38:33 +0000 (12:38 -0400)]
rgw: prevent crash in `radosgw-admin bucket object shard ...`
This subcommand is used to ask radosgw-admin which bucket index shard
a given object in a given bucket would have its bucket index entry
on. The user is required to supply the number of shards (i.e., the
command doesn't look that up). If 0 is provided it would result in a
divide by zero runtime exception. Values less than or equal to zero
are now protected.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Fix stray example command block leftover from rebase in
cloud-transition.rst.
Remove extra character > in cloud-sync-module.rst.
Add missing formatting char ` in cloud-sync-module.rst.
Remove extra empty line between example commands that
resulted in a line with just a "#" prompt.
Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
rbd-mirror: release lock before calling m_async_op_tracker.finish_op()
m_async_op_tracker.finish_op() in InstanceReplayer::start_image_replayers
may invoke a completion that re-enters code paths that attempt to acquire
the same mutex (m_lock), violating the non-recursive lock constraint.
This can be fixed by releasing the lock before calling
m_async_op_tracker.finish_op().
Merge pull request #62818 from ronen-fr/wip-rf-iocnt-plus
osd/scrub: performance counters: count I/Os, use unlabeled counters
Reviewed-by: Alex Ainscow <aainscow@uk.ibm.com> Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com> Reviewed-by: Bill Scales <bill_scales@uk.ibm.com> Reviewed-by: Samuel Just <sjust@redhat.com>
Ville Ojamo [Thu, 10 Apr 2025 10:34:57 +0000 (17:34 +0700)]
doc/radosgw: Promptify CLI, cosmetic fixes
Use the more modern prompt block for CLI commands
and use right one $ vs #.
Fix indentation on JSON example outputs and
some CLI command switches.
Add some arguably missing comma in JSON example output.
Add a full stop at the end of a one-sentence paragraph.
Remove extra comma mid-sentence in another.
Fix missing backslashes or typo at end of multiline commands.
Lines under section headings as long as heading text.
Fix hyperlinks. Fix list items prefixed with - insted of *.
Format configuration syntax in the middle of text as code.
Fix typo "PI" to "API" and remove extra space.
Remove colons at the end of section headers in a few places.
Use Title Case in section titles consistently with short words lowercase.
Possibly controversial: don't add whitespace before and
after main title section header text.
Possibly controversial: don't indent line continuation
backslashes, leave only 1 space before them.
Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>
osd/scrub: a single counters selection mechanism - step 1
Following the preceeding PR, the Scrubber now employs
two methods for selecting the specific subset of performance
counters to update (the replicated pool set or the EC one).
The first method is using labeled counters, with 4 optional labels
(Primary/Replica X Replicated/EC Pool). The second method
is by naming the specific OSD counters to use in ScrubIoCounterSet
objects, then selecting the appropriate set based on the pool type.
This commit is the first step on the path to unifying the two
methods - discarding the use of labeled counters, and only
naming OSD counters.
osd/scrub: perf-counters for I/O performed by the scrubber
Define two sets of performance counters to track I/O performed
by the scrubber - one set to be used when scrubbing a PG
in a replicated pool, and one - for EC PGs.
https://github.com/ceph/ceph/pull/62080 tested version was **different**
from the one that got merged.
The untested change was changing the boolean returned from start_recovery_ops.
While the seastar::repeat loop in BackgroundRecoveryT<T>::start() was changed accordingly,
other do_recovery() return cases were not considered.
See Tested / Merged here: https://github.com/Matan-B/ceph/pull/2/files
start_recovery_ops used by do_recovery should return whether the iteration (i.e recovery) keep going.
Direct users to upgrade only to Squid v19.2.2, and warn readers not to
upgrade to Squid 19.2.1. This PR is raised in response to a request from
Neha Ojha.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
rgw: metadata and data sync fairness notifications to retry upon any error case
This is a complementary fix to the earlier one described at #62156.
When the sync shard notification fails due to any failures including timeout,
this change keeps the loop going for both metadata and data sync.
The 'delay_ready_t' parameter was used in the past to
control whether, when a change in the scrub scheduling inputs
occurs (e.g. a configuration change), even those scheduling targets
that are already ripe for scrubbing will have their schedule recomputed.
This parameter, however, is ignored: all "regular-periodic"
scrubbing targets are always rescheduled when the scheduling inputs
change.
The commit removes the 'delay_ready_t' parameter from the codebase.
The ceph_ll_io_info structure has recently been extended to support
zerocopy operations. The proxy was initializing just the known members,
so, after the zerocopy support, it was passing garbage in some fields,
causing failures.
This patch completely clears the whole structure to be sure that
everything is initialized to its default value.
Naman Munet [Thu, 10 Apr 2025 11:40:02 +0000 (17:10 +0530)]
mgr/dashboard: fix bucket rate limit API on owner change
Fixes: https://tracker.ceph.com/issues/70874
PR covers & fixes below scenarios:
Whenever we change the owner of bucket from non-tenanted to tenanted and
vice-versa with the rate-limit changes, there was issue in sending bucket name
Scenario 1: Changing the bucket owner from tenanted to non-tenanted
Scenario 2: Changing the bucket owner from non-tenanted to tenanted
Scenario 3: Keeping the owner(tenanted) same and changing only rate limit
Xuehan Xu [Thu, 27 Feb 2025 05:54:49 +0000 (13:54 +0800)]
crimson/os/seastore/cache: do `prepare_commit` before retiring extents
Linked tree nodes in logical trees need to take parents from the prior
instances when being rewritten, which has to be done before the prior
instances are retired.
mgr/dashboard: fix typo in User Management form Fixes: https://tracker.ceph.com/issues/70719
-Corrected the label from 'logon' to 'login' in the User Management form
osd/scrub: additional configuration params to trigger scrub reschedule
Adding the following parameters to the (small) set of configuration
options that, if changed, trigger re-computation of the next scrub
schedule:
- osd_scrub_interval_randomize_ratio,
- osd_deep_scrub_interval_cv, and
- osd_deep_scrub_interval (which was missing in the list of
parameters watched by the OSD).
projects / ceph-ci.git / log