Instead of having _discard_start() and _discard_stop() partially or
completely duplicate functionality in handle_conf_change(), have a
single _discard_update_threads() that can handle all three. Loops are
tidied slightly, the unnecessary target_discard_threads class variable
has been removed, and now handle_conf_change() will respect
support_discard.
blk/KernelDevice: Fix several issues with stopping discard threads
1. In _discard_stop(), the wait for !discard_threads.empty() was there
from a prior implementation where there could theoretically be a race
between _discard_start() and _discard_stop(). If that race does
exist, this check won't help; not only is _discard_stop() not called
if discard_threads is empty, but discard_threads won't be populated
until _discard_start() runs and thus this won't detect such a race.
2. Calling _discard_stop() from handle_conf_change() is a guaranteed
deadlock because discard_lock is already held, so don't do that. Use
the same flow whether we're stopping a subset of threads or all
threads.
3. Asking a subset of discard threads to stop was not guaranteed to take
effect, since if they continued to find contents in discard_queue
then they would continue to run indefinitely. Add additional logic to
_discard_thread() to have threads stop if they have been requested to
stop and other threads exist to continue draining discard_queue.
4. Make the flow of _discard_stop() and handle_conf_change() more
similar.
blk/KernelDevice: React to bdev_enable_discard changes in handle_conf_change()
This fixes two issues that were introduced by 755f3e03b5bf547cfd940b52a53833f66285062b:
1. After an OSD boots, discard threads were not stopped when
bdev_enable_discard was set to false, whereas that was the intent of
that commit.
2. If bdev_enable_discard or bdev_async_discard_threads are configured
with a mask that can't be evaluated at OSD boot (e.g. a device
class), then async discard won't be enabled until a later config
change to bdev_async_discard_threads.
Limit private discarded queue for threads to a small items count.
On fast-shutdown take over the main discarded queue copying it to the allocator and only wait for the threads to commit their small private discarded queues
On graceful shutdown we will wait for discard queue to drain before storing the allocator.
ON fast shutdown we will simply copy the discard queue entries to the allocator
os/BlueStore: NCB fix for leaked space when bdev_async_discard is enabled
Fix calls bdev->discard_drain() before calling store_allocator() to make sure all freed space is reflected in the allocator before destaging it
The fix set a timeout for the drain call (500msec) and if expires will not store the allocator (forcing a recovery on the next startup) Fixes: https://tracker.ceph.com/issues/65298 Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>
(cherry picked from commit 3aa891dbf6dafd3b1983fbe2efcbfd1d11d52b40)
some tests are currently failing when `lvm2` isn't installed:
```
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestList::test_empty_device_json_zero_exit_status - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestList::test_empty_device_zero_exit_status - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_no_ceph_lvs - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_ceph_data_lv_reported - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_ceph_journal_lv_reported - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_ceph_wal_lv_reported - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_physical_2nd_device_gets_reported[journal] - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_physical_2nd_device_gets_reported[db] - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_physical_2nd_device_gets_reported[wal] - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_not_a_ceph_lv - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_a_ceph_lv - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_a_ceph_journal_device - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_just_block_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_just_data_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_just_block_wal_and_db_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_data_and_journal_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_nonexistent_osd_id - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_a_ceph_lv_with_no_matching_devices - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_migrate.py::TestNew::test_newdb_not_target_lvm - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_nothing_is_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_journals_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_dbs_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_wals_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_backing_devs_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/objectstore/test_lvmbluestore.py::TestLvmBlueStore::test_activate_all_osd_is_active - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
```
Everything should be actually mocked. This commit addresses that.
ceph-volume: do not convert LVs's symlink to real path
This commit:
- Adds a new function `get_lvm_mappers` in `ceph_volume/util/disk.py`
to retrieve a list of LVM device mappers.
- Updates the `is_lv` property in `ceph_volume/util/device.py`
to use the new `get_lvm_mappers` function for better accuracy.
- Modifies the symlink handling in `Device` class to properly
identify LVM logical volumes.
- Adds a new test `test_reject_lv_symlink_to_device` to ensure
LVM symlinks are correctly identified and handled.
- Updates relevant tests to cover the changes in LVM device detection.
These changes improve the reliability and accuracy of LVM device detection
and handling, ensuring that symlinks to LVM logical volumes are
correctly processed.
Fixes: https://tracker.ceph.com/issues/61597 Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com> Co-Authored-by: Jerry Pu <jerrypu@qnap.com>
(cherry picked from commit 729c5de4f852f1a1ee90e76b71157e9070af7d99)
Igor Fedotov [Fri, 31 May 2024 14:05:29 +0000 (17:05 +0300)]
ceph-volume: do source devices zapping if they're detached.
One needs to zap source device(s) after DB/WAL migration.
Original imlementation removes LVM tags only which leaves device(s) in a
state where "ceph-volume raw activate" still reconginizes them as
attached to OSD due to information preserved in bdev label.
Hence the need to do more zapping. Fixes: https://tracker.ceph.com/issues/66315 Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
(cherry picked from commit ae5ef432845dcf9b061258357ffd97f4eae59a63)
Fixes: https://tracker.ceph.com/issues/66286
(Line added by Gabriel)
In RadosStore, the source and dest objects in the copy_object() call
used to share an obj_ctx. When obj_ctx was removed from the SAL API,
they each got their own, but RGWRados::copy_obj() still assumed they
shared one.
Pass in each one separately, and use the correct one for further calls.
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net> Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>
(cherry picked from commit 111c54a19dc12b84cda785feddb0a0ba483b1f77)
Fixes: https://tracker.ceph.com/issues/66286
Improve display of ref_count in the rados commandline utility
New test cases were added to detect behavior after server side copy in the following cases:
1) delete original only
2) delete destination only
3) delete original then delete destination (this will lead to orphaned tail-objects without the changes made in this PR)
d) delete destination then delete original (this will lead to orphaned tail-objects without the changes made in this PR)
Add call to GC between tests to help control the used disk space since we keep writing huge files of 5GB each Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>
(cherry picked from commit d496d20c803590d41d711e446feab41476c0f20c)
RemoteApplier::load_acct_info() and create_account() decide whether to
add the implicit tenant. store the resulting rgw_user for use in
get_aclowner() and get_tenant()
Nitzan Mordechai [Tue, 25 Jun 2024 09:06:45 +0000 (09:06 +0000)]
crimson/osd: adding osdmap subscribe
when committed osdmap is complete, it will check if should restart.
in case we shouldn't restart but we are still active, we need
the next osdmap to continue the process.
hualong feng [Fri, 14 Jun 2024 07:50:53 +0000 (15:50 +0800)]
rgw: fixup compressor_message didn't store in some cases
When I upload a object to RGW by multipart, the head object
xattr(user.rgw.compression) don't have compressor_message
when the value should be valid and part object xattr
have the value.
hualong feng [Thu, 6 Jun 2024 07:53:03 +0000 (15:53 +0800)]
compressor: Change data formt to QZ_DEFLATE_GZIP_EXT for QAT zlib
QAT zlib 'QZ_DEFLATE_RAW' data format cannot decompress
by QAT hardware. So here we replace 'QZ_DEFLATE_GZIP_EXT' data
format with 'QZ_DEFLATE_RAW'.
'QZ_DEFLATE_GZIP_EXT' data format need to add gz_header
by deflateSetHeader() in QATzip. And it leads multi stream
in one compression for hardware buffer. So the windows bit
is important information for decompression, which related to
if the inflate remove header.
All temp objects are added *only* to PGBackend::temp_content.
cleaning RecoveryBackend::temp_contents (which is always empty) instead
of PGBackend::temp_contents is wrong.
projects / ceph.git / log