Ilya Dryomov [Fri, 2 Aug 2024 07:27:42 +0000 (09:27 +0200)]
librbd/migration: make ImageDispatch handle encryption for non-native formats
With NativeFormat now being handled via dispatch, handling encryption
for non-native formats (i.e. mapping to raw image extents and performing
decryption/mapping back on completion) in the migration layer is really
straightforward.
Note that alignment doesn't need to be performed in the migration layer
because it happens on the destination image -- the "align and resubmit"
logic in C_UnalignedObjectReadRequest should kick in before the call to
read_parent().
Fixes: https://tracker.ceph.com/issues/53674 Co-authored-by: Or Ozeri <oro@il.ibm.com> Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 0000c3447407772039121bb4499f243df1c889da)
librbd/migration: close source image in OpenSourceImageRequest
Currently, on errors in FormatInterface::open(), RawFormat disposes
of src_image_ctx, but QCOWFormat doesn't, which is a leak. Rather than
having each format do it internally, do it in OpenSourceImageRequest.
librbd/migration: don't instantiate NativeFormat, handle it via dispatch
Trying to shoehorn NativeFormat under FormatInterface doesn't really
work. It fundamentally doesn't fit in:
- Unlike for RawFormat and QCOWFormat, src_image_ctx for NativeFormat
is not dummy -- it's an ImageCtx for a real RBD image. Pre-creating
it in OpenSourceImageRequest with the expectation that placeholder
values would be overridden later forces NativeFormat to reach into
ImageCtx guts, duplicating the logic in the constructor. This also
necessitates calling snap_set() in a separate step, since snap_id
isn't known at the time ImageCtx is created.
- Unlike for RawFormat and QCOWFormat, get_image_size() and
get_snapshots() implementations for NativeFormat are dummy.
- read() and list_snaps() implementations for NativeFormat are
inconsistent: read() passes through io::ImageDispatch layer, but
list_snaps() doesn't. Both can be passing through, meaning that in
essence these are also dummy.
All of this is with today's code. Additional complications arise with
planned support for migrating from external clusters where src_image_ctx
would require more invasive patching to "move" to an IoCtx belonging to
an external cluster's CephContext and also with other work.
With the above in mind, NativeFormat actually consists of:
1. Code that parses the "type: native" source spec
2. Code that patches ImageCtx, working around the fact that it's
pre-created in OpenSourceImageRequest
3. A bunch of dummy implementations for FormatInterface
With this change, (1) is wrapped into a static method that also creates
ImageCtx after all required parameters are known and (2) and (3) go away
entirely. NativeFormat no longer implements FormatInterface and doesn't
get instantiated at all.
In preparation for not instantiating NativeFormat and losing a copy of
the source spec JSON object in m_json_object, refactor the parsing code
to use only const methods (which std::map's operator[] isn't) and local
variables where possible.
librbd/migration/NativeFormat: do pool lookup instead of creating io_ctx
A Rados instance is sufficient to map the pool name to the pool ID,
no need to involve an IoCtx instance as well. While at it, report
distinctive errors for a non-existing pool and an invalid JSON value
for pool_name key cases.
librbd/migration: make SourceSpecBuilder::parse_source_spec() static
In preparation for divorcing NativeFormat from FormatInterface and
changing when/how src_image_ctx is created, make parse_source_spec()
independent of src_image_ctx. The "invalid source-spec JSON" error is
duplicated by the "failed to parse migration source-spec" error, so
just get rid of the former to spare having to pass CephContext to
parse_source_spec().
Add missing spaces, don't use the word stream when reporting errors
on POSIX file operations (open() and lseek64()) and fix a cut-and-paste
typo in RawSnapshot.
osd/scrub: exempt only operator scrubs from max_scrubs limit
Existing code exempts all 'high priority' scrubs, including for example
'after_repair' and 'mandatory on invalid history' scrubs from the limit.
PGs that do not have valid last-scrub data (which is what we have when
a pool is first created) - are set to shallow-scrub immediately.
Unfortunately - this type of scrub is (in the low granularity implemented
in existing code) is 'high priority'.
Which means that a newly created pool will have all its PGs start
scrubbing, regardless of concurrency (or any other) limits.
Fixes: https://tracker.ceph.com/issues/67253 Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
(cherry picked from commit babd65e412266f5c734f7a2b57d87657d3470c47)
conflict resolution:
- eliminating irrelevant 'main' code that was picked into this branch.
- the code to set the scrub_job's flag moved to osd_scrub_sched.cc,
where the corresponding function is.
some tests are currently failing when `lvm2` isn't installed:
```
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestList::test_empty_device_json_zero_exit_status - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestList::test_empty_device_zero_exit_status - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_no_ceph_lvs - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_ceph_data_lv_reported - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_ceph_journal_lv_reported - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_ceph_wal_lv_reported - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_physical_2nd_device_gets_reported[journal] - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_physical_2nd_device_gets_reported[db] - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestFullReport::test_physical_2nd_device_gets_reported[wal] - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_not_a_ceph_lv - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_a_ceph_lv - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_a_ceph_journal_device - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_just_block_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_just_data_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_just_block_wal_and_db_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_osd_id_for_data_and_journal_dev - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_by_nonexistent_osd_id - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_listing.py::TestSingleReport::test_report_a_ceph_lv_with_no_matching_devices - FileNotFoundError: [Errno 2] No such file or directory: 'pvs'
FAILED ceph_volume/tests/devices/lvm/test_migrate.py::TestNew::test_newdb_not_target_lvm - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_nothing_is_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_journals_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_dbs_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_wals_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/devices/lvm/test_zap.py::TestEnsureAssociatedLVs::test_multiple_backing_devs_are_found - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
FAILED ceph_volume/tests/objectstore/test_lvmbluestore.py::TestLvmBlueStore::test_activate_all_osd_is_active - FileNotFoundError: [Errno 2] No such file or directory: 'lvs'
```
Everything should be actually mocked. This commit addresses that.
ceph-volume: do not convert LVs's symlink to real path
This commit:
- Adds a new function `get_lvm_mappers` in `ceph_volume/util/disk.py`
to retrieve a list of LVM device mappers.
- Updates the `is_lv` property in `ceph_volume/util/device.py`
to use the new `get_lvm_mappers` function for better accuracy.
- Modifies the symlink handling in `Device` class to properly
identify LVM logical volumes.
- Adds a new test `test_reject_lv_symlink_to_device` to ensure
LVM symlinks are correctly identified and handled.
- Updates relevant tests to cover the changes in LVM device detection.
These changes improve the reliability and accuracy of LVM device detection
and handling, ensuring that symlinks to LVM logical volumes are
correctly processed.
Fixes: https://tracker.ceph.com/issues/61597 Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com> Co-Authored-by: Jerry Pu <jerrypu@qnap.com>
(cherry picked from commit 729c5de4f852f1a1ee90e76b71157e9070af7d99)
Igor Fedotov [Fri, 31 May 2024 14:05:29 +0000 (17:05 +0300)]
ceph-volume: do source devices zapping if they're detached.
One needs to zap source device(s) after DB/WAL migration.
Original imlementation removes LVM tags only which leaves device(s) in a
state where "ceph-volume raw activate" still reconginizes them as
attached to OSD due to information preserved in bdev label.
Hence the need to do more zapping. Fixes: https://tracker.ceph.com/issues/66315 Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
(cherry picked from commit ae5ef432845dcf9b061258357ffd97f4eae59a63)
test/crimson/seastore/test_seastore.cc: should not return a value
clang++-14:
```
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/crimson/seastore/test_seastore.cc:86:5: error: void function 'do_transaction' should not return a value [-Wreturn-type]
return sharded_seastore->do_transaction(
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/crimson/seastore/test_seastore.cc:94:5: error: void function 'set_meta' should not return a value [-Wreturn-type]
return seastore->write_meta(key, value).get();
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.
```
Fixes: https://tracker.ceph.com/issues/66286
(Line added by Gabriel)
In RadosStore, the source and dest objects in the copy_object() call
used to share an obj_ctx. When obj_ctx was removed from the SAL API,
they each got their own, but RGWRados::copy_obj() still assumed they
shared one.
Pass in each one separately, and use the correct one for further calls.
Signed-off-by: Daniel Gryniewicz <dang@fprintf.net> Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>
(cherry picked from commit 111c54a19dc12b84cda785feddb0a0ba483b1f77)
Fixes: https://tracker.ceph.com/issues/66286
Improve display of ref_count in the rados commandline utility
New test cases were added to detect behavior after server side copy in the following cases:
1) delete original only
2) delete destination only
3) delete original then delete destination (this will lead to orphaned tail-objects without the changes made in this PR)
d) delete destination then delete original (this will lead to orphaned tail-objects without the changes made in this PR)
Add call to GC between tests to help control the used disk space since we keep writing huge files of 5GB each Signed-off-by: Gabriel BenHanokh <gbenhano@redhat.com>
(cherry picked from commit d496d20c803590d41d711e446feab41476c0f20c)
RemoteApplier::load_acct_info() and create_account() decide whether to
add the implicit tenant. store the resulting rgw_user for use in
get_aclowner() and get_tenant()
projects / ceph.git / log