Ilya Dryomov [Fri, 2 Aug 2024 07:27:42 +0000 (09:27 +0200)]
librbd/migration: make ImageDispatch handle encryption for non-native formats
With NativeFormat now being handled via dispatch, handling encryption
for non-native formats (i.e. mapping to raw image extents and performing
decryption/mapping back on completion) in the migration layer is really
straightforward.
Note that alignment doesn't need to be performed in the migration layer
because it happens on the destination image -- the "align and resubmit"
logic in C_UnalignedObjectReadRequest should kick in before the call to
read_parent().
Fixes: https://tracker.ceph.com/issues/53674 Co-authored-by: Or Ozeri <oro@il.ibm.com> Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Leonid Chernin [Tue, 17 Oct 2023 13:25:07 +0000 (13:25 +0000)]
mon: add NVMe-oF gateway monitor and HA
- gateway submodule
Fixes: https://tracker.ceph.com/issues/64777
This PR adds high availability support for the nvmeof Ceph service. High availability means that even in the case that a certain GW is down, there will be another available path for the initiator to be able to continue the IO through another GW. High availability is achieved by running nvmeof service consisting of at least 2 nvmeof GWs in the Ceph cluster. Every GW will be seen by the host (initiator) as a separate path to the nvme namespaces (volumes).
The implementation consists of the following main modules:
- NVMeofGWMon - a PaxosService. It is a monitor that tracks the status of the nvmeof running services, and take actions in case that services fail, and in case services restored.
- NVMeofGwMonitorClient – It is an agent that is running as a part of each nvmeof GW. It is sending beacons to the monitor to signal that the GW is alive. As a part of the beacon, the client also sends information about the service. This information is used by the monitor to take decisions and perform some operations.
- MNVMeofGwBeacon – It is a structure used by the client and the monitor to send/recv the beacons.
- MNVMeofGwMap – The map is tracking the nvmeof GWs status. It also defines what should be the new role of every GW. So in the events of GWs go down or GWs restored, the map will reflect the new role of each GW resulted by these events. The map is distributed to the NVMeofGwMonitorClient on each GW, and it knows to update the GW with the required changes.
It is also adding 3 new mon commands:
- nvme-gw create
- nvme-gw delete
- nvme-gw show
The commands are used by the ceph adm to update the monitor that a new GW is deployed. The monitor will update the map accordingly and will start tracking this GW until it is deleted.
Signed-off-by: Leonid Chernin <lechernin@gmail.com> Signed-off-by: Alexander Indenbaum <aindenba@redhat.com>
librbd/migration: close source image in OpenSourceImageRequest
Currently, on errors in FormatInterface::open(), RawFormat disposes
of src_image_ctx, but QCOWFormat doesn't, which is a leak. Rather than
having each format do it internally, do it in OpenSourceImageRequest.
librbd/migration: don't instantiate NativeFormat, handle it via dispatch
Trying to shoehorn NativeFormat under FormatInterface doesn't really
work. It fundamentally doesn't fit in:
- Unlike for RawFormat and QCOWFormat, src_image_ctx for NativeFormat
is not dummy -- it's an ImageCtx for a real RBD image. Pre-creating
it in OpenSourceImageRequest with the expectation that placeholder
values would be overridden later forces NativeFormat to reach into
ImageCtx guts, duplicating the logic in the constructor. This also
necessitates calling snap_set() in a separate step, since snap_id
isn't known at the time ImageCtx is created.
- Unlike for RawFormat and QCOWFormat, get_image_size() and
get_snapshots() implementations for NativeFormat are dummy.
- read() and list_snaps() implementations for NativeFormat are
inconsistent: read() passes through io::ImageDispatch layer, but
list_snaps() doesn't. Both can be passing through, meaning that in
essence these are also dummy.
All of this is with today's code. Additional complications arise with
planned support for migrating from external clusters where src_image_ctx
would require more invasive patching to "move" to an IoCtx belonging to
an external cluster's CephContext and also with other work.
With the above in mind, NativeFormat actually consists of:
1. Code that parses the "type: native" source spec
2. Code that patches ImageCtx, working around the fact that it's
pre-created in OpenSourceImageRequest
3. A bunch of dummy implementations for FormatInterface
With this change, (1) is wrapped into a static method that also creates
ImageCtx after all required parameters are known and (2) and (3) go away
entirely. NativeFormat no longer implements FormatInterface and doesn't
get instantiated at all.
In preparation for not instantiating NativeFormat and losing a copy of
the source spec JSON object in m_json_object, refactor the parsing code
to use only const methods (which std::map's operator[] isn't) and local
variables where possible.
librbd/migration/NativeFormat: do pool lookup instead of creating io_ctx
A Rados instance is sufficient to map the pool name to the pool ID,
no need to involve an IoCtx instance as well. While at it, report
distinctive errors for a non-existing pool and an invalid JSON value
for pool_name key cases.
librbd/migration: make SourceSpecBuilder::parse_source_spec() static
In preparation for divorcing NativeFormat from FormatInterface and
changing when/how src_image_ctx is created, make parse_source_spec()
independent of src_image_ctx. The "invalid source-spec JSON" error is
duplicated by the "failed to parse migration source-spec" error, so
just get rid of the former to spare having to pass CephContext to
parse_source_spec().
The "Client::mds_check_access" expects the target_path without
leading '/' as it eventually calls the "MDSCapMatch::match_path"
which expects the target_path passed to be with out leading '/'
as well.
The single leading '/' was being removed. But absolute path
constructed did have leading '//', so removing all the leading
'/' was necessary.
This causes the clients not to be able to access a particular
path even though it has a rw permission on the specific path.
Add missing spaces, don't use the word stream when reporting errors
on POSIX file operations (open() and lseek64()) and fix a cut-and-paste
typo in RawSnapshot.
This is a rework of the POSIXDriver. It refactors out the actual posix
parts into a set of classes that provide access to underlying
directory/file/symlink, etc. These primatives are used to build up full
support for Bucket, Object, Multipart, and VersionedObjects.
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
```
/home/jenkins-build/build/workspace/ceph-pull-requests/src/fmt/include/fmt/core.h:1756:3: error: static_assert failed due to requirement 'formattable' "Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt"
static_assert(
^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/fmt/include/fmt/core.h:1777:10: note: in instantiation of function template specialization 'fmt::detail::make_value<fmt::basic_format_context<fmt::appender, char>, const hobject_t &>' requested here
return make_value<Context>(val);
^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/fmt/include/fmt/core.h:1899:23: note: in instantiation of function template specialization 'fmt::detail::make_arg<true, fmt::basic_format_context<fmt::appender, char>, fmt::detail::type::custom_type, const hobject_t &, 0>' requested here
data_{detail::make_arg<
^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/fmt/include/fmt/core.h:1918:10: note: in instantiation of function template specialization 'fmt::format_arg_store<fmt::basic_format_context<fmt::appender, char>, hobject_t, unsigned int, unsigned int, bool, unsigned long>::format_arg_store<const hobject_t &, unsigned int &, unsigned int &, const bool &, unsigned long &>' requested here
return {FMT_FORWARD(args)...};
^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/fmt/include/fmt/core.h:3206:28: note: in instantiation of function template specialization 'fmt::make_format_args<fmt::basic_format_context<fmt::appender, char>, const hobject_t &, unsigned int &, unsigned int &, const bool &, unsigned long &>' requested here
return vformat(fmt, fmt::make_format_args(args...));
^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/crimson/common/tri_mutex.h:136:14: note: in instantiation of function template specialization 'fmt::format<const hobject_t &, unsigned int, unsigned int, const bool &, unsigned long>' requested here
os << fmt::format("tri_mutex {} writers {} readers {}"
^
```
projects / ceph.git / log