Joshua Baergen [Thu, 9 Nov 2023 16:43:22 +0000 (09:43 -0700)]
librbd: Append one journal event per image request
In the case where an image request is split across multiple object
extents and journaling is enabled, multiple journal events are appended.
Prior to this change, all object requests would wait for the last
journal event to complete, since journal events complete in order and
thus the last one completing implies that all prior journal events were
safe at that point.
The issue with this is that there's nothing stopping that last journal
event from being cleaned up before all object requests have stopped
referring to it. Thus, it's entirely possible for the following sequence
to occur:
1. An image request gets split into two image extents and two object
requests. Journal events are appended (one per image extent).
2. The first object request gets delayed due to an overlap, but the
second object request gets submitted and starts waiting on the last
journal event (which also causes a C_CommitIOEvent to be instantiated
against that journal event).
3. Journaling completes, and the C_CommitIOEvent fires. The
C_CommitIOEvent covers the entire range of data that was journaled in
this event, and so the event is cleaned up.
4. The first object request from above is allowed to make progress; it
tries to wait for the journal event that was just cleaned up which
causes the assert in wait_event() to fire.
As far as I can tell, this is only possible on the discard path today,
and only recently. Up until 21a26a752843295ff946d1543c2f5f9fac764593
(librbd: Fix local rbd mirror journals growing forever), m_image_extents
always contained a single extent for all I/O types; this commit changed
the discard path so that if discard granularity changed the discard
request, m_image_extents would be repopulated, and if the request
happened to cross objects then there would be multiple m_image_extents.
It appears that the intent here was that there should be one journal
event per image request and the pending_extents kept track of what had
completed thus far. This commit restores that 1:1 relationship.
Vallari Agrawal [Thu, 26 Oct 2023 07:55:44 +0000 (13:25 +0530)]
qa: add rbd/nvmeof test
A basic test for ceph-nvmeof[1] where
nvmeof initiator is created.
It requires use of a new task "nvmeof_gateway_cfg"
under cephadm which shares config information
between two remote hosts.
Zac Dover [Sun, 3 Dec 2023 12:17:46 +0000 (13:17 +0100)]
doc/rados: repair stretch-mode.rst
Remove a section of doc/rados/operations/stretch-mode.rst that I wrongly
re-included after its removal. The request for this (re)-removal is
here: https://github.com/ceph/ceph/pull/54689#discussion_r1413007655.
Zac Dover [Sat, 2 Dec 2023 05:38:28 +0000 (06:38 +0100)]
doc/radosgw: fix formatting
Repair the formatting of a string that had a string inside backticks
that itself was inside double asterisks. The presence of the asterisks
around the entire string caused the backticks to appear in the rendered
documentation.
Ronen Friedman [Fri, 1 Dec 2023 14:48:49 +0000 (08:48 -0600)]
tests/scrub: deactivate osd-scrub-dump stand-alone test
as the scrub reservation changes had made it obsolete.
Note - it is not an issue of fixing the test, but rather
that the tested functionality is no longer there.
Casey Bodley [Tue, 14 Nov 2023 01:05:47 +0000 (20:05 -0500)]
common: add versioned encodings for std::variant
adds two encoding strategies for `std::variant<>` under the namespaces
`ceph::versioned_variant` and `ceph::converted_variant`
these versioned encodings allow the variant to be extended with new
types, provided that they're always added to the end without changing
or removing existing types. because of this requirement, no default
encoding is provided for `std::variant`. callers must opt in to one
namespace or the other
the `converted_variant` encoding requires the variant's first type T
to use versioned encoding, and guarantees that the variant's encoding
is backward-compatible with T's
Nizamudeen A [Wed, 18 Oct 2023 17:46:09 +0000 (23:16 +0530)]
mgr/dashboard: cephfs subvolume list snapshots
Added a tab for displaying the subvolume snapshots
- this tab will show an info alert when there are no subvolumes present
- if the subvolume is present, then it'll be auto-selected by default
Implemented a filter to search the groups and subvolumes by its name.
Also added a scrollbar when there are too many items in the nav list
Modified the REST APIs to fetch only the names of the resources and
fetch the info when an API call is requesting for it.
Added unit tests
Fixes: https://tracker.ceph.com/issues/63237 Signed-off-by: Nizamudeen A <nia@redhat.com>
John Mulligan [Thu, 9 Nov 2023 19:26:35 +0000 (14:26 -0500)]
cephadm: convert all deploy tests to use funkypatch fixture
During the refactor of various daemon type classes some of the tests had
been converted to funkypatch in order to deal with imports occuring over
multiple files. However, this conversion was done piece by piece in
order to make clear what was changing. This left the functions in this
file inconsistent. Change all the remaining function to use funkypatch
for consistency.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 9 Nov 2023 18:46:04 +0000 (13:46 -0500)]
cephamd: update tests to use should_log_to_journald from context_getters
Update tests to import should_log_to_journald from context_getters - the
module that actually defines that function. This makes later refactoring
easier.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 9 Nov 2023 00:21:10 +0000 (19:21 -0500)]
cephadm: use funkypatch for setting up common patches in deploy tests
Add a shim function and convert to the use of the FunkyPatcher class in
the test_deploy.py test functions. Use a shim as to not have to change
all the tests (yet).
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 8 Nov 2023 19:31:12 +0000 (14:31 -0500)]
cephadm: convert test_mon_crush_location to use funkypatch fixture
The test_mon_crush_location test always seems to have me tinkering
with it during refactoring. Re-do the fixtures to use funkpatch instead
of mock.patch and normal monkeypatch. This looks nicer (IMO) and should
avoid having to frequently mess with it when moving functions during future
refactoring.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 20 Aug 2023 17:50:00 +0000 (13:50 -0400)]
cephadm: add a new funkypatch fixture based on mock.patch and pytest
This fixture acts like a combination of mock.patch and pytest's
monkeypatch fixture. It has the additional feature of automatically
finding and patching the same object imported in other modules. If you
have 'from x import y', where y is a function or class, in both a.py and
b.py it will patch both instances (so long as both a and b are already
imported).
This behavior is useful for cephadm because of the heavy use of the
`from x import y` idiom and how cephadm is being actively refactored.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sun, 5 Nov 2023 21:03:34 +0000 (16:03 -0500)]
cephadm: add a make_run_dir function
This function is roughly the same as make_var_run only it doesn't rely
on shelling out to the install command. Eventually, it will be used
to replace make_var_run in certain locations.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Casey Bodley [Sat, 18 Nov 2023 16:27:50 +0000 (11:27 -0500)]
rgw/acl: ACLGrant uses variant for grantee types
use of `ACLGrant::get_id()` was awkward because most grantee types
returned nothing, but emails were returned as `struct rgw_user`. change
the internal representation into a variant, and expose getters for each
grantee type so callers can handle each type specifically. the encoded
format of `ACLGrant` remains unchanged
Casey Bodley [Sat, 18 Nov 2023 15:22:20 +0000 (10:22 -0500)]
rgw/acl: req_state stores ACLs by value instead of unique_ptr
we no longer rely on polymorphism for the s3/swift variants of
`RGWAccessControlPolicy`, so `req_state` can store `bucket_acl`,
`object_acl` and `user_acl` by value
most functions now take these acls by const- or mutable reference
instead of pointers since they won't be nullptr
some code paths won't initialize some of these bucket/object/user acl
variables, and we rely on `RGWAccessControlPolicy::verify_permissions()`
to return false for those because we won't match an empty owner or
array of grants
in only one case, `verify_user_permissions()` has to return true when
`user_acl` is uninitialized, because S3 doesn't have user acls so
uninitialized user acls should not deny access
Casey Bodley [Sat, 18 Nov 2023 02:29:25 +0000 (21:29 -0500)]
rgw/acl/s3: parse_policy() as free function
s3 acl parsing classes no longer inherit from the acl classes
themselves, and are all encapsulated in rgw_acl_s3.cc behind a single
rgw::s3::parse_policy() function
Ronen Friedman [Thu, 30 Nov 2023 06:35:29 +0000 (00:35 -0600)]
osd/scrub: handling unexpected scrub requests
if arriving while still handling a previous chunk request,
the handling of of the previous chunk request will be aborted.
No response is sent. The scrubber resets, then handles the new request.
Ronen Friedman [Mon, 13 Nov 2023 07:09:45 +0000 (01:09 -0600)]
osd/scrub: decouple being reserved from handling scrub requests
For a replica, following this change:
* 'ReplicaActive' captures the state of the scrubber when
acting as a replica, from peering to interval change;
* "being reserved" is just a flag maintained by ReplicaActive, and
is no longer a prerequisite for handling scrub requests.
* each scrub request is now associated with its own 'token' value.
and the following minor simplification:
* the 'should we wait for pushes' decision is now part of the
code executed on the transition from ReplicaIdle into ReplicaActiveOp.
StartReplicaNoWait can now be discarded.
Venky Shankar [Thu, 30 Nov 2023 05:58:10 +0000 (11:28 +0530)]
Merge PR #53431 into main
* refs/pull/53431/head:
qa: add test cases to verify error reporting works as expected
mgr: fix some doc strings in object_format.py
mgr/tests: test returning error status works as expected
mgr: make object_format's Responder class capable of responding err status
mgr/nfs: report proper errno with err status
Reviewed-by: John Mulligan <jmulligan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
projects / ceph.git / log