Patrick Donnelly [Mon, 11 Dec 2023 13:37:39 +0000 (08:37 -0500)]
Merge PR #54726 into main
* refs/pull/54726/head:
PendingReleaseNotes: announce cephfs-shell avail. on rhel9
qa: test fs:shell on all distros
qa: add cephfs-shell to installed rpm packages
ceph.spec.in: enable support for cephfs-shell by default via EPEL9
Ilya Dryomov [Sat, 9 Dec 2023 20:00:42 +0000 (21:00 +0100)]
test/librbd: avoid config-related crashes in DiscardWithPruneWriteOverlap
For reasons that I think no longer apply today, set_val() and
set_val_or_die() refuse to set "type: str" config options that aren't
marked as "can be changed at runtime" -- set_val() returns an error and
set_val_or_die() terminates the process. What is and isn't marked as
"can be changed at runtime" seems to be pretty much random both within
and outside of RBD, so let's just refactor how config is set here.
While at it, I realized that reproducer config is underspecified:
- for rbd_cache_policy and rbd_cache_writethrough_until_flush settings
to matter, rbd_cache must be set to true and rbd_cache_max_dirty must
be set to a positive number
- order should be set explicitly, because rbd_default_order can be as
low as 12 (for 4096-byte objects), interfering with the logic of the
test
Zac Dover [Sat, 9 Dec 2023 03:46:00 +0000 (04:46 +0100)]
doc/radosgw: format POST statements
Format the POST methods so that they appear in the rendered text as
examples of POST API calls and not as plain old unformatted text, which
is how they looked before this commit. The content of these API calls
remains to be tested and confirmed to work, but this is a first step.
Casey Bodley [Thu, 7 Dec 2023 14:11:12 +0000 (09:11 -0500)]
vstart: add --rgw_store option for rados|dbstore|posix
enables dbstore for rgw_backend_store and rgw_config_store, allowing
vstart to run without any mons or osds. database files are put under
the dev subdirectory
when rgw_store=posix, the posix filter is added on top of dbstore
Mark Kogan [Tue, 28 Nov 2023 12:34:31 +0000 (14:34 +0200)]
rgw: d3n: fix valgrind reported leak related to libaio worker threads
which sporadically reproduces on teuthology ubuntu instances
happens because a race between RGW shutdown occurring before
the libaio worker threads had terminated
to fix, reduced the libaio threads inactivity shutdown time
ref:
man aio_init
...
aio_idle_time
This field specifies the amount of time in seconds that a worker thread
should wait for further requests before terminating, after having
completed a previous request. The
default value is 1.
...
Fixes: https://tracker.ceph.com/issues/63445 Signed-off-by: Mark Kogan <mkogan@ibm.com>
Zac Dover [Sat, 2 Dec 2023 05:32:26 +0000 (06:32 +0100)]
doc/radosgw: add gateway starting command
Add a command that properly starts (or restarts) the RADOS gateway after
RGW settings have been changed. This commit has been added in response
to an issue reported anonymously on
https://pad.ceph.com/p/Report_Documentation_Bugs.
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
Joshua Baergen [Thu, 9 Nov 2023 16:43:22 +0000 (09:43 -0700)]
librbd: Append one journal event per image request
In the case where an image request is split across multiple object
extents and journaling is enabled, multiple journal events are appended.
Prior to this change, all object requests would wait for the last
journal event to complete, since journal events complete in order and
thus the last one completing implies that all prior journal events were
safe at that point.
The issue with this is that there's nothing stopping that last journal
event from being cleaned up before all object requests have stopped
referring to it. Thus, it's entirely possible for the following sequence
to occur:
1. An image request gets split into two image extents and two object
requests. Journal events are appended (one per image extent).
2. The first object request gets delayed due to an overlap, but the
second object request gets submitted and starts waiting on the last
journal event (which also causes a C_CommitIOEvent to be instantiated
against that journal event).
3. Journaling completes, and the C_CommitIOEvent fires. The
C_CommitIOEvent covers the entire range of data that was journaled in
this event, and so the event is cleaned up.
4. The first object request from above is allowed to make progress; it
tries to wait for the journal event that was just cleaned up which
causes the assert in wait_event() to fire.
As far as I can tell, this is only possible on the discard path today,
and only recently. Up until 21a26a752843295ff946d1543c2f5f9fac764593
(librbd: Fix local rbd mirror journals growing forever), m_image_extents
always contained a single extent for all I/O types; this commit changed
the discard path so that if discard granularity changed the discard
request, m_image_extents would be repopulated, and if the request
happened to cross objects then there would be multiple m_image_extents.
It appears that the intent here was that there should be one journal
event per image request and the pending_extents kept track of what had
completed thus far. This commit restores that 1:1 relationship.
Casey Bodley [Tue, 5 Dec 2023 21:12:56 +0000 (16:12 -0500)]
osd/scrubber: fix signed comparison warning
[681/1140] Building CXX object src/osd/CMakeFiles/osd.dir/scrubber/scrub_resources.cc.o
src/osd/scrubber/scrub_resources.cc: In member function ‘bool Scrub::ScrubResources::inc_scrubs_remote(pg_t)’:
src/osd/scrubber/scrub_resources.cc:84:18: warning: comparison of integer expressions of different signedness: ‘long unsigned int’ and ‘const int64_t’ {aka ‘const long int’} [-Wsign-compare]
84 | if (pre_op_cnt < conf->osd_max_scrubs) {
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
Zac Dover [Tue, 5 Dec 2023 19:46:26 +0000 (20:46 +0100)]
doc/radosgw: update link in rgw-cache.rst
Update link in doc/radosgw/rgw-cache.rst. The link updated here is a
link to all the Nginx configuration files. The old link was broken. This
update comes to us from an anonymous report on
https://pad.ceph.com/p/Report_Documentation_Bugs.
Casey Bodley [Tue, 5 Dec 2023 17:21:18 +0000 (12:21 -0500)]
common: use inline for monostate dencoders
fix a 'multiple definition' error when included by multiple sources:
src/common/versioned_variant.h:31: multiple definition of `ceph::encode(std::monostate const&, ceph::buffer::v15_2_0::list&)';
rgw_main.cc.o:src/common/versioned_variant.h:31: first defined here
Ilya Dryomov [Fri, 1 Dec 2023 17:29:12 +0000 (18:29 +0100)]
test/librbd: drop DiffIterateTest.DiffIterateRegression6926
This was added to test [1]. It's duplicated by several cases in
DiffIterateTest.DiffIterateDeterministicPP now. Specifically, the
issue could be reproduced by any of:
(8) beginning of time -> snap2
(9) snap1 -> snap2
(10) beginning of time -> snap1
Ilya Dryomov [Fri, 1 Dec 2023 17:54:19 +0000 (18:54 +0100)]
test/librbd: drop TestLibRBD.SnapDiff
This was added to integration test [1], separate from the fix which
went in only with unit test adjustments. It's duplicated by several
cases in DiffIterateTest.DiffIterateDeterministic now. Specifically,
the issue could be reproduced by any of:
(3) snap2 -> HEAD
(4) snap3 -> HEAD
(7) snap2 -> snap3
scribble()-based DiffIterate tests are too weak: at least two
regressions that should been caught by DiffIterate.DiffIterate or
DiffIterate.DiffIterateStress were missed [1][2]. Aside from the
randomness which can be both a good and a bad thing, asserts there
ensure only that the returned diff covers all changes that were made.
If the returned diff is too excessive or otherwise bogus, this isn't
detected [3].
Add a deterministic test to systematically cover the most common cases
that don't involve discards. A similar test for discards will be added
with the fix for [4].
Comment out debug log in vector_iterate_cb() like it's done in
iterate_cb().
Rishabh Dave [Tue, 5 Dec 2023 05:43:16 +0000 (11:13 +0530)]
cephfs: add a release note for PR #53899
With merging of PR #53899 it is mandatory now for CephFS to be offline
and refuse_client_session to be set before an attempt to rename the
CephFS is made. Add a release note for this.
Fixes: https://github.com/ceph/ceph/pull/53899 Fixes: https://tracker.ceph.com/issues/63154 Signed-off-by: Rishabh Dave <ridave@redhat.com>
John Mulligan [Mon, 4 Dec 2023 18:20:02 +0000 (13:20 -0500)]
cephadm: simplify loading jinja2 templates from zipapp
The saga of failures in teuthology brought on by the combination of
zipapp and jinja2 template loading continues in this episode. Remove
some of the path handling that was taken from the original jinja2
implementation and try to replace it with something inherently simpler
such that we always construct paths relative to the zip archive and pass
that to the zipimporter. The hope here is that by doing fewer
manipulations of the "path" we are more likely to match the items
tracked inside the zipimporter class.
We do not worry about some of the concerns that regular jinja2 has as we
never expect cephadm to be run on something other than linux and we have
control over what sources we expect to load templates from. We do take
one precaution and that is to reject any paths that contain "." and ".."
as components. Less to avoid malicious attempts to read files it should
not (but that would be nice) and more to nudge toward simple template
references recorded in our Templates enum.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Vallari Agrawal [Thu, 26 Oct 2023 07:55:44 +0000 (13:25 +0530)]
qa: add rbd/nvmeof test
A basic test for ceph-nvmeof[1] where
nvmeof initiator is created.
It requires use of a new task "nvmeof_gateway_cfg"
under cephadm which shares config information
between two remote hosts.
Zac Dover [Sun, 3 Dec 2023 12:17:46 +0000 (13:17 +0100)]
doc/rados: repair stretch-mode.rst
Remove a section of doc/rados/operations/stretch-mode.rst that I wrongly
re-included after its removal. The request for this (re)-removal is
here: https://github.com/ceph/ceph/pull/54689#discussion_r1413007655.
projects / ceph.git / log