Zac Dover [Fri, 17 Jan 2025 12:33:49 +0000 (22:33 +1000)]
doc/cephfs: edit disaster-recovery-experts (2 of x)
Edit the third and fourth sections of
doc/cephfs/disaster-recovery-experts.rst in preparation for adding
deeper explanations of the contexts in which one should use the various
commands listed on that page.
Follows https://github.com/ceph/ceph/pull/61426
https://tracker.ceph.com/issues/69557
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 4f3a69eb919fc0d99cdf943f095ca3a951c82897)
This commit updates RGW Config Reference - Lifecycle Settings section. In particular it addresses an incorrect suggestion to decrease parallel threads in the workers pool for a more aggressive/accelerated per-bucket lifecycle processing. A more aggressive lifecycle processing for a bucket containing higher number of objects is achieved by increasing, not decreasing parallel threads.
Current suggestion is miss-leading.
Zac Dover [Thu, 16 Jan 2025 11:51:46 +0000 (21:51 +1000)]
doc/cephfs: edit disaster-recovery-experts
Edit the first two sections of doc/cephfs/disaster-recovery-experts.rst
in preparation for adding deeper explanations of the contexts in which
one should use the various commands listed on that page.
Zac Dover [Thu, 16 Jan 2025 15:49:04 +0000 (01:49 +1000)]
squid: src/exporter: revert incorrect lines
Revert part of 99f4eccb6bef444f86e6d6cbf579dc3ba975af98. The lines
removed here were reported to me by Anthony D'Atri, who says that Juan
Miguel Olmo told him that they should not be in the Squid branch.
See https://github.com/ceph/ceph/pull/61331#issuecomment-2596058701
Adam Kupczyk [Fri, 10 Jan 2025 08:26:54 +0000 (08:26 +0000)]
os/bluestore: Fix BlueFS::truncate()
In `struct bluefs_fnode_t` there is a vector `extents` and
the vector `extents_index` that is a log2 seek cache.
Until modifications to truncate() we never removed extents from files.
Modified truncate() did not update extents_index.
For example 10 extents long files when truncated to 0 will have:
0 extents, 10 extents_index.
After writing some data to file:
1 extents, 11 extents_index.
Now, `bluefs_fnode_t::seek` will binary search extents_index,
lets say it located seek at item #3.
It will then jump up from #0 extent (that exists) to #3 extent which
does not exist at.
The worst part is that code is now broken, as #3 != extent.end().
There are 3 parts of the fix:
1) assert in `bluefs_fnode_t::seek` to protect against
jumping outside extents
2) code in BlueFS::truncate to sync up `extents_index` with `extents`
3) dampening down assert in _replay to give a way out of cases
where incorrect "offset 12345" (12345 is file size) instead of
"offset 20000" (allocations occupied) was written to log.
Fixes: https://tracker.ceph.com/issues/69481 Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
(cherry picked from commit 7f3601089d41bfc23f530c7bf3fb7efad2d055ec)
Adam Kupczyk [Fri, 10 Jan 2025 10:07:18 +0000 (10:07 +0000)]
os/bluestore: bluefs unittest for truncate bug
Unittest showing 2 different flavours of problems:
1) bluefs log corruption
2) bluefs sigsegv
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
(cherry picked from commit f2b5e2fa0a9274c1667fccafa597fff9be7a74b1)
+fixup for bad usage of std::string's fill constructor
This commit adds:
1. workflow summary in the first section along with an image.
2. sub-section "Pushing to ceph-ci repository" to second section.
3. file doc/dev/developer_guide/testing_integration_tests/workflow.png
Naman Munet [Fri, 22 Nov 2024 09:57:44 +0000 (15:27 +0530)]
mgr/dashboard: Administration > Configuration > Some of the config options are not updatable at runtime
Fixes: https://tracker.ceph.com/issues/68976
Fixes Includes:
1) by-passing 'can_update_at_runtime' flag for 'rgw' related configurations as the same can be updated at runtime via CLI.
Also implemented a warning popup for user to make force edit to rgw related configurations.
2) when navigated to Administration >> Configuration, modified configuration will be seen as we see in cli "ceph config dump",
instead of configuration with filter level:basic
Ronen Friedman [Thu, 26 Dec 2024 13:06:10 +0000 (07:06 -0600)]
osd/scrub: register for 'osd_max_scrubs' config changes
Since https://github.com/ceph/ceph/pull/55340, the
osd_max_scrubs (also) affects the parameters of the
async scrub reserver used by the replicas. Thus,
the code must notice and acknowledge changes to this config.
Ilya Dryomov [Fri, 20 Dec 2024 10:16:58 +0000 (11:16 +0100)]
rbd: drop --pool option from "rbd group image {add,rm}"
It stopped working with removal of get_special_pool_group_names() in
commit 3e8624f157a1 ("rbd: add support for namespaces") over six years
ago. Given how much time has passed, stop accepting this option.
Ilya Dryomov [Tue, 17 Dec 2024 15:06:17 +0000 (16:06 +0100)]
rbd: handle --{group,image}-namespace in "rbd group image {add,rm}"
Currently only passing the namespace as part of the group or image spec
works. If --group-namespace or --image-namespace options are used, the
namespace isn't picked up.
Inject an object map with all possible inconsistencies before
flattening to ensure that something similar to commit 40af4f87b64f
("librbd: flatten operation should use object map") doesn't reappear
in a different form.
Ilya Dryomov [Wed, 27 Nov 2024 10:11:16 +0000 (11:11 +0100)]
librbd: avoid data corruption on flatten when object map is inconsistent
By making flatten skip copyup in case the object is marked
OBJECT_EXISTS or OBJECT_EXISTS_CLEAN, commit 40af4f87b64f ("librbd:
flatten operation should use object map") introduced a critical
regression. If the object map becomes inconsistent (e.g. because
flatten gets interrupted by killing "rbd flatten" process or a client
running on the clone crashes after updating the object map but before
writing to the image), the following attempt to flatten would corrupt
the clone if the copyup is actually still needed.
By design, it's impossible to tell whether the object is "known to
exist" based on the object map -- only telling whether the object is
"known to NOT exist" is possible (i.e. only OBJECT_NONEXISTENT state
is reliable). Negating OBJECT_NONEXISTENT tells that the object "may
exist", not that the object is "known to exist". This is reflected in
the name of object_may_exist() helper that was introduced together with
the object map implementation. Something like object_may_not_exist()
simply can't be constructed given the rest of librbd.
This effectively reverts commits 4c86bccf07b8 ("librbd: add
object_may_not_exist helper") and 40af4f87b64f ("librbd: flatten
operation should use object map").
A bogus change introduced as part of PR#54363 (commit fbb7d73) changed multiple 'scrub' commands to 'scheduled-scrub'.
In this one instance - that was wrong.
Afreen [Wed, 6 Mar 2024 20:22:16 +0000 (01:52 +0530)]
mgr/dashboard: handle infinite values for pools
Fixes https://tracker.ceph.com/issues/64724
Issue:
======
Json parsing is failing because of Infinity values present in pools
meteadata. "read_balance": {"score_acting": Infinity, "score_stable":
Infinity,}
Due to this entire pool list is not rendered.
Fix:
====
Added a handler for checking "inf" values and replacing them with a
string "Infinity" so that json parsing does not fail on frontend.
Ilya Dryomov [Thu, 12 Dec 2024 20:32:39 +0000 (21:32 +0100)]
librbd/migration/HttpClient: socket isn't shut down on some state transitions
If shut_down() gets delayed until a) the state transition from
STATE_RESET_CONNECTING completes and the reconnect is unsuccessful or
b) the state transition from STATE_RESET_DISCONNECTING completes (i.e.
next_state is STATE_UNINITIALIZED or STATE_RESET_CONNECTING), the
socket needs to be shut down before m_on_shutdown is invoked. The line
of thought here is the same as for the corresponding state transitions
that don't involve STATE_SHUTTING_DOWN.
Ilya Dryomov [Wed, 11 Dec 2024 15:25:13 +0000 (16:25 +0100)]
librbd/migration/HttpClient: avoid hitting an assert in advance_state()
If the shutdown gets delayed until the state transition from
STATE_RESET_CONNECTING completes and the reconnect is successful
(i.e. next_state is STATE_READY), we eventually hit "unexpected
state transition" assert in advance_state(). The reason is that
advance_state() would update m_state and call disconnect() under
STATE_READY instead of STATE_SHUTTING_DOWN. After the disconnect
maybe_finalize_shutdown() would enter advance_state() again with
STATE_SHUTDOWN as next_state, but the transition to that from
STATE_READY is invalid.
Plug this by not transitioning to next_state if current_state is
STATE_SHUTTING_DOWN.
Ilya Dryomov [Mon, 9 Dec 2024 10:19:57 +0000 (11:19 +0100)]
librbd/migration/HttpClient: ignore stream_truncated when shutting down SSL
Propagate ec to handle_disconnect() and use it to suppress
stream_truncated errors. Here is a quote from Beast documentation [1]:
// Gracefully shutdown the SSL/TLS connection
error_code ec;
stream.shutdown(ec);
// Non-compliant servers don't participate in the SSL/TLS shutdown process and
// close the underlying transport layer. This causes the shutdown operation to
// complete with a `stream_truncated` error. One might decide not to log such
// errors as there are many non-compliant servers in the wild.
if(ec != net::ssl::error::stream_truncated)
log(ec);
... and a commit that made ignoring stream_truncated safe [2]:
// ssl::error::stream_truncated, also known as an SSL "short read",
// indicates the peer closed the connection without performing the
// required closing handshake
// [...]
// When a short read would cut off the end of an HTTP message,
// Beast returns the error beast::http::error::partial_message.
// Therefore, if we see a short read here, it has occurred
// after the message has been completed, so it is safe to ignore it.
Ilya Dryomov [Sat, 7 Dec 2024 12:52:41 +0000 (13:52 +0100)]
librbd/migration/HttpClient: drop SslHttpSession::m_ssl_enabled
The remaining callers of disconnect() call it only when m_ssl_enabled
is set to true (i.e. after the handshake is completed):
- shut_down(), in STATE_READY
- maybe_finalize_reset(), very shortly after transitioning out of
STATE_READY as part of performing a reset
- advance_state(), on a transition to STATE_READY that is intercepted
by a previously delayed shut down
m_ssl_enabled isn't used outside of disconnect() and on top of that
is never cleared.
Ilya Dryomov [Sat, 7 Dec 2024 11:22:52 +0000 (12:22 +0100)]
librbd/migration/HttpClient: don't call disconnect() in handle_handshake()
With m_ssl_enabled set to false, disconnect() is a no-op. Since
m_ssl_enabled is flipped to true only when the handshake succeeds,
calling disconnect() on "failed to complete handshake" error is bogus
(as would be attempting to shut down SSL there).
Ilya Dryomov [Fri, 6 Dec 2024 15:51:51 +0000 (16:51 +0100)]
librbd/migration/HttpClient: avoid reusing ssl_stream after shut down
ssl_stream objects can't be reused after shut down: despite
a successful reconnect and handshake, any attempt to read or write
fails with "end of stream" (beast.http:1) or "protocol is shutdown"
(asio.ssl:337690831) error respectively. This doesn't appear to be
documented, but Beast and ASIO authors both mention that the stream
must be destroyed and recreated [1][2].
This was missed because the only integration test with a big enough
image used http instead of https.
Ilya Dryomov [Fri, 6 Dec 2024 13:42:55 +0000 (14:42 +0100)]
librbd/migration/HttpClient: don't shut down socket in resolve_host()
resolve_host() is called from init() and issue() when transitioning out
of STATE_UNINITIALIZED and from advance_state() right after the call to
shutdown_socket(). In all three cases the socket should get closed, so
drop the redundant call and place asserts in connect() implementations
instead.
Ronen Friedman [Sun, 25 Aug 2024 17:05:11 +0000 (12:05 -0500)]
test/scrub: only instruct clean PGs to scrub
Recent changes to the scrub scheduling mechanism, especially
regarding the 'must_scrub' flag, cause operator scrub commands
issued on a not-clean PG to be rejected - and forgotten.
This commit changes the tests to issue a scrub command only
after the target PG is clean.
Zac Dover [Fri, 13 Dec 2024 06:12:49 +0000 (16:12 +1000)]
doc/cephfs: edit 3rd 3rd of mount-using-kernel-driver
Edit the third third of doc/cephfs/mount-using-kernel-driver.rst in
preparation for correcting mount commands that may not work in Reef as
described in this documentation.
This commit edits only English-language strings in
doc/cephfs/mount-using-kernel-driver.rst. No technical content (that is,
no commands and no settings) have been altered in this commit.
Technical alterations to this file will be made only after the English
is unambiguous.
This PR follows the following two PRs:
https://github.com/ceph/ceph/pull/61048 - 1st 3rd
https://github.com/ceph/ceph/pull/61049 - 2nd 3rd
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com> Signed-off-by: Zac Dover <zac.dover@proton.me>
(cherry picked from commit 9c7580a2935511d009c9e66885e76635aa504ee8)
J. Eric Ivancich [Fri, 22 Nov 2024 17:40:24 +0000 (12:40 -0500)]
rgw: optimize bucket listing to skip past regions of namespaced entries
When listing a bucket and the parameters are such that we're not
listing namespaced entries, this commit adds an optimization to
advance the marker such that we skip past a whole region of namespaced
entries rather than evaluating each entry one-by-one.
Prashant D [Mon, 8 Aug 2022 14:55:23 +0000 (10:55 -0400)]
mon/LogMonitor: Use generic cluster log level config
We do not control the verbosity of the LogEntry
which is getting logged to stderr, graylog and
journald. This causes excessive flooding of logs
to /var/log, making a filesystem to fill up quickly.
Also we have different config variables namely
mon_cluster_log_file_level and mon_cluster_log_to_syslog_level
to control verbosity at cluster log file and
syslog level respectively. Add a generic cluster log
level config variable which controls cluster log
verbosity for all external entities.
Additionally, this patch addresses the regression of
`mon_cluster_log_file_level` option which doesn't take effect
because of code refactoring of LogMonitor::update_from_paxos
(commit : 7c84e06).
Fixes: https://tracker.ceph.com/issues/57061 Fixes: https://tracker.ceph.com/issues/57049 Signed-off-by: Prashant D <pdhange@redhat.com>
(cherry picked from commit cb0d4a57bd25d32919f7ed44815b4301f41e9508)
projects / ceph.git / log