Adam C. Emerson [Wed, 30 Nov 2022 22:43:15 +0000 (17:43 -0500)]
cls/fifo: Journal is flat_set, not multimap
We don't really need the overhead and complexity of a multimap.
Fixes: https://tracker.ceph.com/issues/57562 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit fcaa45d32d22653c6c8272a7a43af41bf3dc7e88)
Conflicts:
src/rgw/cls_fifo_legacy.cc
src/cls/fifo/cls_fifo_types.h
- Upstream C++20
- Have to write our own less-than since upstream uses the Spaceship Operator Fixes: https://tracker.ceph.com/issues/58403 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 14 Dec 2022 18:12:48 +0000 (13:12 -0500)]
cls/fifo: Move version inc into `apply_update()`
Only increment the version if we make an actual change.
Return whether we have changed something or not so the OSD side can
skip writing if there's no change.
Fixes: https://tracker.ceph.com/issues/57562 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit 32b514c52b37374202f20f588ef444144ba0e2c4) Fixes: https://tracker.ceph.com/issues/58403 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Wed, 30 Nov 2022 04:21:51 +0000 (23:21 -0500)]
cls/fifo: Remove all use of part tags
Part tags make part creation and setting the head non-idempotent,
leading to issues where racing RGWs may get confused about the correct
tag for a part. (Or worse, potentially have the metadata header hold
different value for a part than the part's header.)
Consistently only requires that all nodes agree on the number.
Fixes: https://tracker.ceph.com/issues/57562 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit f89a6d1833f45220ce558919de656f99cd1c9342)
Conflicts:
src/neorados/cls/fifo.cc
src/neorados/cls/fifo.h
src/test/cls_fifo/bench_cls_fifo.cc
src/test/cls_fifo/test_cls_fifo.cc
- Upstream deleted version has changes not in downstream
src/cls/fifo/cls_fifo_ops.h
- Upstream std::
src/cls/fifo/cls_fifo_types.h
- Upstream friend operator
- Upstream std::
src/rgw/CMakeLists.txt
- Downstream unneeded library dependency
src/rgw/cls_fifo_legacy.h
- Upstream dpp
src/rgw/cls_fifo_legacy.cc
- Upstream C++20
- Upstream dpp
Fixes: https://tracker.ceph.com/issues/58403 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Adam C. Emerson [Fri, 11 Nov 2022 01:00:38 +0000 (20:00 -0500)]
cls/fifo: Fix `same_or_later()` comparison
Fix logic error where disjunction was used instead of conjunction.
Fixes: https://tracker.ceph.com/issues/57562 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit dae8e781452311a2ec32fbf443814e43337294d8) Fixes: https://tracker.ceph.com/issues/58403 Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Zac Dover [Sun, 8 Jan 2023 08:04:43 +0000 (18:04 +1000)]
doc/glossary: Clean up "Ceph Object Storage"
Remove redundant material under the "Ceph Object Storage" headword and
add a "See 'Ceph Object Store'" link. A future PR will provide a couple
of sentences that explain how object storage is what's really supporting
both CephFS and RBD.
Zac Dover [Fri, 6 Jan 2023 16:24:39 +0000 (02:24 +1000)]
doc/css: Add scroll-margin-top to h2 html element
Add "scroll-margin-top: 4em;" to the h2 html element's definition in
custom.css. This moves the text under all h2 html elements out of the
way of the sticky-header-style top bar, which previously obscured the
text.
Zac Dover [Fri, 6 Jan 2023 12:51:47 +0000 (22:51 +1000)]
doc/man: define --num-rep, --min-rep and --max-rep
Explain the "--num-rep", "--min-rep", and "--max-rep" options, which are
required when running "crushtool" commands with the "--show-mappings"
flag. Originally reported by Brad Fitzpatrick.
Zac Dover [Thu, 5 Jan 2023 12:25:43 +0000 (22:25 +1000)]
doc/css: add scroll-margin-top to dt elements
add "scroll-margin-top: em3;" to custom.css so that the header bar
doesn't obscure the text of headwords in glossary.rst. Note that this
applies only to elements in the documentation that are rendered into
HTML with the dt (which stands for "description term" or "description
list") tag. Other modifications will be necessary in order to ensure
that the anchor points of non-dt elements are not obscured by the header
bar.
Zac Dover [Sun, 1 Jan 2023 12:06:54 +0000 (22:06 +1000)]
doc/start: add link-related metadocumentation
Add two kinds of link-related metadocumentation (documentation about how
to write documentation) to the "Documenting Ceph" section of the "Intro
to Ceph" document: 1. metadocumentation about external links, and 2.
metadocumentation about internal links.
Zac Dover [Sat, 31 Dec 2022 04:22:26 +0000 (14:22 +1000)]
doc/glossary: capitalize "DAS" correctly
Correctly capitalize "Direct-Attached Storage" in the glossary. (And
test the "Quincy" branch, which seems lately not to have picked up any
docs backports.)
Zac Dover [Fri, 30 Dec 2022 01:32:31 +0000 (11:32 +1000)]
doc/glossary: collate "releases" entries
Collect the "Releases"-related entries together under the "Releases"
headword, in order to give readers a sense at a glance of how the
different kinds of releases relate to one another.
Xiubo Li [Wed, 23 Nov 2022 05:24:38 +0000 (13:24 +0800)]
qa: switch to https protocol for repos' server
Since the git:// is not reachable any more and have switch to
https://.
The git archive does not support the https protocol, so we couldn't
user the git archive to retrieve the tar ball any more, will split
this into 3 steps:
1, clone the whole ceph repo
2, checkout the commit/tag/branch
3, then change directory to qa/workunits/.
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit 89177d65988c56324916de8394089b6e4b38aab7)
Conflicts:
- qa/workunits/fs/snaps/snaptest-git-ceph.sh: minor conflicts
- qa/machine_types/schedule_subset.sh: no need to fix this
- qa/tasks/cephfs/xfstests_dev.py: minor confilicts
Kamoltat [Wed, 14 Dec 2022 19:54:00 +0000 (19:54 +0000)]
mon/Monitor.cc: notify_new_monmap() skips removal of non-exist rank
Problem:
In RHCS the user can choose to manually remove a monitor rank
before shutting the monitor down. Causing inconsistency in monmap.
for example we remove mon.a from the monmap, there is a short period
where mon.a is still operational and will try to remove itself from
monmap but we will run into an assertion in
ConnectionTracker::notify_ranks_removed().
Solution:
In Monitor::notify_new_monmap() we prevent the func
from going into removing our own rank, or
ranks that doesn't exists in monmap.
FYI: this is an RHCS problem only, in ODF,
we never remove a monitor from monmap
before shutting it down.
--mon-initial-members does nothing but causes monmap
to populate ``removed_ranks`` because the way we start
monitors in standalone tests uses ``run_mon $dir $id ..``
on each mon. Regardless of --mon-initial-members=a,b,c, if
we set --mon-host=$MONA,$MONB,$MONC (which we do every single tests),
everytime we run a monitor (e.g.,run mon.b) it will pre-build
our monmap with
Now, with --mon-initial-members=a,b,c we are letting
monmap know that we should have initial members name:
a,b,c, which we only have `b` as a match. So what
``MonMap::set_initial_members`` do is that it will
remove noname-a and noname-c which will
populate `removed_ranks`.
Solution:
remove all instances of --mon-initial-members
in the standalone test as it has no impact on
the nature of the tests themselves.
When upgrading the monitors (include booting up),
we check if `peer_tracker` is dirty or not. If
so, we clear it. Added some functions in `Elector` and
`ConnectionTracker` class to
check for clean `peer_tracker`.
Moreover, there could be some cases where due
to startup weirdness or abnormal circumstances,
we might get a report from our own rank. Therefore,
it doesn't hurt to add a sanity check in
`ConnectionTracker::report_live_connection` and
`ConnectionTracker::report_dead_connection`.
In `notify_clear_peer_state()` we another
mechanism in reseting our `peer_tracker.rank`
to match our own monitor.rank.
This is added so there is a way for us
to recover from a scenrio where `peer_tracker.rank`
is messed up from adjusting the ranks or removing
ranks.
`notifiy_clear_peer_state()` can be triggered
by using the command:
`ceph connection scores reset`
Also in `clear_peer_reports`, besides
reassigning my_reports to an empty object,
we also have to make `my_reports` = `rank`
from `peer_tracker`, such that we don't get
-1 as a rank in my_reports.
Kamoltat [Wed, 2 Nov 2022 01:59:52 +0000 (01:59 +0000)]
mon: change how we handle removed_ranks
when a new monitor joins, there is a chance that
it will recive a monmap that recently removed
a monitor and ``removed_rank`` will have some
content in it. A new monitor that joins
should never remove rank in peer_tracker but
rather call ``notify_clear_peer_state()``
to reset the `peer_report`.
In the case when it is a monitor that
has joined quorum before and is only 1
epoch behind the newest monmap provided
by the probe_replied monitor. We can
actually remove and adjust ranks in `peer_report`
since we are sure that if there is any content in
removed_ranks, then it has to be because in the
next epoch we are removing a rank, since every
update of an epoch we always clear the removed_ranks.
There is no point in keeping the content
of ``removed_ranks`` after monmap gets updated
to the epoch.
Therefore, clear ``removed_ranks`` every update.
When there is discontinuity between
monmaps for more 1 epoch or the new monitor never joined quorum before,
we always reset `peer_tracker`.
Moreover, beneficial for monitor log to also log
which rank has been removed at the current time
of the monmap. So add removed_ranks to `print_summary`
and `dump` in MonMap.cc.
In `ConnectionTracker::receive_peer_report`
we loop through ranks which is bad when
there is `notify_rank_removed` before this and
the ranks are not adjusted yet. When we rely
on the rank in certain scenarios, we end up
with extra peer_report copy which we don't
want.
SOLUTION:
In `ConnectionTracker::receive_peer_report`
instead of passing `report.rank` in the function
`ConnectionTracker::reports`, we pass `i.first`
instead so that trim old ranks properly.
We also added a assert in notify_rank_removed(),
comparing expected rank provided by the monmap
against the rank that we adjust ourself to as
a sanity check.
We edited test/mon/test_election.cc
to reflect the changes made in notify_rank_removed().
Ilya Dryomov [Thu, 22 Dec 2022 15:32:44 +0000 (16:32 +0100)]
qa: switch to curl for qemu-xfstests
This is a follow-up for commit 631899ffeb84 ("qa: switch back to git
protocol for qemu-xfstests"), needed for the same "ancient execution
environment" reason.
Kefu Chai [Sun, 18 Dec 2022 12:18:44 +0000 (20:18 +0800)]
pybind/mgr/tox.ini: add commas in "modules" variable
since tox v4.0.13, it parses the variables differently, so the newlines
in a variable are passed right to the command referencing it. so we now
have failure like:
```
flake8: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr> flake8 --config=tox.ini alerts
flake8: commands[1] /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr> balancer
flake8: exit 2 (0.00 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/src/pybind/mgr> balancer
flake8: FAIL ✖ in 3.33 seconds
```
so we have to add comma as line continuation separator to address
this problem.
Ilya Dryomov [Mon, 19 Dec 2022 17:54:08 +0000 (18:54 +0100)]
qa: switch back to git protocol for qemu-xfstests
As noted in commit 89177d65988c ("qa: switch to https protocol for
repos' server"), git.ceph.com mirror doesn't make git:// available
anymore. However, run_xfstests-obsolete.sh has "obsolete" in its
name for a reason -- due to an ancient execution environment, git://
is the only viable option:
$ git clone https://git.ceph.com/xfstests-dev.git
Cloning into 'xfstests-dev'...
error: gnutls_handshake() failed: A TLS fatal alert has been received. while accessing https://git.ceph.com/xfstests-dev.git/info/refs
fatal: HTTP request failed
Kefu Chai [Sun, 18 Dec 2022 12:16:02 +0000 (20:16 +0800)]
pybind/mgr: s/setup(self)/setup_method(self)/
avoid pytest warnings like:
4: pg_autoscaler/tests/test_cal_final_pg_target.py::TestPgAutoscaler::test_even_pools_one_meta_three_bulk
4: /home/kefu/dev/ceph/src/pybind/mgr/.tox/py3/lib/python3.10/site-packages/_pytest/fixtures.py:900: PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
4: pg_autoscaler/tests/test_cal_final_pg_target.py::TestPgAutoscaler::test_even_pools_one_meta_three_bulk is using nose-specific method: `setup(self)`
4: To remove this warning, rename it to `setup_method(self)`
4: See docs: https://docs.pytest.org/en/stable/deprecations.html#support-for-tests-written-for-nose
4: fixture_result = next(generator)
Kefu Chai [Sun, 18 Dec 2022 12:15:06 +0000 (20:15 +0800)]
pybind/mgr/prometheus: avoid using distutils
to silence warnings like:
4: prometheus/module.py:35
4: /var/ssd/ceph/src/pybind/mgr/prometheus/module.py:35: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
4: v = StrictVersion(cherrypy.__version__)
Adam Kupczyk [Wed, 14 Dec 2022 16:43:47 +0000 (16:43 +0000)]
os/bluestore: BlueFS: harmonize log read and writes modes
BlueFS log has always been written in non-buffered mode.
Reading of it depends on bluefs_buffered_io option.
It is strongly suspected that this causes some wierd problems.
It is targetted directly to pacific.
Ultimately same fix will go to all versions.
This problem is severe, but happens very infrequently, mostly on contenerized
environments. We have a lot of problems in tracker that we suspect are caused by this.
To find them, we have "problem-detection" PR #49198 in main.
Then we will apply equivalent solution there too.
projects / ceph.git / log