Ilya Dryomov [Thu, 22 Dec 2022 15:32:44 +0000 (16:32 +0100)]
qa: switch to curl for qemu-xfstests
This is a follow-up for commit 631899ffeb84 ("qa: switch back to git
protocol for qemu-xfstests"), needed for the same "ancient execution
environment" reason.
Ilya Dryomov [Mon, 19 Dec 2022 17:54:08 +0000 (18:54 +0100)]
qa: switch back to git protocol for qemu-xfstests
As noted in commit 89177d65988c ("qa: switch to https protocol for
repos' server"), git.ceph.com mirror doesn't make git:// available
anymore. However, run_xfstests-obsolete.sh has "obsolete" in its
name for a reason -- due to an ancient execution environment, git://
is the only viable option:
$ git clone https://git.ceph.com/xfstests-dev.git
Cloning into 'xfstests-dev'...
error: gnutls_handshake() failed: A TLS fatal alert has been received. while accessing https://git.ceph.com/xfstests-dev.git/info/refs
fatal: HTTP request failed
Kamoltat [Wed, 14 Dec 2022 19:54:00 +0000 (19:54 +0000)]
mon/Monitor.cc: notify_new_monmap() skips removal of non-exist rank
Problem:
In RHCS the user can choose to manually remove a monitor rank
before shutting the monitor down. Causing inconsistency in monmap.
for example we remove mon.a from the monmap, there is a short period
where mon.a is still operational and will try to remove itself from
monmap but we will run into an assertion in
ConnectionTracker::notify_ranks_removed().
Solution:
In Monitor::notify_new_monmap() we prevent the func
from going into removing our own rank, or
ranks that doesn't exists in monmap.
FYI: this is an RHCS problem only, in ODF,
we never remove a monitor from monmap
before shutting it down.
--mon-initial-members does nothing but causes monmap
to populate ``removed_ranks`` because the way we start
monitors in standalone tests uses ``run_mon $dir $id ..``
on each mon. Regardless of --mon-initial-members=a,b,c, if
we set --mon-host=$MONA,$MONB,$MONC (which we do every single tests),
everytime we run a monitor (e.g.,run mon.b) it will pre-build
our monmap with
Now, with --mon-initial-members=a,b,c we are letting
monmap know that we should have initial members name:
a,b,c, which we only have `b` as a match. So what
``MonMap::set_initial_members`` do is that it will
remove noname-a and noname-c which will
populate `removed_ranks`.
Solution:
remove all instances of --mon-initial-members
in the standalone test as it has no impact on
the nature of the tests themselves.
When upgrading the monitors (include booting up),
we check if `peer_tracker` is dirty or not. If
so, we clear it. Added some functions in `Elector` and
`ConnectionTracker` class to
check for clean `peer_tracker`.
Moreover, there could be some cases where due
to startup weirdness or abnormal circumstances,
we might get a report from our own rank. Therefore,
it doesn't hurt to add a sanity check in
`ConnectionTracker::report_live_connection` and
`ConnectionTracker::report_dead_connection`.
In `notify_clear_peer_state()` we another
mechanism in reseting our `peer_tracker.rank`
to match our own monitor.rank.
This is added so there is a way for us
to recover from a scenrio where `peer_tracker.rank`
is messed up from adjusting the ranks or removing
ranks.
`notifiy_clear_peer_state()` can be triggered
by using the command:
`ceph connection scores reset`
Also in `clear_peer_reports`, besides
reassigning my_reports to an empty object,
we also have to make `my_reports` = `rank`
from `peer_tracker`, such that we don't get
-1 as a rank in my_reports.
Kamoltat [Wed, 2 Nov 2022 01:59:52 +0000 (01:59 +0000)]
mon: change how we handle removed_ranks
when a new monitor joins, there is a chance that
it will recive a monmap that recently removed
a monitor and ``removed_rank`` will have some
content in it. A new monitor that joins
should never remove rank in peer_tracker but
rather call ``notify_clear_peer_state()``
to reset the `peer_report`.
In the case when it is a monitor that
has joined quorum before and is only 1
epoch behind the newest monmap provided
by the probe_replied monitor. We can
actually remove and adjust ranks in `peer_report`
since we are sure that if there is any content in
removed_ranks, then it has to be because in the
next epoch we are removing a rank, since every
update of an epoch we always clear the removed_ranks.
There is no point in keeping the content
of ``removed_ranks`` after monmap gets updated
to the epoch.
Therefore, clear ``removed_ranks`` every update.
When there is discontinuity between
monmaps for more 1 epoch or the new monitor never joined quorum before,
we always reset `peer_tracker`.
Moreover, beneficial for monitor log to also log
which rank has been removed at the current time
of the monmap. So add removed_ranks to `print_summary`
and `dump` in MonMap.cc.
In `ConnectionTracker::receive_peer_report`
we loop through ranks which is bad when
there is `notify_rank_removed` before this and
the ranks are not adjusted yet. When we rely
on the rank in certain scenarios, we end up
with extra peer_report copy which we don't
want.
SOLUTION:
In `ConnectionTracker::receive_peer_report`
instead of passing `report.rank` in the function
`ConnectionTracker::reports`, we pass `i.first`
instead so that trim old ranks properly.
We also added a assert in notify_rank_removed(),
comparing expected rank provided by the monmap
against the rank that we adjust ourself to as
a sanity check.
We edited test/mon/test_election.cc
to reflect the changes made in notify_rank_removed().
Pere Diaz Bou [Fri, 11 Nov 2022 09:43:01 +0000 (10:43 +0100)]
mgr/prometheus: expose daemon health metrics
Until now daemon health metrics were stored without being used. One of
the most helpful metrics there is SLOW_OPS with respect to OSDs and MONs
which this commit tries to expose to bring fine grained metrics to find
troublesome OSDs instead of having a lone healthcheck of slow ops in the
whole cluster.
Aashish Sharma [Fri, 25 Nov 2022 06:07:54 +0000 (11:37 +0530)]
mgr/dashboard: Add metric relative to osd blocklist
It would be good to have a metric with number of clients "blocklisted" of command: "ceph osd blocklist ls". The reason to have this is metric is have an alert when this number raise up.
Kefu Chai [Sun, 18 Dec 2022 12:16:02 +0000 (20:16 +0800)]
pybind/mgr: s/setup(self)/setup_method(self)/
avoid pytest warnings like:
4: pg_autoscaler/tests/test_cal_final_pg_target.py::TestPgAutoscaler::test_even_pools_one_meta_three_bulk
4: /home/kefu/dev/ceph/src/pybind/mgr/.tox/py3/lib/python3.10/site-packages/_pytest/fixtures.py:900: PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
4: pg_autoscaler/tests/test_cal_final_pg_target.py::TestPgAutoscaler::test_even_pools_one_meta_three_bulk is using nose-specific method: `setup(self)`
4: To remove this warning, rename it to `setup_method(self)`
4: See docs: https://docs.pytest.org/en/stable/deprecations.html#support-for-tests-written-for-nose
4: fixture_result = next(generator)
Kefu Chai [Sun, 18 Dec 2022 12:15:06 +0000 (20:15 +0800)]
pybind/mgr/prometheus: avoid using distutils
to silence warnings like:
4: prometheus/module.py:35
4: /var/ssd/ceph/src/pybind/mgr/prometheus/module.py:35: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
4: v = StrictVersion(cherrypy.__version__)
Aashish Sharma [Wed, 7 Dec 2022 05:25:24 +0000 (10:55 +0530)]
mgr/dashboard: Rbd Mirroring improvements
1.RBD images in dashboard shows default mirroring as journal
2.snapshot based mirroring schedule Interval got disabled to edit
3.unable to create snapshot of an image using dashboard
4.provide snapshot schedule info in a new column
5.dashboard doesn't allow importing peer bootstrap key to be imported for subsequent pools
Aashish Sharma [Tue, 18 Oct 2022 11:12:32 +0000 (16:42 +0530)]
mgr/dashboard: Fix CephPoolGrowthWarning alert
Prometheus reports an error - many-to-many matching not allowed: matching labels must be unique on one side for CephPoolGrowthWarning if we have same pool ids on two different instances.
qa: Allow tests to override recovery configs with mClock scheduler enabled
Set osd_mclock_override_recovery_settings option to true for tests that
modify recovery/backfill configuration options. This prevents logging of
the cluster warning when modifying recovery/backfill limits.
qa/standalone: Add/Modify tests to verify mclock recovery/backfill limits
- Consolidate all mclock standalone tests under
qa/standalone/misc/mclock-config.sh.
- Revert existing tests in ceph-helpers.sh that verified the earlier hard
override of recovery/backfill limits.
- Add new tests to verify the procedure to change the recovery/backfill
limits with mclock scheduler.
doc: Update mClock config reference doc to reflect new max recovery limits
Document the following:
- New max backfill/recovery defaults for mClock.
- Steps to modify the backfill/recovery defaults.
- Modify defaults using new osd_mclock_override_recovery_settings option
- Steps to mitigate unrealistic OSD bench results to set OSD capacity.
- New capacity threshold options for ssd/hdd
With the earlier cost per io and cost per byte parameters, the
client ops performance was good but the recovery ops were taking
much longer to complete. The new cost parameters optimizes this
further so that the recovery ops progress faster while
maintaining similar client throughput.
This optimization is tested on both a small cluster and a larger
cluster in a scaled environment.
osd: Ignore osd bench results for mclock if it exceeds the set threshold
The osd bench may in some cases show unrealistic results. The IOPS capacity
reported is used for QoS specific calculations in the mClock scheduler. If
the IOPS capacity is unrealistic, it could result is a performance impact
to the ops handled by mClock scheduler.
To avoid this, do not consider the results of the osd bench if it breaches
a threshold set for hdd and ssd media. The existing default setting will
be enforced as a safety measure. A cluster log warning is raised for the
osd.
osd: Reduce default max backfill/recoveries for mclock to realistic limits
The earlier max number of backfill and recovery limits were overly
optimistic and caused recovery ops to overwhelm client ops regardless of
the mclock profile type.
Set the max backfill limit to 10. Set the max recovery limit to 10 for
osds having HDD backing device. For osds having SSD backing device, set
the recovery limit to 20.
Introduce and new config option, 'osd_mclock_override_recovery_settings',
that when set (false by default) allows the modification of the max
backfill and recovery options with mclock scheduler enabled.
It is important to note that with 'osd_mclock_override_recovery_settings'
disabled, the config subsystem will still allow the changes made to the
recovery and backfill settings when using the cli commands as the level at
which the change is made is always greater than CONF_DEFAULT. To counter this,
the osd will forcibly revert the recovery and backfill limits to the mClock
defaults by removing the existing setting from the config store and raises a
cluster log warning indicating the revert. This should force the user to set
the 'osd_mclock_override_recovery_settings' option first and then make changes
to the recovery and backfill limits.
Nizamudeen A [Fri, 25 Nov 2022 14:53:31 +0000 (20:23 +0530)]
mgr/dashboard: allow cross origin when the url is set
Allow CORS when the cross_origin_url is set in the config opt.
you have to update the cross_origin_url setting with the url of the
requesting entity.
The request needs to have the header `Access-Control-Allow-Origin`
with the origin URL
The url can be set using this command
`ceph config set mgr mgr/dashboard/cross_origin_url
http://localhost:4200`
multiple urls can be set as
`ceph config set mgr mgr/dashboard/cross_origin_url
http://localhost:4200,http://localhost:4201`
If multiple url is provided in the configuration option, then whatever
url is there in the Access-Control-Allow-Origin request header will be
allowed for CORS
Once the URL is set you have to restart the dashboard module to restart
the cherrypy server with the new CORS policies
Kefu Chai [Thu, 8 Dec 2022 16:49:37 +0000 (00:49 +0800)]
pybind/mgr: do not test with py37
as we always test with ubuntu jammy, which does not provide python3.7:
py37: skipped because could not find python interpreter with spec(s): py37
so there is no point testing with python3.7.
also, in tox v4, it is not able to handle "key = value" anymore, where
value has newlines in it. so we need to find a better way passing
command line options to the test command.
Adam King [Fri, 9 Dec 2022 16:10:36 +0000 (11:10 -0500)]
mgr/pybind: fix mypy arg parsing
On the new tox version it is treating each line as a new command
so it will do something like "mypy --config-file=../../mypy.ini"
as one command and then "-m balancer" as a totally separate command.
The first one immediately fails as it doesn't include any modules
to test. Adding backslashes to the ends of the lines gets it to
handle the lines as one long command
Kefu Chai [Thu, 8 Dec 2022 06:42:42 +0000 (14:42 +0800)]
qa: set locale to C.UTF-8 in tox.ini
as ansible is using UTF-8 encoded characters in the file names, so,
to avoid failures like:
File "/home/jenkins-build/build/workspace/ceph-pull-requests/qa/.tox/py3/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 217, in untar_file
with open(path, "wb") as destfp:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 137-140: ordinal not in range(256)
we have to set a locale which is able to handle UTF-8.
see also https://github.com/ceph/teuthology/pull/1671
Kefu Chai [Thu, 8 Dec 2022 10:25:20 +0000 (18:25 +0800)]
pybind/mgr: drop cython from requires
cython is not required for running tox commands.
this should address the test failure like:
ROOT: will run in automatically provisioned tox, host /home/jenkins-build/build/workspace/ceph-pull-requests/build/mgr-virtualenv/bin/python3.10 is missing [requires (has)]: cython
Kefu Chai [Thu, 8 Dec 2022 06:53:33 +0000 (14:53 +0800)]
*: s/whitelist_externals/allowlist_externals/
as allowlist_externals was introduced in
tox v4.0. see
https://github.com/tox-dev/tox/commit/5e33fda1a40ffb4973de3d607a572891eb3cb2d2 , but
this option was backported to 3.18 as an alias of whitelist_externals, so we don't need
to specify the minversion to 4.0 in this change.
as we started using tox 4.0 and up (v4.0.2 in specific). tox complains
and fails like:
alerts-lint: failed with promtool is not allowed, use allowlist_externals to allow it
alerts-lint: FAIL code 1 (9.25 seconds)
see https://tox.wiki/en/latest/faq.html#tox-4-removed-tox-ini-keys
and https://tox.wiki/en/latest/config.html#allowlist_externals
it'd be nice to use a more inclusive language also. so, in this change,
s/whitelist_externals/allowlist_externals/ in all tox.ini in this
project.
projects / ceph.git / log