Volker Theile [Wed, 9 Feb 2022 08:37:48 +0000 (09:37 +0100)]
mgr/dashboard: "Please expand your cluster first" shouldn't be shown if cluster is already meaningfully running
This PR will assume that a cluster is already up and fully running. If this should not be the expected behaviour, deployment tools have to set 'INSTALLED' explicitly. Without this assumption it might happen that upgraded and fully running clusters, e.g. Octopus -> Pacific, will show the 'Expand Cluster' on first log in.
cephadm will take care that the bootstrap phase will write the necessary key to show the 'Expand cluster' page.
Laura Flores [Fri, 11 Feb 2022 19:37:26 +0000 (19:37 +0000)]
mgr/telemetry: handle empty device report when "send" is triggered
On certain environments, such as the "ceph-dev-docker" environment
(https://github.com/ricardoasmarques/ceph-dev-docker), the mgr
module is unable to fetch device metrics. As a result, the device
report generated by "gather_device_report()" returns an empty dict.
This causes an AssertionError when the "send" function is triggered
(i.e. by running `ceph telemetry status` or `ceph telemetry send`),
and the module crashes.
The fix in this commit checks that the generated device report
contains metrics before trying to send it. If the device report
does not contain metrics (it returns an empty dict), the module
will log an appropriate message in the mgr log and not send the
device report.
If this scenario happens when running the `ceph telemetry send` command,
the user will additionally see this message:
```
Ceph report sent to https://telemetry.ceph.com/report
Unable to send device report: channel is on, but generated report was empty.
```
I also added a few more debug messages in gather_device_report() to make
future debugging easier.
Fixes: https://tracker.ceph.com/issues/54250 Signed-off-by: Laura Flores <lflores@redhat.com>
(cherry picked from commit 54e0e58f1b3f431281df0e2dd2b258f85cbade19)
Adam Kupczyk [Wed, 9 Feb 2022 15:19:56 +0000 (16:19 +0100)]
os/bluestore/bluefs: Fix improper vselector tracking in _flush_special()
Moves vselector size tracking outside _flush_special().
Function _compact_log_async...() updated sizes twice.
Problem could not be solved by making second modification of size just update,
as it will possibly disrupt vselector consistency check (_vselector_check()).
Feature to track vselector consistency relies on the fact that either log.lock or nodes.lock
are taken when the check is performed. Which is not true for _compact_log_async...().
Now _flush_special does not update vselector sizes by itself but leaves the update to
the caller.
Fixes: https://tracker.ceph.com/issues/54248 Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 4bc0f61d23299724fad2d8e6f2858734f1db6e5a)
both a < b and b < a evaluate to true. This violates STL strict weak
ordering requirements which is a problem because GroupSnapshotNamespace
is used as a key in std::map (ictx->snap_ids at least), etc.
Ilya Dryomov [Tue, 8 Feb 2022 09:11:49 +0000 (10:11 +0100)]
rbd: mark optional positional arguments as such in help output
Currently at least five commands have optional positional arguments.
Overloading po::value<std::string>()->default_value("") for this
is a bit sneaky but nothing better fits into the existing Shell.cc
framework.
Note that strictly speaking "[<interval>] [<start-time>]" should be
"[<interval> [<start-time>]]" but we aren't doing that here because
"ceph" command doesn't do it either.
Kamoltat [Fri, 14 Jan 2022 02:44:16 +0000 (02:44 +0000)]
pybind/mgr/progress: disable pg recovery event by default
The progress module disabled the pg recovery event by default
since the event is expensive and has interrupted other serviceis
when there is OSDs being marked in/out from the the cluster.
To turn the event on manually:
ceph config set mgr mgr/progress/allow_pg_recovery_event true
Updated qa/tasks/mgr/test_progress.py to enable
the pg recovery event when testing the progress module.
cephadm/ingress: make frontend stat bind on localhost
The current configuration of keepalived makes it do
a curl on localhost:9999 in order to check the endpoint is alive.
Given the endpoint only binds on the vip addr, that doesn't work.
Laura Flores [Mon, 24 Jan 2022 03:19:50 +0000 (21:19 -0600)]
mgr/telemetry: collect what we can from heap stats, mempools, and osd histograms
If we run into a problem collecting heap stats, mempools,
or osd histograms from a particular osd (i.e. the osd is down),
we should continue to collect what we can from other osds rather
than exiting and returning an empty JSON object.
Some log messages are also refined.
Fixes: https://tracker.ceph.com/issues/53985 Signed-off-by: Laura Flores <lflores@redhat.com>
(cherry picked from commit c617b78f7bb589314b3c377496a9bb3914cbb2ba)
Patrick Donnelly [Fri, 23 Jul 2021 15:19:48 +0000 (08:19 -0700)]
qa: stop overriding distro for k-testing
This is a continuation of previous commit
qa: only use RHEL for workload testing
We don't want to test fs:workload with centos/ubuntu to avoid packaging
issues and to reduce the matrix of distros we're running workloads on.
Also, the testing kernel should install fine on the distros we test with
"supported" random distros.
Patrick Donnelly [Thu, 15 Jul 2021 17:35:41 +0000 (10:35 -0700)]
qa: only use RHEL for workload testing
It's not useful testing workloads with different distributions; it just
adds to the maintenance burden of this qa suite as distro upgrades often
break compilation of various tests.
Nizamudeen A [Tue, 8 Feb 2022 06:20:29 +0000 (11:50 +0530)]
mgr/dashboard: set appropriate baseline branch for applitools
All the dashboard PRs are checked against a baseline branch called
'default' in the visual regresstion testing. This will cause issues when
testing PRs in different branches. For eg: currently our master and
pacific has to save two different screenshots since the two of them
differ slightly.
Disabling the applitools logs as well because its too 'noisy'
Nizamudeen A [Mon, 7 Feb 2022 10:53:29 +0000 (16:23 +0530)]
cephadm: change shared_folder directory for prometheus and grafana
After https://github.com/ceph/ceph/pull/44059 the monitoring/prometheus
and monitoring/grafana/dashboards directories are changed to
monitoring/ceph-mixins. That broke the shared_folders in the cephadm
bootstrap script.
Changed all the instances of monitoring/prometheus and
monitoring/grafana/dashboards to monitoring/ceph-mixins
Also, renaming all the instances of prometheus_alerts.yaml to
prometheus_alerts.yml.
Alfonso Martínez [Fri, 21 Jan 2022 14:57:39 +0000 (15:57 +0100)]
mgr/dashboard: cephadm e2e job: display info on error & other improvements
- Fix: ensure that on_error trap is called (display more info on error).
- Set static IPs to VMs.
- Remove domain in cluster definition to avoid side effects of potential dns misconfiguration.
- Minor improvements.
Fixes: https://tracker.ceph.com/issues/53991 Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 39af61efb24dac6f41ba0752944882d35ad287db)
Build jsonnet and jb in the testso that we can build ceph without
internet access and still be able to run the test needed for monitoring
using jsonnet tools.
spec: debian: monitoring: build jsonnet from source to use 0.18.0
As this new version is recently released it's still not in every distro
we use. We now build jsonnet from source so that we can use this new
version of jsonnet. This commit could be reverted later on when the new
version would be available everywhere.
mgr/dashboard: monitoring: refactor into ceph-mixin
Mixin is a way to bundle dashboards, prometheus rules and alerts into
jsonnet package. Shifting to mixin will allow easier integration with
monitoring automation that some users may use.
This commit moves `/monitoring/grafana/dashboards` and
`/monitoring/prometheus` to `/monitoring/ceph-mixin`. Prometheus alerts
was also converted to Jsonnet using an automated way (from yaml to json
to jsonnet). This commit minimises any change made to the generated files
and should not change neithers the dashboards nor the Prometheus alerts.
In the future some configuration will also be added to jsonnet to add
more functionalities to the dashboards or alerts (i.e.: multi cluster).
Fixes: https://tracker.ceph.com/issues/53374 Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
(cherry picked from commit 98236e3a1d2855c95d86640645c2984efa83791f)
Benoît Knecht [Mon, 3 Jan 2022 15:18:39 +0000 (16:18 +0100)]
monitoring/grafana: Update radosgw dashboards
With the `ceph_daemon` label now replaced by `instance_id` on all `ceph_rgw_*`
metrics, we need to update Grafana dashboards get that label back from
`ceph_rgw_metadata` using this type of construct:
```
ceph_rgw_req * on (instance_id) group_left(ceph_daemon) ceph_rgw_metadata
```
Benoît Knecht [Fri, 12 Nov 2021 14:10:52 +0000 (15:10 +0100)]
pybind/mgr/prometheus: Add instance_id metadata for rgw
In order to get the `ceph_daemon` label for `rgw` metrics corresponding to the
value before #40220, we need to add the `instance_id` label to the
`ceph_rgw_metadata` metric.
This way, the old `ceph_daemon` label can be added to any `ceph_rgw_*` metric
using the following PromQL query, for instance:
```
ceph_rgw_req * on (instance_id) group_left(ceph_daemon) ceph_rgw_metadata
```
Benoît Knecht [Fri, 12 Nov 2021 14:07:35 +0000 (15:07 +0100)]
pybind/mgr/mgr_module.py: Set instance_id label for rgw
Now that the RadosGW returns its instance ID instead of its daemon name,
replace the `ceph_daemon` label with an `instance_id` label on the `rgw`
metrics.
Benoît Knecht [Thu, 28 Oct 2021 16:49:07 +0000 (18:49 +0200)]
mgr/ActivePyModules: Add metadata id in dump_server()
The `DaemonStateCollection` used to always contain the daemon name in its
`DaemonKey`, but since #40220 (or more specifically afc33758e076761b8d4ec004c8f9c49b80a48770), the RadosGW registers with its
instance ID instead (`rados.get_instance_id()`).
As a result, the `ceph_rgw_*` metrics returned by `ceph-mgr` through the
`prometheus` module have their `ceph_daemon` label include that ID instead of
the daemon name, e.g.
Adam Kupczyk [Wed, 2 Feb 2022 19:28:14 +0000 (20:28 +0100)]
os/bluestore/bluefs: Make volume selector operations atomic
Make all RocksDBBlueFSVolumeSelector files/extents/size tracking atomic.
It used to be synchronized by BlueFS global lock.
Now, in Fine Grain Locking era, it is necessary to prevent corruption.
Fixes: https://tracker.ceph.com/issues/53906 Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit 372bda350966624d5081635e659f7c46947980c2)
Adam Kupczyk [Thu, 20 Jan 2022 12:44:35 +0000 (13:44 +0100)]
os/bluestore/bluefs: Code for volume selector check
Adds ability to verify that volume selector properly tracks disk usage.
Creates options:
- bluefs_check_volume_selector_on_umount
- bluefs_check_volume_selector_often
that can be used to validate that vselector does not diverge from
values it should have.
Rishabh Dave [Mon, 7 Feb 2022 18:44:42 +0000 (00:14 +0530)]
monitoring: mention PyYAML only once in requirements
Following error occurs while running "sudo install-deps.sh" -
ERROR: Double requirement given: PyYAML==6.0 (from -r requirements-lint.txt (line 5)) (already in pyyaml (from -r requirements-alerts.txt (line 1)), name='PyYAML')
PyYAML is mentioned twice as a requirement. It is mentioned once in both
the following files -
monitoring/ceph-mixin/requirements-lint.txt
monitoring/ceph-mixin/requirements-alerts.txt
os/BlueStore: NCB fixes recovery code with shared blobs
Replaces the BitmapAllocator used by NCB Recovery code with a dedicated SimpleBitmap.
The SimpleBitmap allows for bits to be set multiple times without any adverse effect.
This is needed beacuse shared-blobs will report the same allocation multiple times.
Fixes: https://tracker.ceph.com/issues/53678 Signed-off-by: Gabriel Benhanokh <gbenhano@redhat.com>
(cherry picked from commit 8868894491c5e4df6d77fb78ed22702a493fe4f8)
Ilya Dryomov [Mon, 31 Jan 2022 13:08:26 +0000 (14:08 +0100)]
qa/suites/krbd: add legacy+rxbounce and crc+rxbounce coverage
For basic, rbd and rbd-nomount subsuites, replace legacy and crc
facets with "legacy or legacy+rxbounce" and "crc or crc+rxbounce"
facets (chosen at random).
For fsx, singleton and thrash subsuites, add legacy+rxbounce and
crc+rxbounce facets and drop prefer-crc facet. The expected behaviour
of the latter depends on cluster configuration and should be tested
separately.
Ilya Dryomov [Sat, 29 Jan 2022 14:01:27 +0000 (15:01 +0100)]
qa/suites/rbd: add cram-based mon command API test
With mon (rbd_support mgr module in this case) command definitions
generated automatically by @CLI{Read,Write}Command decorator, it's
very easy to accidentally break the external facing API.
Ilya Dryomov [Sat, 29 Jan 2022 14:01:27 +0000 (15:01 +0100)]
mgr/rbd_support: level_spec is optional for schedule list/status
Commit fea6fdff4c74 ("mgr/rbd_support: level_spec passed to some
commands is not optional") is wrong. While it is true that a valid
level_spec is needed to create a LevelSpec instance, an empty string
is very much a valid level spec -- it signifies "all levels".
This wasn't caught because within Ceph these commands are wrapped by
rbd CLI which injects an empty string in get_level_spec_args().
Ilya Dryomov [Fri, 28 Jan 2022 22:01:08 +0000 (23:01 +0100)]
mgr/rbd_support: "trash remove" takes image_id_spec, not image_spec
Because of @CLIWriteCommand, the parameter name has to adhere to
the mon command API. Commit dcb51b067a49 ("mgr/rbd_support: define
commands using CLICommand") accidentally changed image_id_spec to
image_spec, breaking external users such as go-ceph.
J. Eric Ivancich [Wed, 12 Jan 2022 18:41:42 +0000 (13:41 -0500)]
rgw: fix bucket index list minor calculation bug
When "bucket index list" traverses the different regions in the bucket
index assembling the output, it miscalculates how many entries to ask
for at one point. This fixes that.
This fixes previous "rgw: bucket index list can produce I/O errors".
Credit for finding this bug goes to Soumya Koduri <skoduri@redhat.com>.
Patrick Donnelly [Mon, 31 Jan 2022 22:58:55 +0000 (17:58 -0500)]
mds: add inline feature to MDS bootstrap incompat
File systems that had inline data enabled at some point would have this
bit in the CompatSet "incompat" set. This would conflict during upgrade
with the default v16.2.4 CompatSet assigned to existing (16.2.4-) MDS.
Subsequently, this would cause an assertion in FSMap::sanity during
pending map creation.
This bit will get added anyway during the upgrade process so might as
well add it to the MDS CompatSet during bootstrap.
Fixes: https://tracker.ceph.com/issues/54081 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d341c5b7734b45568cd986f05d41e91e4cf4a4f7)
projects / ceph.git / log