John Mulligan [Mon, 11 Apr 2022 19:32:42 +0000 (15:32 -0400)]
pybind/mgr: add a wrapper exception for use with Responder
In order to best get a "real" exception converted to something
that can be cleanly sent to the mgr response, this new exception
type can be invoked directly, or with the wrap method to automatically
pull as many properties as possibly from the original exception.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 11 Apr 2022 19:16:34 +0000 (15:16 -0400)]
pybind/mgr: add format arg to Responder's extra args
To ensure that the Responder can make use of a user provided `--format=`
parameter even if the programmer doesn't explicitly add one to the
args of an endpoint function we set the `extra_args` attribute on
our wrapper function so that CLICommand can later extract it.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 11 Apr 2022 19:03:12 +0000 (15:03 -0400)]
pybind/mgr: enhance CLICommand to fetch extra args from wrapped funcs
Previously, the CLICommand decorator "assumed" that the decorator was
applied directly to a mgr module api endpoint function. Now that we plan
on adding the Responder decorator into the mix we need a way of
properly fetching the arguments of the endpoint function. In addition,
the decorator itself needs to provide extra arguments to the mgr
(in cases where the endpoint function doesn't explicitly ask for it).
Thus we add a helper function to find the endpoint function when
wrapped as well as extract extra arguments when "walking" the stack
of __wrapped__ functions.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 11 Apr 2022 18:46:37 +0000 (14:46 -0400)]
pybind/mgr: change to private _load_func_metadata classmethod
The load_func_metadata had exactly one use in the codebase, the
store_func_metadata method. It was also a staticmethod that referred to
a property of it's class.
This change makes the function "private" by renaming it to
_load_func_metadata, removing it from the public "surface area" of the
type. It changes it to a classmethod so that it would work correctly
if used from a subclass of CLICommand.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 9 Apr 2022 19:19:37 +0000 (15:19 -0400)]
pybind/mgr: add a Responder decorator type
The Responder is the decorator that future endpoint functions in the mgr can
use to automatically handle conversions of returned types to serilaized
data (JSON, YAML, etc) as well as automatically convert exceptions into
error responses.
The Responder makes use of format and return-value adapter types,
previously added to the module, to convert a returned value into a mgr
response. This change adds some exception types to return error
responses to the clients.
Simple customizations can be done by passing an alternate format adapter
type when the Responder is being constructed. Additional customization
can be done by subclassing the Responder.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 9 Apr 2022 19:13:41 +0000 (15:13 -0400)]
pybind/mgr: add CommonFormatter type and valid_formats method
A type that has a valid_formats method, and thus meets the
CommonFormatter protocol, supports distinguishing between formats
that are known but unsupported for a given API vs. unknown (possibly a typo).
To make working with the format names easier this also makes the Format
enum inherit from str.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 9 Apr 2022 18:46:50 +0000 (14:46 -0400)]
pybind/mgr: add a ReturnValueAdapter type to object_format.py
The ReturnValueAdapter type fulfills a similar role to the
ObjectFormatAdapter but instead of serializing data for the
body of a mgr response, extracts an return value (error code)
to reply with.
Most of the time it is totally unnecessary to provide an explicit
return value because if you have are returning a valid object (as
opposed to raising an exception) the return value will be zero
(success). However, in the off chance an type need to directly
communicate a return value for the mgr response it can provide
the `mgr_return_value` method and the adapater will discover
and use it.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Sat, 9 Apr 2022 18:29:25 +0000 (14:29 -0400)]
pybind/mgr: add ObjectFormatAdapter type to object_format.py
The ObjectFormatAdapter fills the role for bridging between types
that can return a simplified representation of themselves and
actually formatting objects as JSON and YAML.
Note that we do not want generally want types that serialize themselves
to JSON/YAML strings. That approach makes it harder to standardize on
the final output formatting (indentation, multiple yaml docs, etc).
Additionally, we do not want the types to need to specialize between
JSON and YAML. So, by default, we try to use a method `to_simplified`
which is not specific to any serialization format. However, for
backwards compatibility with types that already have methods *that
return dicts/lists/etc* under the names `to_json` or `to_yaml` we
support using the `compatible` flag to enable the use of those methods.
If the adaptor fails to find a conversion method on the object,
serialization of the object itself is attempted - this way return values
of simple lists, dicts, etc also works.
An earlier version of this patch tried to share the JSON/YAML
serialization logic found in src/pybind/mgr/orchestrator/module.py.
However, this approach was deemed too complicated and we also preferred
to use yaml safe dumping whenever possible. This does lead to a level
of code duplication. Dealing with this duplication is a task left for
the future.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 8 Apr 2022 15:15:55 +0000 (11:15 -0400)]
pybind/mgr: reformat quoting in format enum
Whenever possible I use 'black' to reformat the python code.
It's strict and its formatting is superset of what ceph's
formatting tools require. This change updates the code that was
moved into this file so that future uses of 'black' don't
reformat this section too.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 14 Mar 2022 15:29:50 +0000 (11:29 -0400)]
pybind/mgr: start a new object_format.py for general formatting
Currently, there's some auto-formatting logic in the orchestrator
module and a lot of ad-hoc formatting scattered around the mgr modules.
This new module aims to bring some of that together in a central
location.
Start by moving the Format enum from the orchestrator.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Nizamudeen A [Wed, 11 May 2022 10:47:00 +0000 (16:17 +0530)]
mgr/dashboard: avoid overriding the configuration in package.json
the warning in npm run build is `Option "configuration" was already specified with value "en-US". The
new value "production" will override it.`
This causes the dist for production env to be created in dist/ whereas
it should be dist/en-US (or whatever npm_config_locale. That prevented
from dashboard to load and cause the cephadm e2e checks to fail.
cephfs-shell: get cmd must get both path and should validate them
Description:
- While using `get` command, `local_path` parameter is optional. Changing it
to mandatory.
- Rationale: Till now, there used to be a default path of `local_path` as
`default='.'` but wasn't mentioned anywhere. It led to confusion.
On top of it, considering get command to be a ssh inspired utlity,
or any other CLI tool that copies file between filesystems, source
and destination path are always mandatory. Therefore in order to
simulate this behavior in cephfs-shell`s command(s), my opinion is
to make get command accept both the paths.
- Added checks to make sure:
1) File does exist at `remote_path`
2) File with the same name doesn't exist in `local_path`
3) Removed code that would run through the directory and if it finds
nothing in `root_src_dir`, then it will try to do:
`os.makedirs(root_dst_dir + b'/' + root_src_dir)`, but it will
never be empty as 1) takes care of it.
cephfs-shell: get command should never create a directory on its own
Descrption: When using the get command on a single file, it would append the remote path to
source path and create directories that shouldn't be created. For instance,
file 'foo.txt' resides at `/dir1/dir2/` and `get` command is used to copy it to
/tmp/foo then it would do `/tmp/dir1/dir2/foo.txt` which is not the expected
behavior. Therefore this PR intends to correct this behavior.
dparmar18 [Wed, 4 May 2022 11:26:48 +0000 (16:56 +0530)]
qa: removed test_without_target_dir
Rationale: get and put now demand both the paths mandatorily.
Also testing of get and put without target paths
have been take of in other tests in class TestGetAndPut().
cephfs-shell: put cmd must accept both paths and validate local_path
Description:
- `put` command didn't display any error when file at local_path was not
found. This PR intends to add that check.
- Rationale: Till now, there used to be a default path of `remote_path` as
`default='.'` but wasn't mentioned anywhere. It could lead to confusion.
On top of it, considering put command to be a ssh inspired utlity,
or any other CLI tool that copies file between filesystems, source
and destination path are always mandatory. Therefore in order to
simulate this behavior in cephfs-shell`s command(s), my opinion is
to make put command accept both the paths.
Nizamudeen A [Sun, 8 May 2022 14:27:34 +0000 (19:57 +0530)]
mgr/dashboard: smart data for devices with scsi protocol
In the dashboard, we've been showing smart data for hdd devices with ata
protocol only. Otherwise we show a No Smart Data found error which is
clearly misleading since Smart Data is returned even in the api call.
So this PR is trying to show the smart data for hdd devices
that uses scsi protocol too.
Fixes: https://tracker.ceph.com/issues/55574 Signed-off-by: Nizamudeen A <nia@redhat.com>
Rishabh Dave [Thu, 5 May 2022 13:17:08 +0000 (18:47 +0530)]
qa/vstart_runner: minor bug fix and minor adjustments
The message regarding deletion of helper tools is printed for every
command. This message should be printed only when applicable.
Besides -
* Move XXX comments to _do_run() since it increases visibility of
these messages.
* Move omission of arguments stuff to new method clear up the clutter.
* And remove shell as a parameter from _perform_checks_and_adjustments
since it's redundant.
Samuel Just [Sat, 7 May 2022 07:02:58 +0000 (07:02 +0000)]
crimson/common/operation: trim AggregateBlockingEvent::events entries
PglogBasedRecovery and BackfillRecovery reuse the same Operation
until their respective operations are complete. Each recovery
operation adds an entry to AggregateBlockingEvent::events. This
way, we only retain entries that are currently blocking.
Volker Theile [Mon, 9 May 2022 13:31:15 +0000 (15:31 +0200)]
mgr/dashboard: Creating and editing Prometheus AlertManager silences is buggy
When creating a new monitoring silence the form is pre-filled with the wrong alert data. It is always used the alert data from the very first object in the list of the API response but not the specified alert identified by the 'fingerprint' property.
The same problem applies to editing silences. The selected silence is not edited, it's always the first one in the list returned API response but not that with the specified 'id' property.
The main problem of the origin implementation is that the Prometheus Alertmanager API endpoints /api/v1/[alerts/silences] do not support querying. To fix that, filtering is done in the frontend.
dparmar18 [Mon, 2 May 2022 11:03:30 +0000 (16:33 +0530)]
cephfs-shell: make onecmd() print proper error msg
Rationale: Whenever a python exception occurred in cephfs-shell,
it would often only be the exception message but doesn't
say anything about the type of exception. For example if
`ZeroDivisionError: division by zero` occurred, the onecmd()
would print `division by zero` but will omit the type of
exception. In this case it's easy to understand but let's
say an `KeyError` exception occurred for a key `9999` which
is not existent in the dictionary, onecmd() would print
just `9999` in this scenario and it would be very difficult
to interpret what type of error it is.
qa/cephfs/test_cephfs_shell: Fixed pep8 formatting violations
NOTE: Although most of the issues are fixed but a few function
and variable names are unchanged in order to prevent
ambiguity and preserve their meaning.
They are:
- functions: setUp(), test_ls_H_prints_human_readable_file_size(),
- variables: ls_H_output, ls_H_file_size
Nizamudeen A [Fri, 6 May 2022 15:19:18 +0000 (20:49 +0530)]
mgr/dashboard: fix smart data error
the error in the log was this
```
"/usr/share/ceph/mgr/dashboard/services/ceph_service.py", line 253, in _get_smart_data_by_device
May 06 07:38:39 occldlr750-1.occl208.lab conmon[2142938]: svc_type, svc_id = daemon.split('.')
May 06 07:38:39 occldlr750-1.occl208.lab conmon[2142938]: ValueError: too many values to unpack (expected 2)
```
on the cluster, the output of `ceph device ls-by-host` looks like this
the first device is mon and its name is mon.occldlr750-1.occl208.lab.
In our dashboard code, when fetching the smart data we have a line like
this
`svc_type, svc_id = daemon.split('.')`
so for the mon the output of `daemon.split('.') will be ['mon', 'occldlr750-1', 'occl208', 'lab']. The svc_id gets split into three because of the split. I am changing that and giving the criteria as splitting only on the first occurence of the dot and the considering everything that comes after the dot as the svc_id of the device.
Fixes: https://tracker.ceph.com/issues/55571 Signed-off-by: Nizamudeen A <nia@redhat.com>
Nizamudeen A [Thu, 5 May 2022 17:43:38 +0000 (23:13 +0530)]
mgr/dashboard: devices with same UID causes multiselection
In the Physical Disks page, the uids for multiple devices are coming in
as same and that causes the selection to go berserk and select multiple
rows with same UID. The uid is generated in the frontend service call
itself. I just added some more parameters to it inorder to make it more
unique.
The second issue is the number of selected number getting multiplied
exponentially. Its because each time the table is updated or refreshed,
we push the row with the number of selected items we had before and that
causes the number of selection to multiply.
Fixes: https://tracker.ceph.com/issues/55523 Signed-off-by: Nizamudeen A <nia@redhat.com>
projects / ceph-ci.git / log