John Mulligan [Mon, 26 Jun 2023 16:08:34 +0000 (12:08 -0400)]
docs/cephadm: document new extra_{container,entrypoint}_args behavior
Document the new behaviors of extra_container_args and
extra_entrypoint_args. Document that current (previous in code terms)
behavior of splitting strings on spaces and document the recently added
support for ArgumentSpec objects and how they are used.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Mon, 29 May 2023 20:52:31 +0000 (16:52 -0400)]
mgr/cephadm: rework how _process_ls_output builds a DaemonDescription
Previously, the _process_ls_output function would create an
DaemonDescription and then assign values to attributes one-by-one. This
prevents the class from doing any data type/value conversions unless we
resort to more complex approaches like descriptors or `__setattr__` for
example. Update the _process_ls_output function to build a
DaemonDescription using keyword args so that we have the opportunity
later to convert data passed to it in `__init__`.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 17 May 2023 17:50:19 +0000 (13:50 -0400)]
python-common: replace types for extra_*_args with ArgSpec based types
Use the ArgSpec, ArgumentList, and GeneralArgList types throughout
the deployment module to support extra_container_args and
extra_entrypoint_args. The GeneralArgList type supports all possible
input forms while ArgumentList reflects the internal representation
of the argument lists.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 17 May 2023 17:49:23 +0000 (13:49 -0400)]
python-common: add an ArgumentSpec type for handling args
Add a new ArgumentSpec type to the deployment package. The
ArgumentSpec serves a dual purpose:
* to allow more advanced configuration of extra arguments
* to keep backwards compatibility with string-based args
The previous versions of cephadm supported supplying extra container and
entrypoint arguments for fine-tuning of services and creating custom
containers. However, this mode assumed that spaces in an argument
always meant that the argument should be split into two parts:
"--foo bar" becomes `["--foo", "bar"]`. In some cases there's a good
reason to keep spaces as in "--title=My Little Cluster". When
an argument is expressed as a single string the ArgumentSpec is
designed to retain the existing behavior. When an argument is
expressed as a JSON object then you can explicitly express if
you want the argument split on spaces or not (not split is the default).
The alternative was to keep using strings but add some level of
shell-style quoting. This was discussed but deemed complex and
difficult to read in YAML. Round tripping that data is also challenging.
The JSON object approach also allows for future fields to be added
providing for possible extensibility.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 25 May 2023 23:17:02 +0000 (19:17 -0400)]
mgr/orchestrator: reformat ceph.deployment.service_spec import line
Quoting https://peps.python.org/pep-0008/#maximum-line-length
```
The preferred way of wrapping long lines is by using Python’s
implied line continuation inside parentheses, brackets and braces. Long
lines can be broken over multiple lines by wrapping expressions in
parentheses. These should be used in preference to using a backslash for
line continuation.
```
I also think it is much more readable and is similar to the style
applied by automated code formatting tools like `black`.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Thu, 25 May 2023 22:54:05 +0000 (18:54 -0400)]
mgr/cephadm: reformat ceph.deployment.service_spec import line
Quoting https://peps.python.org/pep-0008/#maximum-line-length
```
The preferred way of wrapping long lines is by using Python’s
implied line continuation inside parentheses, brackets and braces. Long
lines can be broken over multiple lines by wrapping expressions in
parentheses. These should be used in preference to using a backslash for
line continuation.
```
I also think it is much more readable and is similar to the style
applied by automated code formatting tools like `black`.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Fri, 26 May 2023 15:40:51 +0000 (11:40 -0400)]
python-common: reformat ceph.deployment.service_spec import line
Quoting https://peps.python.org/pep-0008/#maximum-line-length
```
The preferred way of wrapping long lines is by using Python’s
implied line continuation inside parentheses, brackets and braces. Long
lines can be broken over multiple lines by wrapping expressions in
parentheses. These should be used in preference to using a backslash for
line continuation.
```
I also think it is much more readable and is similar to the style
applied by automated code formatting tools like `black`.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
John Mulligan [Wed, 17 May 2023 18:30:18 +0000 (14:30 -0400)]
python-common: add tests for service spec extra args handling
There were no existing tests for how the service spec(s) handled the
extra_container_args and extra_entrypoint_args values. Add a short
parametrized test function to assert some basic properties of how the
arguments are currently handled. In particular, it asserts that the
values can appear at the top-level of the YAML and under spec.
This is in preparation for adding a more sophisticated argument
type in the future.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
cephfs-journal-tool: disambiguate usage of all keyword (in tool help).
The fs:all for rank option description was confusing. It seemd
like the fs was optional, but it is mandatory. This change modifies the
help message to reflect the correct way to use all in the --rank option.
Fixes: https://tracker.ceph.com/issues/61753 Signed-off-by: Manish M Yathnalli <myathnal@redhat.com>
Note to the documentation team: This is not a line-edit. This commit
includes nothing but the removal of pipes added to the left of much of
the text in this file. Several future commits will line-edit this file
and correct its formatting.
Xinyu Huang [Fri, 9 Jun 2023 07:25:48 +0000 (15:25 +0800)]
crimson/os/seastore: fix bug in check_node
EXIST_CLEAN and EXIST_MUTATION_PENDING shuold not be treated as
CLEAN in check_node because they are transaction private and the
leafnode has been duplicated for write.
Venky Shankar [Fri, 23 Jun 2023 05:10:51 +0000 (10:40 +0530)]
Merge PR #51974 into main
* refs/pull/51974/head:
doc: fix grammar in cephfs/standby
Reviewed-by: Anthony D Atri <anthony.datri@gmail.com> Reviewed-by: Jos Collin <jcollin@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Laura Flores [Thu, 22 Jun 2023 22:09:33 +0000 (17:09 -0500)]
doc/rados/operations: change file back to original name
The name of this file was changed in 4fcab2e7fc9f7ac170ede21cb07912b79926ccb9,
but on second thought, this could cause 404 situations.
Reverting the file name back to the original name.
Casey Bodley [Thu, 22 Jun 2023 12:47:06 +0000 (08:47 -0400)]
qa/s3tests: make extra_attrs additive
the s3tests.py task is filtering out several attrs by default. but
when dbstore uses `extra_attrs` to add 'not fails_on_dbstore', it
overwrites those other filters
John Mulligan [Mon, 19 Jun 2023 16:54:24 +0000 (12:54 -0400)]
cephadm: dont set ctx.image if json deploy image is unset
If no image has been provided in the "deploy from" json do not set
ctx.image to it (empty-string or None) as we may have had a valid
value passed on the --image CLI option.
Signed-off-by: John Mulligan <jmulligan@redhat.com>
Yuval Lifshitz [Fri, 16 Jun 2023 15:10:19 +0000 (15:10 +0000)]
rgw/amqp: remove possible race conditions with the amqp connections
* simplify memory management of the connection by not using a unique_ptr
* simplify the logic by handling all issues inside the amqp manager
* fix iterator invalidation issue with miltiple n/acks
* allow different connections with different exchanges
* modify the unit tests according to the new behavior
Ilya Dryomov [Fri, 16 Jun 2023 12:01:52 +0000 (14:01 +0200)]
qa/workunits/rbd: make continuous export-diff test actually work
The current version is pretty useless:
- "rbd bench" writes the same byte (0xff) over and over again, so
almost all checksumming is in vain
- snapshots are taken in a steady state (i.e. not under I/O), so no
race conditions can get exposed
- even with these caveats, it's not wired up into the suite
Redo this workunit to be a reliable reproducer for the issue fixed
in the previous commit and wire it up for both krbd and rbd-nbd.
Venky Shankar [Tue, 20 Jun 2023 10:04:30 +0000 (15:34 +0530)]
Merge PR #51500 into main
* refs/pull/51500/head:
test/libcephfs: add test case for revoking caps
client: issue a cap release immediately if no cap exists
mds: add the revoking caps back to _revokes list
mds: move confirm_receipt() to Capability.cc
Jos Collin [Mon, 22 May 2023 04:31:39 +0000 (10:01 +0530)]
mds: display sane hex value (0x0) for empty feature bit
Print a valid hex (0x0) during empty feature bit, so that the clients could recognize it.
When the _vec size becomes 0, print() function creates an invalid hex (0x) and 'perf stats'
crashes with the below error:
"
File "/opt/ceph/src/pybind/mgr/stats/fs/perf_stats.py", line 177, in notify_cmd
metric_features = int(metadata[CLIENT_METADATA_KEY]["metric_spec"]["metric_flags"]["feature_bits"], 16)
ValueError: invalid literal for int() with base 16: '0x'
"
This patch creates a valid hex (0x0), when _vec size is 0.
Fixes: https://tracker.ceph.com/issues/59551 Signed-off-by: Jos Collin <jcollin@redhat.com>
Ilya Dryomov [Tue, 13 Jun 2023 11:36:02 +0000 (13:36 +0200)]
librbd: stop passing IOContext to image dispatch write methods
This is a major footgun since any value passed e.g. at the API layer
may be stale by the time we get to object dispatch. All callers are
passing the IOContext returned by get_data_io_context() for their
ImageCtx anyway, highlighting that the parameter is fictitious.
Only the read method can meaningfully take IOContext.
Ilya Dryomov [Mon, 12 Jun 2023 19:45:03 +0000 (21:45 +0200)]
librbd: use an up-to-date snap context when owning the exclusive lock
By effectively moving capturing of the snap context to the API layer,
commit 1d0a3b17f590 ("librbd: pass IOContext to image-extent IO
dispatch methods") introduced a nasty regression. The snap context can
be captured only after exclusive lock is safely held for the duration
of dealing with the image request and even then must be refreshed if
a snapshot creation request is accepted from a peer. This is needed to
ensure correctness of the object map in general and fast-diff states in
particular (OBJECT_EXISTS vs OBJECT_EXISTS_CLEAN) and object deltas
computed based off of them. Otherwise the object map that is forked
for the snapshot isn't guaranteed to accurately reflect the contents of
the snapshot when the snapshot is taken under I/O (as in disabling the
object map may lead to different results being returned for reads).
The regression affects mainly differential backup and snapshot-based
mirroring use cases with object-map and/or fast-diff enabled: since
some object deltas may be incomplete, the destination image may get
corrupted.
This commit represents a reasonable minimal fix: IOContext passed
through to ImageDispatch is effected only for reads and just gets
ignored for writes. The next commit cleans up further by undoing the
passing of IOContext through the image dispatch layers for writes.
projects / ceph.git / log