Sage Weil [Thu, 12 Mar 2020 03:39:32 +0000 (22:39 -0500)]
Merge PR #33817 into octopus
* refs/pull/33817/head:
mgr/dashboard: Adapt tests to new DriveGroupSpec
fixup mgr/test_orchestrator: validate drive group matches anything.
mgr/orch: CLI: No Tracebacks for ServiceSpecValidationError
mgr/test_orchestrator: validate drive group matches anything.
python-common: don't run flake8 on tests.
python-common: Add support for legacy serialization format for Drive Groups
doc: Move Move ServiceSpec to python-common
python-common: Add `host_pattern` to `PlacementSpec.from_string()`
cephadm: add host_pattern to supported scheduling
python-common: Joined ServiceSpec and DriveGroupSpec from_json()
python-common: Make DriveGroupSpec a sub type of ServiceSpec
pybind/mgr: Move ServiceSpec to python-common: Fix imports
python-common, orch: Move ServiceSpec to python-common: Fix imports
python-common, orch: Move ServiceSpec tests to python-common
python-common: Move ServiceSpec to python-common: fix linting
python-common, orch: Move ServiceSpec (+deps) to python-common
Sage Weil [Fri, 6 Mar 2020 23:43:33 +0000 (17:43 -0600)]
cephadm: update unit.* atomically
Some of these are run as bash scripts, which means that updating them
can lead to the running bash picking up at a weird position mid-script
when it goes to the next command. This produces weird errors like
bash[9321]: /var/lib/ceph/f1758250-639e-11ea-9a42-001a4aab830c/mon.c/unit.run: line 2: -to-stderr=true: command not found
Ernesto Puerta [Wed, 4 Mar 2020 20:31:50 +0000 (21:31 +0100)]
mgr/dashboard: add feature toggle for NFS
- New NFS feature toggle added.
- Fixed regression which broke FeatureToggles in the main menu.
- Added extensive unit testing to NavigationComponent
- Added ng-mocks to improve test isolation and performance (~7x, 139s to
20s)
- Added ng-bullet package to improve unit testing performance (2x, 20s
to 9s)
- Added Rxjs Schedulers to implement NgZone.runOutsideAngular in a Rxjs
friendly way (based on
https://stackoverflow.com/questions/43121400/run-ngrx-effect-outside-of-angulars-zone-to-prevent-timeout-in-protractor)
Minor issues found from exhaustive unit testing:
- Missing permissions in Cluster menu:
- `permissions.log.read` and `permissions.prometheus.read`
- Missing classes:
- Block -> Images: `tc_submenuitem_block_images`
- Block -> iSCSI: `tc_submenuitem_block_iscsi`
- Typos:
- class `tc_submenuitem_hosts` assigned to OSD menu (instead of
`tc_submenuitem_osd`)
- `tc_menuitem_cephs` -> `tc_menuitem_cephfs`
Minor changes:
- Previously, Cluster Map -> CRUSH Map required both OSD and Host
permissions. Now it only requires OSD permissions (there are no
references to hosts in CRUSH Map page). Nevertheless, all system roles
setting OSD permission also set Host's, so no impact expected.
- Previously, Cluster -> Monitoring menu was hidden when both Prometheus
or Alertmanager weren't configured. Now it's displayed and when clicked
on it shows the helper banner indicating that further configuration is
required. This change removes the dependency on the PrometheusService.
Sage Weil [Wed, 11 Mar 2020 13:55:51 +0000 (08:55 -0500)]
Merge PR #33830 into octopus
* refs/pull/33830/head:
qa/tasks/cephadm: no default mon|mgr|crash service specs
qa/suites/rados/cephadm/upgrade: upgrade start point that supports the no-spec option
cephadm: create initial mon and mgr service specs too
cephadm: no need to pregenerate a crash key for the bootstrap host
mgr/cephadm: do not complain when we don't have enough hosts
mgr/cephadm: remove orphan daemons
mgr/cephadm: report size=0 for fabricated ServiceDescription
mgr/cephadm: safety check to prevent removing all mon|mgr daemons
mgr/cephadm: prevent scaling mon|mgr below count=1
mgr/cephadm: do not remove daemons from remove_service
Sage Weil [Wed, 11 Mar 2020 12:12:11 +0000 (07:12 -0500)]
Merge PR #33620 into octopus
* refs/pull/33620/head:
mgr/dashboard: Crush rule modal
mgr/dashboard: Preserve rule selection on pool type change
mgr/dashboard: Crush rule is only send during replicated pool creation
mgr/dashboard: Explicit returns in pool form
mgr/dashboard: Removes fork join in pool form
mgr/dashboard: Hide ECP actions during ec pool edit
mgr/dashboard: Pool form erasure/replicated boolean
mgr/dashboard: Change pool info API endpoint
mgr/dashboard: Moves ECP info endpoint to UI-API
Kefu Chai [Wed, 11 Mar 2020 08:08:51 +0000 (16:08 +0800)]
cephadm: add "assert foo is not None" for mypy check
it's legit to pass file objects to fcntl(), but `Popen.stdout` and
`Popen.stderr` properies are not necessarily file objects -- they could be None.
this cannot be deduced at compile-time. even we can ensure this,
as we do pass `subprocess.PIPE` to the constructor. so mypy just
complains at seeing this:
```
cephadm:429: error: Argument 1 to "fcntl" has incompatible type "Optional[IO[Any]]"; expected "Union[int, HasFileno]"
cephadm:430: error: Argument 1 to "fcntl" has incompatible type "Optional[IO[Any]]"; expected "Union[int, HasFileno]"
cephadm:431: error: Argument 1 to "fcntl" has incompatible type "Optional[IO[Any]]"; expected "Union[int, HasFileno]"
cephadm:432: error: Argument 1 to "fcntl" has incompatible type "Optional[IO[Any]]"; expected "Union[int, HasFileno]"
cephadm:455: error: Item "None" of "Optional[IO[Any]]" has no attribute "fileno"
cephadm:465: error: Item "None" of "Optional[IO[Any]]" has no attribute "fileno"
cephadm:475: error: Item "None" of "Optional[IO[Any]]" has no attribute "fileno"
```
to silence this warning, insert `assert process.stdout is not None`
before accessing `process.stdout` to appease the strict optional
checking of mypy.
Sage Weil [Tue, 10 Mar 2020 22:20:48 +0000 (17:20 -0500)]
Merge PR #33771 into octopus
* refs/pull/33771/head:
common/ceph_timer: Pass reference to waited time on stack
common/ceph_timer: Add test
common/ceph_timer: Use unique_function, allowing noncopyable events
common/ceph_timer: Couple cleanups
common/ceph_timer: Fix namespaces
common/ceph_timer: Add missing includes
common/ceph_timer.h: Don't indent contents of a namespace
Sage Weil [Tue, 10 Mar 2020 14:28:57 +0000 (09:28 -0500)]
cephadm: bootstrap: wait for mgr to restart after enabling a module
It was possible to enable a module (mon updates mgrmap) and then
do a mgr command and have that command reach the mgr before it got the
latest mgrmap and restarted.
Fixes: https://tracker.ceph.com/issues/44531 Signed-off-by: Sage Weil <sage@redhat.com>
Sage Weil [Mon, 9 Mar 2020 18:39:04 +0000 (13:39 -0500)]
mgr/cephadm: do not complain when we don't have enough hosts
This gets rid of INFO level log events like
2020-03-09T13:37:20.980993-0500 mgr.x [WRN] Failed to apply mds.foo spec ServiceSpec({'placement': PlacementSpec(count:2), 'service_type': 'mds', 'service_id': 'foo'}): List of host candidates is empty
Sage Weil [Mon, 9 Mar 2020 01:38:59 +0000 (20:38 -0500)]
mgr/cephadm: fix upgrade order
Create two variables, CEPH_TYPES and CEPH_UPGRADE_ORDER. In reality they
are both the same, but this way the meaning is clear, and they lists
won't get out of sync (they should always have the same elements).
Sage Weil [Mon, 9 Mar 2020 17:26:06 +0000 (12:26 -0500)]
ceph.in: only shut down rados on clean exit
If we exit due to a timeout, then calling rados shutdown can lead to all
sorts of problems, because we may still have another thread that is
trying to call rados_connect and/or do some work, and rados_connect
and rados_shutdown don't (and can't!) really behave well when racing
against each other.
Note that shutdown here isn't that important--the process is about to
exit anyway. It's only useful to exercise the shutdown code path more
often.
Fixes: https://tracker.ceph.com/issues/44526 Signed-off-by: Sage Weil <sage@redhat.com>
Adam C. Emerson [Fri, 6 Mar 2020 03:14:47 +0000 (22:14 -0500)]
common/ceph_timer: Pass reference to waited time on stack
std::condition_variable::wait_until takes a const reference to a
time_point. It may access this reference after relinquishing the
mutex, creating a potential use-after-free error if the first event is
shut down.
So, just copy the time onto the stack, so we have a reference that
won't disappear.
https://tracker.ceph.com/issues/44373
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Sage Weil [Mon, 9 Mar 2020 13:28:57 +0000 (08:28 -0500)]
Merge PR #33793 into master
* refs/pull/33793/head:
qa/suites/rados/cephadm/upgrade: new start point
qa/tasks/cephadm: put bootstrap config etc directly in /etc/ceph
cephadm: shell: default to config and keyring in /etc/ceph, if present
Sage Weil [Mon, 9 Mar 2020 13:28:37 +0000 (08:28 -0500)]
Merge PR #33808 into master
* refs/pull/33808/head:
mgr/cephadm: apply: fill in default placement if none is provided
mgr/cephadm: make placement truly optional (default to count=1)
mgr/cephadm: allow count == 0
mgr/cephadm: remove magic labels
Stephan Müller [Tue, 3 Mar 2020 14:39:32 +0000 (15:39 +0100)]
mgr/dashboard: Preserve rule selection on pool type change
Now if the pool type is changed from replicated to erasure in the pool
form and you have multiple rules, your selection is preserved and not
overwritten by null, which caused an error message to be shown
(crush rule is required).
Fixes: https://tracker.ceph.com/issues/44371 Signed-off-by: Stephan Müller <smueller@suse.com>
Stephan Müller [Mon, 2 Mar 2020 11:26:48 +0000 (12:26 +0100)]
mgr/dashboard: Crush rule is only send during replicated pool creation
The problem was that the crush rule setting was send during the creation
of a pool - the type didn't matter, but the setting is only used if a
replicated pool is created. This hasn't created any problems yet, but to
prevent that it's now omitted for erasure code pool creations.
Fixes: https://tracker.ceph.com/issues/44371 Signed-off-by: Stephan Müller <smueller@suse.com>
Stephan Müller [Mon, 2 Mar 2020 10:48:17 +0000 (11:48 +0100)]
mgr/dashboard: Hide ECP actions during ec pool edit
Hides erasure profile actions during erasure pool edit, as all
actions are disabled anyway as they can't be used in edit mode.
This commit makes also sure that the used crush rule will be shown
during edit of an erasure pool and no crush rule selection is shown
during creation of an erasure code pool, as in most cases a new crush
rule will be created for the ec pool.
Fixes: https://tracker.ceph.com/issues/44371 Signed-off-by: Stephan Müller <smueller@suse.com>
Stephan Müller [Mon, 2 Mar 2020 10:39:48 +0000 (11:39 +0100)]
mgr/dashboard: Pool form erasure/replicated boolean
Now a boolean will be set if the pool type is changed. The question
which type is currently set is raised a lot of times inside the pool
form therefor there is a speed advance if it's just a boolean instead of
getting the form control from the form in the first place and than
compare two strings.
Fixes: https://tracker.ceph.com/issues/44371 Signed-off-by: Stephan Müller <smueller@suse.com>
Stephan Müller [Mon, 2 Mar 2020 10:08:23 +0000 (11:08 +0100)]
mgr/dashboard: Change pool info API endpoint
Moves the "_info" endpoint of pool into an equivalent
UI-API call with the name "info".
Added three more attributes to the info dict which enables the dashboard
to only call info to get all the needed data, currently three calls will
be used to do that.
Removed pool_name parameter as the outcome was not used.
Updated the tests and related angular files accordingly.
Fixes: https://tracker.ceph.com/issues/44371 Signed-off-by: Stephan Müller <smueller@suse.com>
Currently, the removal report is updated after entering the main serve()
loop. This tiny window makes clients (like Dashboard) fail to poll the
result. Refresh the report data immediately after scheduling OSD for
removal.
The report structure is changed from Dict to Set because:
- The key is `OSDRemoval` instance, which make it hard to use.
- More consistent with orchestrator interfaces: most calls return a list.
Kefu Chai [Mon, 9 Mar 2020 03:48:07 +0000 (11:48 +0800)]
crimson/mgr: close() in background
as per Yingxin,
application code is not required to wait for the `close()` future, it
would be safe to ignore it, because:
- `close()` will shutdown its socket synchronously;
- `close()` will create an internal `ConnectionRef` when it's closing;
- `Messenger` will wait for all connections closed during `shutdown()`;
Kefu Chai [Sun, 8 Mar 2020 06:00:53 +0000 (14:00 +0800)]
qa/tasks/ceph_manager: use StringIO for capturing COT output
there are couple factors we should consider when choosing between
BytesIO and StringIO:
- if the producer is producing binary
- if we are expecting binary
- if the layers in between them are doing the decoding/encoding
automatically.
in our case, the producer is either the ChannelFile instances returned
by paramiko.SSHClient or subprocess.CompletedProcess insances returned
by subprocess.run(). the former are file-like objects opened in "r" mode,
but their contents are decoded with utf-8 when reading if
ChannelFile.FLAG_BINARY is not specified. that's why we always try to
add this flag in orchestra/run.py when collecting the stdout and stderr
from paramiko.SSHClient after executing a command.
back in python2, this works just fine. as we don't differentiate bytes
from str by then.
but in python3, we have to make a decision. in the case of
ceph-objectstore-tool (COT for short), it does not produce binary and
we don't check its output with binary, so, if neither Remote.run() nor
LocalRemote.run() decodes/encodes for us, it's fine.
so it boils down to `copy_to_log()`:
i think we we should respect the consumer's expectation, and only decode
the output if a StringIO is passed in as stdout or stderr.
as we always log the output with logging we could either set
`ChannelFile.FLAG_BINARY` depending on the type of `capture` or not.
if it's not set, paramiko will return str (bytes) on python2, and str on
python3. if it's not set paramiko will return str (bytes) on python2,
and bytes on python3.
if there is non-ASCII in the output, logging will bail fail with
`UnicodeDecodeError` exception. and paramiko throws the same exception
when trying to decode for us if `ChannelFile.FLAG_BINARY` is not
specified.
so to ensure that we always have logging messages no matter if the
producer follows the rule of "use StringIO if you only emit text" or
not, we have to use `ChannelFile.FLAG_BINARY`, and force paramiko
to send us the bytes. but we still have the luxury to use StringIO
and do the decode when the caller asks for str explicitly. that'd save
the pain of using `str.decode()` or `six.ensure_str()` everywhere
even if we can assure that the program does not write binary.
projects / ceph.git / log