Kotresh HR [Fri, 5 Feb 2021 18:05:22 +0000 (23:35 +0530)]
qa: Fix a few mgr/volume test cases
Recovering dirty auth metadata file might not retain the order,
fixed the comparison in 'test_recover_auth_metadata_during_authorize'
and 'test_recover_auth_metadata_during_deauthorize'.
Venky Shankar [Tue, 18 Aug 2020 09:32:38 +0000 (05:32 -0400)]
test: optionally create a backup filesystem on startup
Also filter out client-id's starting with "mirror" when
cleaning leftover auth-ids since teuthology would be
configured to create client.mirror and client.mirror_remote
clients before executing mirroring tests.
Ronen Friedman [Thu, 28 Jan 2021 17:04:35 +0000 (19:04 +0200)]
osd: fix the scrubber behavior on multiple preemption attempts
Latest scrub code creates a time window in which a specific scrub
is marked as "preempted", but future preemptions are prohibited.
Write operations handled are then blocked but not restarted on time.
Yuri Weinstein [Wed, 3 Feb 2021 18:14:56 +0000 (10:14 -0800)]
Merge pull request #39165 from sebastian-philipp/pacific-backport-39035-38935-39028-39087-38998-38854-38982-39101-39043-39113-
pacific: cephadm: Batch backport January (2)
Reviewed-by: Jan Fajerski <jfajerski@suse.com> Reviewed-by: Neha Ojha <nojha@redhat.com> Reviewed-by: Laura Paduano <lpaduano@suse.com> Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Jan Fajerski [Tue, 22 Dec 2020 13:29:02 +0000 (14:29 +0100)]
ceph-volume: add some flexibility to bytes_to_extents
For batch we want to report the projected sizes before creating any LVM
structures, we use the device size for that. This means we pass this
projected size to lvm/api.py::create_lv. This could result in situations
where we request more extents for the new LV than are actually
available, as LVM takes up some available space on the block device
(plus alignment).
We fix this by adjusting the extent count downwards if its less than 1% off the
requested value.
Fixes: https://tracker.ceph.com/issues/47758 Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ece6e7eb9a92cb944b2f201a05a5acfb877c2dba)
Neha Ojha [Tue, 2 Feb 2021 17:43:56 +0000 (17:43 +0000)]
qa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests
EnvLibradosMutipoolTest.DBBulkLoadKeysInRandomOrder can overload OSDs and cause
heartbeat timeouts. Tests in test_envlibrados_for_rocksdb also generate slow
requests on OSDs. Use osd_client_message_cap to prevent this.
Since this option is disabled by default, this may be a good way to exercise it.
Kamoltat [Thu, 7 Jan 2021 15:39:19 +0000 (15:39 +0000)]
mgr/pg_autoscaler: avoid scale-down until there is pressure
The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.
Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools
Edited workunit script to reflect the change
of how pgs are calculated and distrubted.
Ilya Dryomov [Sun, 31 Jan 2021 14:24:32 +0000 (15:24 +0100)]
qa/tasks/rbd: don't wait for krbd symlink to be created/removed
Commit 3754c665a11e (":qa/tasks/rbd: test qemu on top of rbd
encryption") broke dev_create() for krbd by messing up the "wait
for the symlink to be created by udev" loop. The rbd tool has been
synchronizing with udev internally since 2014, so rather than fixing
let's just drop it.
Ilya Dryomov [Sun, 31 Jan 2021 13:27:52 +0000 (14:27 +0100)]
qa/tasks/rbd: dev_create() expects a properties dict
In order to add encryption_format property, commit 3754c665a11e
(":qa/tasks/rbd: test qemu on top of rbd encryption") changed the
dev_create() contract. It now expects a properties dict, but the
main task routine wasn't updated and still passes role_images.
On top of that, role_images itself got broken. It is supposed
to be a mapping from role to a corresponding image name, not to
a dict with an image_name key. This affected generic_mount().
Sage Weil [Fri, 29 Jan 2021 20:03:43 +0000 (14:03 -0600)]
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
Normally the git_url is git://git.ceph.com/ceph-ci.git, which mirrors
upstream ceph-ci.git. However, the release branches aren't present there.
Allow a custom git_url so we can pull these from the main ceph.git.
Kefu Chai [Thu, 28 Jan 2021 12:35:36 +0000 (20:35 +0800)]
cmake: install rook-client-python using ExternalProject
so we don't need to rerun the generate_rook_ceph_client.sh script
everytime when building the script. cmake creates a stamp file for
tracking the dependencies and the time of modification of dependencies.
Sage Weil [Wed, 27 Jan 2021 21:44:21 +0000 (15:44 -0600)]
python-common: fix test_datetime_to_str_2 on non-UTC hosts
The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string. Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.
Paul Cuzner [Wed, 27 Jan 2021 20:40:58 +0000 (09:40 +1300)]
cephadm:add missing kernel_security property
The propery decorator had gone missing which meant
the dump of host facts was missing the kernel security
(LSM) settings. This patch just adds the @property
decorator back
Kefu Chai [Sun, 24 Jan 2021 06:58:51 +0000 (14:58 +0800)]
cephadm: refactor call() using asyncio.asyncio.StreamReader
simpler this way, also fix a couple issues:
* create a child watcher explicitly, see
https://bugs.python.org/issue35621
* use StringIO for collecting outputs for better performance,
instead of appending the lines to an existing str
* catch ValueError when reading from the stream reader,
because StreamReader.readline() could raise ValueError when
it reaches the buffer limit while looking for a separator.
in this case, we should try again, in hope that the spawned
process can feed the reader with more data which contains
the separator (i.e., b'\n').
* backport ThreadedChildWatcher from Python 3.8 so we can
run create_subprocess_exec() in non-main threads.
iSCSI daemons need to execute <ceph service status> command
Fixes: https://tracker.ceph.com/issues/48925 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit 9b9934f75b648a9ee01848710a6f7150a371e26e)
Adam King [Mon, 11 Jan 2021 18:23:04 +0000 (13:23 -0500)]
mgr/cephadm: force flag for ok-to-stop and ok-to-stop for monitoring stack
Daemons that could cause data loss when stopped will always block.
Daemons that will only cause loss in availability should block but have
a workaround in the form of a force flag if the user is okay with the service
being down.
Also implements ok-to-stop for monitoring stack daemons that uses this system
of blocking on availability loss unless force flag is provided
Signed-off-by: Adam King <adking@redhat.com> Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
(cherry picked from commit aeeffb07f1bb226b64a362de4eecd7d63fc0b0d1)
The current bionice version triggers a podman/conmon bug that
truncates output, affecting both cephadm bootstrap when 'mgr dump' is
large, and teuthology 'pg dump' when it is large.
Kefu Chai [Tue, 19 Jan 2021 17:56:02 +0000 (01:56 +0800)]
pybind/mgr/dashboard: use _get_localized_key for composing store key
Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)
As suggested by Ernesto Puerta <epuertat@redhat.com>
Kefu Chai [Tue, 19 Jan 2021 17:24:01 +0000 (01:24 +0800)]
pybind/mgr/cephadm/inventory: cast variable to expected type
to silence mypy warnings like:
cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
Kefu Chai [Tue, 19 Jan 2021 06:49:07 +0000 (14:49 +0800)]
pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()
it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.
Kefu Chai [Sun, 17 Jan 2021 08:36:35 +0000 (16:36 +0800)]
mgr/cephadm: use dict instead of defaultdict to silence mypy warning
otherwise we have:
1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"
Kefu Chai [Sun, 17 Jan 2021 07:29:33 +0000 (15:29 +0800)]
mgr/progress: pass id to constructor of Event
to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:
progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"
Kefu Chai [Sun, 17 Jan 2021 01:35:36 +0000 (09:35 +0800)]
pybind/mgr/iostat: use correct types
for better readability and for the sake of correctness.
* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str
Michael Wodniok [Tue, 19 Jan 2021 12:15:32 +0000 (13:15 +0100)]
doc: changed commands as suggested by @sebastian-philipp
Signed-off-by: Michael Wodniok <wodniok@wor.net> Co-authored-by: Sebastian Wagner <sebastian@spawnhost.de>
(cherry picked from commit d97731969ab384909e64afd60d766e5cb06e4b5e)
projects / ceph.git / log