Sage Weil [Tue, 26 Jan 2021 22:10:13 +0000 (16:10 -0600)]
mgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades
For octopus -> pacific, as with other recent releases, we need to scale
down the MDS cluster(s) to a single daemon before upgrading. (This is
because the MDS intra-cluster protocols aren't fully versioned.)
Sage Weil [Wed, 27 Jan 2021 14:54:00 +0000 (08:54 -0600)]
mgr/cephadm/upgrade: match against any repo_digest, not image_id
The image id can vary across hosts and (most notably) docker vs podman.
Instead, use the repo_digest as an image identifier.
Unfortunately, a single image may have multiple digests, even within the
same registry, so keep a list of the digests for the image we are
upgrading to, and ensure that each container has a digest that matches at
least one of them.
This allows upgrade to proceed in mixed docker+podman clusters. However,
it does not yet address a cluster with mixed CPU architectures, because
the container image will have different digest(s) for each architecture
build.
Paul Cuzner [Thu, 14 Jan 2021 22:08:48 +0000 (11:08 +1300)]
cephadm: install doc updated to include cluster-network parameter
Install guide updated to include a description of the --cluster-network
parameter. The text also links to the complete definition for cluster-network
on the rados/configuration/network-config-ref page.
Venky Shankar [Tue, 18 Aug 2020 09:32:38 +0000 (05:32 -0400)]
test: optionally create a backup filesystem on startup
Also filter out client-id's starting with "mirror" when
cleaning leftover auth-ids since teuthology would be
configured to create client.mirror and client.mirror_remote
clients before executing mirroring tests.
Ronen Friedman [Thu, 28 Jan 2021 17:04:35 +0000 (19:04 +0200)]
osd: fix the scrubber behavior on multiple preemption attempts
Latest scrub code creates a time window in which a specific scrub
is marked as "preempted", but future preemptions are prohibited.
Write operations handled are then blocked but not restarted on time.
Yuri Weinstein [Wed, 3 Feb 2021 18:14:56 +0000 (10:14 -0800)]
Merge pull request #39165 from sebastian-philipp/pacific-backport-39035-38935-39028-39087-38998-38854-38982-39101-39043-39113-
pacific: cephadm: Batch backport January (2)
Reviewed-by: Jan Fajerski <jfajerski@suse.com> Reviewed-by: Neha Ojha <nojha@redhat.com> Reviewed-by: Laura Paduano <lpaduano@suse.com> Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Neha Ojha [Tue, 2 Feb 2021 17:43:56 +0000 (17:43 +0000)]
qa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests
EnvLibradosMutipoolTest.DBBulkLoadKeysInRandomOrder can overload OSDs and cause
heartbeat timeouts. Tests in test_envlibrados_for_rocksdb also generate slow
requests on OSDs. Use osd_client_message_cap to prevent this.
Since this option is disabled by default, this may be a good way to exercise it.
Kamoltat [Thu, 7 Jan 2021 15:39:19 +0000 (15:39 +0000)]
mgr/pg_autoscaler: avoid scale-down until there is pressure
The autoscaler will start out with scaling each
pools to have a full complements of pgs from the start
and will only decrease it when pools need more due to
increased usage.
Introduced a unit test that tests only the
function get_final_pg_target_and_ratio() which
deals with the distrubtion of pgs amongst the
pools
Edited workunit script to reflect the change
of how pgs are calculated and distrubted.
Ilya Dryomov [Sun, 31 Jan 2021 14:24:32 +0000 (15:24 +0100)]
qa/tasks/rbd: don't wait for krbd symlink to be created/removed
Commit 3754c665a11e (":qa/tasks/rbd: test qemu on top of rbd
encryption") broke dev_create() for krbd by messing up the "wait
for the symlink to be created by udev" loop. The rbd tool has been
synchronizing with udev internally since 2014, so rather than fixing
let's just drop it.
Ilya Dryomov [Sun, 31 Jan 2021 13:27:52 +0000 (14:27 +0100)]
qa/tasks/rbd: dev_create() expects a properties dict
In order to add encryption_format property, commit 3754c665a11e
(":qa/tasks/rbd: test qemu on top of rbd encryption") changed the
dev_create() contract. It now expects a properties dict, but the
main task routine wasn't updated and still passes role_images.
On top of that, role_images itself got broken. It is supposed
to be a mapping from role to a corresponding image name, not to
a dict with an image_name key. This affected generic_mount().
Sage Weil [Fri, 29 Jan 2021 20:03:43 +0000 (14:03 -0600)]
qa/tasks/cephadm: allow custom git_url for cephadm_branch pull
Normally the git_url is git://git.ceph.com/ceph-ci.git, which mirrors
upstream ceph-ci.git. However, the release branches aren't present there.
Allow a custom git_url so we can pull these from the main ceph.git.
Kefu Chai [Thu, 28 Jan 2021 12:35:36 +0000 (20:35 +0800)]
cmake: install rook-client-python using ExternalProject
so we don't need to rerun the generate_rook_ceph_client.sh script
everytime when building the script. cmake creates a stamp file for
tracking the dependencies and the time of modification of dependencies.
Sage Weil [Wed, 27 Jan 2021 21:44:21 +0000 (15:44 -0600)]
python-common: fix test_datetime_to_str_2 on non-UTC hosts
The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string. Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.
Paul Cuzner [Wed, 27 Jan 2021 20:40:58 +0000 (09:40 +1300)]
cephadm:add missing kernel_security property
The propery decorator had gone missing which meant
the dump of host facts was missing the kernel security
(LSM) settings. This patch just adds the @property
decorator back
Kefu Chai [Sun, 24 Jan 2021 06:58:51 +0000 (14:58 +0800)]
cephadm: refactor call() using asyncio.asyncio.StreamReader
simpler this way, also fix a couple issues:
* create a child watcher explicitly, see
https://bugs.python.org/issue35621
* use StringIO for collecting outputs for better performance,
instead of appending the lines to an existing str
* catch ValueError when reading from the stream reader,
because StreamReader.readline() could raise ValueError when
it reaches the buffer limit while looking for a separator.
in this case, we should try again, in hope that the spawned
process can feed the reader with more data which contains
the separator (i.e., b'\n').
* backport ThreadedChildWatcher from Python 3.8 so we can
run create_subprocess_exec() in non-main threads.
iSCSI daemons need to execute <ceph service status> command
Fixes: https://tracker.ceph.com/issues/48925 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit 9b9934f75b648a9ee01848710a6f7150a371e26e)
Adam King [Mon, 11 Jan 2021 18:23:04 +0000 (13:23 -0500)]
mgr/cephadm: force flag for ok-to-stop and ok-to-stop for monitoring stack
Daemons that could cause data loss when stopped will always block.
Daemons that will only cause loss in availability should block but have
a workaround in the form of a force flag if the user is okay with the service
being down.
Also implements ok-to-stop for monitoring stack daemons that uses this system
of blocking on availability loss unless force flag is provided
Signed-off-by: Adam King <adking@redhat.com> Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
(cherry picked from commit aeeffb07f1bb226b64a362de4eecd7d63fc0b0d1)
The current bionice version triggers a podman/conmon bug that
truncates output, affecting both cephadm bootstrap when 'mgr dump' is
large, and teuthology 'pg dump' when it is large.
Kefu Chai [Tue, 19 Jan 2021 17:56:02 +0000 (01:56 +0800)]
pybind/mgr/dashboard: use _get_localized_key for composing store key
Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)
As suggested by Ernesto Puerta <epuertat@redhat.com>
Kefu Chai [Tue, 19 Jan 2021 17:24:01 +0000 (01:24 +0800)]
pybind/mgr/cephadm/inventory: cast variable to expected type
to silence mypy warnings like:
cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
Kefu Chai [Tue, 19 Jan 2021 06:49:07 +0000 (14:49 +0800)]
pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()
it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.
projects / ceph.git / log