Ilya Dryomov [Mon, 8 Feb 2021 16:01:47 +0000 (17:01 +0100)]
librbd: don't hold owner_lock for validate_image_removal()
handle_exclusive_lock() and handle_shut_down_exclusive_lock() call
validate_image_removal() without owner_lock held, so holding it in
shut_down_exclusive_lock() appears to be redundant.
Ilya Dryomov [Sun, 7 Feb 2021 14:09:24 +0000 (15:09 +0100)]
librbd: treat EROFS as expected in handle_acquire_lock()
If the peer refuses to release exclusive lock (e.g. in case automatic
exclusive lock transitions are disabled), EROFS is retured. Suppress
a rather confusing "Read-only file system" error message -- this case
is no different from EBUSY or EAGAIN.
Ilya Dryomov [Sun, 7 Feb 2021 12:46:15 +0000 (13:46 +0100)]
librbd: refuse to release exclusive lock when removing
Commit 25c2ffe145be ("librbd: acquire exclusive lock from peer when
removing") changed PreRemoveRequest to request exclusive lock from the
peer instead of giving up and proceeding without exclusive lock. This
caused one of the test cases that sometimes runs concurrent "rbd rm"
against the same image to fail intermittently, most often on assert
because exclusive lock is now automatically transitioned to another
"rbd rm" on its request.
The root cause is older and probably goes back to when synchronous
librbd::remove() which held owner_lock across all operations including
trim_image() was converted to a set of state machines. Since then, any
peer that requests exclusive lock (instead of trying once and backing
off) is able to mess with image removal.
Install StandardPolicy to disable automatic exclusive lock transitions
during image removal.
Yuri Weinstein [Wed, 3 Feb 2021 18:14:56 +0000 (10:14 -0800)]
Merge pull request #39165 from sebastian-philipp/pacific-backport-39035-38935-39028-39087-38998-38854-38982-39101-39043-39113-
pacific: cephadm: Batch backport January (2)
Reviewed-by: Jan Fajerski <jfajerski@suse.com> Reviewed-by: Neha Ojha <nojha@redhat.com> Reviewed-by: Laura Paduano <lpaduano@suse.com> Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Ilya Dryomov [Sun, 31 Jan 2021 14:24:32 +0000 (15:24 +0100)]
qa/tasks/rbd: don't wait for krbd symlink to be created/removed
Commit 3754c665a11e (":qa/tasks/rbd: test qemu on top of rbd
encryption") broke dev_create() for krbd by messing up the "wait
for the symlink to be created by udev" loop. The rbd tool has been
synchronizing with udev internally since 2014, so rather than fixing
let's just drop it.
Ilya Dryomov [Sun, 31 Jan 2021 13:27:52 +0000 (14:27 +0100)]
qa/tasks/rbd: dev_create() expects a properties dict
In order to add encryption_format property, commit 3754c665a11e
(":qa/tasks/rbd: test qemu on top of rbd encryption") changed the
dev_create() contract. It now expects a properties dict, but the
main task routine wasn't updated and still passes role_images.
On top of that, role_images itself got broken. It is supposed
to be a mapping from role to a corresponding image name, not to
a dict with an image_name key. This affected generic_mount().
Kefu Chai [Thu, 28 Jan 2021 12:35:36 +0000 (20:35 +0800)]
cmake: install rook-client-python using ExternalProject
so we don't need to rerun the generate_rook_ceph_client.sh script
everytime when building the script. cmake creates a stamp file for
tracking the dependencies and the time of modification of dependencies.
Sage Weil [Wed, 27 Jan 2021 21:44:21 +0000 (15:44 -0600)]
python-common: fix test_datetime_to_str_2 on non-UTC hosts
The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string. Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.
Paul Cuzner [Wed, 27 Jan 2021 20:40:58 +0000 (09:40 +1300)]
cephadm:add missing kernel_security property
The propery decorator had gone missing which meant
the dump of host facts was missing the kernel security
(LSM) settings. This patch just adds the @property
decorator back
Kefu Chai [Sun, 24 Jan 2021 06:58:51 +0000 (14:58 +0800)]
cephadm: refactor call() using asyncio.asyncio.StreamReader
simpler this way, also fix a couple issues:
* create a child watcher explicitly, see
https://bugs.python.org/issue35621
* use StringIO for collecting outputs for better performance,
instead of appending the lines to an existing str
* catch ValueError when reading from the stream reader,
because StreamReader.readline() could raise ValueError when
it reaches the buffer limit while looking for a separator.
in this case, we should try again, in hope that the spawned
process can feed the reader with more data which contains
the separator (i.e., b'\n').
* backport ThreadedChildWatcher from Python 3.8 so we can
run create_subprocess_exec() in non-main threads.
iSCSI daemons need to execute <ceph service status> command
Fixes: https://tracker.ceph.com/issues/48925 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit 9b9934f75b648a9ee01848710a6f7150a371e26e)
Adam King [Mon, 11 Jan 2021 18:23:04 +0000 (13:23 -0500)]
mgr/cephadm: force flag for ok-to-stop and ok-to-stop for monitoring stack
Daemons that could cause data loss when stopped will always block.
Daemons that will only cause loss in availability should block but have
a workaround in the form of a force flag if the user is okay with the service
being down.
Also implements ok-to-stop for monitoring stack daemons that uses this system
of blocking on availability loss unless force flag is provided
Signed-off-by: Adam King <adking@redhat.com> Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
(cherry picked from commit aeeffb07f1bb226b64a362de4eecd7d63fc0b0d1)
The current bionice version triggers a podman/conmon bug that
truncates output, affecting both cephadm bootstrap when 'mgr dump' is
large, and teuthology 'pg dump' when it is large.
Kefu Chai [Tue, 19 Jan 2021 17:56:02 +0000 (01:56 +0800)]
pybind/mgr/dashboard: use _get_localized_key for composing store key
Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)
As suggested by Ernesto Puerta <epuertat@redhat.com>
Kefu Chai [Tue, 19 Jan 2021 17:24:01 +0000 (01:24 +0800)]
pybind/mgr/cephadm/inventory: cast variable to expected type
to silence mypy warnings like:
cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
Kefu Chai [Tue, 19 Jan 2021 06:49:07 +0000 (14:49 +0800)]
pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()
it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.
Kefu Chai [Sun, 17 Jan 2021 08:36:35 +0000 (16:36 +0800)]
mgr/cephadm: use dict instead of defaultdict to silence mypy warning
otherwise we have:
1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"
Kefu Chai [Sun, 17 Jan 2021 07:29:33 +0000 (15:29 +0800)]
mgr/progress: pass id to constructor of Event
to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:
progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"
Kefu Chai [Sun, 17 Jan 2021 01:35:36 +0000 (09:35 +0800)]
pybind/mgr/iostat: use correct types
for better readability and for the sake of correctness.
* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str
Michael Wodniok [Tue, 19 Jan 2021 12:15:32 +0000 (13:15 +0100)]
doc: changed commands as suggested by @sebastian-philipp
Signed-off-by: Michael Wodniok <wodniok@wor.net> Co-authored-by: Sebastian Wagner <sebastian@spawnhost.de>
(cherry picked from commit d97731969ab384909e64afd60d766e5cb06e4b5e)
Add the status field to the definition of the host to ensure an
apply_spec honours hosts that should be defined in maintenance
during a bootstrap based deployment.
Jason Dillaman [Tue, 26 Jan 2021 05:18:45 +0000 (00:18 -0500)]
librbd/migration: compute QCOW snapshot deltas via L2 table deltas
Since each entry in a L2 table represents a cluster block within the
QCOW image, we can compute deltas by comparing L2 tables. This fixes
an issue where snapshots were not being properly thin-provisioned
during the live-migration execute process.
Jason Dillaman [Tue, 26 Jan 2021 03:17:25 +0000 (22:17 -0500)]
librbd/migration: allow retrieving QCOW L2 table from L2 cache
This will be used in the next commit where the list-snaps state
machine is tweaked to compate L2 tables between snapshot revisions
to determine deltas.
Jason Dillaman [Tue, 26 Jan 2021 01:45:40 +0000 (20:45 -0500)]
librbd/migration: refactor lookup table handling for QCOW format
This will be used by a refactored list-snaps state machine to compute
the L2 table cluster for image offsets. The L1 and L2 lookup tables
can now be represented in a single structure.
also use updated build with sha1 of 1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.
and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.
projects / ceph.git / log