Lucian Petrut [Wed, 3 Feb 2021 08:59:24 +0000 (08:59 +0000)]
win32*.sh: move debug symbols to separate files
This patch simplifies releasing Windows binaries along with debug
symbols.
By default, we're going to provide minimum debug information (-g1).
The symbols are extracted from the binaries and placed in separate
files in the ".debug" folder, which is used by gdb implicitly.
This is more convenient than having separate versions of the binaries,
with or without debug symbols.
Lucian Petrut [Fri, 29 Jan 2021 11:03:20 +0000 (11:03 +0000)]
rbd: propagate WNBD start errors
This change will propagate the errors that WNBD may return when
spinning up the IO workers.
Also, we'll avoid removing the registry record for failed
non-persistent mappings. Those will be cleaned up when the service
restarts or when explicitly unmapped.
Lucian Petrut [Fri, 29 Jan 2021 09:54:10 +0000 (09:54 +0000)]
rbd: improve Windows remap failure handling
At the moment, if an image can't be remapped when the centralized
RBD service starts, the service will stop and already started
daemons will continue running.
This change adds a new option: "--remap-failure-fatal". If set,
when an image can't be remmaped, the service stops AND cleans up
the running daemons. By default, an error will be logged and the
service will continue running.
Lucian Petrut [Thu, 28 Jan 2021 14:08:29 +0000 (14:08 +0000)]
rbd: add image map timeouts on Windows
This change adds configurable timeouts used when starting the centralized RBD
service and mapping images.
This change also fixes an issue where the service would wait indefinitely for
a failed mapping. This issue was caused by the fact that multiple child
processes were inheriting pipe handles.
Note that we can't use timeouts with Windows anonymous pipes, which is why
we're going to use unique named pipes.
Lucian Petrut [Wed, 27 Jan 2021 12:58:48 +0000 (12:58 +0000)]
rbd: restrict Windows service exec calls
The centralized Ceph Windows service is responsible of managing
rbd-wnbd daemons. When starting, it's respawns the daemons using the
command line saved in the Windows registry. Also, for new mappings,
the command line is passed through a named pipe.
While writing to the according named pipe and windows registry entries
requires admin privileges, it's better to avoid running arbitrary
commands.
This patch will drop the executable from the commands that the
Ceph service accepts. Instead, it will only accept arguments that
are passed to the the binary that was used to start the service
(rbd-wnbd.exe).
Lucian Petrut [Wed, 27 Jan 2021 09:06:19 +0000 (09:06 +0000)]
rbd: allow non persistent Windows mappings
At the moment, all Windows RBD mappings are persistent, being
recreated when the Ceph service starts.
This change adds the "--non-persistent" flag to allow skipping
certain images.
Note that even for non-persistent mappings, we're still storing
data in the Windows registry, allowing us to retrieve image
details such as the rbd pool as well as the admin socket path.
When the daemons stop or the Ceph service starts, non-persistent
entries are cleaned up from the Windows registry.
Lucian Petrut [Tue, 26 Jan 2021 14:13:22 +0000 (14:13 +0000)]
rbd: delegate map requests to the Windows service
At the moment, rbd-wnbd daemons are tied to the Windows
session. When the user logs off or the WinRM remote session
terminates, the daemons are killed.
In order to avoid such issues, we'll delegate the daemon
initialization to the centralized Ceph Windows service.
We're using a named pipe for the Ceph service communication.
We're not using Ceph admin sockets for now since unix sockets
aren't available on Windows Server 2016.
Worth mentioning that the Ceph Windows service will restart
registered mappings when the host reboots based on the command
lines saved in the Windows registries. Writing to the registry
key as well as the service named pipe requires admin privileges.
Yuri Weinstein [Wed, 3 Feb 2021 18:14:56 +0000 (10:14 -0800)]
Merge pull request #39165 from sebastian-philipp/pacific-backport-39035-38935-39028-39087-38998-38854-38982-39101-39043-39113-
pacific: cephadm: Batch backport January (2)
Reviewed-by: Jan Fajerski <jfajerski@suse.com> Reviewed-by: Neha Ojha <nojha@redhat.com> Reviewed-by: Laura Paduano <lpaduano@suse.com> Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Ilya Dryomov [Sun, 31 Jan 2021 14:24:32 +0000 (15:24 +0100)]
qa/tasks/rbd: don't wait for krbd symlink to be created/removed
Commit 3754c665a11e (":qa/tasks/rbd: test qemu on top of rbd
encryption") broke dev_create() for krbd by messing up the "wait
for the symlink to be created by udev" loop. The rbd tool has been
synchronizing with udev internally since 2014, so rather than fixing
let's just drop it.
Ilya Dryomov [Sun, 31 Jan 2021 13:27:52 +0000 (14:27 +0100)]
qa/tasks/rbd: dev_create() expects a properties dict
In order to add encryption_format property, commit 3754c665a11e
(":qa/tasks/rbd: test qemu on top of rbd encryption") changed the
dev_create() contract. It now expects a properties dict, but the
main task routine wasn't updated and still passes role_images.
On top of that, role_images itself got broken. It is supposed
to be a mapping from role to a corresponding image name, not to
a dict with an image_name key. This affected generic_mount().
Kefu Chai [Thu, 28 Jan 2021 12:35:36 +0000 (20:35 +0800)]
cmake: install rook-client-python using ExternalProject
so we don't need to rerun the generate_rook_ceph_client.sh script
everytime when building the script. cmake creates a stamp file for
tracking the dependencies and the time of modification of dependencies.
Sage Weil [Wed, 27 Jan 2021 21:44:21 +0000 (15:44 -0600)]
python-common: fix test_datetime_to_str_2 on non-UTC hosts
The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string. Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.
Paul Cuzner [Wed, 27 Jan 2021 20:40:58 +0000 (09:40 +1300)]
cephadm:add missing kernel_security property
The propery decorator had gone missing which meant
the dump of host facts was missing the kernel security
(LSM) settings. This patch just adds the @property
decorator back
Kefu Chai [Sun, 24 Jan 2021 06:58:51 +0000 (14:58 +0800)]
cephadm: refactor call() using asyncio.asyncio.StreamReader
simpler this way, also fix a couple issues:
* create a child watcher explicitly, see
https://bugs.python.org/issue35621
* use StringIO for collecting outputs for better performance,
instead of appending the lines to an existing str
* catch ValueError when reading from the stream reader,
because StreamReader.readline() could raise ValueError when
it reaches the buffer limit while looking for a separator.
in this case, we should try again, in hope that the spawned
process can feed the reader with more data which contains
the separator (i.e., b'\n').
* backport ThreadedChildWatcher from Python 3.8 so we can
run create_subprocess_exec() in non-main threads.
iSCSI daemons need to execute <ceph service status> command
Fixes: https://tracker.ceph.com/issues/48925 Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
(cherry picked from commit 9b9934f75b648a9ee01848710a6f7150a371e26e)
Adam King [Mon, 11 Jan 2021 18:23:04 +0000 (13:23 -0500)]
mgr/cephadm: force flag for ok-to-stop and ok-to-stop for monitoring stack
Daemons that could cause data loss when stopped will always block.
Daemons that will only cause loss in availability should block but have
a workaround in the form of a force flag if the user is okay with the service
being down.
Also implements ok-to-stop for monitoring stack daemons that uses this system
of blocking on availability loss unless force flag is provided
Signed-off-by: Adam King <adking@redhat.com> Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
(cherry picked from commit aeeffb07f1bb226b64a362de4eecd7d63fc0b0d1)
The current bionice version triggers a podman/conmon bug that
truncates output, affecting both cephadm bootstrap when 'mgr dump' is
large, and teuthology 'pg dump' when it is large.
Kefu Chai [Tue, 19 Jan 2021 17:56:02 +0000 (01:56 +0800)]
pybind/mgr/dashboard: use _get_localized_key for composing store key
Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)
As suggested by Ernesto Puerta <epuertat@redhat.com>
Kefu Chai [Tue, 19 Jan 2021 17:24:01 +0000 (01:24 +0800)]
pybind/mgr/cephadm/inventory: cast variable to expected type
to silence mypy warnings like:
cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
Kefu Chai [Tue, 19 Jan 2021 06:49:07 +0000 (14:49 +0800)]
pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()
it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.
Kefu Chai [Sun, 17 Jan 2021 08:36:35 +0000 (16:36 +0800)]
mgr/cephadm: use dict instead of defaultdict to silence mypy warning
otherwise we have:
1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"
Kefu Chai [Sun, 17 Jan 2021 07:29:33 +0000 (15:29 +0800)]
mgr/progress: pass id to constructor of Event
to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:
progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"
Kefu Chai [Sun, 17 Jan 2021 01:35:36 +0000 (09:35 +0800)]
pybind/mgr/iostat: use correct types
for better readability and for the sake of correctness.
* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str
Michael Wodniok [Tue, 19 Jan 2021 12:15:32 +0000 (13:15 +0100)]
doc: changed commands as suggested by @sebastian-philipp
Signed-off-by: Michael Wodniok <wodniok@wor.net> Co-authored-by: Sebastian Wagner <sebastian@spawnhost.de>
(cherry picked from commit d97731969ab384909e64afd60d766e5cb06e4b5e)
Add the status field to the definition of the host to ensure an
apply_spec honours hosts that should be defined in maintenance
during a bootstrap based deployment.
Jason Dillaman [Tue, 26 Jan 2021 05:18:45 +0000 (00:18 -0500)]
librbd/migration: compute QCOW snapshot deltas via L2 table deltas
Since each entry in a L2 table represents a cluster block within the
QCOW image, we can compute deltas by comparing L2 tables. This fixes
an issue where snapshots were not being properly thin-provisioned
during the live-migration execute process.
Jason Dillaman [Tue, 26 Jan 2021 03:17:25 +0000 (22:17 -0500)]
librbd/migration: allow retrieving QCOW L2 table from L2 cache
This will be used in the next commit where the list-snaps state
machine is tweaked to compate L2 tables between snapshot revisions
to determine deltas.
Jason Dillaman [Tue, 26 Jan 2021 01:45:40 +0000 (20:45 -0500)]
librbd/migration: refactor lookup table handling for QCOW format
This will be used by a refactored list-snaps state machine to compute
the L2 table cluster for image offsets. The L1 and L2 lookup tables
can now be represented in a single structure.
also use updated build with sha1 of 1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.
and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.
projects / ceph.git / log