Jason Dillaman [Fri, 29 Jan 2021 15:44:38 +0000 (10:44 -0500)]
librbd/deep_copy: skip snap list if object is known to be clean
If the fast-diff indicates that the destination object should exist
and that it hasn't changed, there shouldn't be a need to issue the
snap list operation. Instead, just update the destination object map
to indicate the existence of the object.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Fri, 29 Jan 2021 02:42:09 +0000 (21:42 -0500)]
librbd/deep_copy: object-copy state machine must update object map
If there was no data to copy, the object-copy state machine was bypassing
the object-map update states and prematurely completing. Since the
object-map is default-initialized to all non-existent objects, this results
in incorrect state for OBJECT_EXISTS_CLEAN objects.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Wed, 3 Feb 2021 18:21:34 +0000 (13:21 -0500)]
librbd/io: track object non-existence when computing snapshot deltas
Re-use the existing DNE state to track whether or not the object
already exists when computing snapshot deltas from an arbitrary
set of snapshots. Previously, the non-existence of the object was
only computed for snap id 0 for tracking whiteouts. In a future
commit, the deep-copy object-copy state machine will be able to
properly update the object-map state to indicate exists clean
vs non-existent state.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Wed, 3 Feb 2021 15:13:28 +0000 (10:13 -0500)]
librbd/io: only track initial diff extents if no diffs exists
The purpose of the initial diff extents ({0, 0}) was to help track
whether or not objects exists for read-from-parent / whiteout
tracking. Once we have at least one set of diffs on the object, we
actually have enough information to know about the object state.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Jason Dillaman [Thu, 28 Jan 2021 23:30:16 +0000 (18:30 -0500)]
librbd/object_map: diff state machine should track object existence
The deep-copy snapshot-create state machine initializes the object-map
state to non-existent for all objects. There was an assumption that the
deep-copy object-copy state machine would always update the object map
but that was being skipped for clean objects as an optimization. This
change will support a future commit to run the object-copy state machine
for existing objects.
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Casey Bodley [Wed, 3 Feb 2021 20:17:19 +0000 (15:17 -0500)]
cmake/rgw: forward spawn's compile options to rgw_common object library
since rgw_common is an OBJECT library, we can't use
target_link_libraries() for its dependency on spawn. we add its
include directories manually already with
$<TARGET_PROPERTY:spawn,INTERFACE_INCLUDE_DIRECTORIES>, but this didn't
pull in the compile definitions. this ultimately prevented the
WITH_BOOST_VALGRIND option from passing the BOOST_USE_VALGRIND
definition attached to boost::context
Sage Weil [Wed, 3 Feb 2021 15:38:49 +0000 (10:38 -0500)]
Merge PR #39069 into master
* refs/pull/39069/head:
mgr/cephadm/upgrade: tolerate pre-pacific upgrade state
mgr/cephadm/upgrade: scale down MDS cluster(s) for major version upgrades
mgr/cephadm: fix capitalization, level; drop elipses of log msgs
mgr/cephadm/upgrade: match against any repo_digest, not image_id
cephadm: return repo_digests (plural) in pull/inspect output
mgr/cephadm: include container_image_digests in inventory
cephadm: include image_digests list in 'ls' output
vstart.sh: only extract first container digest
mgr/cephadm: move release -> major translation to helper
mgr/cephadm/upgrade: tolerate old upgrade_state.target_versoin
mgr/cephadm/upgrade: set require-osd-release when done with OSDs
mgr: add lookup_release_name(int) to mgr interface
mgr/cephadm: verify container image version after we pull it
mgr/cephadm: only save version portion of version string
cephadm: fix 'inspect' and 'pull'
mgr/cephadm/upgrade: implement N-2 version checks on upgrade start
Casey Bodley [Wed, 3 Feb 2021 14:46:33 +0000 (09:46 -0500)]
cmake: partial revert of BOOST_USE_VALGRIND when ALLOCATOR=libc
the WITH_SYSTEM_BOOST binaries are not built with BOOST_USE_VALGRIND, so
it probably isn't safe to define for the headers only
this flag is needed for teuthology testing, and the shaman builds use
WITH_SYSTEM_BOOST=OFF. so the better fix is to enable WITH_BOOST_VALGRIND
so BuildBoost.cmake will build the libraries with valgrind support and add
-DBOOST_USE_VALGRIND to the necessary targets
this change was merged in https://github.com/ceph/ceph-build/pull/1736
Lucian Petrut [Wed, 3 Feb 2021 08:59:24 +0000 (08:59 +0000)]
win32*.sh: move debug symbols to separate files
This patch simplifies releasing Windows binaries along with debug
symbols.
By default, we're going to provide minimum debug information (-g1).
The symbols are extracted from the binaries and placed in separate
files in the ".debug" folder, which is used by gdb implicitly.
This is more convenient than having separate versions of the binaries,
with or without debug symbols.
Kefu Chai [Fri, 29 Jan 2021 16:48:36 +0000 (00:48 +0800)]
pybind/mgr/hello: add typing annotation
also, use Option and CLIReadCommand to define options and cli
commands. this module serves as a "hello world" example for
developers of mgr modules. so it's important to use the more
convenient and safer way to implement the module
Lucian Petrut [Fri, 29 Jan 2021 11:03:20 +0000 (11:03 +0000)]
rbd: propagate WNBD start errors
This change will propagate the errors that WNBD may return when
spinning up the IO workers.
Also, we'll avoid removing the registry record for failed
non-persistent mappings. Those will be cleaned up when the service
restarts or when explicitly unmapped.
Lucian Petrut [Fri, 29 Jan 2021 09:54:10 +0000 (09:54 +0000)]
rbd: improve Windows remap failure handling
At the moment, if an image can't be remapped when the centralized
RBD service starts, the service will stop and already started
daemons will continue running.
This change adds a new option: "--remap-failure-fatal". If set,
when an image can't be remmaped, the service stops AND cleans up
the running daemons. By default, an error will be logged and the
service will continue running.
Lucian Petrut [Thu, 28 Jan 2021 14:08:29 +0000 (14:08 +0000)]
rbd: add image map timeouts on Windows
This change adds configurable timeouts used when starting the centralized RBD
service and mapping images.
This change also fixes an issue where the service would wait indefinitely for
a failed mapping. This issue was caused by the fact that multiple child
processes were inheriting pipe handles.
Note that we can't use timeouts with Windows anonymous pipes, which is why
we're going to use unique named pipes.
Lucian Petrut [Wed, 27 Jan 2021 12:58:48 +0000 (12:58 +0000)]
rbd: restrict Windows service exec calls
The centralized Ceph Windows service is responsible of managing
rbd-wnbd daemons. When starting, it's respawns the daemons using the
command line saved in the Windows registry. Also, for new mappings,
the command line is passed through a named pipe.
While writing to the according named pipe and windows registry entries
requires admin privileges, it's better to avoid running arbitrary
commands.
This patch will drop the executable from the commands that the
Ceph service accepts. Instead, it will only accept arguments that
are passed to the the binary that was used to start the service
(rbd-wnbd.exe).
Lucian Petrut [Wed, 27 Jan 2021 09:06:19 +0000 (09:06 +0000)]
rbd: allow non persistent Windows mappings
At the moment, all Windows RBD mappings are persistent, being
recreated when the Ceph service starts.
This change adds the "--non-persistent" flag to allow skipping
certain images.
Note that even for non-persistent mappings, we're still storing
data in the Windows registry, allowing us to retrieve image
details such as the rbd pool as well as the admin socket path.
When the daemons stop or the Ceph service starts, non-persistent
entries are cleaned up from the Windows registry.
Lucian Petrut [Tue, 26 Jan 2021 14:13:22 +0000 (14:13 +0000)]
rbd: delegate map requests to the Windows service
At the moment, rbd-wnbd daemons are tied to the Windows
session. When the user logs off or the WinRM remote session
terminates, the daemons are killed.
In order to avoid such issues, we'll delegate the daemon
initialization to the centralized Ceph Windows service.
We're using a named pipe for the Ceph service communication.
We're not using Ceph admin sockets for now since unix sockets
aren't available on Windows Server 2016.
Worth mentioning that the Ceph Windows service will restart
registered mappings when the host reboots based on the command
lines saved in the Windows registries. Writing to the registry
key as well as the service named pipe requires admin privileges.
It appears that commit 6eb8f30a238 broke the test utility and
its failure was masked by the test case that expected a failure
due to a timeout force-killing the app.
Fixes: https://tracker.ceph.com/issues/49117 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Neha Ojha [Tue, 2 Feb 2021 17:43:56 +0000 (17:43 +0000)]
qa/*/test_envlibrados_for_rocksdb: use osd_client_message_cap to prevent slow requests
EnvLibradosMutipoolTest.DBBulkLoadKeysInRandomOrder can overload OSDs and cause
heartbeat timeouts. Tests in test_envlibrados_for_rocksdb also generate slow
requests on OSDs. Use osd_client_message_cap to prevent this.
Since this option is disabled by default, this may be a good way to exercise it.
myoungwon oh [Tue, 2 Feb 2021 06:43:14 +0000 (15:43 +0900)]
osd: fix to call nullptr when cancel_manifest_ops
Segmentation fault can occur because
cancel_manifest_ops must call callback without checking
nullptr even though ManifestOp can be used with null callback
in start_dedup()
projects / ceph-ci.git / log