Lucian Petrut [Thu, 7 Jan 2021 13:14:00 +0000 (13:14 +0000)]
cmake,win32*.sh: Windows DLL support
At the moment, the Windows binaries can only be linked statically.
This is less than ideal since:
* the resulting binaries can be quite large, especially when
including debug symbols
* librados and librbd cannot be used directly by 3rd party apps
* the build duration is increased
In order to do a dynamic build, we'll:
* add an option to win32_build.sh
* fix link order
* dynamically link boost
* disable the "-pie" flag when using Mingw, which generates incorrect
executable entry points
* by default, cmake generates import libs for executables. The issue
is that for rados.exe, it generates librados.dll.a, thus overriding
the librados.dll import library, which breaks the build process.
We'll configure cmake to skip generating import libs for executables.
Marcus Watts [Wed, 13 Jan 2021 05:17:38 +0000 (00:17 -0500)]
qa/tasks/vault.py: unzip: try harder to find a working unzip.
The existing logic uses "python -m zipfile" to unzip files.
This will (most likely) fail on CentOS 8-Stream , where python
defaults to 'unset' (see man unversioned-python).
So: try harder: try unzip, python3, and python in that order,
to find something that can unzip files.
Fixes: https://tracker.ceph.com/issues/48921 Signed-off-by: Marcus Watts <mwatts@redhat.com>
Sage Weil [Tue, 19 Jan 2021 22:49:08 +0000 (16:49 -0600)]
mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state
If cephadm daemons are not happy we should raise a warning. Aside from
being an important part of the user experience, this will also help us
catch teuthology test errors.
Fixes: https://tracker.ceph.com/issues/45628 Signed-off-by: Sage Weil <sage@newdream.net>
Or Ozeri [Thu, 14 Jan 2021 15:03:42 +0000 (17:03 +0200)]
:qa/tasks/rbd: test qemu on top of rbd encryption
This commit adds new qemu xfstests workloads that run on top of librbd luks1/luks2 encryption.
This is currently done via nbd, instead of the qemu rbd driver.
Right now, we're violating the Liskov substitution principle by deriving from `Orchestrator` but `process` takes a sub class of `Completion`:
See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
The idea is to make Orchestrator a type constructor with `CompletionT` as argument, but this is not supported by mypy: https://github.com/python/typing/issues/548
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Venky Shankar [Tue, 18 Aug 2020 09:32:38 +0000 (05:32 -0400)]
test: optionally create a backup filesystem on startup
Also filter out client-id's starting with "mirror" when
cleaning leftover auth-ids since teuthology would be
configured to create client.mirror and client.mirror_remote
clients before executing mirroring tests.
osd: Remove override for osd_async_recovery_min_cost for mclock profiles
Overriding osd_async_recovery_min_cost as part of enabling a built-in
mclock profile has the undesirable side effect of peers not choosing
the correct async recovery targets if osds are using mixed schedulers
(this could happen during upgrades or if "debug_random" is set for
osd_op_queue config option). Due to the above, osds get into a
"choose_acting" loop during peering.
The solution is to remove the override of osd_async_recovery_min_cost.
osd: Set recovery specific Ceph options for mclock profiles to work.
Set and disable relevant recovery specific Ceph options for mclock
profiles to work as expected. Broadly,
- Set low value for recovery min cost
- High values for max active recoveries and max backfills
- Disable recovery sleep{_hdd, _ssd, _hybrid}
osd: Handle configuration changes to mclock config options
Handle configuration changes to the mclock cost per io, the max
capacity options and the mclock profile. Handle the case where the
profile is changed from a built-in profile to the custom profile.
osd: Add mclock profile infrastructure and implement mclock profiles
Define config options to specify the cost per io for an osd (hdd & ssd).
- osd_mclock_cost_per_io_msec
- osd_mclock_cost_per_io_msec_hdd
- osd_mclock_cost_per_io_msec_ssd
Define config options to set max osd capacity (hdd & ssd) to be allocated
between clients of dmclock namely,
- osd_mclock_max_capacity_iops
- osd_mclock_max_capacity_iops_hdd
- osd_mclock_max_capacity_iops_ssd
Define config option "osd_mclock_profile" to specify the built-in profile
to enable.
Also, Set the number of op shards being used in the osd within the mclock
scheduler as well. This is necessary to calculate the per shard limits
within the mclock scheduler.
With the above information, enable the specified mclock profile by
calling the appropriate method to set the profile specific mclock
parameters and Ceph options.
Prior to enqueuing an op, the scheduler performs a calculation to scale
up or down the cost associated for the OpSchedulerItem. This calculation
is done based on the existing item cost, the max osd capacity provided
and an additional cost factor based on underlying device type(hdd/ssd).
Sunny Kumar [Fri, 15 Jan 2021 02:53:05 +0000 (02:53 +0000)]
osd: handle ceph specific config changes for the mclock scheduler
The below ceph parameters are set automatically
while enabling the mclock scheduler with a built-in profile.
The user in this case will not be able to modify these
ceph specific config options during runtime.
a. osd_async_recovery_min_cost
b. osd_recovery_max_active{_hdd,_ssd}
c. osd_max_backfills
d. osd_recovery_sleep{_hdd,_ssd,_hybrid}
If the custom profile is enabled for the mclock scheduler,
the user can modify these parameters.
Igor Fedotov [Sat, 16 Jan 2021 16:54:52 +0000 (19:54 +0300)]
os/bluestore: fix deferred_queue locking
https://github.com/ceph/ceph/pull/30027 introduced a gap in osr
protection (in _deferred_queue()) which could cause improper deferred_pending value while
processing osr from _deferred_aio_finish().
As a result both segmentation fault in _deferred_aio_finish() or deadlock could occur.
Fixes: https://tracker.ceph.com/issues/48776 Signed-off-by: Igor Fedotov <ifedotov@suse.com>
Patrick Donnelly [Mon, 11 Jan 2021 21:39:27 +0000 (13:39 -0800)]
client: wire up alternate_name
Here we're exposing a public Client::walk (aka path_walk) so that the
user can inspect dentries (not something normally possible in POSIX).
We're going to skip exposing such an interface in libcephfs since
there's no reason to do that (who would use it?) except for testing.
Instead, a follow-up PR will add Client tests (for the first time, yay!)
that will exercise this code.
Also, ideally we'd also expose alternate_name via readdir results but
that is a bit more complicated since dirents do not normally refer to
external memory. So, just rely on Client::walk for testing for now.
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Patrick Donnelly [Fri, 15 Jan 2021 03:55:43 +0000 (19:55 -0800)]
mds: fix alternate_name durability
This is a collection of fixes to Xiubo's prior work. Namely:
- Add new mds_alternate_name_max option to limit the size of
alternate_name. Otherwise a Client could trick the MDS into creating
an alternate_name of any size!
- Clean up how alternate_name is assigned to CDentry. In the general
case, this should be assigned as part of creating the dentry. We want
this value to be immutable for the life of the dentry. Even for the
very special case of rename(2) where the destination dentry already
exists. We explicitly check (after discussion with Jeff) that the
target dentry alternate_name already matches what the rename RPC is
giving.
- The MDS is now properly journaling the alternate_name.
- The MDS rejoin phase is properly transmitting each dentry's
alternate_name. I've discovered that this MMDSCacheRejoin message
actually wasn't versioned which I've raised in a tracker [1]. In the
mean time, we'll just bump CEPH_MDS_PROTOCOL as usual.
[1] https://tracker.ceph.com/issues/48886
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Xiubo Li [Mon, 21 Sep 2020 04:55:31 +0000 (12:55 +0800)]
mds: add alternate_name feature support for dentries
This will support the "alternate_name" filename support, and will save
an alternate name for each dentry. This alternate name is not used in
path lookup or for any other usual file system purpose. The name is
simply an added blob of metadata on the dentry that is distributed to
clients so that "long" file names may be supported for clients which
require them. In the case of an fscrypt enhanced kernel mount driver,
the long name may be the cyphertext (exceeding FILENAME_MAX) of a long
file name.
Because this affects only files with long file names, the use of this
feature should be rare but could be common for some unusual
applications.
The client mount should check the CEPHFS_FEATURE_ALTERNATE_NAME feature
bit first to check whether the MDS has support for this feature or not.
The alternate_name is transmitted as part of the message payload in
MClientRequest when setting the alternate_name. The LeaseStat structure
in MClientReply contains the alternate_name.
When executing a metadata mutation RPC, the client will set the
alternate_name (if it exists) as part of the operation. The MDS will
pick that up and set it on the new or mutated dentry.
Fixes: https://tracker.ceph.com/issues/47162 Signed-off-by: Xiubo Li <xiubli@redhat.com> Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Xiubo Li [Wed, 23 Sep 2020 02:54:34 +0000 (10:54 +0800)]
mds: add new CDentry tags support
This will add new tag 'i' for inode and new tag 'l' for remote link
for the CDentry. And at the same time will add one proper dentry
version, which will be helpful to add new features/members in future
for the CDentry.
Fixes: https://tracker.ceph.com/issues/47162 Signed-off-by: Xiubo Li <xiubli@redhat.com>
projects / ceph.git / log