struct PGTempMap in osd/OSDMap.h tracks a number of int32_t pointers
pointing into a buffer list. But that list was generated via encode,
which means int members are bytes-swapped. Fixed by using ceph_le32
pointers instead.
Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).
Note that this updates only those uses of __le16/32/64 which are
part of data structures that are serialized to disk/network
(i.e. Transaction::Op and Transaction::TransactionData).
Also note that code in this file performs combined operations on
little-endian values (in particular ++, +=, and |=) which are not
supported on the ceph_le16/32/64 class, and are therefore replaced
by more primitive operations.
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 7d3cd10edfbdf9de6344430b0e336c351a903bfc)
Changes from cherry-picked commit:
- Duplicated changes into src/crimson/os/Transaction.h and
src/os/ObjectStore.h (are merged into src/os/Transaction.h
in mainline). Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit 62def3ec70434694fcf918c00666ba0e5f9325c2)
Changes from cherry-picked commit:
- Omitted changes to src/crimson/net/ProtocolV2.cc
(file does not yet exist in nautilus). Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
(cherry picked from commit ede1e851604bb55db4cc29f56ae9520c5d1ec727)
Changes from cherry-picked commit:
- Additional changes in src/msg/xio/XioMsg.h were necessary
(the file was deleted in mainline). Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).
Note that I've also changes cephx_calc_client_server_challenge to
use ceph_le64 instead of manually byte-swapping by mswab. (This
is a no-op, but it seems more consistent to use the ceph_le types
throughout.)
Use ceph_le16/32/64 instead of __le16/32/64 (which are no-op outside
of kernel code).
Note that I've changed the Alg::calc routines to return the
init_value_t type instead of value_t, to avoid having to
introduce explicit byte-swapping calls to init_le16/32/64 in
many places. (This way, the byte-swapping happens implicitly
at the very end, wher the init_value_t value is assigned via
a value_t pointer.)
include: Fix new-style encoding routines on big-endian
The new-style encoding routines (denc.h) are broken on big-endian
systems. While there is a lot of infrastucture in place to
recognize data types that need to be byte-swapped during encoding
and decoding on big-endian systems, nothing is actually ever swapped.
Fixed by using ceph_le16/32/64 instead of __le16/32/64 (which are
no-ops outside of kernel code).
include: Simplify usage of init_le16/32/64 routines
These routines currently just return plain __u16/32/64. This patch
changes them to return ceph_le16/32/64 types instead. This has a
number of benefits, in particular it allows the routines to now be
used to directly initialize variables of ceph_le16/32/64 type, as
one would expect from the names of those routines.
This doesn't make much of a difference in the current code base,
but it simplifies future patches to fix endian issues.
include: Endian fix for shared kernel/user headers
Endian swapping is done differently in kernel space vs. user space,
but a few header files are shared between those two user cases.
Current code attempts to handle this by re-defining __le16/32/64
before pulling those headers into user space, but this is not
consistenly done: when ceph_fs.h is included via types.h, the
redefinition happens, but when ceph_fs.h is directly included,
the redefinition does not happen.
Fix this by performing the same redefinition directly *in* those
shared headers, when included by user space.
Note that the redefines where also in effect for rbd_types.h,
which is *not* shared with the kernel, so in the file I'm simply
replacing __le16/32/64 with ceph_le16/32/64 in-line.
Also note that it is now no longer possible to include any of
the three files using the redefined macros in C code (as they
make use of C++ features). However, this currently happens in
exactly one file, src/mds/locks.c, which only uses a few CEPH_CAP_...
constants from the header. To fix this, I've simply duplicated
those definitions, which are unchangeable ABI constants anyway.
Jeff Layton [Mon, 3 Jun 2019 14:55:37 +0000 (10:55 -0400)]
msg: fix addr2 encoding for sockaddrs
Currently, the sockaddr sa_family field is being sent across the wire
without any sort of endianness conversion.
Redefine the new-style entity_addr_t encoding to encode the sa_family
field as little-endian. This will allow LE machines in the field to
keep limping along, but note that BE machines may break until they are
all fixed.
To do this, use the code that was initially written for BSD/Apple
machines to encode the different fields separately.
Fixes: http://tracker.ceph.com/issues/40114 Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 7b4b70b6853e036379a0aababec598d59edcecee) Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Kefu Chai [Wed, 7 Aug 2019 09:46:13 +0000 (17:46 +0800)]
admin/build-doc: use python3
to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.
in this change:
* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
stable.
ceph-volume: do not fail when trying to remove crypt mapper
In a containerized context, at some point, need to run `simple scan` on a device
from a separate container (not the existing and running corresponding container
to that device), but this can't work because when it tries to remove the
mapper which is still in use by the corresponding running osd container,
it fails.
This can be a bit more permissive and simply throw a warning.
Patrick Donnelly [Mon, 23 Sep 2019 12:15:19 +0000 (05:15 -0700)]
Merge PR #29926 into nautilus
* refs/pull/29926/head:
mgr/volumes: return string type to ceph-manager
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size
mgr/volumes: drop unnecessary size
mgr/volumes: cleanup FS subvolume or subvolume group path
mgr/volumes: give useful error message
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
The 'volumes' ceph-mgr module returns a byte type instead of a string
type as the stdout result of commands, `fs subvolume getpath` and
`fs subvolumegroup getpath` to the ceph-mgr. This causes the commands
to hang and the ceph-mgr to utilize 100% CPU. Fix the 'volumes'
ceph-mgr module to return string type to ceph-mgr as the stdout
result of the commands.
Alfonso Martínez [Thu, 22 Aug 2019 13:33:02 +0000 (15:33 +0200)]
mgr/dashboard: run-backend-api-tests.sh CI improvements
As there is now a jenkins job to run this script
(see https://github.com/ceph/ceph-build/pull/1351),
this refactoring adapt the script to be run in a jenkins job as well as locally.
otherwise we have
```
Running setup.py egg_info for package apache-libcloud
libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
warnings.warn("Missing backports.ssl_match_hostname package")
Traceback (most recent call last):
...
File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated
...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```
and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.
This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing
projects / ceph.git / log