Nathan Cutler [Wed, 2 Oct 2019 12:19:04 +0000 (14:19 +0200)]
qa/rados/upgrade/jewel-x-singleton: do not run on Bionic
The test was failing because it did not explicitly set "os_type" and
"os_version", and teuthology defaulted to Ubuntu Bionic.
Jewel is not supported on Ubuntu Bionic, so we should not try to run tests on
that Ubuntu version.
NOTE: This commit is not cherry-picked from master because the fix is
luminous-specific (there is no "rados/upgrade/jewel-x-singleton" test after
luminous).
qa/rados/upgrade/jewel-x-singleton: set RBD_FEATURES
This commit cannot be cherry-picked from master because luminous is the only
non-EOL stable branch that still includes the
qa/rados/upgrade/jewel-x-singleton suite.
A recent PR made sure that a bucket listing could not request too many
entries at once. It also did a minimum computation for number of
entries. For ordered listing the minimum was 0, as required to pass
all unit tests. However the minimum for unordered listing was left at
1. In order to make ordered and unordered listing behave the same --
with the exception of ordering -- the minimum for unordered listing is
modified to 0.
J. Eric Ivancich [Fri, 19 Jul 2019 20:10:59 +0000 (16:10 -0400)]
rgw: mitigate bucket list with max-entries excessively high
When listing a bucket with radosgw-admin, the user can specify the
maximum number of entries. That number can be unreasonably large, and
can affect the performance and memory availability. For example:
radosgw-admin bucket list --bucket mybucket1 --max-entries=10000000
This has the potential for creating large data structures at multiple
levels in the the call stack of the radosgw(-admin) process,
potentially causing the process to run out of memory. This change
limits the maximum number of entries requested in all but the high
level code to help mitigate this issue.
Conflicts:
src/rgw/rgw_rados.cc
- in luminous, RGWRados::Bucket::List::list_objects_ordered does not take
optional_yield parameter
- no std::string_view in C++11
Jason Dillaman [Thu, 22 Aug 2019 00:20:10 +0000 (20:20 -0400)]
common/config: hold lock while accessing mutable container
The 'call_gate_leave' method was accessing the 'obs_call_gate' map
without holding the required lock. This data structure could be
manipulated by another thread underneath the observer callback
thread context.
Fixes: https://tracker.ceph.com/issues/41354 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0cc2dd7bd1021dd8b5e22850495ce97da7d98716)
Conflicts:
src/common/config_proxy.h: logic incorporated into config.cc
Kefu Chai [Wed, 7 Aug 2019 09:46:13 +0000 (17:46 +0800)]
admin/build-doc: use python3
to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.
in this change:
* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
stable.
ceph-volume: do not fail when trying to remove crypt mapper
In a containerized context, at some point, need to run `simple scan` on a device
from a separate container (not the existing and running corresponding container
to that device), but this can't work because when it tries to remove the
mapper which is still in use by the corresponding running osd container,
it fails.
This can be a bit more permissive and simply throw a warning.
Currently when parsing host in requests, we try to case match against supplied
hostnames set, which violates dns hostnames being case insensitive. Do a case
insensitive comparision instead
This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing
Brad Hubbard [Thu, 22 Nov 2018 00:07:22 +0000 (10:07 +1000)]
install-deps.sh: Remove CR repo
Remove the continuous release repos for CentOS and Virtuozzo 7 as they
should no longer be needed since http://tracker.ceph.com/issues/13997 is
no longer relevant and the newer versions of selinux packages pulled in
by the build system are causing problems for systems without CR repos
enabled.
Ilya Dryomov [Wed, 20 Feb 2019 21:30:29 +0000 (22:30 +0100)]
osdc/Objecter: invalidate crcs on preallocated rx buffers
Both simple and async messengers use c_str() when copying the data from
the socket into the receive buffer, going behind bufferlist's back. If
the receive buffer is preallocated, we need to invalidate its crc cache
by hand to avoid possible data crc mismatches on the client side.
Conflicts:
src/test/librados/io_cxx.cc
- In master, 3730d10623650ce8569be96b28cbba599a9a0db6 renamed this file from
src/test/librados/io.cc but that commit is not being backported to luminous.
Manually cherry-picked the test mods into src/test/librados/io.cc.
Josh Durgin [Sat, 24 Aug 2019 02:52:52 +0000 (22:52 -0400)]
os/bluestore: record gifts < expected size in superblock
Returning 0 from _balance_bluefs_freespace() skips recording
allocations in the superblock, so we fail the consistency check on
startup. The elseif branch handles this case already, so just remove
it from the first branch. This is luminous/mimic specific, since
bluefs extents are not recorded in the superblock in later releases.
Enming Zhang [Sat, 23 Jun 2018 09:09:54 +0000 (17:09 +0800)]
qa: fix for s3tests.functional.test_s3.test_bucket_get_location failed
Fixes: https://tracker.ceph.com/issues/41401 Signed-off-by: Enming Zhang <enming.zhang@umcloud.com>
(cherry picked from commit ad6edd4dd2d6e57682ff82ad0ad0bfcde5942d66) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Enming Zhang [Wed, 30 May 2018 02:54:42 +0000 (10:54 +0800)]
rgw: fix issue for CreateBucket with BucketLocation param under default zonegroup
Because there is not a default realm, the has_zonegroup_api always returns false
when create bucket with the BucketLocation parameter under default zonegroup,
which always leads to the CreateBucket with BucketLocation parameter request failed.
Kefu Chai [Tue, 29 May 2018 07:51:07 +0000 (15:51 +0800)]
mds,osd,mon,msg: use intrusive_ptr for holding Connection::priv
See-also: http://tracker.ceph.com/issues/20924 Signed-off-by: Kefu Chai <kchai@redhat.com> Signed-off-by: Samuel Just <sjust@redhat.com>
(cherry picked from commit 72883956c26fdc4345324e9b27b45c3dfac17fa2)
Sage Weil [Mon, 19 Aug 2019 15:09:36 +0000 (10:09 -0500)]
os/bluestore: try to gift large extents, then fall back to small extents
First try to gift using the larger alloc_size (normally bluefs_alloc_size,
but here max(bluefs_alloc_size,bluefs_shared_alloc_size) just in case the
settings are weird. If that fails, then try the shared_alloc_size. If
that fails, fail and complain as before, with an more accurate error
message.
Sage Weil [Mon, 19 Aug 2019 13:46:09 +0000 (08:46 -0500)]
os/bluestore/BlueFS: apply shared_alloc_size to shared device
Keep an alloc_size vector so that we have this value handy at all times.
Allow bluestore to fetch this value directly instead of looking at the
bluefs_* config options since this encapsulates things a bit better, and
also isn't vulnerable to the config setting changing at runtime.
Sage Weil [Fri, 16 Aug 2019 21:44:57 +0000 (16:44 -0500)]
os/bluestore: cleanup around allocator calls
Both stupid and bitmap allocator returs -ENOSPC if they're
unable to allocate any space. Existing callers aren't always
respect this - hence doing some cleanup.
Kefu Chai [Sun, 24 Mar 2019 09:11:31 +0000 (17:11 +0800)]
ceph-volume: terminal: encode unicode when writing to stdout
python determins the encoding of stdout and stderr based on the LC_CTYPE
and PYTHONIOENCODING env variable, by default, python3's sys.stdout uses
'utf-8' as its encoding, so it will be able to write unicode string even
the stdout is not attached to a tty device. but when it comes to
python2, it will default to ascii if neither of these variabls is set.
so, if we are writing unicode using `_Write` in an environment where
LC_CTYPE and/or PYTHONIOENCODING are using non UTF-8 encoding, it chokes
by raising `UnicodeEncodeError` exception.
in this change, we add a wrapper around `_Write._writer` so it is able
to write unicode string in such a non-unicode-friendly environment.
for more info related the encoding of stdout and stderr, see
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONIOENCODING .
projects / ceph.git / log