Kefu Chai [Thu, 10 Oct 2019 01:54:50 +0000 (09:54 +0800)]
ceph.spec.in: use python_provide macro
our python3 bindings are now named `python3-<modname>` after python3 is
now maintained by RHEL/CentOS instead EPEL. to help the users using
`python36-<modname>`, we should "Provide" `python36-<modname>`.
the `python_provide` macro is offered by `python-rpm-macros` package,
which is in turn required by python*-devel. and we do install
`python36-devel` in install-deps.sh, and install `python3-devel` in
ceph-*build/build/setup_rpm
see also
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_provides
Kefu Chai [Fri, 30 Aug 2019 11:49:28 +0000 (19:49 +0800)]
ceph.spec.in: s/pkgversion/version_nodots/
`python3_pkgversion` is now defined as 3, while we don't have packages
like python3-Cython yet in EPEL7. but we do have `python36-Cython`. so
let's use `python3_version_nodots` instead.
tl;dr: this change addresses the failures of "make check" runs on arm64
builders when they try to build `mgr-dashboard-test-venv` target.
long story: without this change, we will fail to pull in
setuptools >= 36, and as a result pip will fail to import
`setuptools.build_meta` in `pip/_vendor/pep517/_in_process.py`. and will
a `BackendUnavailable` exception thrown by `_call_hook()` in
`pip/_vendor/pep517/wrappers.py`. since the issue addressed by 30ce5e55
has been addressed since setuptools >= 36.0.1, we should be safe to
upgrade to the latest setuptools now.
Kefu Chai [Wed, 19 Dec 2018 08:37:55 +0000 (16:37 +0800)]
install-deps.sh: skip unavailable repos
as centos-sclo-rh-source leads us to 404 at this moment. and we are not
using the source repo for building ceph. so we can just skip any
unavailable repo.
Kefu Chai [Sat, 10 Nov 2018 21:33:43 +0000 (13:33 -0800)]
install-deps: install setuptools before upgrading virtualenv
this should address the failures when running install-deps.sh, like
Downloading/unpacking virtualenv
Running setup.py egg_info for package virtualenv
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'
warnings.warn(msg)
error in virtualenv setup command: 'extras_require' must be a
dictionary whose values are strings or lists of strings containing valid
project/version requirement specifiers.
Complete output from command python setup.py egg_info:
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'
warnings.warn(msg)
error in virtualenv setup command: 'extras_require' must be a dictionary
whose values are strings or lists of strings containing valid
project/version requirement specifiers.
this only happens on very old virtualenv shipped with RHEL7.4
Nathan Cutler [Wed, 1 Aug 2018 10:33:58 +0000 (12:33 +0200)]
build/ops: unify command substitution in install-deps.sh
The $() form is preferable to `` because folks (like me) might be using
` as a keyboard shortcut to GNU Screen, causing havoc to ensue whenever
copy-pasting the ` character.
install-deps.sh: install `python*-devel` for python*rpm-macros
in 087ea813, we installed '*rpm-macros' for the macros, so we can have
access to the latest python packaging related macros for preparing the
build dependencies.
but we could run into https://bugs.centos.org/view.php?id=16379, if
we already have an old version of python-devel installed. as the newer
version of python-rpm-macros conflicts with it.
it was a chicken-and-egg problem, as we don't know the exact name of
*rpm-macros packages. that's why we chose to install all of them. but
we have to upgrade the existing python-devel package to resolve the
conflict. but the since there is no python3-devel in RHEL7/CentOS7,
what they have is python36-devel. so we have to hardwire the
`%{python3_pkgversion}` to "36" even before we have access to this
macro, and upgrade the python36-devel package beforehand. but this
renders installing the rpm-macro package less useful -- we intend to
use the macro offered by the package to figure out "36".
as a workaround, we pretend that we know the "main" version of python3
in current RHEL/CentOS. and always install python36-devel for
python-rpm-macros. as the former requires the latter.
once all python3*-devel on all builders are upgraded, we will be safe
to install '*rpm-macros' again without installing python36-devel first.
by then, we could revert this change, or continue installing
python36-devel until the distro bumps up the "main" python version to 3.7
Brad Hubbard [Thu, 22 Nov 2018 00:07:22 +0000 (10:07 +1000)]
install-deps.sh: Remove CR repo
Remove the continuous release repos for CentOS and Virtuozzo 7 as they
should no longer be needed since http://tracker.ceph.com/issues/13997 is
no longer relevant and the newer versions of selinux packages pulled in
by the build system are causing problems for systems without CR repos
enabled.
Nathan Cutler [Wed, 1 Aug 2018 10:52:45 +0000 (12:52 +0200)]
build/ops: refrain from installing/using lsb_release in install-deps.sh
Unfortunately the mapping between release number and codename (which is only
relevant for Debian and Ubuntu btw) is not available from /etc/os-release.
In that one respect, lsb_release was "better".
However, when I weigh the advantages of obtaining that mapping from an external
tool, with the (substantial) risk that the external dependency might cause
trouble on one or more supported distros (to say nothing of the non- or
semi-/pseudo-supported ones), against the work involved in maintaining a
hard-coded mapping (negligible), the needle on my scale immediately swings
toward eliminating the dependency.
Also, I see this commit as part of the longer-term effort to completely expunge
lsb_release from our codebase. See git log --grep lsb_release.
For another example of an external distro-detection tool (albeit one that was
included in Python 2) gone awry, see http://tracker.ceph.com/issues/18163.
Kefu Chai [Thu, 28 Dec 2017 02:18:11 +0000 (10:18 +0800)]
install-deps.sh: check if have access to stdout
$- is "hB" if launched via command line, and it is "himBH" only when
we are actually *in* an interactive shell. so checking "test -t 1" is
what we want.
Kefu Chai [Fri, 8 Dec 2017 08:34:59 +0000 (16:34 +0800)]
install-deps.sh: use DTS on centos if GCC is too old
please note, run-make-check.sh sources install-deps.sh here to import
the $PATH and other environmental variables, which could be changed by
the the DTS "enable" script.
Nathan Cutler [Wed, 2 Oct 2019 12:19:04 +0000 (14:19 +0200)]
qa/rados/upgrade/jewel-x-singleton: do not run on Bionic
The test was failing because it did not explicitly set "os_type" and
"os_version", and teuthology defaulted to Ubuntu Bionic.
Jewel is not supported on Ubuntu Bionic, so we should not try to run tests on
that Ubuntu version.
NOTE: This commit is not cherry-picked from master because the fix is
luminous-specific (there is no "rados/upgrade/jewel-x-singleton" test after
luminous).
qa/rados/upgrade/jewel-x-singleton: set RBD_FEATURES
This commit cannot be cherry-picked from master because luminous is the only
non-EOL stable branch that still includes the
qa/rados/upgrade/jewel-x-singleton suite.
A recent PR made sure that a bucket listing could not request too many
entries at once. It also did a minimum computation for number of
entries. For ordered listing the minimum was 0, as required to pass
all unit tests. However the minimum for unordered listing was left at
1. In order to make ordered and unordered listing behave the same --
with the exception of ordering -- the minimum for unordered listing is
modified to 0.
J. Eric Ivancich [Fri, 19 Jul 2019 20:10:59 +0000 (16:10 -0400)]
rgw: mitigate bucket list with max-entries excessively high
When listing a bucket with radosgw-admin, the user can specify the
maximum number of entries. That number can be unreasonably large, and
can affect the performance and memory availability. For example:
radosgw-admin bucket list --bucket mybucket1 --max-entries=10000000
This has the potential for creating large data structures at multiple
levels in the the call stack of the radosgw(-admin) process,
potentially causing the process to run out of memory. This change
limits the maximum number of entries requested in all but the high
level code to help mitigate this issue.
Conflicts:
src/rgw/rgw_rados.cc
- in luminous, RGWRados::Bucket::List::list_objects_ordered does not take
optional_yield parameter
- no std::string_view in C++11
Jason Dillaman [Thu, 22 Aug 2019 00:20:10 +0000 (20:20 -0400)]
common/config: hold lock while accessing mutable container
The 'call_gate_leave' method was accessing the 'obs_call_gate' map
without holding the required lock. This data structure could be
manipulated by another thread underneath the observer callback
thread context.
Fixes: https://tracker.ceph.com/issues/41354 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0cc2dd7bd1021dd8b5e22850495ce97da7d98716)
Conflicts:
src/common/config_proxy.h: logic incorporated into config.cc
Kefu Chai [Wed, 7 Aug 2019 09:46:13 +0000 (17:46 +0800)]
admin/build-doc: use python3
to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.
in this change:
* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
stable.
ceph-volume: do not fail when trying to remove crypt mapper
In a containerized context, at some point, need to run `simple scan` on a device
from a separate container (not the existing and running corresponding container
to that device), but this can't work because when it tries to remove the
mapper which is still in use by the corresponding running osd container,
it fails.
This can be a bit more permissive and simply throw a warning.
Currently when parsing host in requests, we try to case match against supplied
hostnames set, which violates dns hostnames being case insensitive. Do a case
insensitive comparision instead
This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing
Zengran Zhang [Tue, 20 Aug 2019 07:06:09 +0000 (15:06 +0800)]
osd: clear PG_STATE_CLEAN when repair object
there is a race be found, when we repair object on clean state,
we queue a DoRecovery peering event, but before the peering event
dequeue,a snaptrim event on the missing object's snap dequeue,
then we will get pass the context< SnapTrimmer >().can_trim()
and go to get the context of the missing object(snapdir)
we can avoid this by clear clean state when we found missing..
projects / ceph.git / log