Patrick Donnelly [Mon, 23 Sep 2019 12:15:19 +0000 (05:15 -0700)]
Merge PR #29926 into nautilus
* refs/pull/29926/head:
mgr/volumes: return string type to ceph-manager
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size
mgr/volumes: drop unnecessary size
mgr/volumes: cleanup FS subvolume or subvolume group path
mgr/volumes: give useful error message
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
The 'volumes' ceph-mgr module returns a byte type instead of a string
type as the stdout result of commands, `fs subvolume getpath` and
`fs subvolumegroup getpath` to the ceph-mgr. This causes the commands
to hang and the ceph-mgr to utilize 100% CPU. Fix the 'volumes'
ceph-mgr module to return string type to ceph-mgr as the stdout
result of the commands.
Alfonso Martínez [Thu, 22 Aug 2019 13:33:02 +0000 (15:33 +0200)]
mgr/dashboard: run-backend-api-tests.sh CI improvements
As there is now a jenkins job to run this script
(see https://github.com/ceph/ceph-build/pull/1351),
this refactoring adapt the script to be run in a jenkins job as well as locally.
otherwise we have
```
Running setup.py egg_info for package apache-libcloud
libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
warnings.warn("Missing backports.ssl_match_hostname package")
Traceback (most recent call last):
...
File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated
...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```
and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.
This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing
This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing
cmake: enforce C++17 instead of relying on cmake-compile-features
this change is a follow-up of #30114
in FindBoost.cmake, we try to deduce the C++ flags by checking the C++
features offered by different C++ standard like C++98, C++11 and C++14
against the ones required by different boost component. then choose the
C++ dialect based on the `CXX_STANDARD` and `COMPILE_FEATURES`.
in the case of nautilus, since cmake 3.5.1 does not offer
CXX_STANDARD=17 support by then, we have to add `-std=c++17` to
`CMAKE_CXX_FLAGS` manually. but unfortunately, the
`cmake-compile-features` machinary does not check `CMAKE_CXX_FLAGS` for
the default C++ standard required by the project. so it falls back to
the minimal C++ standard required by the target.
since Boost::fiber and Boost::context share the same C++ feature set
dependencies which can be fulfilled by C++11, and by default,
`CXX_EXTENSIONS` is enabled, `--std=gnu++11` is used as we are using gcc
in our building hosts. so we have CXX_FLAGS like:
-std=c++17 ... -std=gnu++11
when compiling targets linked against Boost::context. and it seems the
last one takes effect. that's why some targets failed to compile if
the makefiles were generated with cmake 3.5.1.
in this change, `INTERFACE_COMPILE_FEATURES <features>` is replaced with
`CXX_STANDARD 17`. this renders this setting a no-op, as CMake 3.5.1
does not translate this setting to any CXX_FLAGS. but this does not
hurt, as we use C++17 project-wide, and C++17 is able to offer all
compile features required by Boost::context.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts: this change is not cherry-picked from master, because in
master, we use cmake 3.10 and up which offers the support of
`CXX_STANDARD=17`, hence cmake is able to bump up the C++ standard
to C++17 instead of falling back to C++11.
Conflicts: dmclock
this change is not cherry-picked from master, because master is using
a different dmclock version. so in this change, only the minimum fix
is included.
so fmt won't fallback to C++11 if `CMAKE_CXX_STANDARD` is not set
Signed-off-by: Kefu Chai <kchai@redhat.com>
this change is not cherry-picked from master, because
- the seastar submodule in nautilus is not in sync with that in
master
- master is not using fmt submodule included by seastar anymore
- master is using cmake 3.10, so there is no need to address
this issue in master on fmt front.
Matt Benjamin [Thu, 8 Aug 2019 20:21:24 +0000 (16:21 -0400)]
rgw_file: dont deadlock in advance_mtime()
Fixes: https://tracker.ceph.com/issues/41173 Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 0d3338c2d2271bbf6f4c7774ad175f7ee42d07f6)
If we allocate a new extent that is contiguous with the last extent,
just extend it. This avoids having long vectors of continguous extents
when a single large extent would suffice--especially with log files.
Sage Weil [Fri, 9 Aug 2019 17:25:59 +0000 (12:25 -0500)]
os/bluestore: do not set osd_memory_target default from cgroup limit
On the aarch64 box I'm testing, this gives us a value of 7378697629483768832, which is not what we want.
I think we are better off relying on this limit being explicitly set via
environment variables (POD_* by kuberentes/rook) or via the command line.
This partially reverts 5c6b533697814af8acfd9e731a2599b2294687ef, but not
all of it, since we wan to keep the option itself, as it is now used by
common/config.cc when dealing with the POD_MEMORY_LIMIT env var.
Jason Dillaman [Thu, 13 Jun 2019 19:22:54 +0000 (15:22 -0400)]
librbd: tweaks to improve throughput for journaled IO
Increase the per-object append flush bytes to 1MiB to mimic the 16MiB
in-memory writeback cache. Finally, default enable the new journal
writethrough until flush option.
Performance results comparing no journal, pre-change, and post-change:
no journal original modified
4K 40310.84 7223.73 27427.27
32K 21530.37 3256.39 10284.05
256K 3159.67 564.73 1627.16
4M 223.03 48.03 91.91
Fixes: http://tracker.ceph.com/issues/40072 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 5f3b043896762fa97eb35648f8639004c45f5978)
Mykola Golub [Thu, 13 Jun 2019 11:05:17 +0000 (12:05 +0100)]
journal: wait for in flight advance sets on stopping recorder
Before object overflow detection optimization (83461c42b) the
overflow was detected when trying to send data, so when an object
set was advanced we always had some data and flush always
completed later. But now we detect overflow on client side, it
may happen when buffer is empty and flush may complete
when "advance object set" is still in flight.
Mykola Golub [Fri, 24 May 2019 06:07:51 +0000 (07:07 +0100)]
journal: optimize object overflow detection
Previously to detect overflow we were sending journal append
requests until -EOVERFLOW is returned by osd. This means that we
had at least one waste (rejected) request per object set (though
there may be more if the number of in-flight appends is not
limited).
We can easily predict when the osd will start to return
-EOVERFLOW and avoid such additional requests.
Jason Dillaman [Tue, 16 Jul 2019 17:54:16 +0000 (13:54 -0400)]
mon/mgr: add 'rbd_support' to list of always-on mgr modules
The 'rbd_support' mgr module provides many background functions that
are required by the 'rbd' CLI performance queries, as well as future
background long-running maintenance tasks.
Fixes: http://tracker.ceph.com/issues/40790 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7231aab4b844729c1c35da37515dce58026c26a2)
Jason Dillaman [Fri, 2 Aug 2019 15:57:29 +0000 (11:57 -0400)]
pybind/mgr/rbd_support: use image ids to detect duplicate tasks
This helps to to avoid the case where new tasks were not being scheduled
when an image name was re-used after having a task created under the
same name.
Fixes: https://tracker.ceph.com/issues/41032 Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7adb17f83106b2cba87cf343c23fd6d2d26ae0b3)
Jason Dillaman [Thu, 25 Jul 2019 16:36:19 +0000 (12:36 -0400)]
pybind/mgr: handle duplicate rbd task commands
The 'ceph' CLI will duplicate commands within teuthology to test
the MONs idempotency. This shouldn't be required for the MGR module,
but we can keep a fixed set of completed tests to handle this
possible command replay.
projects / ceph.git / log