Sage Weil [Tue, 16 Apr 2019 13:53:56 +0000 (08:53 -0500)]
ceph_test_objectstore: add very_large_write test
Write 2GB to verify the blockdevice aio splitting.
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1974f011137976edf277833d68bdcc4ff22779b0)
- fixed get_val/set_val calls
- add_block_device doesn't take third arg in luminous
On Linux, write() (and similar system calls) will transfer at most
0x7ffff000 (2,147,479,552) bytes, it will cap data if aio pwritev
more than 0x7ffff000, so we have the split the data to more aio submit.
Sage Weil [Wed, 10 Jul 2019 15:21:12 +0000 (10:21 -0500)]
bluestore: print aio in batch
KernelDevice::aio_{submit,write,read}() are critical paths. calling
cct->_conf->subsys.should_gather() multi-times is not optimal. the
downside of this issue is that if the aio is printed, the size of
buffer in PrebufferedStreambuf could be large if the number of iov is
large, that could be a heavy load to the memory subsystem.
Conflicts:
src/mon/OSDMonitor.cc:
1. replace g_conf() with g_conf, g_conf was replaced
with g_conf() to prepare for the crimson project. and that change was
not backported to luminous. so restore all `g_conf()` to `g_conf` in
this backport.
2. replace std::min() and std::max() with MIN() and
MAX() macros, as we modernize it in an unrelated change. and that
change was not backported to luminous. so let's be conservative, and
restore to the old style macros.
Conflicts:
src/common/config_values.h: the changes in this file is applied
to src/common/config.h instead. because, src/common/config_values.h
is extracted in 4718b7cb2fac65b2ac7014f42ba1a10181350f0b. which
happened after luminous, and that commit was not backported.
also, the OPT_SIZE is changed to OPT_U64, as we don't have OPT_SIZE by
then, the size_t option support was added after luminous, and that
change was not backported. so let's use the uint64_t as an alternative,
as other *bytes settings are also using OPT_U64. and OSDMonitor.cc
will be accessing that setting using the legacy way instead of using
conf.get_val<>() interface, so this change will have no impact on the
fix.
simon gao [Tue, 28 May 2019 02:36:32 +0000 (22:36 -0400)]
mds: avoid sending too many osd requests at once after mds restarts Fixes: http://tracker.ceph.com/issues/40028 Signed-off-by: simon gao <simon29rock@gmail.com>
(cherry picked from commit 78484352f553d6b67f7d5c82384a186455f05537)
Yan, Zheng [Sat, 8 Jun 2019 05:08:21 +0000 (13:08 +0800)]
mds: fix corner case of replaying open sessions
Marking a session dirty may flush all existing dirty sessions. MDS
calls Server::finish_force_open_sessions() for log event that opens
multiple sessions. The function marks sessions dirty one by one. So
sessions opened by a log event may get flushed partially.
When replaying a log event that opens multiple sessions, mds need to
check if some of these sessions have already been flushed.
Kefu Chai [Tue, 11 Jun 2019 15:17:46 +0000 (23:17 +0800)]
qa: install python3-{cephfs,rados} instead of python34-*
we install the latest python-rpm-macros on all builders since
https://github.com/ceph/ceph-build/pull/1283 . now that we started
building python36-* after that change, for testing the python3 packages on
CentOS/RHEL 7, we need to install python36-* instead of python34-*.
and after the change of 8ae1947, python36-* now "Provides" python3-*, we
can just install python3-* for fulfill the requirement for testing
python3 cephfs bindings.
Fixes: http://tracker.ceph.com/issues/39164 Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts: this change is not cherry-picked from master, because,
in master, we don't install python3 packages after 7e5c85b604.
(cherry picked from commit 6790821afc749b14b1ddac68a0889059419eebb3)
qa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests
EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283
rpm: add "Provides: python3-*" for python packages
so user can install python3-rados, instead of python36-rados, without
specifying the minor version of python. also, we should not break our
teuthology tests with this naming scheme change. for instance, our
cephfs qa suite installs `python3-cephfs` for testing the `cephfs-shell`
some of our centos7 jenkins builders are failing to build ceph master and
nautilus branches. because EPEL7 recently switched from python3.4 to
python3.6 as the native python3. see
https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedoraproject.org/message/EGUMKAIMPK2UD5VSHXM53BH2MBDGDWMO/
and one of our BuildRequires, cmake3,
was offered by EPEL7. it also followed the python3.6 switch-over to
rebuild against python3.6. as a result, the cmake3-data-3.13.4-2.el7
started to depend on /usr/bin/python3.6, which is in turn offered by
python36 package. after installing python36 as a dependency of the
updated cmake3. but in cmake, we originally checks for the latest
python3 interpreter if WITH_PYTHON3 is enabled, that's why these
builders which happen to install these updated packages started to fail
when detecting the existence of python3.6 related build dependencies.
as a fix, in d1e83082,
python%{python3_pkgversion}-{devel,setuptools,Cython} are listed as
BuildRequires to reflect this change in EPEL7. before d1e83082, we
hardwired them to python34-*.
but as following analysis puts, there are cases where `yum-builddep`
is inconsistent with `rpmbuild`. as `yum-builddep` changes the how
`python3_pkgversion` and `python3_version` macros are expanded:
- none of the packages installed by `yum-builddep` installs the python3
related rpm macros, so the system stays with whatever python3 it was
using. in this case, `rpmbuild` won't complain, as the
`python3_pkgversion` and `python_version` are consistent before and
after `yum-builddep`.
- system has python3.4 installed before `yum-builddep`. but
`yum-builddep` installed python3.6 and also the updated
`python-rpm-macros` packages, which points `python3_version` and
`python3_pkgversion` to 3.6 and 36 respectively. in this case,
`rpmbuild` will complain, because when we run `yum-builddep`,
`python3_version` was still "3.4".
- system does not have python3 installed before `yum-builddep`. so
it was using python34 for preparing the "BuildRequires". but some
of the packages installed by `yum-builddep` installs python36, and
also the updated `python-rpm-macros` packages, which points
`python3_version` and `python3_pkgversion` to 3.6 and 36 respectively.
in this case, `rpmbuild` will complain, because the python36 related
dependencies are missing. what the system has is python34
dependencies.
- system does not have python3 installed before `yum-builddep`. so
it was using python34 for preparing the "BuildRequires". but some
of the packages installed by `yum-builddep` installs python34, and
also the updated `python-rpm-macros` packages, which points
`python3_version` and `python3_pkgversion` to 3.4 and 34 respectively.
in this case, `rpmbuild` won't complain, as the
`python3_pkgversion` and `python_version` are also consistent before and
after `yum-builddep`.
as we cannot tell if the system has python3 or what the python3 version
the system has before `yum-builddep`, so what we can do is to ensure
`rpmbuild` has what it needs to build Ceph. so let's just stick with
python3.6.
to force cmake to use the python3 and python3 modules for building
python3 bindings
on the debian side, it's okay to continue using "-DWITH_PYTHON3=ON", as
- cmake does normalize "ON" to 3
- debian's cmake extension lives on /usr/lib/python3/dist-packages/
not in a specific /usr/lib/python3.x/dist-packages directory
use might have multiple python3 installed, some of them has/have all
dependencies installed and is good enough for building Ceph. we should
not always use the latest python installed in the system and complain that
there is missing dependencies, even if user has installed all the
python3 dependencies for the older python3.
put in other words, if user only installs cython module for python3.4, but
she has both python3.6 and python3.4 in her system. we should not force
her to uninstall python3.6 for installing Ceph.
this change also aligns with MGR_PYTHON_VERSION. i am not applying the
same change to WITH_PYTHON2, because python2 is already stablized. and distros
are not likely to release new python2 releases.
Conflicts:
src/CMakeLists.txt: in luminous, WITH_PYTHON3 was "CHECK" by
default. as it's complicatd to support this behavior. it is changed to
"ON" in this change to be consistent with mimic and up. since we always
specify -DWITH_PYTHON3=ON explicitly when building rpm and deb packages,
this change is not visible to our CI or package builders.
huanwen ren [Wed, 20 Mar 2019 15:46:08 +0000 (23:46 +0800)]
mds: there is an assertion when calling Beacon::shutdown()
If you run MDSDaemon::init(), an exception occurs that causes MDSDaemon::suicide()--->Beacon::shutdown()--->sender.join();
this problem occurs because sender is just Default-constructed is in Beacon.h(std::thread sender;), there is no call
to Beacon::init() to construct the sender, so the sender has no "joinable"
projects / ceph.git / log