]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
6 years agoceph-volume: skip missing interpreters when running tox tests 28519/head
Andrew Schoen [Fri, 10 May 2019 16:04:35 +0000 (11:04 -0500)]
ceph-volume: skip missing interpreters when running tox tests

This will skip tests for versions of python that don't exist on the node
that's running the tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f1db0a864814d9b81192d89b6eebb6c624ddec3b)

6 years agoceph-volume: use the Device.rotational property instead of sys_api
Andrew Schoen [Fri, 10 May 2019 14:02:08 +0000 (09:02 -0500)]
ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 8ebff475dfa136c88101ef7364820fe3c5cee547)

6 years agoMerge pull request #27453 from pdvian/wip-38958-luminous
Yuri Weinstein [Wed, 12 Jun 2019 15:41:30 +0000 (08:41 -0700)]
Merge pull request #27453 from pdvian/wip-38958-luminous

luminous: rgw-admin: fix data sync report for master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #27668 from smithfarm/wip-39360-luminous
Yuri Weinstein [Wed, 12 Jun 2019 15:40:27 +0000 (08:40 -0700)]
Merge pull request #27668 from smithfarm/wip-39360-luminous

luminous: rgw: failed to pass test_bucket_create_naming_bad_punctuation in s3test

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #27881 from pdvian/wip-39227-luminous
Yuri Weinstein [Wed, 12 Jun 2019 15:39:44 +0000 (08:39 -0700)]
Merge pull request #27881 from pdvian/wip-39227-luminous

luminous: rgw_file: save etag and acl info in setattr

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28314 from smithfarm/wip-39180-luminous
Yuri Weinstein [Wed, 12 Jun 2019 15:39:03 +0000 (08:39 -0700)]
Merge pull request #28314 from smithfarm/wip-39180-luminous

luminous: rgw: orphans find perf improvments

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #28493 from tchaikov/luminous-39164
Yuri Weinstein [Tue, 11 Jun 2019 23:08:51 +0000 (16:08 -0700)]
Merge pull request #28493 from tchaikov/luminous-39164

luminous: qa: install python36-{cephfs,rados} instead of python34-*

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoqa: install python3-{cephfs,rados} instead of python34-* 28493/head
Kefu Chai [Tue, 11 Jun 2019 15:17:46 +0000 (23:17 +0800)]
qa: install python3-{cephfs,rados} instead of python34-*

we install the latest python-rpm-macros on all builders since
https://github.com/ceph/ceph-build/pull/1283 . now that we started
building python36-* after that change, for testing the python3 packages on
CentOS/RHEL 7, we need to install python36-* instead of python34-*.

and after the change of 8ae1947, python36-* now "Provides" python3-*, we
can just install python3-* for fulfill the requirement for testing
python3 cephfs bindings.

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts: this change is not cherry-picked from master, because,
  in master, we don't install python3 packages after 7e5c85b604.
(cherry picked from commit 6790821afc749b14b1ddac68a0889059419eebb3)

6 years agoinstall-deps.sh: install '*rpm-macros'
Kefu Chai [Thu, 11 Apr 2019 16:11:42 +0000 (00:11 +0800)]
install-deps.sh: install '*rpm-macros'

so `yum-builddep` can have access to the latest macros for preparing the
build dependencies

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 087ea813a061a1d6858aeae57950e90965f4ae15)

Conflicts:
install-deps.sh: trivial resolution

6 years agoqa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests
Kefu Chai [Thu, 11 Apr 2019 02:02:32 +0000 (10:02 +0800)]
qa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests

EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 42167b87909c20c9bb4d32bac02b1e01ec94af13)

6 years agoPendingReleaseNotes: note on python3.6 changes
Kefu Chai [Wed, 10 Apr 2019 10:08:04 +0000 (18:08 +0800)]
PendingReleaseNotes: note on python3.6 changes

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 77c5ee06311d610f9accd94edf14becdf5820542)

Conflicts:
PendingReleaseNotes: the next release of luminous is 12.2.13
so adapt the cherry-picked change accordingly.

6 years agorpm: add "Provides: python3-*" for python packages
Kefu Chai [Tue, 9 Apr 2019 15:22:57 +0000 (23:22 +0800)]
rpm: add "Provides: python3-*" for python packages

so user can install python3-rados, instead of python36-rados, without
specifying the minor version of python. also, we should not break our
teuthology tests with this naming scheme change. for instance, our
cephfs qa suite installs `python3-cephfs` for testing the `cephfs-shell`

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8ae1947728837bf4a4bcc8e9325aea69ff14b706)

6 years agorpm: always use 3 as the default python3_{pkgversion,version}
Kefu Chai [Tue, 9 Apr 2019 15:15:16 +0000 (23:15 +0800)]
rpm: always use 3 as the default python3_{pkgversion,version}

as we have the rpm macros installed in
https://github.com/ceph/ceph-build/pull/1283 .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 11f0b8d5a8951cb50437f5e87055f3b9405f4556)

6 years agorpm: use python 3.6 as the default python3
Kefu Chai [Sun, 7 Apr 2019 00:43:59 +0000 (08:43 +0800)]
rpm: use python 3.6 as the default python3

some of our centos7 jenkins builders are failing to build ceph master and
nautilus branches. because EPEL7 recently switched from python3.4 to
python3.6 as the native python3. see
https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedoraproject.org/message/EGUMKAIMPK2UD5VSHXM53BH2MBDGDWMO/

and one of our BuildRequires, cmake3,
was offered by EPEL7. it also followed the python3.6 switch-over to
rebuild against python3.6. as a result, the cmake3-data-3.13.4-2.el7
started to depend on /usr/bin/python3.6, which is in turn offered by
python36 package. after installing python36 as a dependency of the
updated cmake3. but in cmake, we originally checks for the latest
python3 interpreter if WITH_PYTHON3 is enabled, that's why these
builders which happen to install these updated packages started to fail
when detecting the existence of python3.6 related build dependencies.

as a fix, in d1e83082,
python%{python3_pkgversion}-{devel,setuptools,Cython} are listed as
BuildRequires to reflect this change in EPEL7. before d1e83082, we
hardwired them to python34-*.

but as following analysis puts, there are cases where `yum-builddep`
is inconsistent with `rpmbuild`. as `yum-builddep` changes the how
`python3_pkgversion` and `python3_version` macros are expanded:

- none of the packages installed by `yum-builddep` installs the python3
  related rpm macros, so the system stays with whatever python3 it was
  using. in this case, `rpmbuild` won't complain, as the
  `python3_pkgversion` and `python_version` are consistent before and
  after `yum-builddep`.
- system has python3.4 installed before `yum-builddep`. but
  `yum-builddep` installed python3.6 and also the updated
  `python-rpm-macros` packages, which points `python3_version` and
  `python3_pkgversion` to 3.6 and 36 respectively. in this case,
  `rpmbuild` will complain, because when we run `yum-builddep`,
  `python3_version` was still "3.4".
- system does not have python3 installed before `yum-builddep`. so
  it was using python34 for preparing the "BuildRequires". but some
  of the packages installed by `yum-builddep` installs python36, and
  also the updated `python-rpm-macros` packages, which points
  `python3_version` and `python3_pkgversion` to 3.6 and 36 respectively.
  in this case, `rpmbuild` will complain, because the python36 related
  dependencies are missing. what the system has is python34
  dependencies.
- system does not have python3 installed before `yum-builddep`. so
  it was using python34 for preparing the "BuildRequires". but some
  of the packages installed by `yum-builddep` installs python34, and
  also the updated `python-rpm-macros` packages, which points
  `python3_version` and `python3_pkgversion` to 3.4 and 34 respectively.
  in this case, `rpmbuild` won't complain, as the
  `python3_pkgversion` and `python_version` are also consistent before and
  after `yum-builddep`.

as we cannot tell if the system has python3 or what the python3 version
the system has before `yum-builddep`, so what we can do is to ensure
`rpmbuild` has what it needs to build Ceph. so let's just stick with
python3.6.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 658ca2b0d79022edbc6cb1e488b243844e87bfbe)

6 years agocmake: should PYTHON3_VERSION_STRING of libpython3
Kefu Chai [Fri, 5 Apr 2019 17:54:15 +0000 (01:54 +0800)]
cmake: should PYTHON3_VERSION_STRING of libpython3

and make sure the version matches exactly the requested one

in future, we should use FindPython.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7b15b682b1201929e0ba9efc332ebb8652be40a8)

Conflicts:
src/CMakeLists.txt: trivial resolution

6 years agorpm: pass %{python3_version} to cmake
Kefu Chai [Thu, 4 Apr 2019 04:07:25 +0000 (12:07 +0800)]
rpm: pass %{python3_version} to cmake

to force cmake to use the python3 and python3 modules for building
python3 bindings

on the debian side, it's okay to continue using "-DWITH_PYTHON3=ON", as

- cmake does normalize "ON" to 3
- debian's cmake extension lives on /usr/lib/python3/dist-packages/
  not in a specific /usr/lib/python3.x/dist-packages directory

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7681a38ce51743a19ec5679d49b362f04e271ddb)

Conflicts:
ceph.spec.in: trivial resolution

6 years agocmake: check for libpython of the same version of interpreter
Kefu Chai [Thu, 4 Apr 2019 03:44:56 +0000 (11:44 +0800)]
cmake: check for libpython of the same version of interpreter

actually cython or python3-cython's dependencies should take care of it.
but we should at least get it right on our side if we check it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c961e002d22ab7ceec20f1019f8c558ec9fcce6c)

6 years agocmake: use specified python3 version if any
Kefu Chai [Thu, 4 Apr 2019 03:37:00 +0000 (11:37 +0800)]
cmake: use specified python3 version if any

use might have multiple python3 installed, some of them has/have all
dependencies installed and is good enough for building Ceph. we should
not always use the latest python installed in the system and complain that
there is missing dependencies, even if user has installed all the
python3 dependencies for the older python3.

put in other words, if user only installs cython module for python3.4, but
she has both python3.6 and python3.4 in her system. we should not force
her to uninstall python3.6 for installing Ceph.

this change also aligns with MGR_PYTHON_VERSION. i am not applying the
same change to WITH_PYTHON2, because python2 is already stablized. and distros
are not likely to release new python2 releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d8c13e86048b7aac458f3eda101f9778e3d45dcf)

Conflicts:
src/CMakeLists.txt: in luminous, WITH_PYTHON3 was "CHECK" by
default. as it's complicatd to support this behavior. it is changed to
"ON" in this change to be consistent with mimic and up. since we always
specify -DWITH_PYTHON3=ON explicitly when building rpm and deb packages,
this change is not visible to our CI or package builders.

6 years agoMerge pull request #27728 from neha-ojha/wip-39420-luminous
Yuri Weinstein [Mon, 10 Jun 2019 22:06:58 +0000 (15:06 -0700)]
Merge pull request #27728 from neha-ojha/wip-39420-luminous

luminous: osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #27858 from ashishkumsingh/wip-39463-luminous
Yuri Weinstein [Mon, 10 Jun 2019 22:06:06 +0000 (15:06 -0700)]
Merge pull request #27858 from ashishkumsingh/wip-39463-luminous

luminous: msg: output peer address when detecting bad CRCs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
6 years agoMerge pull request #28390 from yuriw/wip-yuriw-whitelist-luminous_2
Yuri Weinstein [Tue, 4 Jun 2019 20:11:09 +0000 (13:11 -0700)]
Merge pull request #28390 from yuriw/wip-yuriw-whitelist-luminous_2

qa/tests: whitelisted 'application not enabled' and removed bad dir

6 years agoqa/tests: whitelisted 'application not enabled' and removed bad dir 28390/head
Yuri Weinstein [Tue, 4 Jun 2019 19:50:45 +0000 (12:50 -0700)]
qa/tests: whitelisted 'application not enabled' and removed bad dir

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #28355 from yuriw/wip-yuriw-whitelist-luminous
Yuri Weinstein [Sun, 2 Jun 2019 20:32:41 +0000 (13:32 -0700)]
Merge pull request #28355 from yuriw/wip-yuriw-whitelist-luminous

qa/tests: whitelisted POOL_APP_NOT_ENABLED

6 years agoqa/tests: whitelisted POOL_APP_NOT_ENABLED 28355/head
Yuri Weinstein [Fri, 31 May 2019 21:01:49 +0000 (14:01 -0700)]
qa/tests: whitelisted POOL_APP_NOT_ENABLED

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #28130 from dillaman/wip-39727-luminous
Yuri Weinstein [Thu, 30 May 2019 20:38:49 +0000 (13:38 -0700)]
Merge pull request #28130 from dillaman/wip-39727-luminous

luminous: qa/workunits/rbd: use https protocol for devstack git operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
6 years agorgw-admin: add a detailed mode 28314/head
Abhishek Lekshmanan [Thu, 14 Feb 2019 16:40:44 +0000 (17:40 +0100)]
rgw-admin: add a detailed mode

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 143ff0d835c99b06e57a373bed2c61e84293ccc7)

 Conflicts:
src/rgw/rgw_admin.cc
Trivial merge conflct due to OTP changeset in Mimic

6 years agorgw: orphans find: don't process stale bucket instances
Abhishek Lekshmanan [Wed, 13 Feb 2019 14:32:17 +0000 (15:32 +0100)]
rgw: orphans find: don't process stale bucket instances

As a large bucket might have resharded multiple times, check the cur bucket info
and ensure that no reshard is in progress before we attempt to log bucket index
entries. On a large sized bucket, since a bucket would have undergone reshard
multiple times, this avoids wasteful processing of stale bucket instance entries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 5436ff7fa99061a489f089677c03f63c76bb954b)

 Conflicts:
src/rgw/rgw_orphan.cc
sys_objctx dropped as obj_ctx changes are a part of Nautilus. Also similarly
with the includes, only `rgw_bucket.h` is included

6 years agorgw: orphan: introduce a detailed mode (off by default)
Abhishek Lekshmanan [Wed, 13 Feb 2019 12:45:47 +0000 (13:45 +0100)]
rgw: orphan: introduce a detailed mode (off by default)

We currently stat objects that fit in a head as well and also log them, since we
skip head objects anyway in the rados list output this commit avoids logging
these objects if the object size itself is less than the manifest head size.

Additionally we avoid the stat call itself from the list object output when the
object fits within the chunk size. This behaviour can be unset by setting the
detailed mode which can help in older clusters where the head used to have a
different size.

The old behaviour in both the cases can be turned on by setting the detailed
flag which can be passed on from rgw-admin. Avoiding stat calls and not logging
the head objects significantly reduces the IO activity on clusters which have a
huge percentage of objects that fit in a head.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit ca12ecbfed0f65f206e8ad05125fac93c5a5ed0f)

6 years agorgw: orphan: flush bucket index entries only after map is full
Abhishek Lekshmanan [Wed, 13 Feb 2019 12:32:43 +0000 (13:32 +0100)]
rgw: orphan: flush bucket index entries only after map is full

We otherwise may flush more regularly while we don't have enough entries in the
map

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit e9a406c9b036e4c3190077d57fc51d6ea3762475)

6 years agorgw: orphans tool: align with rgw list bucket min readahead
Abhishek Lekshmanan [Wed, 13 Feb 2019 12:15:02 +0000 (13:15 +0100)]
rgw: orphans tool: align with rgw list bucket min readahead

At rgw::rados layer we read upto `min readahead` entries anyway and then pass on
only the requested amount to the caller. Since this translates down to a cls
call requesting a 1000 omap keys by default, it makes sense not to waste the
entries, and process them

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 4c033bf0fd0429378ac350eecd78530cc1c11b84)

6 years agoMerge pull request #28183 from yuriw/wip-yuriw-fix-distro-luminous
Yuri Weinstein [Tue, 28 May 2019 22:49:51 +0000 (15:49 -0700)]
Merge pull request #28183 from yuriw/wip-yuriw-fix-distro-luminous

qa/tests: cleaned up supported distro

6 years agoqa/tests: cleaned up supported distro 28183/head
Yuri Weinstein [Mon, 20 May 2019 20:11:37 +0000 (13:11 -0700)]
qa/tests: cleaned up supported distro

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #28117 from yuriw/wip-yuriw-clean-luminous
Yuri Weinstein [Tue, 28 May 2019 21:38:07 +0000 (14:38 -0700)]
Merge pull request #28117 from yuriw/wip-yuriw-clean-luminous

luminous: qa/test: clean up

6 years agoMerge pull request #28216 from smithfarm/wip-40004-luminous
Nathan Cutler [Tue, 28 May 2019 09:41:54 +0000 (11:41 +0200)]
Merge pull request #28216 from smithfarm/wip-40004-luminous

luminous: do_cmake.sh: source not found

Reviewed-by: Sebastian Wagner <swagner@suse.com>
6 years agodo_cmake.sh: use bash 28216/head
Nathan Cutler [Mon, 20 May 2019 15:01:40 +0000 (17:01 +0200)]
do_cmake.sh: use bash

Fixes: http://tracker.ceph.com/issues/39981
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit ac8d7c5fd4045d79ab6caacc0b72d64866fcc5c8)

6 years agoqa/workunits/rbd: use https protocol for devstack git operations 28130/head
Jason Dillaman [Thu, 9 May 2019 19:48:30 +0000 (15:48 -0400)]
qa/workunits/rbd: use https protocol for devstack git operations

Fixes: http://tracker.ceph.com/issues/39656
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fb4f9a8a08edd18b1a23e1be4c1285d0ec4d1de6)

6 years agoqa/test: clean up 28117/head
Yuri Weinstein [Wed, 15 May 2019 16:34:59 +0000 (09:34 -0700)]
qa/test: clean up

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #28115 from yuriw/wip-yuriw-clients-upgrades-luminous-2-nautilus 28143/head
Yuri Weinstein [Wed, 15 May 2019 16:27:15 +0000 (09:27 -0700)]
Merge pull request #28115 from yuriw/wip-yuriw-clients-upgrades-luminous-2-nautilus

qa/tests: renamed suite

6 years agoqa/tests: renamed suite 28115/head
Yuri Weinstein [Wed, 15 May 2019 16:20:47 +0000 (09:20 -0700)]
qa/tests: renamed suite

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoqa/tests: renamed suite to clients-upgrades-luminous-nautilus to be consistent
Yuri Weinstein [Fri, 10 May 2019 16:19:52 +0000 (09:19 -0700)]
qa/tests: renamed suite to clients-upgrades-luminous-nautilus to be consistent

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #28066 from yuriw/wip-yuriw-clients-upgrades-luminous-2-nautilus
Yuri Weinstein [Fri, 10 May 2019 16:38:53 +0000 (09:38 -0700)]
Merge pull request #28066 from yuriw/wip-yuriw-clients-upgrades-luminous-2-nautilus

qa/tests: renamed suite to clients-upgrades-luminous-nautilus to be c…

6 years agoqa/tests: renamed suite to clients-upgrades-luminous-nautilus to be consistent 28066/head
Yuri Weinstein [Fri, 10 May 2019 16:19:52 +0000 (09:19 -0700)]
qa/tests: renamed suite to clients-upgrades-luminous-nautilus to be consistent

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #27934 from yuriw/wip-yuriw-clients-upgrades-luminous-2-nautilus
Josh Durgin [Thu, 9 May 2019 22:48:17 +0000 (15:48 -0700)]
Merge pull request #27934 from yuriw/wip-yuriw-clients-upgrades-luminous-2-nautilus

qa/tests: new suite to test `luminous` clients on `nautilus` cluster

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoqa/tests: new suite to test `luminous` clients on `nautilus` cluster 27934/head
Yuri Weinstein [Thu, 2 May 2019 20:17:13 +0000 (13:17 -0700)]
qa/tests: new suite to test `luminous` clients on `nautilus` cluster

This suite will run on `nautilus` relases only and we will keep running older suite `client-upgrade-luminous`
on previous releases

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agoMerge pull request #27910 from yuriw/wip-yuriw-fix-clients-upgrates-luminous 27768/head
Yuri Weinstein [Wed, 1 May 2019 23:38:58 +0000 (16:38 -0700)]
Merge pull request #27910 from yuriw/wip-yuriw-fix-clients-upgrates-luminous

qa/tests:  Excluded  "'python34-cephfs','python34-rados'" for runs on nautilus

6 years agoqa/tests: Excluded "'python34-cephfs','python34-rados'" for runs on nautilus 27910/head
Yuri Weinstein [Wed, 1 May 2019 20:08:38 +0000 (13:08 -0700)]
qa/tests:  Excluded  "'python34-cephfs','python34-rados'" for runs on nautilus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
6 years agorgw_file: save etag and acl info in setattr 27881/head
taoCH [Fri, 15 Feb 2019 03:57:29 +0000 (19:57 -0800)]
rgw_file: save etag and acl info in setattr

Fixes: https://tracker.ceph.com/issues/38327
Signed-off-by: Tao Chen <chentao@umcloud.com>
(cherry picked from commit f9f4e996ebf4cce79a45196d8aa796b07e03b7f7)

6 years agomsg: output peer address when detecting bad CRCs 27858/head
Greg Farnum [Wed, 17 Apr 2019 22:27:22 +0000 (15:27 -0700)]
msg: output peer address when detecting bad CRCs

Fixes: http://tracker.ceph.com/issues/39367
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 19679231fc32daa405cc060847a02ac1cfd51300)

6 years agoosd/OSDMap: Replace get_out_osds with get_out_existing_osds 27728/head
Brad Hubbard [Wed, 17 Apr 2019 06:13:27 +0000 (16:13 +1000)]
osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Fixes: http://tracker.ceph.com/issues/39154
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit adfb6a595363e2c085c6eac3001fcaa2dce007aa)

 Conflicts:
src/osd/OSDMap.h - Trivial resolution - no std::

6 years agorgw: s3: awsv4 drop special handling for x-amz-credential 27668/head
Abhishek Lekshmanan [Mon, 20 Aug 2018 15:45:04 +0000 (17:45 +0200)]
rgw: s3: awsv4 drop special handling for x-amz-credential

While s3 docs mention that every byte must be urlencoded, they are relaxed in
its implementation, when testing this behaviour on aws s3 itself, they seem to
be relaxed in handling aws credentials of the form

X-Amz-Credential=access1/20180817T123456Z/us-east-1...

wherein

X-Amz-Credential=access1%2F2018...

is expected. This allows for clients requesting the urls in both forms to
succeed as aws s3 does at the moment

Fixes: http://tracker.ceph.com/issues/26965
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 25e1d2a32074254e9ebd780f3a41b2ff6469370f)

6 years agorgw: should recode canonical_uri when caculate s3v4 auth
yuliyang [Sat, 7 Apr 2018 10:58:55 +0000 (06:58 -0400)]
rgw: should recode canonical_uri when caculate s3v4 auth

fix: http://tracker.ceph.com/issues/23587

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 919a8976cf40866e7212c5bb2ff38a9c5f8a1f75)

6 years ago12.2.12 v12.2.12
Jenkins Build Slave User [Thu, 11 Apr 2019 12:33:50 +0000 (12:33 +0000)]
12.2.12

6 years agorgw-admin: fix data sync report for master zone 27453/head
cfanz [Wed, 5 Sep 2018 02:57:35 +0000 (10:57 +0800)]
rgw-admin: fix data sync report for master zone

Signed-off-by: cfanz <songxinying@sensetime.com>
(cherry picked from commit 6e3be2461b13e24fa78df39bb0084cbf4742bac0)

6 years agoMerge pull request #26989 from ceph/backport-luminous-26957
Yuri Weinstein [Mon, 8 Apr 2019 19:44:57 +0000 (12:44 -0700)]
Merge pull request #26989 from ceph/backport-luminous-26957

luminous: ceph-volume: look for rotational data in lsblk

Reviewed-by: Alfredo Deza <adeza@redhat.com>
6 years agoMerge pull request #27312 from ifed01/wip-ifed-fix-bmap-alloc-luminous 27364/head
Yuri Weinstein [Wed, 3 Apr 2019 20:22:15 +0000 (13:22 -0700)]
Merge pull request #27312 from ifed01/wip-ifed-fix-bmap-alloc-luminous

luminous: os/bluestore: bitmap allocator might fail to return contiguous chunk despite having enough space

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #26475 from pdvian/wip-38338-luminous
Yuri Weinstein [Wed, 3 Apr 2019 20:18:40 +0000 (13:18 -0700)]
Merge pull request #26475 from pdvian/wip-38338-luminous

luminous: tests: run-standalone.sh set local library location so mgr can find li…

Reviewed-by: David Zafman <dzafman@redhat.com>
6 years agoMerge pull request #27322 from liewegas/wip-valgrind-luminous
Sage Weil [Wed, 3 Apr 2019 14:57:12 +0000 (09:57 -0500)]
Merge pull request #27322 from liewegas/wip-valgrind-luminous

luminous: qa/valgrind.supp: backport valgrind suppressions

6 years agoMerge pull request #26191 from cbodley/wip-38081-luminous
Yuri Weinstein [Tue, 2 Apr 2019 20:04:41 +0000 (13:04 -0700)]
Merge pull request #26191 from cbodley/wip-38081-luminous

luminous: rgw: fetch_remote_obj filters out olh attrs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #26549 from cbodley/wip-38412-luminous
Yuri Weinstein [Tue, 2 Apr 2019 20:03:40 +0000 (13:03 -0700)]
Merge pull request #26549 from cbodley/wip-38412-luminous

luminous: rgw: fix rgw_data_sync_info::json_decode()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #27295 from theanalyst/luminous-sse-c-backport
Yuri Weinstein [Tue, 2 Apr 2019 20:02:38 +0000 (13:02 -0700)]
Merge pull request #27295 from theanalyst/luminous-sse-c-backport

luminous: sse-c-fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #27299 from cbodley/wip-39073
Yuri Weinstein [Tue, 2 Apr 2019 20:01:44 +0000 (13:01 -0700)]
Merge pull request #27299 from cbodley/wip-39073

luminous: rgw multisite: data sync checks empty next_marker for datalog

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #26064 from pdvian/wip-37987-luminous
Yuri Weinstein [Tue, 2 Apr 2019 15:59:28 +0000 (08:59 -0700)]
Merge pull request #26064 from pdvian/wip-37987-luminous

luminous: core: Objecter::calc_op_budget: Fix invalid access to extent union member

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
6 years agoqa/valgrind.supp: backport valgrind suppressions 27322/head
Sage Weil [Tue, 2 Apr 2019 15:57:31 +0000 (10:57 -0500)]
qa/valgrind.supp: backport valgrind suppressions

in the correction location

Signed-off-by: Sage Weil <sage@redhat.com>
6 years agoMerge pull request #27173 from pdvian/wip-38755-luminous
Yuri Weinstein [Tue, 2 Apr 2019 15:44:44 +0000 (08:44 -0700)]
Merge pull request #27173 from pdvian/wip-38755-luminous

luminous: rgw: ldap: fix LDAPAuthEngine::init() when uri !empty()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #27180 from pdvian/wip-38771-luminous
Yuri Weinstein [Tue, 2 Apr 2019 15:44:05 +0000 (08:44 -0700)]
Merge pull request #27180 from pdvian/wip-38771-luminous

luminous : rgw: nfs: skip empty (non-POSIX) path segments

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoos/bluestore: unconditionally cap chunks returned by allocator to 2^31 27312/head
Igor Fedotov [Wed, 20 Mar 2019 18:10:04 +0000 (21:10 +0300)]
os/bluestore: unconditionally cap chunks returned by allocator to 2^31

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 1337443e37ffbfcaf494bdb2bb80db4cf41b97f9)

6 years agoos/bluestore: start using 64-bit intervals for bitmap allocator
Igor Fedotov [Wed, 13 Mar 2019 16:09:40 +0000 (19:09 +0300)]
os/bluestore: start using 64-bit intervals for bitmap allocator

This helps to avoid 4Gb+ values wrapping which resulted in
allocation failures.

Fixes: https://tracker.ceph.com/issues/38761
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit d1466963a60ed2345a6fc159071ec09ce422b656)

6 years agoos/bluestore: make bluestore interval base template.
Igor Fedotov [Wed, 13 Mar 2019 15:51:46 +0000 (18:51 +0300)]
os/bluestore: make bluestore interval base template.

We might need intervals at BlueStore with both 32-bit and 64-bit
lengths. Having base template eliminates the need for copy/paste

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f775d4227fb9432840c795115669766565c5e718)

6 years agotests/fastbmap_alloc: UT to reproduce 4G allocation bug
Igor Fedotov [Wed, 13 Mar 2019 14:02:13 +0000 (17:02 +0300)]
tests/fastbmap_alloc: UT to reproduce 4G allocation bug

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ed8a93ba387458961444913a8695f538595efacf)

6 years agoos/bluestore: os/bluestore: implement dump for bitmap allocator
Igor Fedotov [Tue, 12 Mar 2019 11:04:17 +0000 (14:04 +0300)]
os/bluestore: os/bluestore: implement dump for bitmap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 80310d98a7456afdb2002eaf2891853ea21eaf68)

6 years agorgw: fixup_range() clamps offsets to valid part range 27295/head
Casey Bodley [Wed, 27 Mar 2019 17:52:59 +0000 (13:52 -0400)]
rgw: fixup_range() clamps offsets to valid part range

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 4b3bac20d550cae274c07a1d555751483dde463b)

6 years agotest_rgw_crypto: add invalid ranges beyond obj boundary
Abhishek Lekshmanan [Wed, 27 Mar 2019 12:45:27 +0000 (13:45 +0100)]
test_rgw_crypto: add invalid ranges beyond obj boundary

This is just to assert we're not doing an invalid memory access

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit b34a00de0f6ecea6d97fd80f65d2a2dca0876e6e)

6 years agoMerge pull request #26387 from smithfarm/wip-38188-luminous
Yuri Weinstein [Mon, 1 Apr 2019 21:47:10 +0000 (14:47 -0700)]
Merge pull request #26387 from smithfarm/wip-38188-luminous

luminous: osd/bluestore: deep fsck fails on inspecting very large onodes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
6 years agoMerge pull request #26535 from k0ste/luminous_backports3
Yuri Weinstein [Mon, 1 Apr 2019 21:46:42 +0000 (14:46 -0700)]
Merge pull request #26535 from k0ste/luminous_backports3

luminous: mgr/prometheus: add interface and objectstore to osd metadata

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
6 years agoMerge pull request #26616 from pdvian/wip-38442-luminous
Yuri Weinstein [Mon, 1 Apr 2019 21:21:57 +0000 (14:21 -0700)]
Merge pull request #26616 from pdvian/wip-38442-luminous

luminous: qa/standalone/osd/osd-markdown: disable CLI command dups

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
6 years agoMerge pull request #26751 from noahdesu/cherrypy-ipv6-luminous
Yuri Weinstein [Mon, 1 Apr 2019 21:08:49 +0000 (14:08 -0700)]
Merge pull request #26751 from noahdesu/cherrypy-ipv6-luminous

luminous: mgr/dashboard: fix for using '::' on hosts without ipv6

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
6 years agoMerge pull request #26979 from ifed01/wip-ifed-bmap-alloc-luminous
Yuri Weinstein [Mon, 1 Apr 2019 20:40:55 +0000 (13:40 -0700)]
Merge pull request #26979 from ifed01/wip-ifed-bmap-alloc-luminous

luminous: os/bluestore: backport new bitmap allocator

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agorgw: data sync checks empty next_marker for datalog 27299/head
Casey Bodley [Fri, 29 Mar 2019 14:27:56 +0000 (10:27 -0400)]
rgw: data sync checks empty next_marker for datalog

RGWReadRemoteDataLogShardCR tracks the marker and next_marker
separately, because next_marker will be empty when it reaches the end

this allows RGWDataSyncShardCR to avoid clearing its sync_marker and
restarting datalog listing from the beginning

Fixes: http://tracker.ceph.com/issues/39033
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ff63778342282ab5c676529f6e7d90f30e6da1a)

Conflicts:
src/rgw/rgw_data_sync.cc: no sync tracing

6 years agotests: rgw crypto tests with partlen vector
Abhishek Lekshmanan [Tue, 26 Mar 2019 18:12:15 +0000 (19:12 +0100)]
tests: rgw crypto tests with partlen vector

4 tests are added when either obj_size for an aligned object, and when an object
is not aligned to a boundary (1 B last part), or part_size is not aligned to a
boundary. Comments on boundary case explains the logic on choosing whether to
align to a 4095B boundary or not

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 1bf59f68db608084a1e7a05aca4cca8a8ab76062)
luminous changes: std::make_unique -> ceph::make_unique

6 years agotest_rgw_crypt: BlockCryptNone takes a block_size param
Abhishek Lekshmanan [Tue, 26 Mar 2019 18:11:28 +0000 (19:11 +0100)]
test_rgw_crypt: BlockCryptNone takes a block_size param

So that block sizes like 4096 can be tested

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9e13a42daaba26751f56f2454f9ec314d7d0d460)

6 years agorgw_crypt: make decrypt's parts_len protected
Abhishek Lekshmanan [Tue, 26 Mar 2019 18:10:04 +0000 (19:10 +0100)]
rgw_crypt: make decrypt's parts_len protected

This is so that testing classes can set this and use that to test the branches
in fixup_range tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 78b37a5ccd67c464f3c6e61568e9e827c327ad6d)

6 years agorgw_crypt: take care of start & end boundaries when parsing range
Abhishek Lekshmanan [Mon, 25 Mar 2019 14:24:15 +0000 (15:24 +0100)]
rgw_crypt: take care of start & end boundaries when parsing range

As http range header is inclusive of start and end ranges, this otherwise breaks
at ranges close to multipart boundaries

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 9c92b56d77932283ccc0336645dc3b76bd0fffa2)

6 years agorgw: decrypt filter does not cross multipart boundaries
Casey Bodley [Fri, 22 Mar 2019 13:57:33 +0000 (09:57 -0400)]
rgw: decrypt filter does not cross multipart boundaries

multipart uploads with sse encrypts each part separately, using an
initialization vector based on the part offset

decryption must respect the same part boundaries, and start each part
with a fresh initialization vector. this means that the decrypt filter
must flush data up to part boundaries before starting the next

Fixes: http://tracker.ceph.com/issues/38700
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b782902954abd20e49c0ed1c14cebee4c8ab6f88)
luminous modifications: std::make_unique -> ceph::make_unique

6 years agoMerge pull request #26642 from pdvian/wip-38449-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:59:14 +0000 (06:59 -0700)]
Merge pull request #26642 from pdvian/wip-38449-luminous

luminous: MDSMonitor: do not assign standby-replay when degraded

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #26776 from pdvian/wip-38541-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:58:49 +0000 (06:58 -0700)]
Merge pull request #26776 from pdvian/wip-38541-luminous

luminous : qa: bump fsstress timeout to 6h

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #26805 from pdvian/wip-38543-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:58:18 +0000 (06:58 -0700)]
Merge pull request #26805 from pdvian/wip-38543-luminous

luminous: qa: unmount clients before deleting fs

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #26820 from smithfarm/wip-38545-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:57:38 +0000 (06:57 -0700)]
Merge pull request #26820 from smithfarm/wip-38545-luminous

luminous: qa: Loading libcephfs-jni: Failure!

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #26964 from ashishkumsingh/wip-38735-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:57:14 +0000 (06:57 -0700)]
Merge pull request #26964 from ashishkumsingh/wip-38735-luminous

luminous: qa: extend MDS heartbeat grace for valgrind

Reviewed-by: Venky Shankar <vshankar@redhat.com>
6 years agoMerge pull request #27024 from pdvian/wip-38669-luminous
Yuri Weinstein [Mon, 1 Apr 2019 13:56:42 +0000 (06:56 -0700)]
Merge pull request #27024 from pdvian/wip-38669-luminous

luminous: mds: wait for client to release shared cap when re-acquiring xlock

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
6 years agoMerge pull request #27207 from pdvian/wip-38854-luminous
Yuri Weinstein [Fri, 29 Mar 2019 19:45:31 +0000 (12:45 -0700)]
Merge pull request #27207 from pdvian/wip-38854-luminous

luminous: mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Kefu Chai <kchai@redhat.com>
6 years agoMerge pull request #27224 from xiexingguo/wip-luminous-upmap-backports
Yuri Weinstein [Fri, 29 Mar 2019 19:44:55 +0000 (12:44 -0700)]
Merge pull request #27224 from xiexingguo/wip-luminous-upmap-backports

luminous: osd: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoMerge pull request #27226 from pdvian/wip-38857-luminous
Yuri Weinstein [Fri, 29 Mar 2019 19:44:25 +0000 (12:44 -0700)]
Merge pull request #27226 from pdvian/wip-38857-luminous

luminous: should set EPOLLET flag on del_event()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
6 years agoqa/suites: fix bluestore links 26979/head
Sage Weil [Wed, 5 Dec 2018 16:27:53 +0000 (10:27 -0600)]
qa/suites: fix bluestore links

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 269910fc8b2bdf64b98534ce525318e108aa9409)

6 years agoMerge pull request #26358 from smithfarm/wip-37557-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:45:34 +0000 (08:45 -0700)]
Merge pull request #26358 from smithfarm/wip-37557-luminous

luminous: multisite: es sync null versioned object failed because of olh info

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #26530 from pdvian/wip-38354-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:44:28 +0000 (08:44 -0700)]
Merge pull request #26530 from pdvian/wip-38354-luminous

luminous: rgw: check for non-existent bucket in RGWGetACLs

Reviewed-by: Adam Emerson <aemerson@redhat.com>
6 years agoMerge pull request #26619 from pdvian/wip-38446-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:43:58 +0000 (08:43 -0700)]
Merge pull request #26619 from pdvian/wip-38446-luminous

luminous: rgw: only update last_trim marker on ENODATA

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #26761 from pdvian/wip-38529-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:43:36 +0000 (08:43 -0700)]
Merge pull request #26761 from pdvian/wip-38529-luminous

luminous: rgw: data sync drains lease stack on lease failure

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #26946 from ashishkumsingh/wip-38727-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:43:09 +0000 (08:43 -0700)]
Merge pull request #26946 from ashishkumsingh/wip-38727-luminous

luminous: rgw: bucket limit check misbehaves for > max-entries buckets (usually 1000)

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #27030 from pdvian/wip-38671-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:42:46 +0000 (08:42 -0700)]
Merge pull request #27030 from pdvian/wip-38671-luminous

luminous: rgw: sync module: avoid printing attrs of objects in log

Reviewed-by: Casey Bodley <cbodley@redhat.com>
6 years agoMerge pull request #26516 from yehudasa/wip-rgw-es-fixes-luminous
Yuri Weinstein [Thu, 28 Mar 2019 15:14:03 +0000 (08:14 -0700)]
Merge pull request #26516 from yehudasa/wip-rgw-es-fixes-luminous

 luminous: rgw: multiple es related fixes and improvements

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>