Jenkins Build Slave User [Thu, 25 Jun 2020 17:32:30 +0000 (17:32 +0000)]
14.2.10
Abhishek L [Thu, 25 Jun 2020 17:26:27 +0000 (19:26 +0200)]
Merge pull request #35775 from theanalyst/nautilus-rgw-cors-fixes
nautilus: rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader
Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Casey Bodley [Tue, 26 May 2020 19:03:03 +0000 (15:03 -0400)]
rgw: sanitize newlines in s3 CORSConfiguration's ExposeHeader
the values in the <ExposeHeader> element are sent back to clients in a
Access-Control-Expose-Headers response header. if the values are allowed
to have newlines in them, they can be used to inject arbitrary response
headers
this issue only affects s3, which gets these values from an xml document
in swift, they're given in the request header
X-Container-Meta-Access-Control-Expose-Headers, so the value itself
cannot contain newlines
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Reported-by: Adam Mohammed <amohammed@linode.com>
Yuri Weinstein [Thu, 18 Jun 2020 15:20:14 +0000 (08:20 -0700)]
Merge pull request #35599 from tchaikov/nautilus-el8
nautilus: ceph.spec.in: build on el8
Reviewed-by: David Galloway <dgallowa@redhat.com>
Yuri Weinstein [Thu, 18 Jun 2020 15:19:20 +0000 (08:19 -0700)]
Merge pull request #35488 from jdurgin/wip-radosbench-qa-nautilus
nautilus: qa/tasks/radosbench: use long form of option for compatibility
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Kefu Chai [Tue, 1 Oct 2019 09:31:52 +0000 (17:31 +0800)]
install-deps.sh: enable CRB/PowerTool repo for RHEL/CentOS 8
see also https://fedoraproject.org/wiki/EPEL
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
d301137bb16d4e530745172624ed9800a9b4c5ae)
Brad Hubbard [Fri, 25 Oct 2019 05:29:37 +0000 (15:29 +1000)]
ceph.spec.in: Enable amqp_endpoint on RHEL8 by default
RHEL/CentOS 8 now provide librabbitmq-devel so we can enable it as a
build requirement.
Fixes: https://tracker.ceph.com/issues/38466
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit
c44c140dfab4551eebc532c13865f56557b8378c)
Kefu Chai [Sat, 28 Sep 2019 13:57:29 +0000 (21:57 +0800)]
ceph.spec.in: disable cephfs_java and amqp_endpoint if RHEL8
RHEL/CentOS 8 does not offer junit and sharutils yet, so disable
cephfs_java on rhel8 at this moment.
RHEL/CentOS 8 does not offer librabbitmq-devel yet, so disable
amqp_endpoint on rhel8 at this moment.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
dd133840b9d35ca7d8b1788836f6076b06c33087)
Conflicts:
ceph.spec.in: trivial resolution
Alfonso Martínez [Thu, 23 Jan 2020 10:16:27 +0000 (11:16 +0100)]
ceph.spec.in: fix 'make check' deps for centos8
When running 'FOR_MAKE_CHECK=1 ./install-deps.sh' in CentOS 8
these dependencies were not being installed.
Missing dependencies are provided by
https://copr.fedorainfracloud.org/coprs/ktdreyer/ceph-el8/
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit
e5fa448229646a9dc9e3314389374d43243b06d2)
Conflicts:
ceph.spec.in: trivial resolution
Kefu Chai [Tue, 24 Dec 2019 05:17:55 +0000 (13:17 +0800)]
ceph.spec.in: re-enable "make check" deps for el8
this change partially reverts
e92cb7a0. as these packages are now
available in AppStream, BaseOS or PowerTools in el8, in this change,
they are re-enabled.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
2a27084e1d77919571e338c4634576e4b36dbd02)
Conflicts:
ceph.spec.in: trivial resolution
Nathan Cutler [Fri, 6 Dec 2019 13:51:40 +0000 (14:51 +0100)]
ceph.spec.in: move distro-conditional deps to dedicated section
Post
e92cb7a0336406a2981e9241031497b1749b26aa cleanup. Restore previous
structure of "distro-conditional make check dependencies" section.
Fixes: https://tracker.ceph.com/issues/43171
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit
e80d14a17647c5f6e7c3c7fcc4c8bfb5c8297fcd)
Conflicts:
ceph.spec.in: trivial resolution
Kefu Chai [Fri, 1 Nov 2019 15:18:23 +0000 (10:18 -0500)]
ceph.spec.in: run pathfix.py
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit
97bf619e1c538eea4d884b82c4f316e2da15dc3d)
Kefu Chai [Tue, 1 Oct 2019 10:12:43 +0000 (18:12 +0800)]
ceph.spec.in: install python3-Cython on el8
EL8 has python3 support. no need to wait for python3-Cython to populate
to its mirrors.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
0098b42927be9d965e8377fe1fd769774ff3b27a)
Kefu Chai [Sat, 28 Sep 2019 14:01:31 +0000 (22:01 +0800)]
ceph.spec.in: disable "make check" for rhel8
following packages are not available on RHEL/CentOS 8, they are for
dashboard unit tests. so disable all python packages for dashboard
tests for now:
python-cherrypy
python-coverage
python-pecan
python-routes
python-tox
xmlsec1-devel
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
e92cb7a0336406a2981e9241031497b1749b26aa)
Conflicts:
ceph.spec.in: trivial resolution
luo.runbing [Tue, 17 Sep 2019 06:55:37 +0000 (14:55 +0800)]
rpm,deb: python-requests is not needed for ceph-common
Signed-off-by: luo.runbing <luo.runbing@zte.com.cn>
(cherry picked from commit
998a3025c0e87638ae7b36a8b44710abb30fff8a)
Conflicts:
ceph.spec.in: trivial resolution
Yuri Weinstein [Tue, 16 Jun 2020 15:27:55 +0000 (08:27 -0700)]
Merge pull request #35520 from ajarr/py3-fs-fixes-nautilus
nautilus: qa/tasks/cephfs: py 3 transition fixes
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Rishabh Dave [Mon, 20 May 2019 09:30:07 +0000 (11:30 +0200)]
test_volume_client: simplify test_get_authorized_ids()
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit
79d62dd87461c43dd6df3775f21962be9042a7b1)
Sage Weil [Thu, 19 Dec 2019 17:08:59 +0000 (11:08 -0600)]
qa/tasks/cephfs: os.write takes bytes, not str
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit
05eb2f647d16504a81ff17a8ef732cf0e2d90284)
Conflicts:
qa/tasks/cephfs/test_pool_perm.py: trivial resolution
Sage Weil [Thu, 19 Dec 2019 03:51:52 +0000 (21:51 -0600)]
qa/tasks/cephfs: python3
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit
3c751fea7f48ef613286211797a7eafc4913b36e)
Conflicts:
qa/tasks/cephfs/kernel_mount.py
qa/tasks/cephfs/test_volume_client.py
Ramana Raja [Wed, 10 Jun 2020 11:22:14 +0000 (16:52 +0530)]
qa/tasks/cephfs/kernel_mount: fix incomplete py3 transition backport
Commit
3dad1ac5f2db97cc3445bdf14a80dbb3e11f7d21 wan an incomplete
backport of
875b45f3c903781ba99dbd2fcf4d2fc89001c25b. The missing
commit lines are added here.
Introduced-by: 3dad1ac5f2db97cc3445bdf14a80dbb3e11f7d21
Fixes: https://tracker.ceph.com/issues/45966
Signed-off-by: Ramana Raja <rraja@redhat.com>
Yuri Weinstein [Thu, 11 Jun 2020 15:43:40 +0000 (08:43 -0700)]
Merge pull request #35391 from smithfarm/wip-45679-nautilus
nautilus: cephfs: allow pool names with hyphen and period
Reviewed-by: Ramana Raja <rraja@redhat.com>
Ramana Raja [Thu, 11 Jun 2020 13:49:07 +0000 (19:19 +0530)]
Merge pull request #35535 from ajarr/wip-45960
nautilus: fix TestExports failure
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Patrick Donnelly [Fri, 22 May 2020 21:15:19 +0000 (14:15 -0700)]
qa: return a string via getfattr
Fixes: https://tracker.ceph.com/issues/45666
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit
3c1330795add81126424e9e0a49cc07881fbdf8f)
Sidharth Anupkrishnan [Wed, 29 Apr 2020 13:04:57 +0000 (18:34 +0530)]
qa/test_exports: fix TestExports failure under new python3 compability changes
self.mount_a.client_remote.sh() returns an 'str' object rather than a StringIO object. Hence the p.stdout.getvalue() produces an error. This commit fixes this and also fix str and byte mismatch as byte and string were the same object in Python2 but this is not the case in Python3.
Signed-off-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
(cherry picked from commit
b98f3e388b6a38e91c2ee064466ae8f130a5bcda)
Yuri Weinstein [Wed, 10 Jun 2020 15:32:51 +0000 (08:32 -0700)]
Merge pull request #35393 from smithfarm/wip-45689-nautilus
nautilus: cephfs: client: expose Client::ll_register_callback via libcephfs
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Yuri Weinstein [Wed, 10 Jun 2020 15:32:03 +0000 (08:32 -0700)]
Merge pull request #35392 from smithfarm/wip-45686-nautilus
nautilus: cephfs: mds: take xlock in the order requests start locking
Reviewed-by: Ramana Raja <rraja@redhat.com>
Kotresh HR [Wed, 10 Jun 2020 05:57:02 +0000 (11:27 +0530)]
Merge pull request #35482 from ajarr/wip-45681-nautilus
nautilus: mgr/volumes: minor fixes
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Josh Durgin [Mon, 8 Jun 2020 18:29:43 +0000 (11:29 -0700)]
qa/tasks/radosbench: use long form of option for compatibility
Since the short version of --object-size changed from -o to -O, it
does not work with upgrade tests.
Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit
9673d37f1b72681f7583bf36537575d876da9712)
Yuri Weinstein [Tue, 9 Jun 2020 16:48:15 +0000 (09:48 -0700)]
Merge pull request #35453 from shyukri/wip-45784-nautilus
nautilus: ceph-volume: add and delete lvm tags in a single lvchange call.
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Yuri Weinstein [Tue, 9 Jun 2020 15:01:52 +0000 (08:01 -0700)]
Merge pull request #35461 from badone/wip-44846-nautilus
nautilus: install-deps.sh: Use dnf for rhel/centos 8
Reviewed-by: Kefu Chai <kchai@redhat.com>
Yuri Weinstein [Tue, 9 Jun 2020 14:57:49 +0000 (07:57 -0700)]
Merge pull request #35387 from smithfarm/wip-45780-nautilus
nautilus: qa/workunits/rados/test_envlibrados_rocksdb: Build correct rocksdb
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Rishabh Dave [Fri, 19 Jul 2019 08:18:51 +0000 (13:48 +0530)]
qa/cephfs: make run_shell() accept args as str too
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit
a83ea849d7d2096762a125b8c6dbb8eb9988b2a9)
Conflicts:
qa/tasks/cephfs/mount.py
- in nautilus, run_shell doesn't take a "stdin" argument
Kotresh HR [Mon, 27 Apr 2020 18:31:33 +0000 (00:01 +0530)]
mgr/volumes: Create subvolume with isolated rados namespace
1. Add --namespace-isolated option to 'subvolume create' command
to create subvolume in a separate RADOS namespace
2. Add "pool_namespace" field to 'subvolume info' command
which displays the rados namespace if set else empty string
Fixes: https://tracker.ceph.com/issues/45289
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit
7396880d4096b3a2a813b8ebcdcdf3082ee98b23)
Kotresh HR [Fri, 15 May 2020 14:59:01 +0000 (20:29 +0530)]
mgr/volumes: Fix subvolume create idempotency
After subvolume is created, it can be resized
using subvolume create command. But it was
broken and the same is fixed.
Fixes: https://tracker.ceph.com/issues/45398
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit
db8706aa68887a45f25340a62eaddf62c0ea4f46)
Brad Hubbard [Fri, 25 Oct 2019 05:51:38 +0000 (15:51 +1000)]
install-deps.sh: Use dnf for rhel/centos 8
Also install the correct dnf-utils package rather than yum-utils.
Fixes: https://tracker.ceph.com/issues/42504
Co-Authored-By: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit
6065050022053478ac77aa6c34e3d71339080880)
Conflicts:
install-deps.sh - Changes for install-deps-cache
Jan Fajerski [Tue, 31 Mar 2020 14:07:45 +0000 (16:07 +0200)]
ceph-volume: add and delete lvm tags in a single lvchange call.
Otherwise we can end up in race-y situations when a concurrent c-v calls
sees only one tag but expects all tags to be present. Say if the
ceph.type tag is present, c-v expects ceph.osd_id to be present. By
setting/deleting tags in bulk, we use lvchange (and lvms internal
locking) as a sync mechanism.
Fixes: https://tracker.ceph.com/issues/44852
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit
20ecc309371e53fda5d6a5b6cf6de6110dbe5497)
Yuri Weinstein [Fri, 5 Jun 2020 15:45:18 +0000 (08:45 -0700)]
Merge pull request #35184 from smithfarm/wip-44330-nautilus
nautilus: tests: specify random distros in multimds
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Yuri Weinstein [Fri, 5 Jun 2020 15:44:06 +0000 (08:44 -0700)]
Merge pull request #34356 from smithfarm/wip-44899-nautilus
nautilus: mgr: Improve internal python to c++ interface
Reviewed-by: David Zafman <dzafman@redhat.com>
Kefu Chai [Sun, 7 Jul 2019 08:17:35 +0000 (16:17 +0800)]
common/ceph_mutex: add more ceph_mutex_is_*() helpers
we have caller sites like
ceph_assert(!client->client_lock.is_locked_by_me())
in `src/client/Client.cc` and `src/librados/RadosClient.cc`.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
307ccc42f09421b2bf1df3792a29a4b86bdd2b1a)
Kefu Chai [Sun, 7 Jul 2019 04:32:53 +0000 (12:32 +0800)]
common/ceph_mutex: improve ceph::shared_mutex
add helpers for supporting `ceph_mutex_is_locked()` and
`ceph_mutex_is_wlocked()` macros
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
cc67c34a9e5b32e9372f6e109549619def3b6bc4)
Jeff Layton [Fri, 17 Apr 2020 13:55:41 +0000 (09:55 -0400)]
client: add a new inode release request callback
trim_caps() walks the list of caps on the session, and releases
non-auth caps, and attempts to trim dentries until the cache
size is under the max_caps value requested by MDS.
This is fine for FUSE, but doesn't really match the use-case of
nfs-ganesha. Ganesha typically looks up inodes by inode number, not
by dentry. It's quite possible that after a restart, we may have a
ton of outstanding inodes with no dentries associated with them.
Ganesha holds a reference to each inode, so libcephfs can't release
them, and we don't have a way to request that ganesha do so.
Add a new ino_release_callback and finisher. The intent is to allow
libcephfs to "upcall" to the application and request that it release
references to a specific inode.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit
e3b9df78fa42add2dfea1f8bff1e00c832a35697)
Jeff Layton [Thu, 16 Apr 2020 16:19:59 +0000 (12:19 -0400)]
client: expose ceph_ll_register_callbacks via libcephfs
Allow applications based on libcephfs to register callbacks, like we
do for ceph-fuse.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit
adcf12d6c329a3bd534e089bc0c8b42dfd3e4515)
Jeff Layton [Thu, 16 Apr 2020 16:09:54 +0000 (12:09 -0400)]
client: move callback typedefs and arg struct into ceph_ll_client.h
Put them in a common interface header file. This also allows us to
eliminate the duplicate definition of ceph_deleg_cb_t in Delegation.h.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit
8370f70cacfb0bccc96d121cab687376155e8b8d)
Conflicts:
src/include/cephfs/libcephfs.h
- nautilus version of this file does not have the line:
struct CephContext;
Jeff Layton [Tue, 28 Apr 2020 18:00:13 +0000 (14:00 -0400)]
test: add a new program for testing ino_release_cb
Create a bunch of files and get their inode numbers. Remount, look them
all up by inode number and hold references. Stop looking up inodes as
soon as we get a callback from libcephfs. If we got the callback, return
success. Fail otherwise.
Since this has the same cluster setup as the other client_trim_caps
testcase, we can piggyback onto that task.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit
c0db8a01f0a04cf9e10f3715bfb802d619bc32b9)
Jeff Layton [Thu, 16 Apr 2020 16:01:05 +0000 (12:01 -0400)]
client: rename ceph_statx.h to ceph_ll_client.h
...so we can use it to include other definitions and types that need to
be shared with in-tree code that doesn't want to include libcephfs.h.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit
4436f27a0476e96d10ed3367d738e1679d8b0a38)
Conflicts:
src/client/Client.h
- nautilus version does not have the line
#include "include/common_fwd.h"
which is not relevant to this backport
Jeff Layton [Thu, 16 Apr 2020 14:05:20 +0000 (10:05 -0400)]
client: make client_dentry_callback_t more friendly for C
C doesn't have the string type, and doesn't understand references.
Change client_dentry_callback_t to take separate pointer and length
arguments.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit
79b588211017d8af15a0e49628563dbe45b55ccb)
Jeff Layton [Tue, 21 Apr 2020 12:50:54 +0000 (08:50 -0400)]
client: only override umask_cb with non-NULL values
Client::init sets this, but if we later call ll_register_callbacks again
with a new set of function pointers that has umask_cb set to nullptr,
it'll override the value in the cmount.
Only reset umask_cb if the one in args is not nullptr.
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit
4cb17bb12feaa2b1a78a08612f8e80a191c87e5e)
Yan, Zheng [Mon, 27 Apr 2020 06:48:33 +0000 (14:48 +0800)]
mds: take xlock in the order requests start locking
this avoid assertion in MutaionImpl::finish_locking()
Fix: https://tracker.ceph.com/issues/45261
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit
2e11a35d5b06312e0b2d0aecd83e8eb882ddf719)
Ramana Raja [Thu, 14 May 2020 10:06:22 +0000 (15:36 +0530)]
cephfs: allow pool names with '-' and '.'
... to be set as pool layouts for files and directories.
Fixes: https://tracker.ceph.com/issues/45521
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit
d9f8f8e3d073de8e1cfd5075159420ca8dfdcf2d)
Conflicts:
doc/cephfs/administration.rst
- nautilus has "filesystems" where master has "file systems"
- a difference that is not relevant to this backport
Brad Hubbard [Thu, 9 Apr 2020 04:33:51 +0000 (14:33 +1000)]
qa/workunits/rados/test_envlibrados_rocksdb: Build correct rocksdb
We should be building the version of rocksdb the release is pinned to,
not master. Let's just update the rocksdb submodule and clone that.
Fixes: https://tracker.ceph.com/issues/44981
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit
1361913f5995f8a4c8e1b0f5c661ad09e66fe9aa)
Kefu Chai [Thu, 4 Jun 2020 02:24:47 +0000 (10:24 +0800)]
Merge pull request #35369 from djgalloway/nautilus
nautilus: qa: add .teuthology_branch with `master` contents
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Kyr Shatskyy [Tue, 2 Jun 2020 17:05:30 +0000 (19:05 +0200)]
qa: add .teuthology_branch with `master` contents
This is follow up change for:
teuthology-suite: automate -t argument default value
https://github.com/ceph/teuthology/pull/1490
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit
07cc36dcd264b66a7f3a19687b336737a772c2f5)
Jan Fajerski [Wed, 3 Jun 2020 14:50:09 +0000 (16:50 +0200)]
Merge pull request #35132 from shyukri/wip-45483-nautilus
nautilus: ceph-volume: add ceph.osdspec_affinity tag
Kefu Chai [Wed, 3 Jun 2020 12:56:22 +0000 (20:56 +0800)]
Merge pull request #34171 from tchaikov/nautilus-py3
nautilus: qa: migrate to python3
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Kefu Chai [Mon, 25 May 2020 07:52:04 +0000 (15:52 +0800)]
qa/suites/rgw/tempest: bump up keystone to 17.0.0
Fixes: https://tracker.ceph.com/issues/45692
please note, the change on master also includes changes to use identity
v3. in this change, we only pick the change to use newer keystone to
address the pip install issue
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
9bd3e0ff4027cfbef0c62931b669f20c989e8bb2
Kefu Chai [Sun, 24 May 2020 17:07:09 +0000 (01:07 +0800)]
qa/tasks/tox: use the latest stable tox
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
cd437bc5dfd7b82d2348810a177d56654083a6d2)
Kefu Chai [Sun, 24 May 2020 17:04:37 +0000 (01:04 +0800)]
qa/tasks/tox: use python3 for tox tests
both tempest and keystone have dropped python2 support in their tox
based tests.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
b7152e8e943b713e779aec0c2414986bd556a423)
Kefu Chai [Sun, 24 May 2020 15:57:46 +0000 (23:57 +0800)]
qa/valgrind.supp: less specific when suppressing issues/22052
Fixes: https://tracker.ceph.com/issues/22052
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
a8e9fd71d79f7696a47773acdcc738f47a88d8d8)
Kefu Chai [Sun, 24 May 2020 15:06:08 +0000 (23:06 +0800)]
qa/tasks/tempest.py: install python3 for tempest test
install python3 for tempest's tox based test otherwise the test would
fail if python3.6 or python3.8 is not found.
this change is not cherry-picked from master, as our python bindings
are now built with python3.6, so python3.6 is always used. it's
supported by tempest's test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Kefu Chai [Sun, 24 May 2020 14:51:05 +0000 (22:51 +0800)]
qa/tasks/keystone.py: install python3
install python3 for keystone's tox based test otherwise python34 would
be installed, and keystone's tox based test does not support python34.
this change is not cherry-picked from master, as our python bindings are
now built with python3.6, so python3.6 is always used. it's supported by
keystone's test.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Sage Weil [Tue, 17 Dec 2019 16:52:37 +0000 (10:52 -0600)]
qa/suites/rgw: disable valgrind for tests that require py2/ubuntu
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit
ea21829ae728354b759f117e9af8a20eb8213687)
Conflicts:
qa/suites/rgw/website/overrides.yaml
qa/suites/upgrade/nautilus-p2p/nautilus-p2p-stress-split/7-final-workload/.qa: trivial resolution
Casey Bodley [Fri, 29 Mar 2019 18:18:55 +0000 (14:18 -0400)]
DNM qa/rgw: disable debuginfo packages
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit
d0142a189abfad641145986ca1552c27c870f713)
Sage Weil [Fri, 13 Dec 2019 18:42:57 +0000 (18:42 +0000)]
qa/tasks/ceph_manager.py: do not use python to write a file
/usr/bin/python dne on el8, /usr/bin/python3 dne on el7. But
all we need to do is write a file--we can do that with tee.
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit
d4f4a2cbd81f2c86e281b9220bd5b727c0fd85f3)
Rishabh Dave [Fri, 19 Jul 2019 04:35:18 +0000 (10:05 +0530)]
test_sessionmap: use sudo_write_file() from teuthology.misc
Instead of defining a method to write files within the testsuite, use
sudo_write_file() from teuthology.misc.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit
4b776240705ebe8bdd0d43cd6244491e435a509c)
Patrick Donnelly [Mon, 18 May 2020 19:57:43 +0000 (12:57 -0700)]
qa: use py3 compat list from range
Fixes: https://tracker.ceph.com/issues/45590
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit
8d51b33e5d70feec3b65ac3eb819a3515befcdac)
Kyr Shatskyy [Fri, 21 Feb 2020 19:15:44 +0000 (20:15 +0100)]
qa/tasks/keystone: get rid of cStringIO for py3
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit
dcf85e4328bfba057d30b6720f6e641dacb11e9f)
Casey Bodley [Thu, 5 Sep 2019 18:06:45 +0000 (14:06 -0400)]
qa/keystone: pip install bindep
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit
2144fd8cc7f156264aff958c54c5e35d0e65757c)
Michael Fritch [Mon, 4 May 2020 18:37:11 +0000 (12:37 -0600)]
mgr/insights: fix python3 DeprecationWarning
insights/module.py:167: DeprecationWarning: invalid escape sequence \d
r = "ceph version (?P<release>\d+)\.(?P<major>\d+)\.(?P<minor>\d+)"
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit
41ffb701eca6280df00cd526edfce3a9c851b53b)
Kefu Chai [Fri, 8 May 2020 03:10:56 +0000 (11:10 +0800)]
qa/tasks/cephfs: remove sys.exc_clear() calls
sys.exc_clear() was removed in Python3, see
https://docs.python.org/3/whatsnew/3.0.html#index-22. so we should not
call it.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
702de9590cdfe6c95ef8e87cb74cd05944e380c9)
Rishabh Dave [Fri, 8 May 2020 11:40:24 +0000 (17:10 +0530)]
vstart_runner.py: use tuple instead of set
Using python3 leads to an error that says
"TypeError: unhashable type: 'Raw'".
Fixes: https://tracker.ceph.com/issues/45446
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit
cb0bfc8fab4372b0caf5102823883b0b913d4fcb)
Conflicts:
qa/tasks/vstart_runner.py: trivial resolution
Michael Fritch [Wed, 13 May 2020 23:16:17 +0000 (17:16 -0600)]
qa/tasks: logger.warn() -> logger.warning()
logger.warn() is the same as logger.warning(), but was removed in py3.3
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit
44bcee8d1a637b1429a0c9576397f70f0a660ec2)
Conflicts:
qa/tasks/vstart_runner.py: trivial resolution
Casey Bodley [Mon, 4 May 2020 20:39:59 +0000 (16:39 -0400)]
test/rgw: Cluster.admin() returns output as utf-8 str
fixes the TypeError:
File "rgw_multi/tests.py", line 1107, in test_bucket_sync_disable
check_buckets_sync_status_obj_not_exist(zone, buckets)
File "rgw_multi/tests.py", line 438, in check_buckets_sync_status_obj_not_exist
if log_list.find(':'+bucket+":") >= 0:
TypeError: argument should be integer or bytes-like object, not 'str'
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit
ff5ce2cf1457e1945aea0fe4a023bc48bffe1174)
Kefu Chai [Fri, 1 May 2020 08:52:41 +0000 (16:52 +0800)]
qa/tasks/qemu: open file in text mode
in python3, if a file is opened in 'rb' mode, we will have bytes when
reading from it, as a result, we have following failure like:
```
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 32, in nested
vars.append(enter())
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/github.com_tchaikov_ceph_nautilus-py3/qa/tasks/qemu.py", line 124, in generate_iso
test_setup = ''.join(f.readlines())
TypeError: sequence item 0: expected str instance, bytes found
```
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
0b21c597d438fcba43629d0795b2d4ff6112c9cf)
Kefu Chai [Thu, 30 Apr 2020 10:07:00 +0000 (18:07 +0800)]
qa/tasks: use list comprehension for checking the length
instead of using filter(), use `sum()` for counting its
length, as in Python3, `filter()` actually returns a `filter` object
instead of a list.
in this change, `filter()` calls are replaced with `sum()`
for Python3 compatibility.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit
726c59be58473cfe53fa3ea63cf35b58ed797886)
Xiubo Li [Fri, 24 Apr 2020 01:08:23 +0000 (21:08 -0400)]
qa/tasks/cephfs/test_volumes: to make sure the size is int type
Fix "Error EINVAL: invalid size option '
4915200.0'".
In pytho2 int/2 will get a int type result, but in python3 it maybe
a float type.
Fixes: https://tracker.ceph.com/issues/45247
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
afe19f24baf3983a42f6213c04efc20307c286d2)
Xiubo Li [Thu, 23 Apr 2020 21:31:24 +0000 (17:31 -0400)]
qa/tasks/cephfs/mount: to make sure that the count/seek are int type
To fix "stderr:dd: invalid number: '5.0'".
Fixes: https://tracker.ceph.com/issues/45247
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
65f97690af8ca49e7a8b508621bf47d4fc004853)
Xiubo Li [Thu, 23 Apr 2020 13:23:31 +0000 (09:23 -0400)]
qa/tasks/cephfs/cephfs_test_case: switch 'dict_keys' object to list
Fix the "TypeError: 'dict_keys' object does not support indexing" under
py3.
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
666974e0a663aba6aff37cf78728c44b0e3a08c1)
Ilya Dryomov [Tue, 28 Apr 2020 21:38:05 +0000 (23:38 +0200)]
qa/tasks/rbd_fio: update NamedTemporaryFile usage for python3
Open the file in text mode to avoid 'bytes' objects.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
4e3160a019b5af51e498793aaa24a2a7b82b4b54)
Ilya Dryomov [Tue, 28 Apr 2020 21:22:17 +0000 (23:22 +0200)]
qa/tasks/rbd: update NamedTemporaryFile usage for python3
- open the file in text mode to avoid 'bytes' objects
- 'bufsize' is now spelled 'buffering' and switching buffering off
with 'buffering=0' is only allowed in binary mode. As we need the
file to be either in the page cache or on disk before we copy it
to the remote, call flush().
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
aba9910388cce7c03aa90726b5de1263298ac8ec)
Xiubo Li [Thu, 16 Apr 2020 02:53:23 +0000 (22:53 -0400)]
qa/volume_client: fix TestVolumeClient failing under new py3 runtime
The pybind now has dropped the WITH_PYTHON2 option, and for now only
py3 supported.
Fixes: https://tracker.ceph.com/issues/45103
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
0ed507dfd4dd1e38c358cd6c77f20fac6e3a8343)
Conflicts:
qa/suites/fs/upgrade/snaps/clusters/.qa
qa/tasks/cephfs/test_volume_client.py: trivial resolution
Kefu Chai [Thu, 30 Apr 2020 05:50:39 +0000 (13:50 +0800)]
qa/cephfs: whitelist TOO_FEW_PGS
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 is not backported to mimic, but
we are still running upgrade test from mimic. so in mimic, monitor
still complains at seeing too many pgs per osd.
this change is not cherry-picked from master, because we already have
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 in master.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Kefu Chai [Mon, 27 Apr 2020 09:31:12 +0000 (17:31 +0800)]
qa/tasks: s/string.uppercase/string.ascii_uppercase/
`string.uppercase` is renamed to `string.ascii_uppercase` in Python2
and Python3. so let's update accordingly.
this change is not cherry-picked from master, as the modified file does
not exist in master anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Kefu Chai [Mon, 27 Apr 2020 05:11:56 +0000 (13:11 +0800)]
qa/tasks: s/itervalues/values/ to by py3 compatible
this change is not cherry-picked from master, as the changed files do
not exist in master anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Xiubo Li [Thu, 23 Apr 2020 11:21:46 +0000 (07:21 -0400)]
qa/tasks/cephfs: switch to StringIO to fix byte-like errors
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
dd4fc808a8a26ebbfae5b5e4bc3b6c99bf0f5d37)
Xiubo Li [Thu, 23 Apr 2020 11:39:31 +0000 (07:39 -0400)]
qa/tasks/vstart_runner: remove the stdin str type check
For the "ceph-dencoder" it will use the byte type data.
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
bf10478294e26dc785dd79b0367bf096e9f67b42)
Xiubo Li [Tue, 21 Apr 2020 02:22:55 +0000 (22:22 -0400)]
qa/cephfs: fix AssertionError: CommandFailedError not raised under py3
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
7f57576123b81051a5e64f7c7234153ee11f4062)
Alfonso Martínez [Fri, 24 Apr 2020 08:53:24 +0000 (10:53 +0200)]
mgr/dashboard: fix API tests
* test_perf_counters_mgr_get
* test_selftest_cluster_log
Fixes: https://tracker.ceph.com/issues/45170
Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit
0ac296430f45ddf937b6c4c25607f0b757dcf447)
Kefu Chai [Sat, 25 Apr 2020 01:47:00 +0000 (09:47 +0800)]
qa/tasks/vstart_runner.py: use StringIO for capturing text
to be consistent with
8bfe977854686149c74de3ebaed7b571d43404dd
Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
2de78c3bd9eed9bec235b68e8e52b4635e9add61)
Xiubo Li [Thu, 23 Apr 2020 05:08:04 +0000 (01:08 -0400)]
qa/tasks/cephfs/test_scrub: fix self.assertEqual no attribute '_type_equality_funcs'
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
89d06490886fb5cf840cadde49913dd497c14cf8)
Xiubo Li [Wed, 22 Apr 2020 12:34:05 +0000 (08:34 -0400)]
qa/tasks/cephfs/mount: switch to StringIO to fix TypeErrors
TypeError: cannot use a string pattern on a bytes-like object
Fixes: https://tracker.ceph.com/issues/45175
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit
0631342d105575fda009e9e97b26859616be2565)
Conflicts:
qa/tasks/cephfs/fuse_mount.py
qa/tasks/cephfs/mount.py: drop the change not applicable to
nautilus.
Kefu Chai [Mon, 27 Apr 2020 03:04:21 +0000 (11:04 +0800)]
qa/tasks: decode bytes returned by base64.b64encode()
we use the return value from `base64.b64encode()` as a string, this works
in python2, but in python3, instead of a string, `base64.b64encode()`
returns bytes. so for instance, if we try to compose command line
arguments using `pipes.quote()` by passing a byte instance, we will have
```
TypeError: cannot use a string pattern on a bytes-like object
```
in this change, the retval is always decoded using ASCII, because base64
only returns ASCII strings when encoding.
this change is not cherry-picked from master, as the changed files do
not exist in master anymore.
Signed-off-by: Kefu Chai <kchai@redhat.com>
Kyr Shatskyy [Sat, 15 Feb 2020 02:36:08 +0000 (03:36 +0100)]
qa/tasks/rados: get rid of itervalues for py3
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>
(cherry picked from commit
dd8bacdb131db74411a07d2952182c7218472769)
Kefu Chai [Fri, 27 Mar 2020 05:57:37 +0000 (13:57 +0800)]
qa/tasks/mgr: use relative import
for better readability, and to ease the pain of developer to track back
to the top level python package for referencing a submodule
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
7d37226548a1d543bd0cdba916f3e00730a91797)
Kefu Chai [Mon, 20 Apr 2020 11:50:02 +0000 (19:50 +0800)]
qa/tasks/cephfs/fuse_mount.py: by python3 compatible
pass `StringIO()` to capture stdout whose value will be interpreted as a
string later on
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
c5bd318de65866b4992314c2fb0e82090b4d2979)
Kefu Chai [Mon, 20 Apr 2020 11:13:52 +0000 (19:13 +0800)]
pybind/mgr/dashboard/run-backend-api-tests.sh: use python3 by default
since vstart_runner.py is now py3, let's use python3 by default
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
d54f3ea43e05c2823c813e9687194b0b1fe15ffe)
Conflicts:
src/pybind/mgr/dashboard/run-backend-api-tests.sh: trivial
resolution
Kefu Chai [Mon, 20 Apr 2020 10:27:58 +0000 (18:27 +0800)]
qa/tasks/vstart_runner.py: be python3 compatible
differentiate `str` and `bytes` instances, and drop python2 support from
vstart_runner.py, as we've moved to python3 already
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
bedea85019076999a205c421bbf593ab8ab0641b)
Conflicts:
qa/tasks/vstart_runner.py: trivial resolution
Ilya Dryomov [Wed, 8 Apr 2020 10:24:51 +0000 (12:24 +0200)]
qa/tasks/rbd_fio: unbreak after the conversion from StringIO
Fix a bad typo in commit
db7ae8eff60a ("qa/tasks/rbd_fio: get rid of
StringIO for py3").
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit
c3f4f1d6604749a51c552b868ef373f321139005)
Kefu Chai [Wed, 8 Apr 2020 11:22:06 +0000 (19:22 +0800)]
qa/tasks/openssl_keys.py: sort cert configs before creating certs
we cannot rely on the order in which items are arranged in a dict, the
order varies from version to another. in Python2, it happens to work,
and we can always have the self-signed cert added first. but in Python3,
it does not. and an exception is thrown
```
teuthology.exceptions.ConfigError: ssl: ca root not found for
certificate rgw.client.0
```
in this change, before creating certs, the settings are reordered so
that the self-signed ones are created first.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
f28a5fef3b8ddb97962f91cc78174fd6e1431fed)
Kefu Chai [Wed, 8 Apr 2020 04:32:54 +0000 (12:32 +0800)]
qa/tasks/ceph_objectstore_tool.py: use str.startswit
in Python3, string module does not offer `string.find()` anymore, let's
use `str.find()` method instead.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
77ec9ce88df54b7758e56eeef1b3cf3f14952513)
Kefu Chai [Wed, 8 Apr 2020 02:50:21 +0000 (10:50 +0800)]
test/rgw: upload using a NamedTemporaryFile
in boto, it tries to figure out the MIME type of a file by its name, if
the file-like objects has an attribute of "name". in Python2, the "name"
is always "<fdopen>", fortunately. while in Python3, `TemporaryFile` also
have a "name" which is its fd, and it is an integer now. so we have following
error when sending a `TemporaryFile` using
`upload_part_from_file()`:
```
2020-04-08T02:25:34.660 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
2020-04-08T02:25:34.661 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-qa-py3/qa/tasks/rgw_multi/tests_ps.py", line 2567, in test_ps_creation_triggers
2020-04-08T02:25:34.662 INFO:tasks.rgw_multisite_tests: uploader.upload_part_from_file(fp, 1)
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/multipart.py", line 260, in upload_part_from_file
2020-04-08T02:25:34.663 INFO:tasks.rgw_multisite_tests: query_args=query_args, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.664 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 750, in send_file
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: chunked_transfer=chunked_transfer, size=size)
2020-04-08T02:25:34.665 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-py3/virtualenv/lib/python3.5/site-packages/boto/s3/key.py", line 920, in
_send_file_internal
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: self.content_type = mimetypes.guess_type(self.path)[0]
2020-04-08T02:25:34.666 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 289, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: return _db.guess_type(url, strict)
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/mimetypes.py", line 114, in guess_type
2020-04-08T02:25:34.667 INFO:tasks.rgw_multisite_tests: scheme, url = urllib.parse.splittype(url)
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: File "/usr/lib/python3.5/urllib/parse.py", line 881, in splittype
2020-04-08T02:25:34.668 INFO:tasks.rgw_multisite_tests: match = _typeprog.match(url)
2020-04-08T02:25:34.669 INFO:tasks.rgw_multisite_tests:TypeError: expected string or bytes-like object
```
to address this issue, in this change, a `NamedTemporaryFile` is used
instead of `TemporaryFile`. the former does have a "name" which is a
`str`.
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit
523c623b282ce5cf4e4bb7bcf2b35c4009d89b3a)
Conflicts:
src/test/rgw/rgw_multi/tests_ps.py: trivial resolution