qa/packages/packages: el8 has granular -debuginfo

Also add a bunch of missing packages.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cbt: include py2 deps on ubuntu for now

...until cbt is converted to py3

Signed-off-by: Sage Weil <sage@redhat.com>

src/test: misc python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits: env python -> env python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS

"2019-12-19T20:42:43.020748+0000 mon.b (mon.0) 2771 : cluster [WRN] Health check failed: noout flag(s) set (OSDMAP_FLAGS)" in cluster log

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: ping to specific centos

The simple

 os_type: centos

in valgrind.yaml doesn't pick a particular centos, and we end up with
the teuthology default (currently 7.6).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools

We need it for snappy-devel

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_pool_access.sh: python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: fix signature for py3

With python3 the sig.digest() is bytes, so c is an int, not a char.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31142 from zhangsw/fix-rgw-reshard-list-error

rgw: don't print error log when list reshard result is not truncated.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32335 into master

* refs/pull/32335/head:
qa/tasks/s3tests_java: move to gradle 6.0.1

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32343 into master

* refs/pull/32343/head:
doc: update '--force' flag to be precise

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31186 from zhangsw/clean-rgw-check-buckets-shard

rgw: remove unused bucket parameter in check_bucket_shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31150 from zhangsw/cleanup-rgw-reshard-last_run

rgw: remove unused 'last_run' in reshard thread entry

Merge PR #32339 into master

* refs/pull/32339/head:
qa/tasks/cephfs: python3
qa/workunits/fs: python3
qa/tasks/cephfs/fuse_mount: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32344 from kotreshhr/boost_cmake_py3_fix

cmake/Boost: Fix python3 version

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake/Boost: Fix python3 version

Signed-off-by: Kotresh HR <khiremat@redhat.com>

doc: update '--force' flag to be precise

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #31525 from tchaikov/wip-no-py2-in-make-check

install-deps.sh,src: drop python2 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/run-rbd-tests: s/nosetests/python3 -m nose/

different distro use different names for nosetests3, so let's just go
with `python3 -m nose`

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: s/nosetests/python3/

different distros package python3-nose in different ways by adding
different postfix to "/usr/bin/nosetests" to differentiate it from
its python2 counterpart.

* on bionic, python3-nose offers "nosetests3"
* on el8, python3-nose offers "nosetests-3" and "nosetests-3.6"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: drop Find{Python,Python2}.cmake

since we are python3 only, there is no need to check Python2 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: do not echo version of git

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: silence popd and pushd

otherwise they will echo the pushed/popped directory

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: drop the compatibility with python2

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: revert a5a44cf4

we don't prepare wheels for python2 anymore, and we don't test for
python2. so no need to test for python2

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: stop preparing wheels for python2

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: remove py27 from envlist in tox.ini

Signed-off-by: Kefu Chai <kchai@redhat.com>

python-common: remove py27 from envlist in tox.ini

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume/test: check using python3 instead of python

Signed-off-by: Kefu Chai <kchai@redhat.com>

src,qa: install python bindings into lib/cython_modules/lib.3

instead of checking for the WITH_PYTHON* options, just hardwire to lib.3

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: python3

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: drop WITH_PYTHON2 option

* drop WITH_PYTHON2 option
* print warning message if WITH_PYTHON3 is disabled
* drop MGR_PYTHON_VERSION option, as we don't support use different
  python version for python binding and ceph-mgr embedded python
  interpreter anymore. as before switching to python3-only build,
  we can build python3 and python2 python bindings, and ceph-mgr
  can use either of them. but after switching to python3-only
  build, ceph-mgr has to use whatever python version used to
  build python binding.
* move WITH_PYTHON3 option to $top_srcdir/CMakeLists.txt, as ceph-mgr
  and python binding will share this option.
* hardware ${PYTHON_VERSION} to 3
* hardware ${Python${PYTHON_VERSION}_VERSION_MAJOR} to 3
* only build boost library with python3
* s/Python_EXECUTABLE/Python3_EXECUTABLE/
* update the build scripts and packagings accordingly
* rename all cython${PYTHON_VERSION}_* targets to cython_*
* update distutils_install_module() so it does not take python_version
  parameter anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: CMAKE_ROLE defaults to "PROJECT"

so we can use exported targets like "Python3::Python", which is defined
only if CMAKE_ROLE is "PROJECT". but this global property was introduced
by CMake 3.14, but the minimum required version of CMake is 3.10.2 in
Ceph project at this moment. let's set it if it's not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/fs: python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephfs/fuse_mount: use python3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32323 into master

* refs/pull/32323/head:
spec, debian: cephadm requires lvm2

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/s3tests_java: move to gradle 6.0.1

Also switch to using a stable symlink so that the version is determined by the branch.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32191 into master

* refs/pull/32191/head:
doc: clarify priority use

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32310 into master

* refs/pull/32310/head:
Revert "qa/suites/rgw: ragweed on ubuntu"
Revert "qa/suites/rgw: disable remaining ragweed test"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32121 from yehudasa/wip-42869

rgw: bucket re-creation fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32309 from cbodley/wip-43359

rgw: data sync markers include timestamp from datalog entry

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #32311 from cbodley/wip-43360

radosgw-admin: sync status displays id of shard furthest behind

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #32324 from tchaikov/wip-copr-repo

install-deps.sh: add ken's copr repo for el8 build

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32317 from tchaikov/wip-rpm-pyOpenSSL

ceph.spec.in: always depends on python3.6-pyOpenSSL

Reviewed-by: Alfredo Deza <adeza@redhat.com>

install-deps.sh: add ken's copr repo for el8 build

Signed-off-by: Kefu Chai <kchai@redhat.com>

spec, debian: cephadm requires lvm2

cephadm mounts `/run/lvm` into OSD containers, which only exists,
if `lvm2` is installed on the container host. Otherwise we cannot
start ceph-volume or osd containers.

Fixes: https://tracker.ceph.com/issues/43368
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph.spec.in: always depends on python3.6-pyOpenSSL

since we've moved to py3-only world, there is no need to depend on
pyOpenSSL anymore.

also, _python_buildid was removed in
3bf8b4d7d6012458fc761edd759a4fb3070a30fe .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31527 into master

* refs/pull/31527/head:
vstart: Add note for nfs-ganesha dev packages
stop.sh: Add option to kill Ganesha daemons
vstart: Add option to deploy ganesha daemons

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32303 from tchaikov/wip-build-integration-branch

build-integration-branch: s/prefix/postfix/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #32291 into master

* refs/pull/32291/head:
doc: update 'adding an MDS'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32252 into master

* refs/pull/32252/head:
qa/cephfs/begin: libaio-devel on el8
qa/tasks: nosetests -> python -m nose
qa/tasks/rbd_fio: fio 2.21 -> 3.16
src/test/cli-integration/rbd/snap-diff.t: python -> python
qa/workunits: use nose 3
qa/tasks/cbt: install python3 deps
qa/tasks/ceph_manager.py: do not use python to write a file
test/pybind/test_rados: execute takes a bytes (not str) payload
qa/packages/packages: python[3]-ceph is no more
qa: use python3 for venvs etc
packaging: remove python3-ipaddres, as it is part of the stdlib in py3
qa/packages: python-ceph -> python3-ceph
qa/distros: centos7 -> centos8, rhel7 -> rhel8
spec: remove _python_buildid in favor of python3_pkgversion macro
spec: remove python2 packages and conditions
debian: remove python >= 2.7 requirement
debian: add mgr python versions
debian: explicitly set PYTHON2=OFF to prevent picking up python2 interpreter
debian: update control file to use python3 dependency names
debian: remove all python2 overrides and declarations
debian: remove all python2 install files

Reviewed-by: Alfredo Deza <adeza@redhat.com>

radosgw-admin: sync status displays id of shard furthest behind

'radosgw-admin sync status' gives you a timestamp for the "oldest
incremental change not applied", but doesn't tell you which shard that
came from. add [shard-id] to the output

Fixes: https://tracker.ceph.com/issues/43360
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Revert "qa/suites/rgw: ragweed on ubuntu"

This reverts commit 73c2779524cc13bf69e025bdaed04035f5ba3968.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rgw: disable remaining ragweed test"

This reverts commit 5a80fbe505e203227c4fa0de6141aaa69589d35e.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32146 into master

* refs/pull/32146/head:
mgr/ssh: Fix remove_osds()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32149 into master

* refs/pull/32149/head:
mgr/orchestrator: Rename PlacementSpec.nodes -> PlacementSpec.hosts
mgr/ansible: Added missing type annotations
mgr/orchestrator: Add __str__ to HostSpec
mgr/ansible: Integrate mgr/ansible/tox into mgr/tox

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32278 into master

* refs/pull/32278/head:
qa/suites/rgw: disable valgrind for tests that require py2/ubuntu
qa/suites/rgw: disable remaining ragweed test
qa/suites/rgw: pin swift tests to py2/ubuntu
qa/suites/rgw: ragweed on ubuntu
qa/suites: run s3tests on ubuntu

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/cephfs/begin: libaio-devel on el8

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: data sync markers include timestamp from datalog entry

this corrects the output of 'radosgw-admin data sync status' to show the
timestamp of the last datalog entry applied

Fixes: https://tracker.ceph.com/issues/43359
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32300 into master

* refs/pull/32300/head:
cephadm: drop 'sha256:' prefix on container id

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32302 into master

* refs/pull/32302/head:
cephadm: check for both ntp.service and ntpd.service

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/tasks: nosetests -> python -m nose

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw: disable valgrind for tests that require py2/ubuntu

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32280 from dillaman/wip-librbd-qos-test

qa/workunit/rbd: fixed QoS throughput unit parsing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #32279 into master

* refs/pull/32279/head:
mgr/orchestrator_cli: combine all host into one 'device ls' table
python-common: 'ssd/nvme' -> 'ssd'
mgr/orchestrator_cli: make 'device ls' table formatting consistent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32282 into master

* refs/pull/32282/head:
cephadm: clean up shell containers on exit
cephadm: do not force container removal on unit stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rgw: disable remaining ragweed test

Needs to be py3 since it relies on the python bindings.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32284 into master

* refs/pull/32284/head:
qa/suites/rbd/openstack: use 18.04, not 16.04

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: drop 'sha256:' prefix on container id

Podman doesn't include the prefix--only the (bionic) docker as far as I
can tell.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32273 from changchengx/doc_libvirt

doc/rbd: refine rbd/libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

build-integration-branch: s/prefix/postfix/

prefix is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32296 from rzarzynski/wip-seastar-pickup-futureopt

seastar: pickup the recent future optimizations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: Rename PlacementSpec.nodes -> PlacementSpec.hosts

By definition in the orchestrator's docs, we're using `host` insted of `node`

Fixes wrong usage of PlacementSpec in mgr/ansible

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32301 from tchaikov/wip-build-doc

admin/doc-requirements.txt: bump up Sphinx and breathe

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32285 from liewegas/wip-py3-docs

admin/build-doc: do not use system site-packages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: check for both ntp.service and ntpd.service

Sigh...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32263 from rosinL/fix-vcluster-stop

stop.sh: use bash shell to solve syntax error

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: do not use system site-packages

Don't use site-packages, since the host might have the same version, and
it won't have the ditaa and breathe.

Require Cython to make the venv sufficient.

Signed-off-by: Sage Weil <sage@redhat.com>

admin/doc-requirements.txt: bump Sphinx and breathe

* use the latest Sphinx
* use the latest breathe. also, since there is no need to be compatible
  with python2. we can move to 4.14.0 which is python3 only.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunit/rbd: fixed QoS throughput unit parsing

The 'rbd bench' command was recently modified to print IEC units
instead of bytes/sec. This broke the handling for QoS throughput
tests since it was incorrectly evaluating the available RBD
throughput. Additionally, the QoS tests should use a "<="
comparison operator since the QoS is the upper-bound limit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32293 into master

* refs/pull/32293/head:
script/build-integration-branch: Add usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32295 from tchaikov/wip-stop.sh

stop.sh: do not try to contact mon unless cluster is up

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

cephadm: clean up shell containers on exit

The run_cmd() was already passing --rm; do it for shell_cmd too so that
we don't leave behind a mess from shell commands, ceph-volume, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not force container removal on unit stop

The -f arg for 'podman rm' will remove the container without actually
stopping it, leaving the running containers and processes (conmon and
daemon) running and generally making a mess of things.

Doing 'podman rm' without the -f on a running container will complain
that the container is still running.

So, first stop, then remove.

Fixes: f1de7060762d31e567a277c02b0d8d533fbb4661
Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: use ${CEPH_BIN} and -c ${conf_fn}

to make sure ${CEPH_BIN} and ${CEPH_CONF_PATH} are respected

Signed-off-by: Kefu Chai <kchai@redhat.com>

stop.sh: do not try to contact mon unless cluster is up

otherwise "ceph mon metadata" will hang, when we try to stop an
already-stopped cluster. and `vstart_runner.py` always calls `stop.sh`
before setting up a cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the recent future optimizations.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart: Add note for nfs-ganesha dev packages

Signed-off-by: Varsha Rao <varao@redhat.com>

script/build-integration-branch: Add usage

Also add a --no-date to make it possible to re-use an existing branch name

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: update 'adding an MDS'

Fixes: https://tracker.ceph.com/issues/43162
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #32207 from bk201/wip-43136

mgr/dashboard: Fix OSD IDs are not displayed when using cephadm backend

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32266 from xiexingguo/wip-balancer-efficiency-1

mgr/balancer: avoid pulling pg_dump twice

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/ssh: Fix remove_osds()

Changed signature from `(str) -> ...` to `(List[str]) -> ...`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32237 from tchaikov/wip-cmake-rdkafka

cmake: check version of librdkafka

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

stop.sh: Add option to kill Ganesha daemons

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Add option to deploy ganesha daemons

To create Ganesha daemons with vstart following packages needs to be installed:
nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rados-grace

Version used for these packages 2.7.6-2 on fedora 30 system.

This is an experimental option, it just deploys specified number of ganesha
daemons.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #31144 from badone/wip-use-dnf-on-rhel-or-centos-8

install-deps.sh: Use dnf for rhel/centos 8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31884 into master

* refs/pull/31884/head:
qa/tasks: Fix the volume ls in test_volume_rm
qa/tasks: tests for 'fs volume create' and 'fs volume ls'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>