mon/PGMap.h: disable network stats in dump_osd_stats

Signed-off-by: Neha Ojha <nojha@redhat.com>

osd_stat_t::dump: Add option for ceph-mgr pythonn callers to skip ping network

Due to the overhead of this on large clusters, we removed the network
information was wasn't getting used anyway.

Fixes: https://tracker.ceph.com/issues/43364
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #32362 into master

* refs/pull/32362/head:
qa/packages/packages: el8 has granular -debuginfo
qa/tasks/cbt: include py2 deps on ubuntu for now
src/test: misc python -> python3
qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos
qa/workunits: env python -> env python3
qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS
qa/suites/rados/verify: ping to specific centos
qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools
qa/workunits/rados/test_pool_access.sh: python -> python3
mgr/crash: fix signature for py3

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32374 from zdover23/wip-doc-osd-verb-commit-20-dec-2019

doc: added an "is" where it was needed.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32371 from tchaikov/wip-mgr-dashboard-popd

mgr/dashboard: restore working directory after creating venv

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/vstart_runner.py: use current python executable

python3 is not always the path to executable used by current python
session.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/packages/packages: el8 has granular -debuginfo

Also add a bunch of missing packages.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cbt: include py2 deps on ubuntu for now

...until cbt is converted to py3

Signed-off-by: Sage Weil <sage@redhat.com>

src/test: misc python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits: env python -> env python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS

"2019-12-19T20:42:43.020748+0000 mon.b (mon.0) 2771 : cluster [WRN] Health check failed: noout flag(s) set (OSDMAP_FLAGS)" in cluster log

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: ping to specific centos

The simple

 os_type: centos

in valgrind.yaml doesn't pick a particular centos, and we end up with
the teuthology default (currently 7.6).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools

We need it for snappy-devel

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_pool_access.sh: python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: fix signature for py3

With python3 the sig.digest() is bytes, so c is an int, not a char.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: added an "is" where it was needed.

This tiny commit changes "The foregoing functionality
equivalent to" to "The foregoing functionality is
equivalent to".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/dashboard: restore working directory after creating venv

this change addresses the regression introduced by 79584862f6

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32353 from mgfritch/wip-cephadm-rename-tox

cephadm: Rename tox tests ceph-daemon -> cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32173 from rosinL/wip-fix-coremask

msg/async/dpdk: Fix the overflow while parsing dpdk coremask

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32359 into master

* refs/pull/32359/head:
qa/workunits: python -> python3 throughout
qa/workunits/fs/misc/direct_io.py: fix for py3
qa/tasks/cephfs: os.write takes bytes, not str

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32355 into master

* refs/pull/32355/head:
qa/suites/rados/perf: run on ubuntu

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32356 into master

* refs/pull/32356/head:
qa/suites/rados/thrash-old-clients: centos -> ubuntu

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32357 into master

* refs/pull/32357/head:
qa/tasks/swift: remove swift tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32117 into master

* refs/pull/32117/head:
doc: add config help/get/set section for runtime client configuration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32168 into master

* refs/pull/32168/head:
test_cephfs_shell: fix test_du_works_for_hardlinks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits: python -> python3 throughout

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/fs/misc/direct_io.py: fix for py3

os.write takes bytes, os.read returns bytes

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/swift: remove swift tests

- This is an ancient swift version
- The tempest tests are newer and show provide similar coverage
- It somehow broke with the py3 transition

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: centos -> ubuntu

We can't upgrade packages from el7 to el8, so do this on ubuntu 18.04.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/perf: run on ubuntu

pdsh and collectl packages don't seem to exist on el8.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephfs: os.write takes bytes, not str

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31142 from zhangsw/fix-rgw-reshard-list-error

rgw: don't print error log when list reshard result is not truncated.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32335 into master

* refs/pull/32335/head:
qa/tasks/s3tests_java: move to gradle 6.0.1

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32343 into master

* refs/pull/32343/head:
doc: update '--force' flag to be precise

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31186 from zhangsw/clean-rgw-check-buckets-shard

rgw: remove unused bucket parameter in check_bucket_shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: Rename tox tests ceph-daemon -> cephadm

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31150 from zhangsw/cleanup-rgw-reshard-last_run

rgw: remove unused 'last_run' in reshard thread entry

Merge PR #32339 into master

* refs/pull/32339/head:
qa/tasks/cephfs: python3
qa/workunits/fs: python3
qa/tasks/cephfs/fuse_mount: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32344 from kotreshhr/boost_cmake_py3_fix

cmake/Boost: Fix python3 version

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake/Boost: Fix python3 version

Signed-off-by: Kotresh HR <khiremat@redhat.com>

doc: update '--force' flag to be precise

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #31525 from tchaikov/wip-no-py2-in-make-check

install-deps.sh,src: drop python2 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/run-rbd-tests: s/nosetests/python3 -m nose/

different distro use different names for nosetests3, so let's just go
with `python3 -m nose`

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: s/nosetests/python3/

different distros package python3-nose in different ways by adding
different postfix to "/usr/bin/nosetests" to differentiate it from
its python2 counterpart.

* on bionic, python3-nose offers "nosetests3"
* on el8, python3-nose offers "nosetests-3" and "nosetests-3.6"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: drop Find{Python,Python2}.cmake

since we are python3 only, there is no need to check Python2 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: do not echo version of git

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: silence popd and pushd

otherwise they will echo the pushed/popped directory

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: drop the compatibility with python2

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: revert a5a44cf4

we don't prepare wheels for python2 anymore, and we don't test for
python2. so no need to test for python2

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: stop preparing wheels for python2

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: remove py27 from envlist in tox.ini

Signed-off-by: Kefu Chai <kchai@redhat.com>

python-common: remove py27 from envlist in tox.ini

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume/test: check using python3 instead of python

Signed-off-by: Kefu Chai <kchai@redhat.com>

src,qa: install python bindings into lib/cython_modules/lib.3

instead of checking for the WITH_PYTHON* options, just hardwire to lib.3

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: python3

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: drop WITH_PYTHON2 option

* drop WITH_PYTHON2 option
* print warning message if WITH_PYTHON3 is disabled
* drop MGR_PYTHON_VERSION option, as we don't support use different
  python version for python binding and ceph-mgr embedded python
  interpreter anymore. as before switching to python3-only build,
  we can build python3 and python2 python bindings, and ceph-mgr
  can use either of them. but after switching to python3-only
  build, ceph-mgr has to use whatever python version used to
  build python binding.
* move WITH_PYTHON3 option to $top_srcdir/CMakeLists.txt, as ceph-mgr
  and python binding will share this option.
* hardware ${PYTHON_VERSION} to 3
* hardware ${Python${PYTHON_VERSION}_VERSION_MAJOR} to 3
* only build boost library with python3
* s/Python_EXECUTABLE/Python3_EXECUTABLE/
* update the build scripts and packagings accordingly
* rename all cython${PYTHON_VERSION}_* targets to cython_*
* update distutils_install_module() so it does not take python_version
  parameter anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: CMAKE_ROLE defaults to "PROJECT"

so we can use exported targets like "Python3::Python", which is defined
only if CMAKE_ROLE is "PROJECT". but this global property was introduced
by CMake 3.14, but the minimum required version of CMake is 3.10.2 in
Ceph project at this moment. let's set it if it's not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async/dpdk: Fix the overflow while parsing dpdk coremask

coremask supports up to 256 bits in DPDK19.05, using stoull in
dpdk::eal:init limits coremask up to 64 bits. Parse coremask by hex
character from low to high.

Fixes: https://tracker.ceph.com/issues/43044
Signed-off-by: Hu Ye <yehu5@huawei.com>
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

qa/workunits/fs: python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephfs/fuse_mount: use python3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32323 into master

* refs/pull/32323/head:
spec, debian: cephadm requires lvm2

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/s3tests_java: move to gradle 6.0.1

Also switch to using a stable symlink so that the version is determined by the branch.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32191 into master

* refs/pull/32191/head:
doc: clarify priority use

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32310 into master

* refs/pull/32310/head:
Revert "qa/suites/rgw: ragweed on ubuntu"
Revert "qa/suites/rgw: disable remaining ragweed test"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32121 from yehudasa/wip-42869

rgw: bucket re-creation fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32309 from cbodley/wip-43359

rgw: data sync markers include timestamp from datalog entry

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #32311 from cbodley/wip-43360

radosgw-admin: sync status displays id of shard furthest behind

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #32324 from tchaikov/wip-copr-repo

install-deps.sh: add ken's copr repo for el8 build

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32317 from tchaikov/wip-rpm-pyOpenSSL

ceph.spec.in: always depends on python3.6-pyOpenSSL

Reviewed-by: Alfredo Deza <adeza@redhat.com>

install-deps.sh: add ken's copr repo for el8 build

Signed-off-by: Kefu Chai <kchai@redhat.com>

spec, debian: cephadm requires lvm2

cephadm mounts `/run/lvm` into OSD containers, which only exists,
if `lvm2` is installed on the container host. Otherwise we cannot
start ceph-volume or osd containers.

Fixes: https://tracker.ceph.com/issues/43368
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph.spec.in: always depends on python3.6-pyOpenSSL

since we've moved to py3-only world, there is no need to depend on
pyOpenSSL anymore.

also, _python_buildid was removed in
3bf8b4d7d6012458fc761edd759a4fb3070a30fe .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31527 into master

* refs/pull/31527/head:
vstart: Add note for nfs-ganesha dev packages
stop.sh: Add option to kill Ganesha daemons
vstart: Add option to deploy ganesha daemons

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32303 from tchaikov/wip-build-integration-branch

build-integration-branch: s/prefix/postfix/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #32291 into master

* refs/pull/32291/head:
doc: update 'adding an MDS'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32252 into master

* refs/pull/32252/head:
qa/cephfs/begin: libaio-devel on el8
qa/tasks: nosetests -> python -m nose
qa/tasks/rbd_fio: fio 2.21 -> 3.16
src/test/cli-integration/rbd/snap-diff.t: python -> python
qa/workunits: use nose 3
qa/tasks/cbt: install python3 deps
qa/tasks/ceph_manager.py: do not use python to write a file
test/pybind/test_rados: execute takes a bytes (not str) payload
qa/packages/packages: python[3]-ceph is no more
qa: use python3 for venvs etc
packaging: remove python3-ipaddres, as it is part of the stdlib in py3
qa/packages: python-ceph -> python3-ceph
qa/distros: centos7 -> centos8, rhel7 -> rhel8
spec: remove _python_buildid in favor of python3_pkgversion macro
spec: remove python2 packages and conditions
debian: remove python >= 2.7 requirement
debian: add mgr python versions
debian: explicitly set PYTHON2=OFF to prevent picking up python2 interpreter
debian: update control file to use python3 dependency names
debian: remove all python2 overrides and declarations
debian: remove all python2 install files

Reviewed-by: Alfredo Deza <adeza@redhat.com>

radosgw-admin: sync status displays id of shard furthest behind

'radosgw-admin sync status' gives you a timestamp for the "oldest
incremental change not applied", but doesn't tell you which shard that
came from. add [shard-id] to the output

Fixes: https://tracker.ceph.com/issues/43360
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Revert "qa/suites/rgw: ragweed on ubuntu"

This reverts commit 73c2779524cc13bf69e025bdaed04035f5ba3968.

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rgw: disable remaining ragweed test"

This reverts commit 5a80fbe505e203227c4fa0de6141aaa69589d35e.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32146 into master

* refs/pull/32146/head:
mgr/ssh: Fix remove_osds()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32149 into master

* refs/pull/32149/head:
mgr/orchestrator: Rename PlacementSpec.nodes -> PlacementSpec.hosts
mgr/ansible: Added missing type annotations
mgr/orchestrator: Add __str__ to HostSpec
mgr/ansible: Integrate mgr/ansible/tox into mgr/tox

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32278 into master

* refs/pull/32278/head:
qa/suites/rgw: disable valgrind for tests that require py2/ubuntu
qa/suites/rgw: disable remaining ragweed test
qa/suites/rgw: pin swift tests to py2/ubuntu
qa/suites/rgw: ragweed on ubuntu
qa/suites: run s3tests on ubuntu

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/cephfs/begin: libaio-devel on el8

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: data sync markers include timestamp from datalog entry

this corrects the output of 'radosgw-admin data sync status' to show the
timestamp of the last datalog entry applied

Fixes: https://tracker.ceph.com/issues/43359
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32300 into master

* refs/pull/32300/head:
cephadm: drop 'sha256:' prefix on container id

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32302 into master

* refs/pull/32302/head:
cephadm: check for both ntp.service and ntpd.service

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/tasks: nosetests -> python -m nose

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw: disable valgrind for tests that require py2/ubuntu

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32280 from dillaman/wip-librbd-qos-test

qa/workunit/rbd: fixed QoS throughput unit parsing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #32279 into master

* refs/pull/32279/head:
mgr/orchestrator_cli: combine all host into one 'device ls' table
python-common: 'ssd/nvme' -> 'ssd'
mgr/orchestrator_cli: make 'device ls' table formatting consistent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32282 into master

* refs/pull/32282/head:
cephadm: clean up shell containers on exit
cephadm: do not force container removal on unit stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rgw: disable remaining ragweed test

Needs to be py3 since it relies on the python bindings.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32284 into master

* refs/pull/32284/head:
qa/suites/rbd/openstack: use 18.04, not 16.04

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: drop 'sha256:' prefix on container id

Podman doesn't include the prefix--only the (bionic) docker as far as I
can tell.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32273 from changchengx/doc_libvirt

doc/rbd: refine rbd/libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

build-integration-branch: s/prefix/postfix/

prefix is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32296 from rzarzynski/wip-seastar-pickup-futureopt

seastar: pickup the recent future optimizations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: Rename PlacementSpec.nodes -> PlacementSpec.hosts

By definition in the orchestrator's docs, we're using `host` insted of `node`

Fixes wrong usage of PlacementSpec in mgr/ansible

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32301 from tchaikov/wip-build-doc

admin/doc-requirements.txt: bump up Sphinx and breathe

Reviewed-by: Sage Weil <sage@redhat.com>