qa/suites/rados/thrash-old-clients: configure mons in terms of addrvecs

This is more explicit.  More importantly, the 'mon update' command
can't handle an "ip:port"; it wants either a CIDR, bare IP, or addrvec.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: hammer: fix package list

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: set .conf to cluster config object

ceph.py does this and ceph_manager.py kill_osd relies on it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: archive /var/log/ceph logs too (not just cluster dir)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: client keyring

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: setup thrashers ctx item

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: asok commands via cephadm shell

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: stick to el7

Old distros don't have packages for bionic.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: check cluster log; support log-whitelist

Mostly just lifted from ceph.py

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clienets: python-foo to python3-foo

Jewel and Hammer don't have python-rgw tho.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: add new exclude_packages

Due to 6f5fb9540896af28ad4ce1c14c4ad0a2974b03e3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: use cephadm

- deploy cluster with cephadm so we can run a octopus+ cluster and also
  install client packages that are ancient.
- move client.2 back onto the third node, since packages no longer
  conflict.
- test on centos 7.x (i picked 6), since the old releases all built on
  that release.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: make legacy mon addr/port parseable by legacy code

If it's a v1 addr on the old default 6789 port, print it as a legacy-style
addr (no v1: prefix) so that legacy code can parse it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32384 into master

* refs/pull/32384/head:
qa/tasks/mgr/test_orchestrator_cli: fix device ls test
mgr/orchestrator_cli: no need for colored=False

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32334 into master

* refs/pull/32334/head:
cephadm: add some coarse-grained locking
cephadm: add FileLock class

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32327 into master

* refs/pull/32327/head:
mgr/cephadm: remove old remotes
mgr/cephadm: try various pythons for root mode _run_cephadm
mgr/cephadm: no need to _get_connection from _create_daemon
mgr/cephadm: include host name in check-host result
mgr/cephadm: remove 'cephadm prepare-host' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #27866 into master

* refs/pull/27866/head:
mds: fix deadlock when xlocking policylock
mds: handle link request with zero depth filepath2
mds: enable lock cache for openc/unlink requests
mds: include linkage type in dentry lease
mds: cleanup Server::set_trace_dist()
mds: define lease mask bits
mds: delegete lock cache to client
mds: suppress frozen inode when locks of dir operation is cached.
mds: invalidate lock caches when freezing dirfrag/subtree
mds: invalidate lock caches if they hold conflicting locks
mds: initial code for lock cache
mds: adjust locking for subtree migration
mds: add 'path_locked' flag to MDCache::find_ino_peers()
mds: change MDCache::discover_path()'s wants_xlocked semantic
mds: introduce Server::rdlock_two_paths_xlock_destdn()
mds: make Server::rdlock_path_xlock_dentry take locks
mds: make Server::rdlock_path_pin_ref() take dentry rdlocks
mds: take snaplock and policylock during path traverse.
mds: let Locker::acquire_locks()'s caller choose locking order

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32380 from athanatos/sjust/wip-dmclock-fix

src/dmclock: bring in fixes for indirect_intrusive_heap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: remove old remotes

These are not used.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: try various pythons for root mode _run_cephadm

Some distros don't have the /usr/bin/python link.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix device ls test

No more "$host:" formatting as of 594179f1aae5fcca631b24c2f1e2dbf4a3b5c460

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: no need for colored=False

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32332 into master

* refs/pull/32332/head:
cephadm: bootstrap: force password choice

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32383 into master

* refs/pull/32383/head:
qa/standalone: python -> python3
qa/standalone/special/ceph_objectstore_tool: python3

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32272 into master

* refs/pull/32272/head:
pybind/mgr: Add orchestrator_cli to mgr/tox.ini
mgr/orchestrator_cli: Fix NFS

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32314 into master

* refs/pull/32314/head:
mgr/cephadm: fix listing services by host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32378 into master

* refs/pull/32378/head:
cephadm: python3 shebang

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32381 from athanatos/sjust/wip-read-from-replica-py2

osd: propagate mlcod to replicas and fix problems with read from replica

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32382 from athanatos/sjust/wip-backoff-fix

osd/scheduler/OpSchedulerItem: schedule backoffs as client ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32375 into master

* refs/pull/32375/head:
qa/tasks/cephadm: ceph.git branches are now pushed to quay.io

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/standalone: python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/special/ceph_objectstore_tool: python3

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: python3 shebang

We need to figure out if/how we can make this as tolerant as possible for
the curl users so that it can still run on a python2-only host.

Maybe.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30669 from theanalyst/rgw/user-stats-flush

rgw: radosgw-admin flush user stats output

Merge pull request #30993 from theanalyst/rgw/mp-list-encoding

rgw: support encoding-type param for list bucket multiparts

Merge pull request #31127 from theanalyst/rgw/qa-s3-user-caps

qa: rgw: add user-policy caps for the s3tests users

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Pritha Srivastava <prsivas@redhat.com>

Merge pull request #31124 from theanalyst/rgw/user-policy-args

rgw: iam: add all http args to req_info

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Pritha Srivastava <prsivas@redhat.com>
Reviewed-By: Yuval Lifshitz <yuvalif@yahoo.com>

Merge PR #32362 into master

* refs/pull/32362/head:
qa/packages/packages: el8 has granular -debuginfo
qa/tasks/cbt: include py2 deps on ubuntu for now
src/test: misc python -> python3
qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos
qa/workunits: env python -> env python3
qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS
qa/suites/rados/verify: ping to specific centos
qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools
qa/workunits/rados/test_pool_access.sh: python -> python3
mgr/crash: fix signature for py3

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/cephadm: ceph.git branches are now pushed to quay.io

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32374 from zdover23/wip-doc-osd-verb-commit-20-dec-2019

doc: added an "is" where it was needed.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32371 from tchaikov/wip-mgr-dashboard-popd

mgr/dashboard: restore working directory after creating venv

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/vstart_runner.py: use current python executable

python3 is not always the path to executable used by current python
session.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/packages/packages: el8 has granular -debuginfo

Also add a bunch of missing packages.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cbt: include py2 deps on ubuntu for now

...until cbt is converted to py3

Signed-off-by: Sage Weil <sage@redhat.com>

src/test: misc python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits: env python -> env python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS

"2019-12-19T20:42:43.020748+0000 mon.b (mon.0) 2771 : cluster [WRN] Health check failed: noout flag(s) set (OSDMAP_FLAGS)" in cluster log

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: ping to specific centos

The simple

 os_type: centos

in valgrind.yaml doesn't pick a particular centos, and we end up with
the teuthology default (currently 7.6).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools

We need it for snappy-devel

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_pool_access.sh: python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: fix signature for py3

With python3 the sig.digest() is bytes, so c is an int, not a char.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: added an "is" where it was needed.

This tiny commit changes "The foregoing functionality
equivalent to" to "The foregoing functionality is
equivalent to".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/dashboard: restore working directory after creating venv

this change addresses the regression introduced by 79584862f6

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32353 from mgfritch/wip-cephadm-rename-tox

cephadm: Rename tox tests ceph-daemon -> cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32173 from rosinL/wip-fix-coremask

msg/async/dpdk: Fix the overflow while parsing dpdk coremask

Reviewed-by: Kefu Chai <kchai@redhat.com>

PrimaryLogPG: report ENOENT on missing clone on replica

Can't check is_degraded_or_backfilling etc on replica.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG: drop replica reads from prior intervals

Otherwise, they cause misdirected op warnings.

See comment for details.

Signed-off-by: Samuel Just <sjust@redhat.com>

PrimaryLogPG: drop replica obc cache upon repop

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: bounce writes on objects written since mlcod back to primary

If there are any writes since mlcod on an object with the same head,
our ondisk state may be unstable or may reflect uncommitted state.
Bounce back to primary.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: maintain PeeringState::min_last_complete_ondisk on replica

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend,MOSDRepOp: replace pg_roll_forward_to with mlcod

New clients will ignore the field on header.version <= 2.  New
primaries will send the old variant if there are peers expecting
the old variant.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: always rollforward on repop to version

The primary always sets this field to be at_version, and we're
going to reuse this field to mean min_last_complete_ondisk.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PGBackend: rename submit_transaction field roll_forward_to

This field is actually just a lower bound on committed, client
visible log versions.

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_test_rados: use - not _ in args

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash[-erasure-code]: add misc -{localized,balanced}.yaml jobs

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rados: add {localize,balance}_reads field

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add --localize_reads option

Same as balance_reads, but sets the LOCALIZE_READS op flag on read ops.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/scheduler/OpSchedulerItem: schedule backoffs as client ops

The backoff machinery relies on backoffs being ordered the same
way as client ops.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/dmclock: bring in fixes for indirect_intrusive_heap

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #32359 into master

* refs/pull/32359/head:
qa/workunits: python -> python3 throughout
qa/workunits/fs/misc/direct_io.py: fix for py3
qa/tasks/cephfs: os.write takes bytes, not str

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32355 into master

* refs/pull/32355/head:
qa/suites/rados/perf: run on ubuntu

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32356 into master

* refs/pull/32356/head:
qa/suites/rados/thrash-old-clients: centos -> ubuntu

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32357 into master

* refs/pull/32357/head:
qa/tasks/swift: remove swift tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/cephadm: no need to _get_connection from _create_daemon

This is leftover from ages ago; we don't use conn in this context.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32117 into master

* refs/pull/32117/head:
doc: add config help/get/set section for runtime client configuration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32168 into master

* refs/pull/32168/head:
test_cephfs_shell: fix test_du_works_for_hardlinks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephadm: bootstrap: force password choice

Ignore the password complexity checks, since we might (randomly) generate
something that fails whatever the current criteria are.

INFO:cephadm:Non-zero exit code 22 from /usr/bin/podman run --rm --net=host -e CONTAINER_IMAGE=ceph/daemon-base:latest-master-devel -e NODE_NAME=gnit -v /var/log/ceph/00000000-0000-0000-0000-0000deadbeef:/var/log/ceph:z -v /tmp/ceph-tmp6if8iaha:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4vyvfiio:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph ceph/daemon-base:latest-master-devel dashboard ac-user-create admin oaaacwf4ul administrator
INFO:cephadm:/usr/bin/ceph:stderr Error EINVAL: Password cannot contain repetitive characters.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add some coarse-grained locking

Take per-cluster 'deploy' lock for

 bootstrap
 deploy
 adopt
 rm-daemon
 rm-cluster

No locking for

 version
 pull
 run (this is never actually used anyway)
 shell
 enter
 ceph-volume
 unit (this is a clean pass-through to systemctl)
 logs
 ls
 check-host

The only one I'm not sure about is ceph-volume.  I don't want the c-v
list/inventory commands to block, but perhaps the deploy ones should...
but perhaps not, since there's no reason we can't deploy multiple OSDs
on different devices at the same time.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add FileLock class

This is an abbreviated version of https://github.com/benediktschmitt/py-filelock

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits: python -> python3 throughout

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/fs/misc/direct_io.py: fix for py3

os.write takes bytes, os.read returns bytes

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/swift: remove swift tests

- This is an ancient swift version
- The tempest tests are newer and show provide similar coverage
- It somehow broke with the py3 transition

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: centos -> ubuntu

We can't upgrade packages from el7 to el8, so do this on ubuntu 18.04.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/perf: run on ubuntu

pdsh and collectl packages don't seem to exist on el8.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephfs: os.write takes bytes, not str

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31142 from zhangsw/fix-rgw-reshard-list-error

rgw: don't print error log when list reshard result is not truncated.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32335 into master

* refs/pull/32335/head:
qa/tasks/s3tests_java: move to gradle 6.0.1

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32343 into master

* refs/pull/32343/head:
doc: update '--force' flag to be precise

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31186 from zhangsw/clean-rgw-check-buckets-shard

rgw: remove unused bucket parameter in check_bucket_shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: Rename tox tests ceph-daemon -> cephadm

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31150 from zhangsw/cleanup-rgw-reshard-last_run

rgw: remove unused 'last_run' in reshard thread entry

Merge PR #32339 into master

* refs/pull/32339/head:
qa/tasks/cephfs: python3
qa/workunits/fs: python3
qa/tasks/cephfs/fuse_mount: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32344 from kotreshhr/boost_cmake_py3_fix

cmake/Boost: Fix python3 version

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake/Boost: Fix python3 version

Signed-off-by: Kotresh HR <khiremat@redhat.com>

doc: update '--force' flag to be precise

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #31525 from tchaikov/wip-no-py2-in-make-check

install-deps.sh,src: drop python2 support

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

test/run-rbd-tests: s/nosetests/python3 -m nose/

different distro use different names for nosetests3, so let's just go
with `python3 -m nose`

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: s/nosetests/python3/

different distros package python3-nose in different ways by adding
different postfix to "/usr/bin/nosetests" to differentiate it from
its python2 counterpart.

* on bionic, python3-nose offers "nosetests3"
* on el8, python3-nose offers "nosetests-3" and "nosetests-3.6"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: drop Find{Python,Python2}.cmake

since we are python3 only, there is no need to check Python2 anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>