Merge PR #32390 into master

* refs/pull/32390/head:
cephadm: create /var/run/ceph/$fsid as needed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32281 into master

* refs/pull/32281/head:
mgr/cephadm: fix =[] default arg to _create_daemon
qa/suites/rados/cephadm: enable mgr debugging
cephadm: restart non-ceph daemons on reconfig
mgr/cephadm/test_cephadm: fix expected output
mgr/cephadm: fix remapping of hosts and services
mgr/cephadm: add 'reconfig' service action
cephadm: move mon creation config bits into mgr/cephadm
cephadm: fix bootstrap message 'ssh' -> 'cephadm'
cephadm: add --reconfig option for 'deploy'
mgr/orchestrator: drop 'reload' service[-instance] action

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32338 from trociny/wip-peer-list-default-ns

librbd: look for mirror peers in default namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: fix =[] default arg to _create_daemon

This is a no-no (pass by reference, not value)!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm: enable mgr debugging

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32368 from xiexingguo/wip-balancer-efficiency-4

osd/OSDMap: consider overfull osds only when trying to do upmap

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #32289 from xiexingguo/wip-balancer-efficiency-2

mgr/balancer: eliminate usage of MS infrastructure for upmap mode

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

cephadm: create /var/run/ceph/$fsid as needed

The get_container_mounts method only passed this through if it already
exists so that shell etc commands will work even when this dir isn't
present yet.  Normally the systemd unit creates it.  This may not be the
case on initial bootstrap, of even later if a deploy happens but a unit
hasn't started yet.

Precreate it during bootstrap and other times before we do a deplot so
that the unit.run command has this passed through.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32389 into master

* refs/pull/32389/head:
test_cephadm.sh: pass --fsid to shell command

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: restart non-ceph daemons on reconfig

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/test_cephadm: fix expected output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix remapping of hosts and services

This was nondeterministically remapping services to different hosts
depending on which hosts had stale caches because the 'hosts' list order
didn't match 'results' + 'in_cache'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32286 into master

* refs/pull/32286/head:
cephadm: correct ipv6 support in port open detection

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: add 'reconfig' service action

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: move mon creation config bits into mgr/cephadm

There's no need for mgr/cephadm to pass --mon-* arguments to
/usr/bin/cephadm just so it can add bits to the config file when the
config file is also being passed in from mgr/cephadm.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix bootstrap message 'ssh' -> 'cephadm'

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add --reconfig option for 'deploy'

We could make this a totally separate command from 'deploy', but so much
of the code path is shared that this is a lot simpler.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: drop 'reload' service[-instance] action

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32388 from wjwithagen/wjw-fix-boost_spirit-isblank-assert

test: Do not test unicode if boost::spirit >= 1.72

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Do not test unicode if boost::spirit >= 1.72

Testing with Boost 1.72 unittest_confutil crashes with:
```
[ RUN      ] ConfUtils.ReadFiles2
Assertion failed: (strict_ischar(ch)), function isblank, file /usr/local/include/boost/spirit/home/support/char_encoding/standard.hpp, line 129.
Abort
```

Which is due to the fact that in Boost 1.72 most of the character
testing has been garded with an assert to make sure char holds an
ascii char.
Like:
```
        static bool
        isblank BOOST_PREVENT_MACRO_SUBSTITUTION (int ch)
        {
            BOOST_ASSERT(strict_ischar(ch));
            return (ch == ' ' || ch == '\t');
        }
```
And ReadFile2 explicitly tests for strings with char value > 0x7f.
So that is certainly going to crash.

Now I can imagine that we would like to be able to have users use
different encodings for things like logfiles.
But for now that is not going to work with the boost:spirit parser

Fixes: https://tracker.ceph.com/issues/43406
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test_cephadm.sh: pass --fsid to shell command

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32261 into master

* refs/pull/32261/head:
mgr/orchestrator: use full device path for blinking lights (if available)
mgr: report device by-path paths too
common/blkdev: include by-path path for each device in metadata
common/blkdev: factor get_device_metadata out of mon, osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32333 into master

* refs/pull/32333/head:
qa/quites/rados/singleton-flat/valgrind-leaks: specify centos8
qa/suites/rados: test cephadm on centos and ubuntu both
qa/workunits/cephadm/test_cephadm.sh: detect python3 and python2

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/quites/rados/singleton-flat/valgrind-leaks: specify centos8

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados: test cephadm on centos and ubuntu both

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: detect python3 and python2

Only test on whichever pythons are available on the host.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32384 into master

* refs/pull/32384/head:
qa/tasks/mgr/test_orchestrator_cli: fix device ls test
mgr/orchestrator_cli: no need for colored=False

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32334 into master

* refs/pull/32334/head:
cephadm: add some coarse-grained locking
cephadm: add FileLock class

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32327 into master

* refs/pull/32327/head:
mgr/cephadm: remove old remotes
mgr/cephadm: try various pythons for root mode _run_cephadm
mgr/cephadm: no need to _get_connection from _create_daemon
mgr/cephadm: include host name in check-host result
mgr/cephadm: remove 'cephadm prepare-host' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #27866 into master

* refs/pull/27866/head:
mds: fix deadlock when xlocking policylock
mds: handle link request with zero depth filepath2
mds: enable lock cache for openc/unlink requests
mds: include linkage type in dentry lease
mds: cleanup Server::set_trace_dist()
mds: define lease mask bits
mds: delegete lock cache to client
mds: suppress frozen inode when locks of dir operation is cached.
mds: invalidate lock caches when freezing dirfrag/subtree
mds: invalidate lock caches if they hold conflicting locks
mds: initial code for lock cache
mds: adjust locking for subtree migration
mds: add 'path_locked' flag to MDCache::find_ino_peers()
mds: change MDCache::discover_path()'s wants_xlocked semantic
mds: introduce Server::rdlock_two_paths_xlock_destdn()
mds: make Server::rdlock_path_xlock_dentry take locks
mds: make Server::rdlock_path_pin_ref() take dentry rdlocks
mds: take snaplock and policylock during path traverse.
mds: let Locker::acquire_locks()'s caller choose locking order

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32380 from athanatos/sjust/wip-dmclock-fix

src/dmclock: bring in fixes for indirect_intrusive_heap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: remove old remotes

These are not used.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: try various pythons for root mode _run_cephadm

Some distros don't have the /usr/bin/python link.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix device ls test

No more "$host:" formatting as of 594179f1aae5fcca631b24c2f1e2dbf4a3b5c460

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: no need for colored=False

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32332 into master

* refs/pull/32332/head:
cephadm: bootstrap: force password choice

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32383 into master

* refs/pull/32383/head:
qa/standalone: python -> python3
qa/standalone/special/ceph_objectstore_tool: python3

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32272 into master

* refs/pull/32272/head:
pybind/mgr: Add orchestrator_cli to mgr/tox.ini
mgr/orchestrator_cli: Fix NFS

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32314 into master

* refs/pull/32314/head:
mgr/cephadm: fix listing services by host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32378 into master

* refs/pull/32378/head:
cephadm: python3 shebang

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32381 from athanatos/sjust/wip-read-from-replica-py2

osd: propagate mlcod to replicas and fix problems with read from replica

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32382 from athanatos/sjust/wip-backoff-fix

osd/scheduler/OpSchedulerItem: schedule backoffs as client ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32375 into master

* refs/pull/32375/head:
qa/tasks/cephadm: ceph.git branches are now pushed to quay.io

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/standalone: python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/special/ceph_objectstore_tool: python3

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: python3 shebang

We need to figure out if/how we can make this as tolerant as possible for
the curl users so that it can still run on a python2-only host.

Maybe.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30669 from theanalyst/rgw/user-stats-flush

rgw: radosgw-admin flush user stats output

Merge pull request #30993 from theanalyst/rgw/mp-list-encoding

rgw: support encoding-type param for list bucket multiparts

Merge pull request #31127 from theanalyst/rgw/qa-s3-user-caps

qa: rgw: add user-policy caps for the s3tests users

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Pritha Srivastava <prsivas@redhat.com>

Merge pull request #31124 from theanalyst/rgw/user-policy-args

rgw: iam: add all http args to req_info

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Pritha Srivastava <prsivas@redhat.com>
Reviewed-By: Yuval Lifshitz <yuvalif@yahoo.com>

Merge PR #32362 into master

* refs/pull/32362/head:
qa/packages/packages: el8 has granular -debuginfo
qa/tasks/cbt: include py2 deps on ubuntu for now
src/test: misc python -> python3
qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos
qa/workunits: env python -> env python3
qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS
qa/suites/rados/verify: ping to specific centos
qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools
qa/workunits/rados/test_pool_access.sh: python -> python3
mgr/crash: fix signature for py3

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/cephadm: ceph.git branches are now pushed to quay.io

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32374 from zdover23/wip-doc-osd-verb-commit-20-dec-2019

doc: added an "is" where it was needed.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32371 from tchaikov/wip-mgr-dashboard-popd

mgr/dashboard: restore working directory after creating venv

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/vstart_runner.py: use current python executable

python3 is not always the path to executable used by current python
session.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/packages/packages: el8 has granular -debuginfo

Also add a bunch of missing packages.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cbt: include py2 deps on ubuntu for now

...until cbt is converted to py3

Signed-off-by: Sage Weil <sage@redhat.com>

src/test: misc python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits: env python -> env python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS

"2019-12-19T20:42:43.020748+0000 mon.b (mon.0) 2771 : cluster [WRN] Health check failed: noout flag(s) set (OSDMAP_FLAGS)" in cluster log

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: ping to specific centos

The simple

 os_type: centos

in valgrind.yaml doesn't pick a particular centos, and we end up with
the teuthology default (currently 7.6).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools

We need it for snappy-devel

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_pool_access.sh: python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/crash: fix signature for py3

With python3 the sig.digest() is bytes, so c is an int, not a char.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: added an "is" where it was needed.

This tiny commit changes "The foregoing functionality
equivalent to" to "The foregoing functionality is
equivalent to".

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/orchestrator: use full device path for blinking lights (if available)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: restore working directory after creating venv

this change addresses the regression introduced by 79584862f6

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32353 from mgfritch/wip-cephadm-rename-tox

cephadm: Rename tox tests ceph-daemon -> cephadm

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd/OSDMap: consider overfull osds only when trying to do upmap

By definition, we should only upmap pgs from overfull osds to
underfull osds. Hence we could make a safe exit if we start
to hit underfull osds when searching a target osd for upmapping.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #32173 from rosinL/wip-fix-coremask

msg/async/dpdk: Fix the overflow while parsing dpdk coremask

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: look for mirror peers in default namespace

(when creating primary mirror snapshot)

Signed-off-by: Mykola Golub <mgolub@suse.com>

PrimaryLogPG: report ENOENT on missing clone on replica

Can't check is_degraded_or_backfilling etc on replica.

Signed-off-by: Samuel Just <sjust@redhat.com>

PG: drop replica reads from prior intervals

Otherwise, they cause misdirected op warnings.

See comment for details.

Signed-off-by: Samuel Just <sjust@redhat.com>

PrimaryLogPG: drop replica obc cache upon repop

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: bounce writes on objects written since mlcod back to primary

If there are any writes since mlcod on an object with the same head,
our ondisk state may be unstable or may reflect uncommitted state.
Bounce back to primary.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: maintain PeeringState::min_last_complete_ondisk on replica

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend,MOSDRepOp: replace pg_roll_forward_to with mlcod

New clients will ignore the field on header.version <= 2.  New
primaries will send the old variant if there are peers expecting
the old variant.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: always rollforward on repop to version

The primary always sets this field to be at_version, and we're
going to reuse this field to mean min_last_complete_ondisk.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PGBackend: rename submit_transaction field roll_forward_to

This field is actually just a lower bound on committed, client
visible log versions.

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_test_rados: use - not _ in args

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash[-erasure-code]: add misc -{localized,balanced}.yaml jobs

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rados: add {localize,balance}_reads field

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add --localize_reads option

Same as balance_reads, but sets the LOCALIZE_READS op flag on read ops.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/scheduler/OpSchedulerItem: schedule backoffs as client ops

The backoff machinery relies on backoffs being ordered the same
way as client ops.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/dmclock: bring in fixes for indirect_intrusive_heap

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #32359 into master

* refs/pull/32359/head:
qa/workunits: python -> python3 throughout
qa/workunits/fs/misc/direct_io.py: fix for py3
qa/tasks/cephfs: os.write takes bytes, not str

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32355 into master

* refs/pull/32355/head:
qa/suites/rados/perf: run on ubuntu

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32356 into master

* refs/pull/32356/head:
qa/suites/rados/thrash-old-clients: centos -> ubuntu

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32357 into master

* refs/pull/32357/head:
qa/tasks/swift: remove swift tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: correct ipv6 support in port open detection

Updated the port_in_use function to work better
in ipv4 and ipv6 environments.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/cephadm: no need to _get_connection from _create_daemon

This is leftover from ages ago; we don't use conn in this context.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32117 into master

* refs/pull/32117/head:
doc: add config help/get/set section for runtime client configuration

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32168 into master

* refs/pull/32168/head:
test_cephfs_shell: fix test_du_works_for_hardlinks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephadm: bootstrap: force password choice

Ignore the password complexity checks, since we might (randomly) generate
something that fails whatever the current criteria are.

INFO:cephadm:Non-zero exit code 22 from /usr/bin/podman run --rm --net=host -e CONTAINER_IMAGE=ceph/daemon-base:latest-master-devel -e NODE_NAME=gnit -v /var/log/ceph/00000000-0000-0000-0000-0000deadbeef:/var/log/ceph:z -v /tmp/ceph-tmp6if8iaha:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4vyvfiio:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/ceph ceph/daemon-base:latest-master-devel dashboard ac-user-create admin oaaacwf4ul administrator
INFO:cephadm:/usr/bin/ceph:stderr Error EINVAL: Password cannot contain repetitive characters.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add some coarse-grained locking

Take per-cluster 'deploy' lock for

 bootstrap
 deploy
 adopt
 rm-daemon
 rm-cluster

No locking for

 version
 pull
 run (this is never actually used anyway)
 shell
 enter
 ceph-volume
 unit (this is a clean pass-through to systemctl)
 logs
 ls
 check-host

The only one I'm not sure about is ceph-volume.  I don't want the c-v
list/inventory commands to block, but perhaps the deploy ones should...
but perhaps not, since there's no reason we can't deploy multiple OSDs
on different devices at the same time.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add FileLock class

This is an abbreviated version of https://github.com/benediktschmitt/py-filelock

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits: python -> python3 throughout

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/fs/misc/direct_io.py: fix for py3

os.write takes bytes, os.read returns bytes

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/swift: remove swift tests

- This is an ancient swift version
- The tempest tests are newer and show provide similar coverage
- It somehow broke with the py3 transition

Signed-off-by: Sage Weil <sage@redhat.com>