mgr/test_orchestrator: fix update_mgrs assert

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32232 into master

* refs/pull/32232/head:
qa: no need to exclude ceph-mgr-diskprediction-cloud from package list to be installed
qa/packages: do not install ceph-mgr-diskprediction-cloud by default
ceph.spec.in: add runtime deps for mgr-diskprediction-cloud

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32341 from dillaman/wip-43379

rbd-mirror: do not auto-create peers in non-default namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32202 from xiexingguo/wip-pg-stuck-undersized

osd/PeeringState: transit async_recovery_targets back into acting before backfilling

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32408 from dillaman/wip-devstack

qa/suites/rbd: fixed download path for Ubuntu Bionic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32305 from neha-ojha/wip-43308

osd/PeeringState.cc: don't let num_objects become negative

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32354 from rzarzynski/wip-log-id2name

log: print pthread ID / name mapping in recent events dump.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32364 from rzarzynski/wip-bug-42780

common: fix deadlocky inflight op visiting in OpTracker.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32316 from bk201/wip-43363

mgr/cephadm: fix type of timeout options

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32270 from sebastian-philipp/orchestrator-completion-already-finished

mgr/orchestrator: Raise more expressive Error, if completion already …

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32185 from batrick/interval-set-c++17

include: fix interval_set const_iterator call operator type

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30344 from shenhang/jsoninfo

osd/OSDMap.cc: add more info in json output of osd stat

Reviewed-by: Коренберг Марк <socketpair@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32183 from aclamk/fix-getxattr-leak

librados: fix leak in getxattr and getxattrs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32265 from simon-rock/simon_work_adv

tool: ceph_monstore_tool: --readable=0 => --readable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32222 from toabctl/qa-flake8-py3

qa: Run flake8 on python2 and python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32312 from badone/wip-nature-theme-changes

doc: Modify nature theme

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32366 from liu-chunmei/vstart

vstart.sh: move extra_seastar_args up in vstart.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32391 from tchaikov/wip-boost-1.72

install-deps.sh, cmake: use boost 1.72 on bionic

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32407 into master

* refs/pull/32407/head:
qa/suites/rados: move cephadm_orchestrator to el8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32392 into master

* refs/pull/32392/head:
cephadm: fix v1/v2 ip/addrv handling; explicitly check bind to ip:port

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32377 into master

* refs/pull/32377/head:
qa/suites/rados/thrash-old-clients: configure mons in terms of addrvecs
qa/suites/rados/thrash-old-clients: hammer: fix package list
qa/tasks/cephadm: set .conf to cluster config object
qa/tasks/cephadm: archive /var/log/ceph logs too (not just cluster dir)
qa/tasks/cephadm: client keyring
qa/tasks/cephadm: setup thrashers ctx item
qa/tasks/ceph_manager: asok commands via cephadm shell
qa/suites/rados/thrash-old-clients: stick to el7
qa/tasks/cephadm: check cluster log; support log-whitelist
qa/suites/rados/thrash-old-clienets: python-foo to python3-foo
qa/suites/rados/thrash-old-clients: add new exclude_packages
qa/suites/rados/thrash-old-clients: use cephadm
mon/ConfigMonitor: make legacy mon addr/port parseable by legacy code

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-old-clients: configure mons in terms of addrvecs

This is more explicit.  More importantly, the 'mon update' command
can't handle an "ip:port"; it wants either a CIDR, bare IP, or addrvec.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: hammer: fix package list

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: set .conf to cluster config object

ceph.py does this and ceph_manager.py kill_osd relies on it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: archive /var/log/ceph logs too (not just cluster dir)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: client keyring

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32390 into master

* refs/pull/32390/head:
cephadm: create /var/run/ceph/$fsid as needed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32281 into master

* refs/pull/32281/head:
mgr/cephadm: fix =[] default arg to _create_daemon
qa/suites/rados/cephadm: enable mgr debugging
cephadm: restart non-ceph daemons on reconfig
mgr/cephadm/test_cephadm: fix expected output
mgr/cephadm: fix remapping of hosts and services
mgr/cephadm: add 'reconfig' service action
cephadm: move mon creation config bits into mgr/cephadm
cephadm: fix bootstrap message 'ssh' -> 'cephadm'
cephadm: add --reconfig option for 'deploy'
mgr/orchestrator: drop 'reload' service[-instance] action

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32338 from trociny/wip-peer-list-default-ns

librbd: look for mirror peers in default namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rados: move cephadm_orchestrator to el8

The python3-remoto dependency does not exist on 18.04 (or any ubuntu or
debian AFAICS).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rbd: fixed download path for Ubuntu Bionic

The OS image was changed in a9ee4bcf241 from Xenial to Bionic,
but the Bionic image path is incorrect.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/cephadm: setup thrashers ctx item

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: asok commands via cephadm shell

Signed-off-by: Sage Weil <sage@redhat.com>

make-dist: package boost v1.72 instead of v1.67

to be consistent with the version used by "make check"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build boost v1.72 instead of v1.67

so we can drop two patches which are already included by v1.72

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not use small_vector

if `std::is_nothrow_move_constructible<small_vector<peer_shard_t, 2>>`
also depends on if the allocator throws when moving from a small_vector
to a new one, as we need to allocate memory for the destination
small_vector.

so, in this change, the optimization for small_vector is dropped,
because the default allocator used by small_vector throws.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: use "apt" for checking install status

as "dpkg -s <package-name>" always returns 0 even if the <package> is
removed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,rgw: workaround for boost 1.72

see also https://github.com/boostorg/coroutine/issues/46

will need to remove this change once we require boost >= 1.73

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: fix =[] default arg to _create_daemon

This is a no-no (pass by reference, not value)!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm: enable mgr debugging

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32368 from xiexingguo/wip-balancer-efficiency-4

osd/OSDMap: consider overfull osds only when trying to do upmap

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #32289 from xiexingguo/wip-balancer-efficiency-2

mgr/balancer: eliminate usage of MS infrastructure for upmap mode

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

install-deps.sh: install boost-1.72 for bionic

and uninstall boost-1.67 before installing boost-1.72

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: move extra_seastar_agrs up in vstart.sh

add extra_seastar_args in --mkfs

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

cephadm: create /var/run/ceph/$fsid as needed

The get_container_mounts method only passed this through if it already
exists so that shell etc commands will work even when this dir isn't
present yet.  Normally the systemd unit creates it.  This may not be the
case on initial bootstrap, of even later if a deploy happens but a unit
hasn't started yet.

Precreate it during bootstrap and other times before we do a deplot so
that the unit.run command has this passed through.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix v1/v2 ip/addrv handling; explicitly check bind to ip:port

- Handle the various v1 and v2 cases
- Handle --mon-ip with a port
- Explicitly test bind to the IP + port combination

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32389 into master

* refs/pull/32389/head:
test_cephadm.sh: pass --fsid to shell command

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: restart non-ceph daemons on reconfig

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/test_cephadm: fix expected output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix remapping of hosts and services

This was nondeterministically remapping services to different hosts
depending on which hosts had stale caches because the 'hosts' list order
didn't match 'results' + 'in_cache'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32286 into master

* refs/pull/32286/head:
cephadm: correct ipv6 support in port open detection

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: add 'reconfig' service action

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: move mon creation config bits into mgr/cephadm

There's no need for mgr/cephadm to pass --mon-* arguments to
/usr/bin/cephadm just so it can add bits to the config file when the
config file is also being passed in from mgr/cephadm.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix bootstrap message 'ssh' -> 'cephadm'

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add --reconfig option for 'deploy'

We could make this a totally separate command from 'deploy', but so much
of the code path is shared that this is a lot simpler.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: drop 'reload' service[-instance] action

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32388 from wjwithagen/wjw-fix-boost_spirit-isblank-assert

test: Do not test unicode if boost::spirit >= 1.72

Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: drop trusty and xenial support

since we only build on bionic on ubuntu distros, there is no need
to build on trusty and xenial anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update FindBoost.cmake for 1.72

adapted from
https://github.com/Kitware/CMake/blob/master/Modules/FindBoost.cmake
commit e51232b739b3eeae642be8a7e89079f7c2e8c1b2

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Do not test unicode if boost::spirit >= 1.72

Testing with Boost 1.72 unittest_confutil crashes with:
```
[ RUN      ] ConfUtils.ReadFiles2
Assertion failed: (strict_ischar(ch)), function isblank, file /usr/local/include/boost/spirit/home/support/char_encoding/standard.hpp, line 129.
Abort
```

Which is due to the fact that in Boost 1.72 most of the character
testing has been garded with an assert to make sure char holds an
ascii char.
Like:
```
        static bool
        isblank BOOST_PREVENT_MACRO_SUBSTITUTION (int ch)
        {
            BOOST_ASSERT(strict_ischar(ch));
            return (ch == ' ' || ch == '\t');
        }
```
And ReadFile2 explicitly tests for strings with char value > 0x7f.
So that is certainly going to crash.

Now I can imagine that we would like to be able to have users use
different encodings for things like logfiles.
But for now that is not going to work with the boost:spirit parser

Fixes: https://tracker.ceph.com/issues/43406
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

test_cephadm.sh: pass --fsid to shell command

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32261 into master

* refs/pull/32261/head:
mgr/orchestrator: use full device path for blinking lights (if available)
mgr: report device by-path paths too
common/blkdev: include by-path path for each device in metadata
common/blkdev: factor get_device_metadata out of mon, osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32333 into master

* refs/pull/32333/head:
qa/quites/rados/singleton-flat/valgrind-leaks: specify centos8
qa/suites/rados: test cephadm on centos and ubuntu both
qa/workunits/cephadm/test_cephadm.sh: detect python3 and python2

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/thrash-old-clients: stick to el7

Old distros don't have packages for bionic.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: check cluster log; support log-whitelist

Mostly just lifted from ceph.py

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clienets: python-foo to python3-foo

Jewel and Hammer don't have python-rgw tho.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: add new exclude_packages

Due to 6f5fb9540896af28ad4ce1c14c4ad0a2974b03e3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: use cephadm

- deploy cluster with cephadm so we can run a octopus+ cluster and also
  install client packages that are ancient.
- move client.2 back onto the third node, since packages no longer
  conflict.
- test on centos 7.x (i picked 6), since the old releases all built on
  that release.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/ConfigMonitor: make legacy mon addr/port parseable by legacy code

If it's a v1 addr on the old default 6789 port, print it as a legacy-style
addr (no v1: prefix) so that legacy code can parse it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/quites/rados/singleton-flat/valgrind-leaks: specify centos8

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados: test cephadm on centos and ubuntu both

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: detect python3 and python2

Only test on whichever pythons are available on the host.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32384 into master

* refs/pull/32384/head:
qa/tasks/mgr/test_orchestrator_cli: fix device ls test
mgr/orchestrator_cli: no need for colored=False

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32334 into master

* refs/pull/32334/head:
cephadm: add some coarse-grained locking
cephadm: add FileLock class

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32327 into master

* refs/pull/32327/head:
mgr/cephadm: remove old remotes
mgr/cephadm: try various pythons for root mode _run_cephadm
mgr/cephadm: no need to _get_connection from _create_daemon
mgr/cephadm: include host name in check-host result
mgr/cephadm: remove 'cephadm prepare-host' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #27866 into master

* refs/pull/27866/head:
mds: fix deadlock when xlocking policylock
mds: handle link request with zero depth filepath2
mds: enable lock cache for openc/unlink requests
mds: include linkage type in dentry lease
mds: cleanup Server::set_trace_dist()
mds: define lease mask bits
mds: delegete lock cache to client
mds: suppress frozen inode when locks of dir operation is cached.
mds: invalidate lock caches when freezing dirfrag/subtree
mds: invalidate lock caches if they hold conflicting locks
mds: initial code for lock cache
mds: adjust locking for subtree migration
mds: add 'path_locked' flag to MDCache::find_ino_peers()
mds: change MDCache::discover_path()'s wants_xlocked semantic
mds: introduce Server::rdlock_two_paths_xlock_destdn()
mds: make Server::rdlock_path_xlock_dentry take locks
mds: make Server::rdlock_path_pin_ref() take dentry rdlocks
mds: take snaplock and policylock during path traverse.
mds: let Locker::acquire_locks()'s caller choose locking order

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32380 from athanatos/sjust/wip-dmclock-fix

src/dmclock: bring in fixes for indirect_intrusive_heap

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: Modify nature theme

Apply ceph project customisations to default nature theme.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mgr/cephadm: remove old remotes

These are not used.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: try various pythons for root mode _run_cephadm

Some distros don't have the /usr/bin/python link.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix device ls test

No more "$host:" formatting as of 594179f1aae5fcca631b24c2f1e2dbf4a3b5c460

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: no need for colored=False

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32332 into master

* refs/pull/32332/head:
cephadm: bootstrap: force password choice

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

doc: Update to current version of 'nature' theme

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: Change copyright attribution to ceph authors

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #32383 into master

* refs/pull/32383/head:
qa/standalone: python -> python3
qa/standalone/special/ceph_objectstore_tool: python3

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32272 into master

* refs/pull/32272/head:
pybind/mgr: Add orchestrator_cli to mgr/tox.ini
mgr/orchestrator_cli: Fix NFS

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32314 into master

* refs/pull/32314/head:
mgr/cephadm: fix listing services by host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32378 into master

* refs/pull/32378/head:
cephadm: python3 shebang

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32381 from athanatos/sjust/wip-read-from-replica-py2

osd: propagate mlcod to replicas and fix problems with read from replica

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32382 from athanatos/sjust/wip-backoff-fix

osd/scheduler/OpSchedulerItem: schedule backoffs as client ops

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32375 into master

* refs/pull/32375/head:
qa/tasks/cephadm: ceph.git branches are now pushed to quay.io

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/standalone: python -> python3

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/special/ceph_objectstore_tool: python3

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: python3 shebang

We need to figure out if/how we can make this as tolerant as possible for
the curl users so that it can still run on a python2-only host.

Maybe.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30669 from theanalyst/rgw/user-stats-flush

rgw: radosgw-admin flush user stats output

Merge pull request #30993 from theanalyst/rgw/mp-list-encoding

rgw: support encoding-type param for list bucket multiparts

Merge pull request #31127 from theanalyst/rgw/qa-s3-user-caps

qa: rgw: add user-policy caps for the s3tests users

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Pritha Srivastava <prsivas@redhat.com>

Merge pull request #31124 from theanalyst/rgw/user-policy-args

rgw: iam: add all http args to req_info

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Pritha Srivastava <prsivas@redhat.com>
Reviewed-By: Yuval Lifshitz <yuvalif@yahoo.com>

Merge PR #32362 into master

* refs/pull/32362/head:
qa/packages/packages: el8 has granular -debuginfo
qa/tasks/cbt: include py2 deps on ubuntu for now
src/test: misc python -> python3
qa/suites/rados/singleton-flag/valgrind-leaks: run on latest centos
qa/workunits: env python -> env python3
qa/suites/rados/dashboard/tasks/dashboard: whitelist OSDMAP_FLAGS
qa/suites/rados/verify: ping to specific centos
qa/workunits/rados/test_envlibrados_rocksdb: enable el8 PowerTools
qa/workunits/rados/test_pool_access.sh: python -> python3
mgr/crash: fix signature for py3

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>