Merge PR #34030 into octopus

* refs/pull/34030/head:
cephadm: env over last used
cephadm: fall back to default for infer_image
cephadm: remove outdated check
cephadm: consolidate default image logic
cephadm: only infer image for shell, run, inspect-image, pull, ceph-volume

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge PR #34027 into octopus

* refs/pull/34027/head:
qa/workunits/cephadm/test_cephadm: mark services unmanaged for test
mgr/cephadm: do not reconfig unmanaged services
qa/workunits/cephadm/test_cephadm: output file for pub key

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34045 into octopus

* refs/pull/34045/head:
mgr/progress: fix duration strings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cephadm: env over last used

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34035 from dillaman/wip-rbd-permissions

qa/workunits/rbd: use context managers to control Rados lifespan

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm: fall back to default for infer_image

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: remove outdated check

Either the user passed their own image or we filled in the (monitoring)
default--no need for this check.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: consolidate default image logic

Merge the _parse_args() logic into @default_image.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/progress: fix duration strings

- simplify the code to just calculate the durations when we need them
(I'm not sure why we had those temporary strings!)
- use a nicer time delta format

Fixes: https://tracker.ceph.com/issues/44672
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34014 into octopus

* refs/pull/34014/head:
mgr/balancer: tolerate pgs outside of target weight map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #34001 into octopus

* refs/pull/34001/head:
mgr/cephadm: pytest: Enable SpecStore
mgr/orchestrator: add test for default implementation for apply()
python-common: validate ServiceSpec.service_type
fixup mgr/cephadm: Fix ceph orch apply -i
mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion
mgr/orchestrator: `orch daemon add` accepts a yaml
mgr/cephadm: apply_drivegroups() returns a single Completion
mgr/cephadm: remove `trivial_result()`
mgr/cephadm: Fix `ceph orch apply -i`

Reviewed-by: Sage Weil <sage@redhat.com>

cephadm: only infer image for shell, run, inspect-image, pull, ceph-volume

Use a decorators for infer, require, and default.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34011 into octopus

* refs/pull/34011/head:
qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS
qa/suites/rados/verify/validater/valgrind: less bluestore logging
qa/suites/rados/verify/validater: increase heartbeat grace
Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"
Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"
ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set
ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set
qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/rbd: use context managers to control Rados lifespan

There is a potential race between the expected exceptions being
thrown and Python shutting down racing with librados background
threads. Ensure that librados is properly shut down prior to
exiting Python.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34032 from dillaman/wip-rbd-octopus-docs

doc/releases/octopus: add additional RBD improvements

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/releases/octopus: add additional RBD improvements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/cephadm/test_cephadm: mark services unmanaged for test

We are deploying containers manually.  Mark them unmanaged so that we
do not fight against mgr/cephadm cleaning up orphan daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not reconfig unmanaged services

Fixes: https://tracker.ceph.com/issues/44615
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33981 into octopus

* refs/pull/33981/head:
doc/install: edits
doc/cephadm: more edits
doc/cephadm/install: edits
doc/cephadm/adoption: improvements
doc/cephadm/install: a few edits
doc/cephadm/install: do not install ceph-common on host (by default)
doc/cephadm: drop os recs link
doc/cephadm/upgrade: improvements
doc/cephadm/upgrade: document upgrade
doc/cephadm/install: revamp install docs
doc: reorganize cephadm docs
doc/cephadm/administration: update docs on customizing SSH config
doc/cephadm/administration: add a note about the 'removed' dir

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #34018 from ajarr/octopus-subvolume-clone-cancel

mgr/volumes: add FS subvolume clone cancel

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/workunits/cephadm/test_cephadm: output file for pub key

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33866 into octopus

* refs/pull/33866/head:
mgr/cephadm: limit number of times check host is performed in the serve loop

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34005 into octopus

* refs/pull/34005/head:
python-common: reduce output of ServiceSpec.to_json()
python-common,mgr/cephadm: move assert_valid_host to service_spec
mgr/cephadm: add HostAssignment.validate()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34013 into octopus

* refs/pull/34013/head:
qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pytest: Enable SpecStore

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add test for default implementation for apply()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: validate ServiceSpec.service_type

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

fixup mgr/cephadm: Fix ceph orch apply -i

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: `orch daemon add` accepts a yaml

Fixes: https://tracker.ceph.com/issues/44622
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: apply_drivegroups() returns a single Completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: remove `trivial_result()`

Replaced with `@trivial_completion`, otherwise
`self._apply()` would have needed to accept a list
of specs.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Fix `ceph orch apply -i`

* We had two implementations for `ceph orcha apply`
* Remove yaml pasing in `mgr/cephadm`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33994 from dillaman/wip-librbd-poll-event-race

librbd: defer event socket completion until after callback issued

Reviewed-by: Dehao Shang <dehao.shang@intel.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: document `clone cancel` command

Fixes: http://tracker.ceph.com/issues/44208
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add `clone cancel` tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce "clone cancel" volume command

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: allow canceling a single asynchronous job for a volume

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: helper for looking up a clone entry index

... this would be used when canceling a clone operation to
find the clone index for a given clone.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if clone operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if copy operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce 'canceled' state in clone op state machine

When fetching the next execution state, -EINTR jumps to 'canceled'
state signifying a canceled (interrupted) operation. Also include
a helper routine to check if a given state machine is in initial
state.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater/valgrind: less bluestore logging

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater: increase heartbeat grace

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"

This reverts commit 4f742f200df6c91db87bfee1109c37fad3c0548b.

This was in the wrong file.. see valgrind.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"

This reverts commit 65e81e6eb4f136bf21b67e5de10ab49f028f9e95.

This slows things down too much with valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44062

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44582

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

When running under valgrind (and thrashing) things can be slow.  Tell
tests in case they need to tolerate timeouts.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/install: edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: more edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: edits

Based on review from Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/adoption: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: a few edits

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: do not install ceph-common on host (by default)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: drop os recs link

This mostly doesn't quite apply to cephadm because we use containers.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: document upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: revamp install docs

Signed-off-by: Sage Weil <sage@redhat.com>

doc: reorganize cephadm docs

- reorganized cephadm into a top-level item with a series of sub-items.
- condensed the 'install' page so that it doesn't create a zillion items
in the toctree on the left
- started updating the cephadm/install sequence (incomplete)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/administration: update docs on customizing SSH config

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/administration: add a note about the 'removed' dir

(and other paths)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: tolerate pgs outside of target weight map

We build a target weight map based on the primary crush weights, and
ignore weights that are 0.  However, it's possible that existing PGs are
on other OSDs that have weight 0 because the weight-set weight is >0.
That leads to a KeyError exception when we

                     pgs_by_osd[osd] += 1

and the key isn't present.  Fix by simply populating those keys as we
encounter OSDs.  Drop the old initialization loop.  The net of this is
we may have OSDs outside of target_by_root (won't matter, as far as I can
tell) and we won't have keys for osds with weight 0 (also won't matter,
as far as I can tell).

Fixes: https://tracker.ceph.com/issues/42721
Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack

We're deploying these things manually later, and they use fixed ports.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33974 into octopus

* refs/pull/33974/head:
use quay octopus tip until 15.2 tag is available
update default container images

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33442 into octopus

* refs/pull/33442/head:
cephadm: add allow_ptrace option to enable SYS_PTRACE

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33997 into octopus

* refs/pull/33997/head:
mon/MgrMonitor: make 'mgr fail' work with no arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34000 into octopus

* refs/pull/34000/head:
mgr/dashboard: adapt create_osds interface change

Reviewed-by: Laura Paduano <lpaduano@suse.com>

use quay octopus tip until 15.2 tag is available

Signed-off-by: Sage Weil <sage@redhat.com>

python-common: reduce output of ServiceSpec.to_json()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common,mgr/cephadm: move assert_valid_host to service_spec

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: add HostAssignment.validate()

Fixes: https://tracker.ceph.com/issues/44607
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: adapt create_osds interface change

Support the parameter change of create_osds since
https://github.com/ceph/ceph/pull/33922.

Fixes: https://tracker.ceph.com/issues/44632
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mon/MgrMonitor: make 'mgr fail' work with no arguments

Fail the active mgr so you don't have to check ceph -s and cut&paste it or
do some jq shenanigans with 'mgr dump'.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add allow_ptrace option to enable SYS_PTRACE

In any environments it is helpful to have SYS_PTRACE so that you can
gdb attach or strace a daemon.

Leave this off by default so that the container is more secure.

Enable this in teuthology and vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

update default container images

- For tests, use bleeding-edge octopus branch
- For production defaults, use ceph/ceph:v15.2 tag
- For bootstrap, grab cephadm script from latest octopus branch

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: limit number of times check host is performed in the serve loop

we only need this to happen every 10 min on hosts

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #33961 into octopus

* refs/pull/33961/head:
cephadm: add-repo: add --version
cephadm: rename distro args in repo methods

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33952 into octopus

* refs/pull/33952/head:
qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1
cephadm: add tests
qa/tasks/cephadm: pass -v to bootstrap
mgr/cephadm: only try to place mons on hosts matching public_network
mgr/cephadm: keep track of host networks, ips
cephadm: automatically infer mon public_network, if we can
cephadm: add list-networks command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33990 into octopus

* refs/pull/33990/head:
mgr/orch: add --all-available-devices to 'orch apply osd'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33955 into octopus

* refs/pull/33955/head:
mgr/cephadm: respect 'unmanaged' flag in spec
mgr/orch: orch ls: show <no spec> or <unmanaged> as appropriate
mgr/orch: orch ls: rename SPEC -> PLACEMENT
mgr/orch: add 'unmanaged' property to ServiceSpec
mgr/orch: combine 'orch daemon add <type> ...' into one command
mgr/orch: combine 'orch apply <type> [<placement>]' into one command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33936 into octopus

* refs/pull/33936/head:
cephadm: bootstrap: deploy monitoring stack by default

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/orch: add --all-available-devices to 'orch apply osd'

Provide a super-simple "use any avaialable device" command for
'orch apply osd'.  This will work for many (maybe even most?) users.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: pass -v to bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: only try to place mons on hosts matching public_network

Only try to schedule new mons on hosts that match the configured
public_network, if any.  If we do not have one configured, then don't
try to place new mons at all.

Note that there are other restrictions that ceph-mon supports that we
aren't considering, here: public_network_interface in particular, which
might further limit which IPs we consider binding to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: keep track of host networks, ips

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: automatically infer mon public_network, if we can

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add list-networks command

Summarize CIDR networks and our IPs in them.  Result looks like this:

{
    "10.3.64.0/24": [
        "10.3.64.23",
        "10.3.64.27"
    ],
    "10.88.0.0/16": [
        "10.88.0.1"
    ],
    "192.168.122.0/24": [
        "192.168.122.1"
    ]
}

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: bootstrap: deploy monitoring stack by default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33979 into octopus

* refs/pull/33979/head:
cephadm: implement 'install' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

librbd: defer event socket completion until after callback issued

A change post-Nautilus fixed an issue where multiple threads could
concurrently invoke callbacks to librbd clients. However, it also
introduced the potential that a callback hasn't yet fired by the
time the event socket is completed. This resulted in a crash of
fio under high-throughput testing since it expected both a callback
and the event socket completion, in that order.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33988 from dillaman/wip-44391

rbd-mirror: hold lock while updating local image name

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm: add-repo: add --version

Instead of --release octopus, which would get the latest octopus
version (whatever it might be), or possibly a repo with many build versions
inside, you can instead do --version 15.2.1 to get a repo with a
specific version and that version only.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: respect 'unmanaged' flag in spec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: orch ls: show <no spec> or <unmanaged> as appropriate

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: orch ls: rename SPEC -> PLACEMENT

More accurate, I think.  We only ever show placement here.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add 'unmanaged' property to ServiceSpec

Signed-off-by: Sage Weil <sage@redhat.com>