Merge PR #34001 into octopus

* refs/pull/34001/head:
mgr/cephadm: pytest: Enable SpecStore
mgr/orchestrator: add test for default implementation for apply()
python-common: validate ServiceSpec.service_type
fixup mgr/cephadm: Fix ceph orch apply -i
mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion
mgr/orchestrator: `orch daemon add` accepts a yaml
mgr/cephadm: apply_drivegroups() returns a single Completion
mgr/cephadm: remove `trivial_result()`
mgr/cephadm: Fix `ceph orch apply -i`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34011 into octopus

* refs/pull/34011/head:
qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS
qa/suites/rados/verify/validater/valgrind: less bluestore logging
qa/suites/rados/verify/validater: increase heartbeat grace
Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"
Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"
ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set
ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set
qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34032 from dillaman/wip-rbd-octopus-docs

doc/releases/octopus: add additional RBD improvements

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/releases/octopus: add additional RBD improvements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33981 into octopus

* refs/pull/33981/head:
doc/install: edits
doc/cephadm: more edits
doc/cephadm/install: edits
doc/cephadm/adoption: improvements
doc/cephadm/install: a few edits
doc/cephadm/install: do not install ceph-common on host (by default)
doc/cephadm: drop os recs link
doc/cephadm/upgrade: improvements
doc/cephadm/upgrade: document upgrade
doc/cephadm/install: revamp install docs
doc: reorganize cephadm docs
doc/cephadm/administration: update docs on customizing SSH config
doc/cephadm/administration: add a note about the 'removed' dir

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #34018 from ajarr/octopus-subvolume-clone-cancel

mgr/volumes: add FS subvolume clone cancel

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #33866 into octopus

* refs/pull/33866/head:
mgr/cephadm: limit number of times check host is performed in the serve loop

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34005 into octopus

* refs/pull/34005/head:
python-common: reduce output of ServiceSpec.to_json()
python-common,mgr/cephadm: move assert_valid_host to service_spec
mgr/cephadm: add HostAssignment.validate()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34013 into octopus

* refs/pull/34013/head:
qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pytest: Enable SpecStore

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add test for default implementation for apply()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: validate ServiceSpec.service_type

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

fixup mgr/cephadm: Fix ceph orch apply -i

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: `orch daemon add` accepts a yaml

Fixes: https://tracker.ceph.com/issues/44622
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: apply_drivegroups() returns a single Completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: remove `trivial_result()`

Replaced with `@trivial_completion`, otherwise
`self._apply()` would have needed to accept a list
of specs.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Fix `ceph orch apply -i`

* We had two implementations for `ceph orcha apply`
* Remove yaml pasing in `mgr/cephadm`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33994 from dillaman/wip-librbd-poll-event-race

librbd: defer event socket completion until after callback issued

Reviewed-by: Dehao Shang <dehao.shang@intel.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: document `clone cancel` command

Fixes: http://tracker.ceph.com/issues/44208
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add `clone cancel` tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce "clone cancel" volume command

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: allow canceling a single asynchronous job for a volume

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: helper for looking up a clone entry index

... this would be used when canceling a clone operation to
find the clone index for a given clone.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if clone operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if copy operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce 'canceled' state in clone op state machine

When fetching the next execution state, -EINTR jumps to 'canceled'
state signifying a canceled (interrupted) operation. Also include
a helper routine to check if a given state machine is in initial
state.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater/valgrind: less bluestore logging

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater: increase heartbeat grace

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"

This reverts commit 4f742f200df6c91db87bfee1109c37fad3c0548b.

This was in the wrong file.. see valgrind.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"

This reverts commit 65e81e6eb4f136bf21b67e5de10ab49f028f9e95.

This slows things down too much with valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44062

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44582

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

When running under valgrind (and thrashing) things can be slow.  Tell
tests in case they need to tolerate timeouts.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/install: edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: more edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: edits

Based on review from Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/adoption: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: a few edits

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: do not install ceph-common on host (by default)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: drop os recs link

This mostly doesn't quite apply to cephadm because we use containers.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: document upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: revamp install docs

Signed-off-by: Sage Weil <sage@redhat.com>

doc: reorganize cephadm docs

- reorganized cephadm into a top-level item with a series of sub-items.
- condensed the 'install' page so that it doesn't create a zillion items
in the toctree on the left
- started updating the cephadm/install sequence (incomplete)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/administration: update docs on customizing SSH config

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/administration: add a note about the 'removed' dir

(and other paths)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack

We're deploying these things manually later, and they use fixed ports.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33974 into octopus

* refs/pull/33974/head:
use quay octopus tip until 15.2 tag is available
update default container images

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33442 into octopus

* refs/pull/33442/head:
cephadm: add allow_ptrace option to enable SYS_PTRACE

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33997 into octopus

* refs/pull/33997/head:
mon/MgrMonitor: make 'mgr fail' work with no arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34000 into octopus

* refs/pull/34000/head:
mgr/dashboard: adapt create_osds interface change

Reviewed-by: Laura Paduano <lpaduano@suse.com>

use quay octopus tip until 15.2 tag is available

Signed-off-by: Sage Weil <sage@redhat.com>

python-common: reduce output of ServiceSpec.to_json()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common,mgr/cephadm: move assert_valid_host to service_spec

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: add HostAssignment.validate()

Fixes: https://tracker.ceph.com/issues/44607
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: adapt create_osds interface change

Support the parameter change of create_osds since
https://github.com/ceph/ceph/pull/33922.

Fixes: https://tracker.ceph.com/issues/44632
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mon/MgrMonitor: make 'mgr fail' work with no arguments

Fail the active mgr so you don't have to check ceph -s and cut&paste it or
do some jq shenanigans with 'mgr dump'.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add allow_ptrace option to enable SYS_PTRACE

In any environments it is helpful to have SYS_PTRACE so that you can
gdb attach or strace a daemon.

Leave this off by default so that the container is more secure.

Enable this in teuthology and vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

update default container images

- For tests, use bleeding-edge octopus branch
- For production defaults, use ceph/ceph:v15.2 tag
- For bootstrap, grab cephadm script from latest octopus branch

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: limit number of times check host is performed in the serve loop

we only need this to happen every 10 min on hosts

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #33961 into octopus

* refs/pull/33961/head:
cephadm: add-repo: add --version
cephadm: rename distro args in repo methods

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33952 into octopus

* refs/pull/33952/head:
qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1
cephadm: add tests
qa/tasks/cephadm: pass -v to bootstrap
mgr/cephadm: only try to place mons on hosts matching public_network
mgr/cephadm: keep track of host networks, ips
cephadm: automatically infer mon public_network, if we can
cephadm: add list-networks command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33990 into octopus

* refs/pull/33990/head:
mgr/orch: add --all-available-devices to 'orch apply osd'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33955 into octopus

* refs/pull/33955/head:
mgr/cephadm: respect 'unmanaged' flag in spec
mgr/orch: orch ls: show <no spec> or <unmanaged> as appropriate
mgr/orch: orch ls: rename SPEC -> PLACEMENT
mgr/orch: add 'unmanaged' property to ServiceSpec
mgr/orch: combine 'orch daemon add <type> ...' into one command
mgr/orch: combine 'orch apply <type> [<placement>]' into one command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33936 into octopus

* refs/pull/33936/head:
cephadm: bootstrap: deploy monitoring stack by default

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/orch: add --all-available-devices to 'orch apply osd'

Provide a super-simple "use any avaialable device" command for
'orch apply osd'.  This will work for many (maybe even most?) users.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: pass -v to bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: only try to place mons on hosts matching public_network

Only try to schedule new mons on hosts that match the configured
public_network, if any.  If we do not have one configured, then don't
try to place new mons at all.

Note that there are other restrictions that ceph-mon supports that we
aren't considering, here: public_network_interface in particular, which
might further limit which IPs we consider binding to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: keep track of host networks, ips

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: automatically infer mon public_network, if we can

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add list-networks command

Summarize CIDR networks and our IPs in them.  Result looks like this:

{
    "10.3.64.0/24": [
        "10.3.64.23",
        "10.3.64.27"
    ],
    "10.88.0.0/16": [
        "10.88.0.1"
    ],
    "192.168.122.0/24": [
        "192.168.122.1"
    ]
}

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: bootstrap: deploy monitoring stack by default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33979 into octopus

* refs/pull/33979/head:
cephadm: implement 'install' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

librbd: defer event socket completion until after callback issued

A change post-Nautilus fixed an issue where multiple threads could
concurrently invoke callbacks to librbd clients. However, it also
introduced the potential that a callback hasn't yet fired by the
time the event socket is completed. This resulted in a crash of
fio under high-throughput testing since it expected both a callback
and the event socket completion, in that order.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33988 from dillaman/wip-44391

rbd-mirror: hold lock while updating local image name

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm: add-repo: add --version

Instead of --release octopus, which would get the latest octopus
version (whatever it might be), or possibly a repo with many build versions
inside, you can instead do --version 15.2.1 to get a repo with a
specific version and that version only.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: respect 'unmanaged' flag in spec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: orch ls: show <no spec> or <unmanaged> as appropriate

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: orch ls: rename SPEC -> PLACEMENT

More accurate, I think.  We only ever show placement here.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add 'unmanaged' property to ServiceSpec

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: implement 'install' command

This was present, but a no-op.

By default, install cephadm.

Optionally take a list of packages to install instead (e.g., ceph-common).

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: hold lock while updating local image name

There is a potential for an independently scheduled status update to
request the local image name from the bootstrap state machine during its
initialization.

Fixes: https://tracker.ceph.com/issues/44391
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33987 from tchaikov/wip-octopus-formatting

doc/releases/octopus.rst: formatting tweaks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33980 into octopus

* refs/pull/33980/head:
cephadm: bootstrap: allow --output-dir
cephadm: do not infer image for bootstrap
cephadm: write output files to /etc/ceph by default
cephadm: verify the output files' containing directory exists

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/releases/octopus.rst: use note directive for notes

so the note stands out in the text

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus.rst: add '$' as prompt

to help the BashSession lexer to render the text starting with "#" as comments

see https://pygments.org/docs/lexers/#lexers-for-various-shells

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus: decrease the indent level

for less and better horizontal spacing

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_static/css: fine tune the spacing

* `ul.simple > li` is also used in front page, where the items are too
  sparse now. so we need to restore the default spacing.
* `ul.simple > li > ul > li:last-child` is used to control the spacing
  between nested unordered list. we need have more spacing there.
* `div.section > ul > li > p` is added to decrease the spacing of nested
  list items.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33971 from tchaikov/octopus-doc-release

doc/releases/octopus.rst: format tweaks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33968 into octopus

* refs/pull/33968/head:
mgr/cephadm: do not include 'osd' service in 'orch ls' output

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33970 into octopus

* refs/pull/33970/head:
python-common: 'all:true' -> '*'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33973 into octopus

* refs/pull/33973/head:
mgr/orch,cephadm: elevate --force flag for daemons to interface
cephadm: backup data when we remove stateful daemons (mon, osd, prometheus)
cephadm: do not zap disk after removing an OSD daemon

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33935 into octopus

* refs/pull/33935/head:
mgr/cephadm: fix host connection exception
doc/cephadm: use ssh-copy-id

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc/releases/octopus.rst: reindent top-level items

reduce the indent level, so these items are better formated

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus.rst: formatting tweaks

* use console lexer for highlighting command lines
  otherwise they will be rendered as python code
* use bold font instead italic in subsection headers,
  they look more formal this way
* fix a typo: delineated

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: fix hyper link

Signed-off-by: Kefu Chai <kchai@redhat.com>