mgr/cephadm: clean up client.crash.* container_image settings after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34014 into octopus

* refs/pull/34014/head:
mgr/balancer: tolerate pgs outside of target weight map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #34001 into octopus

* refs/pull/34001/head:
mgr/cephadm: pytest: Enable SpecStore
mgr/orchestrator: add test for default implementation for apply()
python-common: validate ServiceSpec.service_type
fixup mgr/cephadm: Fix ceph orch apply -i
mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion
mgr/orchestrator: `orch daemon add` accepts a yaml
mgr/cephadm: apply_drivegroups() returns a single Completion
mgr/cephadm: remove `trivial_result()`
mgr/cephadm: Fix `ceph orch apply -i`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34011 into octopus

* refs/pull/34011/head:
qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS
qa/suites/rados/verify/validater/valgrind: less bluestore logging
qa/suites/rados/verify/validater: increase heartbeat grace
Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"
Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"
ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set
ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set
qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34032 from dillaman/wip-rbd-octopus-docs

doc/releases/octopus: add additional RBD improvements

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/releases/octopus: add additional RBD improvements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33981 into octopus

* refs/pull/33981/head:
doc/install: edits
doc/cephadm: more edits
doc/cephadm/install: edits
doc/cephadm/adoption: improvements
doc/cephadm/install: a few edits
doc/cephadm/install: do not install ceph-common on host (by default)
doc/cephadm: drop os recs link
doc/cephadm/upgrade: improvements
doc/cephadm/upgrade: document upgrade
doc/cephadm/install: revamp install docs
doc: reorganize cephadm docs
doc/cephadm/administration: update docs on customizing SSH config
doc/cephadm/administration: add a note about the 'removed' dir

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #34018 from ajarr/octopus-subvolume-clone-cancel

mgr/volumes: add FS subvolume clone cancel

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #33866 into octopus

* refs/pull/33866/head:
mgr/cephadm: limit number of times check host is performed in the serve loop

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34005 into octopus

* refs/pull/34005/head:
python-common: reduce output of ServiceSpec.to_json()
python-common,mgr/cephadm: move assert_valid_host to service_spec
mgr/cephadm: add HostAssignment.validate()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34013 into octopus

* refs/pull/34013/head:
qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pytest: Enable SpecStore

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add test for default implementation for apply()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: validate ServiceSpec.service_type

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

fixup mgr/cephadm: Fix ceph orch apply -i

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: `orch daemon add` accepts a yaml

Fixes: https://tracker.ceph.com/issues/44622
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: apply_drivegroups() returns a single Completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: remove `trivial_result()`

Replaced with `@trivial_completion`, otherwise
`self._apply()` would have needed to accept a list
of specs.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Fix `ceph orch apply -i`

* We had two implementations for `ceph orcha apply`
* Remove yaml pasing in `mgr/cephadm`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33994 from dillaman/wip-librbd-poll-event-race

librbd: defer event socket completion until after callback issued

Reviewed-by: Dehao Shang <dehao.shang@intel.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: document `clone cancel` command

Fixes: http://tracker.ceph.com/issues/44208
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add `clone cancel` tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce "clone cancel" volume command

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: allow canceling a single asynchronous job for a volume

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: helper for looking up a clone entry index

... this would be used when canceling a clone operation to
find the clone index for a given clone.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if clone operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if copy operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce 'canceled' state in clone op state machine

When fetching the next execution state, -EINTR jumps to 'canceled'
state signifying a canceled (interrupted) operation. Also include
a helper routine to check if a given state machine is in initial
state.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater/valgrind: less bluestore logging

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater: increase heartbeat grace

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"

This reverts commit 4f742f200df6c91db87bfee1109c37fad3c0548b.

This was in the wrong file.. see valgrind.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"

This reverts commit 65e81e6eb4f136bf21b67e5de10ab49f028f9e95.

This slows things down too much with valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44062

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44582

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

When running under valgrind (and thrashing) things can be slow.  Tell
tests in case they need to tolerate timeouts.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/install: edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: more edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: edits

Based on review from Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/adoption: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: a few edits

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: do not install ceph-common on host (by default)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: drop os recs link

This mostly doesn't quite apply to cephadm because we use containers.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: document upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: revamp install docs

Signed-off-by: Sage Weil <sage@redhat.com>

doc: reorganize cephadm docs

- reorganized cephadm into a top-level item with a series of sub-items.
- condensed the 'install' page so that it doesn't create a zillion items
in the toctree on the left
- started updating the cephadm/install sequence (incomplete)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/administration: update docs on customizing SSH config

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/administration: add a note about the 'removed' dir

(and other paths)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/balancer: tolerate pgs outside of target weight map

We build a target weight map based on the primary crush weights, and
ignore weights that are 0.  However, it's possible that existing PGs are
on other OSDs that have weight 0 because the weight-set weight is >0.
That leads to a KeyError exception when we

                     pgs_by_osd[osd] += 1

and the key isn't present.  Fix by simply populating those keys as we
encounter OSDs.  Drop the old initialization loop.  The net of this is
we may have OSDs outside of target_by_root (won't matter, as far as I can
tell) and we won't have keys for osds with weight 0 (also won't matter,
as far as I can tell).

Fixes: https://tracker.ceph.com/issues/42721
Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack

We're deploying these things manually later, and they use fixed ports.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33974 into octopus

* refs/pull/33974/head:
use quay octopus tip until 15.2 tag is available
update default container images

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33442 into octopus

* refs/pull/33442/head:
cephadm: add allow_ptrace option to enable SYS_PTRACE

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33997 into octopus

* refs/pull/33997/head:
mon/MgrMonitor: make 'mgr fail' work with no arguments

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34000 into octopus

* refs/pull/34000/head:
mgr/dashboard: adapt create_osds interface change

Reviewed-by: Laura Paduano <lpaduano@suse.com>

use quay octopus tip until 15.2 tag is available

Signed-off-by: Sage Weil <sage@redhat.com>

python-common: reduce output of ServiceSpec.to_json()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common,mgr/cephadm: move assert_valid_host to service_spec

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: add HostAssignment.validate()

Fixes: https://tracker.ceph.com/issues/44607
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: adapt create_osds interface change

Support the parameter change of create_osds since
https://github.com/ceph/ceph/pull/33922.

Fixes: https://tracker.ceph.com/issues/44632
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mon/MgrMonitor: make 'mgr fail' work with no arguments

Fail the active mgr so you don't have to check ceph -s and cut&paste it or
do some jq shenanigans with 'mgr dump'.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add allow_ptrace option to enable SYS_PTRACE

In any environments it is helpful to have SYS_PTRACE so that you can
gdb attach or strace a daemon.

Leave this off by default so that the container is more secure.

Enable this in teuthology and vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

update default container images

- For tests, use bleeding-edge octopus branch
- For production defaults, use ceph/ceph:v15.2 tag
- For bootstrap, grab cephadm script from latest octopus branch

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: limit number of times check host is performed in the serve loop

we only need this to happen every 10 min on hosts

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge PR #33961 into octopus

* refs/pull/33961/head:
cephadm: add-repo: add --version
cephadm: rename distro args in repo methods

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33952 into octopus

* refs/pull/33952/head:
qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1
cephadm: add tests
qa/tasks/cephadm: pass -v to bootstrap
mgr/cephadm: only try to place mons on hosts matching public_network
mgr/cephadm: keep track of host networks, ips
cephadm: automatically infer mon public_network, if we can
cephadm: add list-networks command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33990 into octopus

* refs/pull/33990/head:
mgr/orch: add --all-available-devices to 'orch apply osd'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33955 into octopus

* refs/pull/33955/head:
mgr/cephadm: respect 'unmanaged' flag in spec
mgr/orch: orch ls: show <no spec> or <unmanaged> as appropriate
mgr/orch: orch ls: rename SPEC -> PLACEMENT
mgr/orch: add 'unmanaged' property to ServiceSpec
mgr/orch: combine 'orch daemon add <type> ...' into one command
mgr/orch: combine 'orch apply <type> [<placement>]' into one command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33936 into octopus

* refs/pull/33936/head:
cephadm: bootstrap: deploy monitoring stack by default

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/orch: add --all-available-devices to 'orch apply osd'

Provide a super-simple "use any avaialable device" command for
'orch apply osd'.  This will work for many (maybe even most?) users.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm: --skip-mon-network when using 127.0.0.1

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: pass -v to bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: only try to place mons on hosts matching public_network

Only try to schedule new mons on hosts that match the configured
public_network, if any.  If we do not have one configured, then don't
try to place new mons at all.

Note that there are other restrictions that ceph-mon supports that we
aren't considering, here: public_network_interface in particular, which
might further limit which IPs we consider binding to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: keep track of host networks, ips

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: automatically infer mon public_network, if we can

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add list-networks command

Summarize CIDR networks and our IPs in them.  Result looks like this:

{
    "10.3.64.0/24": [
        "10.3.64.23",
        "10.3.64.27"
    ],
    "10.88.0.0/16": [
        "10.88.0.1"
    ],
    "192.168.122.0/24": [
        "192.168.122.1"
    ]
}

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: bootstrap: deploy monitoring stack by default

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33979 into octopus

* refs/pull/33979/head:
cephadm: implement 'install' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

librbd: defer event socket completion until after callback issued

A change post-Nautilus fixed an issue where multiple threads could
concurrently invoke callbacks to librbd clients. However, it also
introduced the potential that a callback hasn't yet fired by the
time the event socket is completed. This resulted in a crash of
fio under high-throughput testing since it expected both a callback
and the event socket completion, in that order.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33988 from dillaman/wip-44391

rbd-mirror: hold lock while updating local image name

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm: add-repo: add --version

Instead of --release octopus, which would get the latest octopus
version (whatever it might be), or possibly a repo with many build versions
inside, you can instead do --version 15.2.1 to get a repo with a
specific version and that version only.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: respect 'unmanaged' flag in spec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: orch ls: show <no spec> or <unmanaged> as appropriate

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: orch ls: rename SPEC -> PLACEMENT

More accurate, I think.  We only ever show placement here.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add 'unmanaged' property to ServiceSpec

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: implement 'install' command

This was present, but a no-op.

By default, install cephadm.

Optionally take a list of packages to install instead (e.g., ceph-common).

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: hold lock while updating local image name

There is a potential for an independently scheduled status update to
request the local image name from the bootstrap state machine during its
initialization.

Fixes: https://tracker.ceph.com/issues/44391
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33987 from tchaikov/wip-octopus-formatting

doc/releases/octopus.rst: formatting tweaks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33980 into octopus

* refs/pull/33980/head:
cephadm: bootstrap: allow --output-dir
cephadm: do not infer image for bootstrap
cephadm: write output files to /etc/ceph by default
cephadm: verify the output files' containing directory exists

Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/releases/octopus.rst: use note directive for notes

so the note stands out in the text

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus.rst: add '$' as prompt

to help the BashSession lexer to render the text starting with "#" as comments

see https://pygments.org/docs/lexers/#lexers-for-various-shells

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases/octopus: decrease the indent level

for less and better horizontal spacing

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_static/css: fine tune the spacing

* `ul.simple > li` is also used in front page, where the items are too
  sparse now. so we need to restore the default spacing.
* `ul.simple > li > ul > li:last-child` is used to control the spacing
  between nested unordered list. we need have more spacing there.
* `div.section > ul > li > p` is added to decrease the spacing of nested
  list items.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33971 from tchaikov/octopus-doc-release

doc/releases/octopus.rst: format tweaks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33968 into octopus

* refs/pull/33968/head:
mgr/cephadm: do not include 'osd' service in 'orch ls' output

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33970 into octopus

* refs/pull/33970/head:
python-common: 'all:true' -> '*'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33973 into octopus

* refs/pull/33973/head:
mgr/orch,cephadm: elevate --force flag for daemons to interface
cephadm: backup data when we remove stateful daemons (mon, osd, prometheus)
cephadm: do not zap disk after removing an OSD daemon

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33935 into octopus

* refs/pull/33935/head:
mgr/cephadm: fix host connection exception
doc/cephadm: use ssh-copy-id

Reviewed-by: Joshua Schmid <jschmid@suse.de>