doc/releases/octopus: final notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34042 into octopus

* refs/pull/34042/head:
mgr/rook: list rgw services
mgr/rook: tolerate timestamps that are None
mgr/orch: add 'subcluster' property to RGWSpec
mgr/rook: do not create radosgw pools
mgr/rook: refactor apply/add for rgw
mgr/cephadm: configure rgw_frontends for rgw service
mgr/orch: accept port and ssl flags to 'apply rgw'
python-common/ceph/deployment/service_spec: add ssl to RGWSpec
mgr/rook: fix 'orch ps' for osds

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #33959 into octopus

* refs/pull/33959/head:
qa: fix all the fsx.sh-invoking yaml files to install dependencies

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34067 into octopus

* refs/pull/34067/head:
mgr/DaemonServer: add explicit check that acting matches for merge

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/DaemonServer: add explicit check that acting matches for merge

Add an explicit check that the PG acting for the source and target
match before merging.

Fixes: https://tracker.ceph.com/issues/44684
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34040 from dillaman/wip-44396-partial-fix

rbd-mirror: snapshot-based mirroring should use image sync throttler

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #34098 into octopus

* refs/pull/34098/head:
cephadm: relabel /etc/ganesha mount

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/rook: list rgw services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: tolerate timestamps that are None

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add 'subcluster' property to RGWSpec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: do not create radosgw pools

First, we don't know how big they should be or what they should look like.
The caller should already know that, and/or radosgw can create the pools
itself.

This depends on https://github.com/rook/rook/pull/5058

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: refactor apply/add for rgw

A few caveats here:

- enforce that realm == zone, since that is all rook does at the moment.
- we force a (bad!) pool configuration, since rook requires that these
be present (instead of allowing radosgw or the caller to create the pools)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34082 into octopus

* refs/pull/34082/head:
cephadm: switch grafana image to the ceph repo

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #34068 into octopus

* refs/pull/34068/head:
mgr/cephadm: clean up client.crash.* container_image settings after upgrade

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: relabel /etc/ganesha mount

Fixes: https://tracker.ceph.com/issues/44701
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34046 into octopus

* refs/pull/34046/head:
qa/suites/rados/cephadm/upgrade: update starting version
mgr/orch: 'ceph upgrade ...' -> 'ceph orch upgrade ...'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34092 into octopus

* refs/pull/34092/head:
doc/releases/octopus.rst: add note about ec recovery below min_size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33719 from ukernel/wip-44416

mds: pass proper MutationImpl::LockOp to Locker::wrlock_start()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

rbd-mirror: leader watcher should not cancel get locker if locker is invalid

When a new leader acquires the lock, it will send out a lock acquired
notification along with periodic heartbeats. The get locker will attempt to
run immediately, but if a heartbeat arrives before it executes the heartbeat
will cancel the timer and reschedule it for the future. This process repeats
for each periodic heartbeat and the locker is never re-read from the OSD.

This is an issue only for namespace replayers due to the delayed fashion in
which the leader instance id is retrieved.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: snapshot sync request needs to check for interruption

If the sync request was locally canceled, we need to resume the paused
shut down logic instead of just notifying the image replayer state
machine of the change -- since it had already requested a shut down and
will not re-request it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: request exclusive lock when moving to trash

Even if the image is in-use, moving it to the trash does not
remove any data. This also solves a race between snapshot-based
mirroring shutting down and being able to move a mirrored image
to the trash.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: basic integration with sync throttling

snapshot-based mirroring did not have any throttling to prevent
too many concurrent syncs from running. Since each sync might need
to iterate over every object of an image, that could potentially
put an extreme burden on the remote cluster.

A future PR will add a more intelligent throttle based on the actual
number of objects needed to be scanned.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: don't prematurely finish snapshot replay loop

The unlink step was being incorrectly skipped if a state machine
shut down was requested.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: pass InstanceWatcher to snapshot Replayer

The replayer will request image syncs via the instance watcher in the next
commit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/releases/octopus.rst: add note about ec recovery below min_size

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: configure rgw_frontends for rgw service

This assumes the beast frontend.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: switch grafana image to the ceph repo

Pull the pre-built grafana image from the ceph org on
docker.io

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #34034 into octopus

* refs/pull/34034/head:
cephadm: make add-repo --release and --version independent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/suites/rados/cephadm/upgrade: update starting version

Need upgrade command to match!

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33540 into octopus

* refs/pull/33540/head:
doc/orchestrator: update rgw creation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34023 into octopus

* refs/pull/34023/head:
mgr/test_orchestrator: fix service filtering when using dummy data
mgr/dashboard: fix adding/removing host errors

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #34044 into octopus

* refs/pull/34044/head:
remove ceph_test_rados_watch_notify

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #34030 into octopus

* refs/pull/34030/head:
cephadm: env over last used
cephadm: fall back to default for infer_image
cephadm: remove outdated check
cephadm: consolidate default image logic
cephadm: only infer image for shell, run, inspect-image, pull, ceph-volume

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc/orchestrator: update rgw creation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/cephadm: clean up client.crash.* container_image settings after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34027 into octopus

* refs/pull/34027/head:
qa/workunits/cephadm/test_cephadm: mark services unmanaged for test
mgr/cephadm: do not reconfig unmanaged services
qa/workunits/cephadm/test_cephadm: output file for pub key

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: make add-repo --release and --version independent

Specify either --release name (to get the latest) or --version x.y.z to
get a specific version.

Adapt to updated locations on download.ceph.com so that we don't need to
know the release name for a specific x.y.z release.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34045 into octopus

* refs/pull/34045/head:
mgr/progress: fix duration strings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cephadm: env over last used

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34035 from dillaman/wip-rbd-permissions

qa/workunits/rbd: use context managers to control Rados lifespan

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/orch: accept port and ssl flags to 'apply rgw'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: 'ceph upgrade ...' -> 'ceph orch upgrade ...'

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fall back to default for infer_image

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: remove outdated check

Either the user passed their own image or we filled in the (monitoring)
default--no need for this check.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: consolidate default image logic

Merge the _parse_args() logic into @default_image.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/progress: fix duration strings

- simplify the code to just calculate the durations when we need them
(I'm not sure why we had those temporary strings!)
- use a nicer time delta format

Fixes: https://tracker.ceph.com/issues/44672
Signed-off-by: Sage Weil <sage@redhat.com>

remove ceph_test_rados_watch_notify

This is an old test, we have good watch/notify coverage in the newer
tests, and it is buggy.

Fixes: https://tracker.ceph.com/issues/43861
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34014 into octopus

* refs/pull/34014/head:
mgr/balancer: tolerate pgs outside of target weight map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #34001 into octopus

* refs/pull/34001/head:
mgr/cephadm: pytest: Enable SpecStore
mgr/orchestrator: add test for default implementation for apply()
python-common: validate ServiceSpec.service_type
fixup mgr/cephadm: Fix ceph orch apply -i
mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion
mgr/orchestrator: `orch daemon add` accepts a yaml
mgr/cephadm: apply_drivegroups() returns a single Completion
mgr/cephadm: remove `trivial_result()`
mgr/cephadm: Fix `ceph orch apply -i`

Reviewed-by: Sage Weil <sage@redhat.com>

python-common/ceph/deployment/service_spec: add ssl to RGWSpec

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: only infer image for shell, run, inspect-image, pull, ceph-volume

Use a decorators for infer, require, and default.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34011 into octopus

* refs/pull/34011/head:
qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS
qa/suites/rados/verify/validater/valgrind: less bluestore logging
qa/suites/rados/verify/validater: increase heartbeat grace
Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"
Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"
ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set
ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set
qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/rbd: use context managers to control Rados lifespan

There is a potential race between the expected exceptions being
thrown and Python shutting down racing with librados background
threads. Ensure that librados is properly shut down prior to
exiting Python.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34032 from dillaman/wip-rbd-octopus-docs

doc/releases/octopus: add additional RBD improvements

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/releases/octopus: add additional RBD improvements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/cephadm/test_cephadm: mark services unmanaged for test

We are deploying containers manually.  Mark them unmanaged so that we
do not fight against mgr/cephadm cleaning up orphan daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not reconfig unmanaged services

Fixes: https://tracker.ceph.com/issues/44615
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33981 into octopus

* refs/pull/33981/head:
doc/install: edits
doc/cephadm: more edits
doc/cephadm/install: edits
doc/cephadm/adoption: improvements
doc/cephadm/install: a few edits
doc/cephadm/install: do not install ceph-common on host (by default)
doc/cephadm: drop os recs link
doc/cephadm/upgrade: improvements
doc/cephadm/upgrade: document upgrade
doc/cephadm/install: revamp install docs
doc: reorganize cephadm docs
doc/cephadm/administration: update docs on customizing SSH config
doc/cephadm/administration: add a note about the 'removed' dir

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #34018 from ajarr/octopus-subvolume-clone-cancel

mgr/volumes: add FS subvolume clone cancel

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/workunits/cephadm/test_cephadm: output file for pub key

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33866 into octopus

* refs/pull/33866/head:
mgr/cephadm: limit number of times check host is performed in the serve loop

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34005 into octopus

* refs/pull/34005/head:
python-common: reduce output of ServiceSpec.to_json()
python-common,mgr/cephadm: move assert_valid_host to service_spec
mgr/cephadm: add HostAssignment.validate()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34013 into octopus

* refs/pull/34013/head:
qa/workunits/cephadm/test_cephadm: --skip-monitoring-stack

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pytest: Enable SpecStore

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add test for default implementation for apply()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: validate ServiceSpec.service_type

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

fixup mgr/cephadm: Fix ceph orch apply -i

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/test_orchestrator: fix service filtering when using dummy data

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: fix adding/removing host errors

Send a HostSpec instance to the Orchestrator when adding a host.

Also, to be consistent with other components:
- Reword from Add/Remove hosts to Create/Delete hosts
- Display a modal when there is no Orchestrator backend enabled

Fixes: https://tracker.ceph.com/issues/44664
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: `orch daemon add` accepts a yaml

Fixes: https://tracker.ceph.com/issues/44622
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: apply_drivegroups() returns a single Completion

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: remove `trivial_result()`

Replaced with `@trivial_completion`, otherwise
`self._apply()` would have needed to accept a list
of specs.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Fix `ceph orch apply -i`

* We had two implementations for `ceph orcha apply`
* Remove yaml pasing in `mgr/cephadm`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33994 from dillaman/wip-librbd-poll-event-race

librbd: defer event socket completion until after callback issued

Reviewed-by: Dehao Shang <dehao.shang@intel.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: document `clone cancel` command

Fixes: http://tracker.ceph.com/issues/44208
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: add `clone cancel` tests

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce "clone cancel" volume command

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: allow canceling a single asynchronous job for a volume

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: helper for looking up a clone entry index

... this would be used when canceling a clone operation to
find the clone index for a given clone.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if clone operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: periodically check if copy operations should be canceled

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: introduce 'canceled' state in clone op state machine

When fetching the next execution state, -EINTR jumps to 'canceled'
state signifying a canceled (interrupted) operation. Also include
a helper routine to check if a given state machine is in initial
state.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater/valgrind: less bluestore logging

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify/validater: increase heartbeat grace

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"

This reverts commit 4f742f200df6c91db87bfee1109c37fad3c0548b.

This was in the wrong file.. see valgrind.yaml

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"

This reverts commit 65e81e6eb4f136bf21b67e5de10ab49f028f9e95.

This slows things down too much with valgrind.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44062

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set

See https://tracker.ceph.com/issues/44582

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

When running under valgrind (and thrashing) things can be slow.  Tell
tests in case they need to tolerate timeouts.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/install: edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: more edits

Based on review by Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: edits

Based on review from Alexandra Settle <asettle@suse.com>

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/adoption: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: a few edits

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/install: do not install ceph-common on host (by default)

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: drop os recs link

This mostly doesn't quite apply to cephadm because we use containers.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: improvements

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/upgrade: document upgrade

Signed-off-by: Sage Weil <sage@redhat.com>