mgr/cephadm: add watch_url to NFS bootstrap config

watch for changes to the per-daemon RADOS conf file

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit ff9265104f69c702630e5069b1e1a2105d1f74fa)

mgr/cephadm: check for an existing NFS config object

assume an existing NFS config if the named rados object already exists

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 75b988f314ab8792cccad70bb9894ff1a50ae3f3)

mgr/cephadm: combine `get_daemon_name` and `get_rados_user` funcs

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit fdf90bf530edbb9608fb8a5d273f7ce38147f534)

mon: allow `auth caps` for `mgr` profile

cephadm orchestrator needs to update the caps during reconfig/redeploy

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit ef8a46a3b13ea6753b8c03121da4c6daff370fd2)

mgr/cephadm: add deployment example to NFS docs

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 77440d91c1be7c872fc91042a0d723b9198f224f)

mgr/cephadm: add utils.py

move `name_to_config_section` and `assert_valid_host` into utils.py

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 322eea8099616b25a091ac66a1253d1991a0da2c)

mgr/cephadm: create/update keyring during nfs config

the keyring might already exist from a prior config/reconfig/redeploy.

attempt to get_or_create the keyring first and than update the keyring
caps afterward

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 084fd4a91ae781ab7bed9e06ceca17b1a5ca5be2)

mgr/cephadm: pass NFSServiceSpec instead of pool/namespace args

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 8bb79a5f275ecf79428dc35a8fc1a76c36569df3)

mgr/cephadm: add nfs apply test

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 78ce6d431806c1f3e8e55e946eff64d450821db1)

mgr/cephadm: fix nfs redeploy/reconfig

search for the `NFSServiceSpec` instead of passing empty
`cephadm_args` during `_create_daemon`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 107885801bb5d7401dead52cbc2abadaa2865a91)

mgr/cephadm: ensure nfs uses the correct entity

mon entity needs to start with `client.`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit fd6322f7ec51f3dc57168bbd08ffab3c832a4598)

mgr/cephadm: fix service name for nfs type

the `nfs` service was incorrectly reported as `mds`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit d9f04ec1d73948e915dda0f51592306542314238)

mgr/cephadm: apply nfs service spec

orch apply nfs <svc_id> [<placement>]

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 76f27894caba2b3bc146c79d86c4335c9c33dcde)

mgr/cephadm: don't create a unique nfs daemon name

only one nfs daemon will be deployed per host

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 81823176af101864524fc611fff346bf9798f0e0)

mgr/cephadm: update docs

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 530c62d76d79c4dd90a966302cd07d1017261686)

mgr/cephadm: add minimal caps for nfs client keyring

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 94600c69e2f547c02017204c909d11d626649066)

mgr/cephadm: add logging to NFSGanesha class

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 26ea33ad4b0e624345d34526dcf04e35176d2c8e)

mgr/cephadm: create empty nfs rados config obj

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 9103f4758beb7e0f0dba523f776f9918c0dbfe6d)

mgr/cephadm: add nfs tests

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit b2b6e016d190d4a5dacfc1a4f27adf5d2d7a7aa0)

mgr/cephadm: add a minimal ganesha.conf

we only need to specify the rados pool and config object url

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 2606cc3543bba8afc67e8f78185286123fd71744)

mgr/cephadm: deploy nfs-ganesha service

orch daemon add nfs <svc_arg> <pool> [<namespace>] [<placement>]

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 770816dc9f5b440fd1fc6cc507a06cd8573f6ea1)

mgr/cephadm: split out `config-and-keyrings` json

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 3be608102834bbc7ba5de927ccc868b5b5fb3cad)

mgr/orchestrator: rearrange interface to match impl. order

rgw impl. happens before rbd impl

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 7976ac745aa83ac664f2628fb9fe7bbec0c07100)

Merge pull request #34125 from smithfarm/wip-cbs-deprecate-octopus

octopus: ceph-backport.sh: add deprecation warning

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34137 from theanalyst/octopus-changelog-1

octopus: doc: add changelog for octopus release

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc: add changelog for octopus release

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit dc7cf3fe8986de7b3e245ea04084eef51c7a0cdf)

Merge PR #34100 into octopus

* refs/pull/34100/head:
doc/cephadm: notes on status/stability
doc/cephadm: some troubleshooting tips

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34115 into octopus

* refs/pull/34115/head:
doc/releases/octopus: drop stray line
doc/releases/octopus: note about repository locations
doc/releases: include octopus in index
doc/install/get-packages: update package install instructions
doc/releases/octopus: final notes

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/releases/octopus: drop stray line

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/octopus: note about repository locations

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: include octopus in index

Signed-off-by: Sage Weil <sage@redhat.com>

15.2.0

doc/install/get-packages: update package install instructions

- include cephadm method
- update to include new version URLs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34119 into octopus

* refs/pull/34119/head:
doc: update octopus release notes for rgw

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge PR #34122 into octopus

* refs/pull/34122/head:
cephadm: default to v15 tag for octopus

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-backport.sh: add deprecation warning

The script is not maintained in the stable versions of Ceph.

This commit is not a cherry-pick from master, because the script is maintained
in master. It is unmaintained only in the stable versions, like
"octopus".

Fixes: https://tracker.ceph.com/issues/42194
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(manual cherry pick of commit 365a93bd3fe62369d33d07993dd246e9b4dd63fd)

Conflicts:
    src/script/ceph-backport.sh

cephadm: default to v15 tag for octopus

Signed-off-by: Sage Weil <sage@redhat.com>

doc: update octopus release notes for rgw

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 984f3ff7e0fe10f1b54da4271a2a0879ffea0ddb)

doc/releases/octopus: final notes

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: notes on status/stability

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34042 into octopus

* refs/pull/34042/head:
mgr/rook: list rgw services
mgr/rook: tolerate timestamps that are None
mgr/orch: add 'subcluster' property to RGWSpec
mgr/rook: do not create radosgw pools
mgr/rook: refactor apply/add for rgw
mgr/cephadm: configure rgw_frontends for rgw service
mgr/orch: accept port and ssl flags to 'apply rgw'
python-common/ceph/deployment/service_spec: add ssl to RGWSpec
mgr/rook: fix 'orch ps' for osds

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #33959 into octopus

* refs/pull/33959/head:
qa: fix all the fsx.sh-invoking yaml files to install dependencies

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34067 into octopus

* refs/pull/34067/head:
mgr/DaemonServer: add explicit check that acting matches for merge

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/DaemonServer: add explicit check that acting matches for merge

Add an explicit check that the PG acting for the source and target
match before merging.

Fixes: https://tracker.ceph.com/issues/44684
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34040 from dillaman/wip-44396-partial-fix

rbd-mirror: snapshot-based mirroring should use image sync throttler

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #34098 into octopus

* refs/pull/34098/head:
cephadm: relabel /etc/ganesha mount

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/rook: list rgw services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: tolerate timestamps that are None

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add 'subcluster' property to RGWSpec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: do not create radosgw pools

First, we don't know how big they should be or what they should look like.
The caller should already know that, and/or radosgw can create the pools
itself.

This depends on https://github.com/rook/rook/pull/5058

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: refactor apply/add for rgw

A few caveats here:

- enforce that realm == zone, since that is all rook does at the moment.
- we force a (bad!) pool configuration, since rook requires that these
be present (instead of allowing radosgw or the caller to create the pools)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34082 into octopus

* refs/pull/34082/head:
cephadm: switch grafana image to the ceph repo

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #34068 into octopus

* refs/pull/34068/head:
mgr/cephadm: clean up client.crash.* container_image settings after upgrade

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: relabel /etc/ganesha mount

Fixes: https://tracker.ceph.com/issues/44701
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34046 into octopus

* refs/pull/34046/head:
qa/suites/rados/cephadm/upgrade: update starting version
mgr/orch: 'ceph upgrade ...' -> 'ceph orch upgrade ...'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34092 into octopus

* refs/pull/34092/head:
doc/releases/octopus.rst: add note about ec recovery below min_size

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33719 from ukernel/wip-44416

mds: pass proper MutationImpl::LockOp to Locker::wrlock_start()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

rbd-mirror: leader watcher should not cancel get locker if locker is invalid

When a new leader acquires the lock, it will send out a lock acquired
notification along with periodic heartbeats. The get locker will attempt to
run immediately, but if a heartbeat arrives before it executes the heartbeat
will cancel the timer and reschedule it for the future. This process repeats
for each periodic heartbeat and the locker is never re-read from the OSD.

This is an issue only for namespace replayers due to the delayed fashion in
which the leader instance id is retrieved.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: snapshot sync request needs to check for interruption

If the sync request was locally canceled, we need to resume the paused
shut down logic instead of just notifying the image replayer state
machine of the change -- since it had already requested a shut down and
will not re-request it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: request exclusive lock when moving to trash

Even if the image is in-use, moving it to the trash does not
remove any data. This also solves a race between snapshot-based
mirroring shutting down and being able to move a mirrored image
to the trash.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: basic integration with sync throttling

snapshot-based mirroring did not have any throttling to prevent
too many concurrent syncs from running. Since each sync might need
to iterate over every object of an image, that could potentially
put an extreme burden on the remote cluster.

A future PR will add a more intelligent throttle based on the actual
number of objects needed to be scanned.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: don't prematurely finish snapshot replay loop

The unlink step was being incorrectly skipped if a state machine
shut down was requested.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: pass InstanceWatcher to snapshot Replayer

The replayer will request image syncs via the instance watcher in the next
commit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/releases/octopus.rst: add note about ec recovery below min_size

Signed-off-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: configure rgw_frontends for rgw service

This assumes the beast frontend.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: switch grafana image to the ceph repo

Pull the pre-built grafana image from the ceph org on
docker.io

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #34034 into octopus

* refs/pull/34034/head:
cephadm: make add-repo --release and --version independent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/suites/rados/cephadm/upgrade: update starting version

Need upgrade command to match!

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: some troubleshooting tips

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33540 into octopus

* refs/pull/33540/head:
doc/orchestrator: update rgw creation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34023 into octopus

* refs/pull/34023/head:
mgr/test_orchestrator: fix service filtering when using dummy data
mgr/dashboard: fix adding/removing host errors

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #34044 into octopus

* refs/pull/34044/head:
remove ceph_test_rados_watch_notify

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #34030 into octopus

* refs/pull/34030/head:
cephadm: env over last used
cephadm: fall back to default for infer_image
cephadm: remove outdated check
cephadm: consolidate default image logic
cephadm: only infer image for shell, run, inspect-image, pull, ceph-volume

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc/orchestrator: update rgw creation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/cephadm: clean up client.crash.* container_image settings after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34027 into octopus

* refs/pull/34027/head:
qa/workunits/cephadm/test_cephadm: mark services unmanaged for test
mgr/cephadm: do not reconfig unmanaged services
qa/workunits/cephadm/test_cephadm: output file for pub key

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: make add-repo --release and --version independent

Specify either --release name (to get the latest) or --version x.y.z to
get a specific version.

Adapt to updated locations on download.ceph.com so that we don't need to
know the release name for a specific x.y.z release.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34045 into octopus

* refs/pull/34045/head:
mgr/progress: fix duration strings

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

cephadm: env over last used

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #34035 from dillaman/wip-rbd-permissions

qa/workunits/rbd: use context managers to control Rados lifespan

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/orch: accept port and ssl flags to 'apply rgw'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: 'ceph upgrade ...' -> 'ceph orch upgrade ...'

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fall back to default for infer_image

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: remove outdated check

Either the user passed their own image or we filled in the (monitoring)
default--no need for this check.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: consolidate default image logic

Merge the _parse_args() logic into @default_image.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/progress: fix duration strings

- simplify the code to just calculate the durations when we need them
(I'm not sure why we had those temporary strings!)
- use a nicer time delta format

Fixes: https://tracker.ceph.com/issues/44672
Signed-off-by: Sage Weil <sage@redhat.com>

remove ceph_test_rados_watch_notify

This is an old test, we have good watch/notify coverage in the newer
tests, and it is buggy.

Fixes: https://tracker.ceph.com/issues/43861
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34014 into octopus

* refs/pull/34014/head:
mgr/balancer: tolerate pgs outside of target weight map

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #34001 into octopus

* refs/pull/34001/head:
mgr/cephadm: pytest: Enable SpecStore
mgr/orchestrator: add test for default implementation for apply()
python-common: validate ServiceSpec.service_type
fixup mgr/cephadm: Fix ceph orch apply -i
mgr/dashbaord: orchestrator service: Revert wait_api_result to a single completion
mgr/orchestrator: `orch daemon add` accepts a yaml
mgr/cephadm: apply_drivegroups() returns a single Completion
mgr/cephadm: remove `trivial_result()`
mgr/cephadm: Fix `ceph orch apply -i`

Reviewed-by: Sage Weil <sage@redhat.com>

python-common/ceph/deployment/service_spec: add ssl to RGWSpec

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: only infer image for shell, run, inspect-image, pull, ceph-volume

Use a decorators for infer, require, and default.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34011 into octopus

* refs/pull/34011/head:
qa/suites/rados/verify/validater/valgrind: tolerate SLOW_OPS
qa/suites/rados/verify/validater/valgrind: less bluestore logging
qa/suites/rados/verify/validater: increase heartbeat grace
Revert "qa/suites/rados/verify: debug_ms = 1, osd_heartbeat_grace = 60"
Revert "qa/suites/rados/verify/validator/valgrind: debug refs = 5"
ceph_test_watch_notify: try notify 10x if ALLOW_TIMEOUTS is set
ceph_test_rados_api_misc: ShutdownRace timeout if ALLOW_TIMEOUTS is set
qa/suites/rados/verify: set ALLOW_TIMEOUTS for workunits

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/workunits/rbd: use context managers to control Rados lifespan

There is a potential race between the expected exceptions being
thrown and Python shutting down racing with librados background
threads. Ensure that librados is properly shut down prior to
exiting Python.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34032 from dillaman/wip-rbd-octopus-docs

doc/releases/octopus: add additional RBD improvements

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/releases/octopus: add additional RBD improvements

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/cephadm/test_cephadm: mark services unmanaged for test

We are deploying containers manually.  Mark them unmanaged so that we
do not fight against mgr/cephadm cleaning up orphan daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not reconfig unmanaged services

Fixes: https://tracker.ceph.com/issues/44615
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33981 into octopus

* refs/pull/33981/head:
doc/install: edits
doc/cephadm: more edits
doc/cephadm/install: edits
doc/cephadm/adoption: improvements
doc/cephadm/install: a few edits
doc/cephadm/install: do not install ceph-common on host (by default)
doc/cephadm: drop os recs link
doc/cephadm/upgrade: improvements
doc/cephadm/upgrade: document upgrade
doc/cephadm/install: revamp install docs
doc: reorganize cephadm docs
doc/cephadm/administration: update docs on customizing SSH config
doc/cephadm/administration: add a note about the 'removed' dir

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #34018 from ajarr/octopus-subvolume-clone-cancel

mgr/volumes: add FS subvolume clone cancel

Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/workunits/cephadm/test_cephadm: output file for pub key

Signed-off-by: Sage Weil <sage@redhat.com>