Merge pull request #34826 from smithfarm/wip-45052-octopus

octopus: rgw: Disable prefetch of entire head object when GET request with range header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34819 from idryomov/wip-krbd-unmap-spacing-octopus

octopus: tests: update unmap.t for table spacing changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34810 from smithfarm/wip-45047-octopus

octopus: rbd-mirror: improved replication statistics

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34425 from smithfarm/wip-44839-octopus

octopus: pybind/rbd: ensure image is open before permitting operations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #34626 from shyukri/wip-45042-octopus

octopus: mgr/PyModule: fix missing tracebacks in handle_pyerror()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34629 from shyukri/wip-45069-octopus

octopus: mgr/telegraf: catch FileNotFoundError exception

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34731 from smithfarm/wip-45215-octopus

octopus: rgw: use DEFER_DROP_PRIVILEGES flag unconditionally

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34758 from callithea/wip-45272-octopus

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34759 from callithea/wip-45280-octopus

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rbd-mirror: skip snapshot image sync if mirror snapshot is marked clean

This is currently only utilized for the case where a newly created image
has mirroring enabled at time of creation, but it could be expanded in the
future if we track writes.

Fixes: https://tracker.ceph.com/issues/44596
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fbede28d1a7713248765cf91136bb19b3b3fdac2)

librbd: EnableRequest now accepts a boolean to indicate a clean image

If the image is clean, it's treated as if it was newly created and
therefore clean since snapshot id 0. The CreateRequest and
CloneRequest state machines pass true for this bool if mirroring
is being enabled during creation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ea64ba5a6f2a73e3440244c0b358b948647670c5)

librbd: pass clean_since_snap_id to CreatePrimaryRequest

This will be stored in the primary MirrorSnapshotNamespace for use by
the rbd-mirror snapshot replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 959d5e60d8a7eec96c71875cf69d1d4a7d80816d)

cls/rbd: added clean_since_snap_id to MirrorSnapshotNamespace

It is only valid for primary snapshots so we can re-use the space
previously used by the primary_snap_id without the need to bump the
version id.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 50702eece0b1d7535d387bdf923b88cd4bbfee37)

rbd: add '--mirror-image-mode' optional to image creation actions

This allows the user to enable mirroring while creating, cloning,
copying, importing, or migrating an image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6da56d798248369f15191d8d5ec3b093f8dca831)

librbd: save requested mirror image mode during migration prepare

Once the migration is committed, the requested mode will be applied.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 25c034f0a0a627b54a1e4b948292890385cb7b6d)

librbd: permit mirroring to be enabled upon image cloning

The librbd clone API's ImageOptions parameter now supports passing
the mirror image mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 407f2e5c172cf5bef49a28b3156099487e7c2bf6)

librbd: add non-primary global image id to mirror enable

The constructor that took an ImageCtx did not support passing the
non-primary global image id and instead defaulted to an empty string.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a90042b2546bdaa056610bd10d7dae0f5e5da18e)

librbd: permit mirroring to be enabled upon image creation

The librbd create API's ImageOptions parameter now supports passing
the mirror image mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 37f6fa8fe95729af845100e28df8b8e0668c2270)

librbd: mirror enable state machine might need to open image

If attempting to create a snapshot-based mirroring primary snapshot,
the image needs to first be opened. If we weren't supplied an image,
open the image, create the snapshot, and close the image again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 931c9f0fd9332136466a8d81c9708b86f597f4be)

librbd: pass bit-flags to image::CreateRequest

The current boolean for skip enable mirror will be able to be changed to
a tri-state to force enable mirror (in addition to the current auto-enable
if in pool-mode).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b0c3ca63c2dff11c823d0d4456a2a07f1594fb9f)

rgw: Disable prefetch of entire head object when GET request with range header

Disable prefetch of entire head object when GET request with range header.
The current behavior for the RGW is getting the whole object although the client asked only for a small bytes offset.
For example: If the client asked for bytes=0-1, The RGW will anyway fetch 0-4194304

Fixes: https://tracker.ceph.com/issues/44508
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
(cherry picked from commit 2be5af0006169cb54547034aa98b7eacb8751d59)

qa/workunits/rbd: fix peer_add helper

We need to temporary disable "exit on error" mode so it does not
abort when `rbd mirror pool peer add` returns "already exists"
error code.

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 1ad816ae0c2d4f0db3d26e0ae993947154bab6c8)

qa/workunits/rbd: retry the addition of a mirror pool peer

fb4311f5 has fixed this for setup, but "remove mirroring pool"
test needs fixing too.

Fixes: https://tracker.ceph.com/issues/44938
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 7eced158a9a3c47cc408b35219b4428e97e018fb)

tests: update unmap.t for table spacing changes

Introduced in 4d42b4c5a0ed ("common/TextTable: default to 2 spaces
separating columns") and 41f003518a07 ("common/TextTable: only pad
between columns").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 314b14aebeef4ef1dfb450d191e2f53eaf063e67)

rbd-mirror: track in-flight start/stop/restart in instance replayer

The shut down waits for in-flight ops to complete but the
start/stop/restart operations were previously not tracked. This
could cause a potential race and crash between an image replayer
operation and the instance replayer shutting down.

Fixes: https://tracker.ceph.com/issues/45072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 31140a940ea1909c4b5d68ef4593cb582a527354)

rbd-mirror: switch to common C_TrackedOp context class

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 64f8d9c30c53077f48107e83ba836897aa99e609)

common: add helper C_TrackerOp context class

This wraps the functionality of starting and finishing a tracked op
into the standard context interface.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4bd9d1501f3832206ef12155cec3f008e3160822)

rbd-mirror: utilize structured binding in 'for' loop

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 847f178c4bfb50ab1c1b21222fcb47759119aa43)

rbd-mirror: unlink from remote snapshot if required

If a previous remote snapshot was synced but the unlink failed,
ensure we retry the unlink so that the remote can cleanup the unused
snapshot.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit cb7b91dc02b64cb15f8d21e830a698bd4173b35a)

rbd-mirror: prune unnecessary non-primary mirror snapshots

Once a non-primary snapshot is no longer required for syncing, delete it
from the image.

Fixes: https://tracker.ceph.com/issues/44105
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 281af0de86b17244a2c64af80db0bf84de6d9819)

rbd-mirror: propagate full snap-seq mapping in non-primary snapshots

Previously only newly created user snapshots were included in the
non-primary snapshot snap-seq mapping table. However, we need to
retain a full history of the mapping table if we want to be able to
prune non-primary snapshots.

Failovers are a special case since we won't have a valid snap seq mapping
so it will need to be rebuilt. Luckily, both sides should be read-only
in the previous state so we can use the snapshot names to find matches.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit cb8187c0dd22ed9098ab51e47777aee5aec471a7)

rbd-mirror: ignore non-primary read-only state for remote images

snapshot-based mirroring needs to be able to potentially delete a
demotion snapshot during the unlink process. Previously, these
snapshots have been left while the read-only error was ignored.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6bf2132cf041f6ab16c86e46b5acd4cf2ce12060)

rbd: fix missing space when listing non-primary mirror snapshots

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0ca7817eceafd7756eee967e7d740afeedd517e7)

librbd: fixed race condition on demotion of snapshot-based mirrored image

A pending refresh could occur after setting the non-primary feature flag but
before the creation of the demotion snapshot. This would prevent the snapshot
from being created and would leave the image in a half-primary state.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a3acdbd069b5c5bd62e528cbf2fbc33fe6f23d92)

librbd: store mirror peer uuids in non-primary demoted snapshots

This will allow a remote rbd-mirror process to have a snapshot to use for
delta sync operations during failover.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0102ce88701dd52208520b8c0ab2e505548ed3e1)

librbd: additional debug logs for mirror snapshot unlink peer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit eed00eb17969791107ddb8df0e909790c17bd573)

test/rbd-mirror: fix gmock warnings during snapshot-based replayer tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6a342bb5e01fea205088592128f62b55261408bf)

rbd-mirror: fixed race condition with snapshot sync and shutdown

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8a18a7fc71944c55e7a5fa47e55572546b4088db)

rbd-mirror: add snapshot-based replay performance metrics

The mirror image status for replaying snapshot-based images now includes
bytes per second and per snapshot, in addition to an estimated number of
seconds until the image is fully synced.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 03aa081435abf12429fbe98b7ff1095a4b37d830)

rbd-mirror: switch to json_spirit formatter for snapshot image status

This will make it cleaner and easier to add additional data fields
to the existing JSON replaying status.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit de445810e97b345f647ffc5f5792dfc2ad589f06)

rbd-mirror: add journal-based replay performance metrics

The mirror image status for replaying journal-based images now includes
bytes and entries per second in addition to an estimated number of seconds
until the image is fully synced.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c0dc96bf137dac6942e14800e173bfb9bda233eb)

rbd-mirror: switch to json_spirit formatter for journal image status

The free-form journal replay status description is now JSON-encoded. The
"master"/"mirror" designators have been changed to "primary"/"non_primary"
to better align with RBD terminology.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e04422f2c60a9ce891da496b71a568264e39233e)

rbd-mirror: periodically poll image replayer status

When metrics are incorporated, there might not be a forced status update
if no new data is available to replicate. However, we will want the metrics
to decrease over time.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 133cce8af50653d147d7147fc4bc7269ad10b5e1)

librbd: pass bytes-read statistics from deep-copy through handler

These simple stats will be utilized by rbd-mirror to compute throughput
metrics for snapshot-based mirroring.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0b6cbdeb9deec8184a43f1b7533c8316b22b9f5d)

librbd: pass specialized handler interface to deep-copy

The initial version mimics the existing ProgressContext callback
interface. Later commits will add additional deep-copy unique methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c7d6e762d19284a05b88daba6c27cbbdbdb210e9)

Merge pull request #34760 from callithea/wip-45281-octopus

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

qa/tasks/vstart_runner.py: use StringIO for capturing text

to be consistent with 8bfe977854686149c74de3ebaed7b571d43404dd

Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 2de78c3bd9eed9bec235b68e8e52b4635e9add61)

qa/tasks: Add missing StringIO import

This is directly fixed in octopus since it's needed
to get the dashboard backend API tests running
based on the previous commits of this PR.
There is a fix in master but this one
uses "from io import StringIO" while
we still need to be python2 compatible in octopus.
So this import is done with "six" (expects str in python2
instead of "io" (expects unicode
in python2).

This import line is not cherry-picked from master
due to the above mentioned reasons.

Signed-off-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: fix API tests

* test_perf_counters_mgr_get
* test_selftest_cluster_log

Fixes: https://tracker.ceph.com/issues/45170
Fixes: https://tracker.ceph.com/issues/45246
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 0ac296430f45ddf937b6c4c25607f0b757dcf447)

Merge pull request #34467 from rhcs-dashboard/wip-44992-octopus

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34708 from rhcs-dashboard/wip-45233-octopus

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34631 from shyukri/wip-44834-octopus

octopus: mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34547 from tspmelo/wip-45083-octopus

octopus: mgr/dashboard: Fix iSCSI's username and password validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34498 from votdev/wip-45020-octopus

octopus: mgr/dashboard: use FQDN for failover redirection

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34211 from s0nea/wip-44712-octopus

octopus: mgr/dashboard: add crush rule test suite

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Fix ServiceDetails and PoolDetails unit tests

While testing the upgrade to Angular 9,
these 2 unit tests were consistently failling.

Fixes: https://tracker.ceph.com/issues/42929
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit d7e7d2b09e463a978d2f881b6b1978fd2606651e)

mgr/dashboard: fixing RBD purge error in backend

Current UTC datetime should be used as `now`, otherwise the expiration
comparison might fail on machines with non-UTC localtime.

Fixes: https://tracker.ceph.com/issues/45149
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 00881adf55cac6116887bfffc506444473f0ee82)

Merge pull request #34750 from yuriw/wip-yuriw-distro-octupus

qa/tests: clean up distro and point to centos 8.1

qa/tests: clean up distro and point to centos 8.1

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34687 from sebastian-philipp/octopus-backport-34558-34216-34415-34555-34556-34490-34529-34346-32951-34584-34612-34565-34568-34621

octopus: cephadm: Batch backport April (2)

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34548 from tspmelo/wip-45084-octopus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34625 from shyukri/wip-45051-octopus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34630 from shyukri/wip-44837-octopus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/rgw: allow 'ceph' user to read vault token

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 58a773b59c2f563caa02ff913d6d00f009f62730)

rgw: use DEFER_DROP_PRIVILEGES flag unconditionally

this flag doesn't take effect unless its set before global_init()

Fixes: https://tracker.ceph.com/issues/44661
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a9d9158233135d43f8c79ea86569f5d8f496d180)

rgw: combine calls to global_init/pre_init

calling global_init() without 'run_pre_init=false' will call
global_pre_init() for us

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d12c1bed88bc93846de022bcdd55d596199247e2)

qa/rgw: add setuser/group to test DEFER_DROP_PRIVILEGES

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 94ca8f4b81f6e1c546a4c50c401d1ffde48be0fb)

Merge pull request #34686 from sebastian-philipp/octopus-backport-34443

octopus: cephadm: Update images used

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34638 from smithfarm/wip-45036-octopus

octopus: rbd: ignore tx-only mirror peers when adding new peers

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: fix tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all

Increased the time for getting 2nd TOTP.

Fixes: https://tracker.ceph.com/issues/44405
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit ee2b2d19781e531770350ad2abe53ce876cebe3b)

Merge pull request #34526 from smithfarm/wip-45034-octopus

octopus: spec: address some warnings raised by RPM 4.15.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34519 from smithfarm/wip-45059-octopus

octopus: qa/suites/rados/rest: don't pass empty dict as data arg

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34395 from votdev/wip-44918-octopus

octopus: monitoring: alert for prediction of disk and pool fill up broken

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #34648 from shyukri/wip-44892-octopus

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34418 from votdev/wip-44955-octopus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

test_cephadm.sh: remove unmaintained test_cephadm.sh

the script has become unmaintained and duplicates logic found in
qa/workunits/cephadm/test_cephadm.sh

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 879e6850992685115d0bd0a38b4391722b4e646f)

qa/workunits/cephadm/test_cephadm.sh: add CLEANUP var

preserve the current test state during EXIT:
`CLEANUP=false ../qa/workunits/cephadm/test_cephadm.sh`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 6b211447230d3b6d6d8d49b780750cf6aeeace6d)

mgr/orch: upgrade: better input validation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 67a29846f80fe8903f2dbce5dca0f5a0134d55b8)

mgr/orch: cleanup python imports

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 62ed783dc243e076e00be93ea8ab2dd1fb262d58)

cephadm: bail out early during adopt if data dir not present

In case someone tries to run this again on an already adopted daemon...

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit bed711d3d1256758bdf7428339316bcca6f9dae0)

cephadm: add --force-start option to adopt

Current behaviour is to only start a newly adopted ceph daemon if it was
already running before the adopt.  Adding a --force-start option allows
the adopt command to start newly adopted daemons that weren't originally
running, to save the user having to manually invoke `systemctl start
ceph-$FSID@$DAEMMON.$ID`.

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 225b25512916ec4a9577ca7f2ab39faaf122827f)

cephadm: don't throw when disabling ceph-volume simple unit

When adopting OSDs, if a ceph-volume simple service is already disabled
(or otherwise missing) the previous implementation would raise an error,
thus killing the adopt.

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 6834ad983762b7cbf950c6a6e71aa44ec6657c32)

cephadm: handle adopting offline OSDSs

The current adopt behavior expects OSDs to be online, in order to read
/var/lib/ceph/osd/ceph-$ID/fsid.  To handle the case where OSDs
are offline, this change first checks to see if that file is present,
and if not, falls back to calling `ceph-volume lvm list` to see if
there's a matching OSD there, and if that doesn't work, it checks
/etc/ceph/osd/*.json to see if there's a matching old-style simple
OSD present.

For LVM OSDs, the only thing we need is the ODS's fsid; the remainer
of the adopt procedure "just works", as the various other files
in /var/lib/ceph/$FSID/osd.$ID are created by magic anyway when the
OSD is activated, so it doesn't matter if they're not present at
adoption time.

For simple (ceph-disk created) OSDs, we actually need all the files under
/var/lib/ceph/osd/ceph-$ID/ to be moved to /var/lib/ceph/$FSID/osd.$ID
so if a simple OSD is found, it's mounted first, so the existing
move_files() a bit further down around line 3200 continues to work.

Fixes: https://tracker.ceph.com/issues/45095
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 9169119bd3c0ac976871e7f3321d9d7a53335c82)

Add troubleshooting guidance for ssh connection failures.

For more thorough information about this commit, see:
https://tracker.ceph.com/issues/44905

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Michael Fritch <mfritch@suse.com>
ibid

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Added mgfritch's suggestion regarding checking to see if the public key
is listed in the authorized_keys file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>
(cherry picked from commit c4765afb81489b1df71d87be32c566ed55d29a28)

mgr/test_orchestrator: fix error when listing services

- Adapting attribute change of ServiceSpec.
- Update dummy data.

Fixes: https://tracker.ceph.com/issues/45108
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 223c8d816f7f1d3d5ea737c540bae4553334aabf)

mgr/orch: Fixes some deserialization errors

Fixes errors when calling `from_json` of these classes:
- InventoryHost: parsing labels
- ServiceDescription: `last_refresh` and `created` fields should be parsed
  to datetime type.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 45fd5ee275aa94409bbe9d9c3874c9aad82850a8)

cephadm: add RGW SSL support

This patch adds SSL support to RGW when using cephadm.
If an SSL certificate is provided, inside the json supplied with:

  cpeh orchestrator rgw create -i rgw.json

Then the SSL cert and/or key will be added to pushed into the mon config-key database
using the key `rgw/cert/<rgw_realm>/<rgw_zone>.[crt|key]`.
Which will then be referenced in the config:

  rgw_frontends = beast port=80 ssl_port=443 ssl_certificate=config://rgw/cert/<rgw_realm>/<rgw_zone>.crt

And if an ssl key is also supplied this becomes something like:

  rgw_frontends = beast port=80 ssl_port=443 ssl_certificate=config://rgw/cert/<rgw_realm>/<rgw_zone>.crt ssl_key=config://rgw/cert/<rgw_realm>/<rgw_zone>.key

Of course you could also just upload the cert and key yourself to
config-key location, and ssl will be enabled as well. But this patch
let's you either supply them via `-i` or as a manual upload step.

Co-Authored-By: Michael Fritch <mfritch@suse.com>
Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit 8204b1d524b155fac56c3937d1aec41a18ab2d52)

mgr/cephadm: add replacement flag to _create_osd

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit e670b1ea83f66d977c919ef956cc4ae0db9cb453)

mgr/cephadm: use --osd-ids instead of --osd-id

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit fa88483f660f2bf79d1c1ca77d7104b3a42ed05d)

mgr/cephadm: add support for osd_id_claims

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 5b32c3e51b517332b7ac26b161b7b6b4fae05b85)

cephadm: warn when no valid version arguments are supplied to add-repo

Currently add-repo will add a broken repository by default, this commit gives a pretty error to instruct the user to use one of the version arguments.

Fixes #45029

Signed-off-by: Maran Hidskes <maran@protonmail.com>
(cherry picked from commit 115b3d7b7f3b4cf4149240fc14e374697a7cd6b1)

cephadm: add-repo: Don't test for podman

Fixes:

```
root@buster:/cephadm# ./cephadm add-repo
Unable to locate any of ['podman', 'docker']
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 0d22708cc2f858491604b09f87e7af742884cb3c)

mgr/cephadm: corrected documentation for starting an upgrade

Signed-off-by: Andreas Haase <andreas.haase@community4you.de>
(cherry picked from commit bf3d06d180135901903251e6e73b66f4fbac289b)

mgr/cephadm: Fixed warn_on_stray_hosts setting

Fixes: https://tracker.ceph.com/issues/45065
Signed-off-by: Andreas Haase <andreas.haase@community4you.de>
(cherry picked from commit b07c2da63f9fb9e70a647a989675bbdf309c14f3)

mgr/orch: Fix DaemonDescription().daemon_id() for RGWs

Turns out, users put dots into their RGW service names.

Fixes: https://tracker.ceph.com/issues/44934
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit 39fdb5c67c4d3834c5157e8d6da20933a6f7bfb4)

mgr/cephadm: enable osd previews

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 462a68ab87446af8299d4f71c5ff22f77dfeea64)

cephadm: use CephContainer during command `enter`

instead of the `get_container` helper func

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 3c762d0ed92d38a2c47bf273bf17600f101ed1bb)

cephadm: Update images used

Print warning when using cephadm from master

See also "use quay octopus tip until 15.2 tag is available"
* a9b15c7e1a0c14376cd66f166370694294398494.

See also  "update default container images"
* 1f05f7578794380f969a7e93db07345626b3e4df.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit de448fafe0fe7d6889f3f75ac37f870cde00cf1c)
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34495 from votdev/wip-44953-octopus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34654 from ceph/wip-yuriw-octopus-client-upgrade-octopus-pacific

qa/tests: initial checkin for clients upgrades tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: initial checkin for clients upgrades tests

for future release 'pacific'

(Note to self: moved from wrongly pushed to master branch)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>