mgr/ssh: add 'osd rm'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: keep inventory and service cache keys in sync

Our host list is implicitly defined as the set of keys for the inventory
(and service) caches.  Ensure they are in sync by unioning and inserting
any missing items.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: fix 'service ls' table format

Make this match the rest of Ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: invalidate service state on create, remove, and service action

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add invalidate() to OutdatableDictMixin

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: cache services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: drop specialized _remove_{mds,mgr,rgw}

_remove_daemon is sufficient

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: change inventory_cache_timeout to seconds (not minutes)

The descriptio didn't match the implementation (which took minutes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: make wait poll every 1s (not 5s)

More responsive!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Add support for device management (#30759)

mgr/dashboard: Add support for device management

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Searchable objects for table (#31199)

mgr/dashboard: Searchable objects for table

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26768 from sebastian-philipp/upstream-pull-26684

mgr/orchestrator: device lights

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31409 into master

* refs/pull/31409/head:
mds: Fix MDCache.h reorder compiler warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31401 into master

* refs/pull/31401/head:
mds: clarify comment

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #31420 into master

* refs/pull/31420/head:
ceph-daemon: do not screw up line splitting
mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: do not screw up line splitting

When we log stdout and stderr, we were artificially adding line splits
due to reading 1024 bytes at a time and assuming it was aligned with a
newline.

Fix by keeping partial lines buffered and only log them once they are
complete (or the call finishes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: disable pg_autoscale for new pools in e2e tests (#31396)

mgr/dashboard: disable pg_autoscale for new pools in e2e tests

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31365 into master

* refs/pull/31365/head:
ceph-daemon: make rm-daemon reset-failed
ceph-daemon: make rm-cluster handle failed unit cleanup

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31303 into master

* refs/pull/31303/head:
ceph-daemon: make systemd unit descriptions Capitalized
ceph-daemon: handle daemon names with multiple .'s
mgr/ssh: rgw: ensure map rgws are mapped to the zone
mgr/ssh: use . as delimiter between service group and id
mgr/ssh: implement 'rgw update'
mgr/orchestrator_cli: add 'rgw update' CLI command
mgr/ssh: add 'rgw rm'
mgr/ssh: implement 'rgw add'
ceph-daemon: handle 'rgw' type daemons
mgr/orchestrator_cli: rgw add: optionally take count and node names

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: make systemd unit descriptions Capitalized

Same style as other units.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: handle daemon names with multiple .'s

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: rgw: ensure map rgws are mapped to the zone

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: use . as delimiter between service group and id

This aligns with . as an entity delimiter in the ceph config.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'rgw update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: add 'rgw update' CLI command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'rgw rm'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'rgw add'

Note that this doesn't correctly confine a daemon to a zone.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: handle 'rgw' type daemons

- weird data dir: /var/lib/ceph/radosgw/ceph-rgw.$id (not
  /var/lib/ceph/rgw/ceph-$id)
- weird auth name: client.rgw.$id (not rgw.$id)
- weird entrypoint: /usr/bin/radosgw (not /usr/bin/ceph-rgw)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rgw add: optionally take count and node names

This avoids the need for JSON in the common case.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31327 into master

* refs/pull/31327/head:
mon/ConfigMap: search nested sections

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31412 into master

* refs/pull/31412/head:
osd: send smart asok result to stdout, not stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31363 from cbodley/wip-qa-rgw-ec-is-fast-again

qa/rgw: remove whitelist for SLOW_OPS against ec pools

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

osd: send smart asok result to stdout, not stderr

Uncross the streams!

Reported-by: Patrick Seidensal <pseidensal@suse.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31321 into master

* refs/pull/31321/head:
ceph-daemon: do not relabel system directories

Merge PR #31342 into master

* refs/pull/31342/head:
qa/standalone/test_ceph_daemon: fix multi-version python test

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/orchestrator: Improve ceph CLI for blink lights

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mds: Fix MDCache.h reorder compiler warnings

Fixes: https://tracker.ceph.com/issues/42642
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #30598 from smithfarm/wip-doc-releases-toc

doc/releases: access main releases page from top-level TOC

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #31291 from liewegas/bug-42578

mgr/ActivePyModules: behave if a module queries a devid that does not exist

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31230 from aclamk/fix-rados-put-truncate

tools/rados: prevent 'put' operation from recreating object when --offset=0

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: check for DEVICE_{IDENT|FAULT}_ON

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Add interface and tests to device lights

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: 'ceph device {fault,ident}-{on,off}', 'ceph device ls-lights'

Add generic CLI commands to turn device lights on and off.

These will call out to orchestrator stubs, once it has the appropriate hook
implemented, which will in turn call out to salt or rook or ssh or whatever
to actually turn a light on or off.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31366 from smithfarm/wip-cbs-4

ceph-backport.sh: implement interactive setup routine and new options

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix iSCSI Target's validation (#30894)

mgr/dashboard: Fix iSCSI Target's validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Fix OSD's safe_to_destroy method (#30499)

mgr/dashboard: Fix OSD's safe_to_destroy method

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: disable pg_autoscale for new pools in e2e tests

The pg_autoscaler might interfere the editing pool e2e test, disable the
global config to avoid this problem.

Fixes: https://tracker.ceph.com/issues/42608
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mds: clarify comment

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30835 into master

* refs/pull/30835/head:
mds: no assert on frozen dir when scrub path

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30915 into master

* refs/pull/30915/head:
mds: Reorganize class members in MDSAuthCaps header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30745 into master

* refs/pull/30745/head:
mds: Reorganize class members in MDCache header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31330 into master

* refs/pull/31330/head:
messages: protect CephFS messages get/put methods
messages: protect MClientCapRelease cons
messages: simplify MMDSCacheRejoin default cons
mds: simplify type declarations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31374 from cbodley/wip-qa-rgw-multisite-sse

qa/rgw: use 'testing' kms backend for multisite tests

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

ceph-backport.sh: increment the version number

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: delete deprecated config file after setup complete

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: implement interactive setup routine and new options

This commit implements several new features:

* a --cherry-pick-only option
* a --force option
* an --existing-pr option
* an interactive setup routine

The --cherry-pick-only option can be used to test whether a backport
cherry-picks cleanly, for example. This is the same as the --prepare
functionality that was provided by an earlier version of the script, and
--prepare is re-introduced as a synonym for --cherry-pick-only.

The --force option can be used to make the script less careful (less
"cowardly"). For example, if the script refuses to do a backport because the
backport tracker issue is assigned to someone else, the script will "cowardly"
refuse to continue. Use --force to override. Be aware that --force will also
blow away an existing wip branch - the script asks for user confirmation in this
case.

The new --existing-pr option can be used to specify the number (ID) of an
existing backport PR that addresses the backport tracker issue given via the
positional argument.

The new "interactive setup routine" should make the setup process much simpler
for the user. If there is a setup issue, the script produces a report and starts
the interactive setup routine, which prompts the user for the needed
information.

Also, the script no longer requires the user to explicitly provide values for
github_user and redmine_user_id. Instead, it divines the correct values from the
GitHub token and the Redmine key, respectively.

Finally, the existing ~/bin/backport_common.sh file is deprecated in favor of
two files:

    ~/.github_token
    ~/.redmine_key

(The latter is already used by Sage's build-integration-branch tool and it
didn't make sense to have two different configuration files for a single
purpose.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/rgw: use 'testing' kms backend for multisite tests

a missing piece from https://github.com/ceph/ceph/pull/30940

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31192 from sebastian-philipp/run_mypy-0.740

script/run_mypy: Support mypy 0.740

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #31341 into master

* refs/pull/31341/head:
ceph-daemon: only set up /var/run/ceph/$fsid if it exists

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31331 into master

* refs/pull/31331/head:
ceph-daemon: Use `shutil` to move/copy files

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31353 into master

* refs/pull/31353/head:
doc/orchestrator: Fix various issues in Orchestrator CLI documentation

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31329 into master

* refs/pull/31329/head:
ceph-daemon: Fix `ls` cmd for legacy confs

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-daemon: Fix `ls` cmd for legacy confs

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31328 into master

* refs/pull/31328/head:
ceph-daemon: raise RuntimeError when CephContainer.run() fails

ceph-daemon: Use `shutil` to move/copy files

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: raise RuntimeError when CephContainer.run() fails

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: make rm-daemon reset-failed

We did this with rm-cluster and forgot to update rm-daemon too.  Also,
switch to call instead of call_throws.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make rm-cluster handle failed unit cleanup

Signed-off-by: Sage Weil <sage@redhat.com>

qa/rgw: remove whitelist for SLOW_OPS against ec pools

now that https://tracker.ceph.com/issues/41834 is resolved, we can
enforce the SLOW_OPS failures again in ec pool testing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Automatically use correct chromedriver version (#31352)

mgr/dashboard: Automatically use correct chromedriver version

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31292 into master

* refs/pull/31292/head:
mgr/ssh: fix remove_mds exception
doc/mgr/orchestrator_cli: remove extra characters after checkmarks
doc/mgr/orchestrator_cli: no {stop,start,restart} for all the random svcs
mgr/ssh: add 'redeploy' operation
ceph-daemon: stop + reset-failed existing unit on daemon deploy
ceph-daemon: make second call arg optional
ceph-daemon: make 'deploy' on mon only to mkfs step is mon doesn't yet exist
mgr/ssh,orchestrator_cli: add 'restart' service action
mgr/ssh: service[-instance] [start|stop]
mgr/ssh: simplify get_services call for 'mds update'
mgr/ssh: make _get_services take service *id* or service *name*
mgr/ssh: fix 'service ls' on stopped services
mgr/ssh: move some code around
doc/mgr/orchestrator_cli: 'service ls' works for ssh
doc/mgr/orchestrator_cli: we already completed mds ops for rook+ssh

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/ssh: fix remove_mds exception

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: remove extra characters after checkmarks

Not sure what those were.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/orchestrator_cli: no {stop,start,restart} for all the random svcs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'redeploy' operation

This will stop, reprovision container, and restart. For most daemons, that
means we can

 - change the container image
 - refresh the config
 - refresh the keyring

For OSDs, it means we also re-run ceph-volume activate.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: stop + reset-failed existing unit on daemon deploy

Normally it won't exist, but it will on a re-deploy.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make second call arg optional

If not specified, pull it from command[0].

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make 'deploy' on mon only to mkfs step is mon doesn't yet exist

This allows us to (re)deploy a mon container like the other daemon types,
which means rewriting the config, keyring, and systemd unit(s).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,orchestrator_cli: add 'restart' service action

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: service[-instance] [start|stop]

Note that 'reload' is meaningless for ssh orch...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31266 from rhcs-dashboard/wip-42565-master

mgr/dashboard: fix LazyUUID4 not serializable

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: CephFS directory component (#30047)

mgr/dashboard: CephFS directory component

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Searchable objects for table

The table can now search through objects, by default it won't search
through them, but it won't fail like before.

The RBD list page is now capable of searching through objects, which
only exist on an RBD that was cloned from a snapshot.

Fixes: https://tracker.ceph.com/issues/42480
Signed-off-by: Stephan Müller <smueller@suse.com>

doc/orchestrator: Fix various issues in Orchestrator CLI documentation

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Automatically use correct chromedriver version

Every Chrome version requires a corresponding chromedriver version for the e2e
test to work.
Since we were using a fixed chromedriver version, each time chrome was updated
the e2e scripts stopped working.
With this new script we will check which version of Chrome is installed and
update chromedriver to the correct version.

Fixes: https://tracker.ceph.com/issues/42607
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
Signed-off-by: Tiago Melo <tmelo@suse.com>

script/run_mypy: Support mypy 0.740

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Co-Authored-By: Ernesto Puerta <37327689+epuertat@users.noreply.github.com>

Merge pull request #31191 from sebastian-philipp/ceph-volume-pr-template

github: Add ceph-volume to list of jenkins commands

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

messages: protect CephFS messages get/put methods

Use SafeMessage inheritance so that only intrusive pointers can modify
the reference count. This is just a defensive programming refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

messages: protect MClientCapRelease cons

This should only be constructed via make_message.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

messages: simplify MMDSCacheRejoin default cons

Trivial initializations can be moved out.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: simplify type declarations

This is a trivial refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31326 from cbodley/wip-cmake-mgr-cap

cmake: define mgr_cap_obj library when WITH_MGR=OFF

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31015 into master

* refs/pull/31015/head:
mgr/orchestrator: Fix raise_if_exception for Python 3

Reviewed-by: Tim Serong <tserong@suse.com>

qa/standalone/test_ceph_daemon: fix multi-version python test

We have to rewrite the shebang line, since it is no longer just
'#/usr/bin/env python' (as of e12ad1b016db818fe20062e4373218890c6f4cbd).

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: only set up /var/run/ceph/$fsid if it exists

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31337 into master

* refs/pull/31337/head:
Fixing typo in ceph-daemon script for the output_pub_ssh_key argument

Reviewed-by: Sage Weil <sage@redhat.com>

Fixing typo in ceph-daemon script for the output_pub_ssh_key argument

Signed-off-by: John McGowan <john@steakfest.com>

Merge pull request #31335 from wjwithagen/wjw-fix-objecstore-bluestore

test: move bluestore dependent code under WITH_BLUESTORE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31312 from changchengx/qa

qa: correct zap disk with ceph-deploy tool

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>