ceph-daemon: make mon container privileged

libudev needs to be privileged in order to query the underlying hardware
devices, as reported by the 'ceph device ...' command set, and to scrape
smart metrics, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31443 into master

* refs/pull/31443/head:
ceph-daemon: fix 'version' field for legacy `ls`

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31458 into master

* refs/pull/31458/head:
ceph-daemon: Only run in the __main__ scope

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31463 into master

* refs/pull/31463/head:
ceph-daemon: switch default image

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31448 into master

* refs/pull/31448/head:
ceph-daemon: make 'ls' log less noisy

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: switch default image

Use this until octopus is ready.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/{dashboard,prometheus}: return FQDN instead of '0.0.0.0' (#31430)

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Use htmllint (#30998)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Set iSCSI disk WWN and LUN number from the UI (#31056)

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-daemon: Only run in the __main__ scope

That makes unit testing easier to setup because the code is not loaded
when ceph-daemon gets imported. Instead it is only loaded when
executed.
For that, the parser also moved to a function instead of being on
module level.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #31070 from sebastian-philipp/dashbaord-run-backend-zsh

mgr/dashboard: Fix zsh support in run-backend-api-tests.sh

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: fix e2e test failure for OSDs (#31451)

mgr/dashboard: fix e2e test failure for OSDs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: fix e2e test failure for OSDs

When clicking a selected row, the row becomes unselected and the detail
component is hidden. Rearrange related tests by selecting the row only
once.

Fixes: https://tracker.ceph.com/issues/42671
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #31350 from chenpan1012/rbd_testcase

cls/rbd: add snapshot limit UINT64_MAX test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-daemon: make 'ls' log less noisy

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31208 from weiyingze-git/master

test:add test_remove_snap2 interface to remove  snap when it's protected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-daemon: fix 'version' field for legacy `ls`

The `ls` command was reporting the git commit id instead of the ceph
version.

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31441 into master

* refs/pull/31441/head:
github/codeowners: Add orchestrator team

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31437 into master

* refs/pull/31437/head:
ceph-daemon: Remove data dir during adopt

Reviewed-by: Sage Weil <sage@redhat.com>

github/codeowners: Add orchestrator team

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #31440 into master

* refs/pull/31440/head:
qa/standalone/test_ceph_daemon: Make container images configurable
qa/standalone/test_ceph_daemon: Allow running from root dir
qa/standalone/test_ceph_daemon: Fix hang when CEPH_DAEMON is not set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31439 into master

* refs/pull/31439/head:
ceph-daemon: fix traceback during `ls` command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31385 into master

* refs/pull/31385/head:
mgr/ssh: add 'osd rm'
mgr/ssh: keep inventory and service cache keys in sync
mgr/orchestrator_cli: fix 'service ls' table format
mgr/ssh: invalidate service state on create, remove, and service action
mgr/orchestrator: add invalidate() to OutdatableDictMixin
mgr/ssh: cache services
mgr/ssh: drop specialized _remove_{mds,mgr,rgw}
mgr/ssh: change inventory_cache_timeout to seconds (not minutes)
mgr/orchestrator: make wait poll every 1s (not 5s)

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/standalone/test_ceph_daemon: Make container images configurable

Instead of hardcoding the images, make them configureable via
environment variables.
That way, downstream can use the script with custom images.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: fix traceback during `ls` command

fixup for 3a1ac3ff6ac2ef6e3e8c361c0c55562b732b9aac

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/standalone/test_ceph_daemon: Allow running from root dir

Allow running the script from root directory via:

./qa/standalone/test_ceph_daemon.sh

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

qa/standalone/test_ceph_daemon: Fix hang when CEPH_DAEMON is not set

When running test_ceph_daemon.sh from the root dir and not setting
$CEPH_DAEMON manually, the call hangs at:

$ ./qa/standalone/test_ceph_daemon.sh
[...]
+ for p in $PYTHONS
+ echo '=== re-running with python3 ==='
=== re-running with python3 ===
++ which python3
+ ln -s /usr/bin/python3 /tmp/tmp.6hneCsNMio/python
+ echo '#!/tmp/tmp.6hneCsNMio/python'
+ cat

Check that there is a ceph-daemon found before continue.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: Remove data dir during adopt

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: add missing test_orchestrator suite (#31198)

mgr/dashboard: add missing test_orchestrator suite

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31422 into master

* refs/pull/31422/head:
mgr/MgrClient: fix open condition fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/ssh: add 'osd rm'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: keep inventory and service cache keys in sync

Our host list is implicitly defined as the set of keys for the inventory
(and service) caches.  Ensure they are in sync by unioning and inserting
any missing items.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: fix 'service ls' table format

Make this match the rest of Ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: invalidate service state on create, remove, and service action

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add invalidate() to OutdatableDictMixin

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: cache services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: drop specialized _remove_{mds,mgr,rgw}

_remove_daemon is sufficient

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: change inventory_cache_timeout to seconds (not minutes)

The descriptio didn't match the implementation (which took minutes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: make wait poll every 1s (not 5s)

More responsive!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/prometheus: return FQDN for default server_addr

Instead of returning '0.0.0.0' the prometheus module returns the FQDN of
the host, as it is the case for '::'.

Fixes: https://tracker.ceph.com/issues/42664
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: return FQDN for default server_addr

Instead of returning '0.0.0.0' the dashboard returns the FQDN of the
host, as it is the case for '::'.

Fixes: https://tracker.ceph.com/issues/42664
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: Use htmllint

This package will replace htmlhint.

Some rules from the previous installed packages were not working that well.
This new package has more rules and seems to work better.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add support for device management (#30759)

mgr/dashboard: Add support for device management

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Searchable objects for table (#31199)

mgr/dashboard: Searchable objects for table

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26768 from sebastian-philipp/upstream-pull-26684

mgr/orchestrator: device lights

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31409 into master

* refs/pull/31409/head:
mds: Fix MDCache.h reorder compiler warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31401 into master

* refs/pull/31401/head:
mds: clarify comment

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #31420 into master

* refs/pull/31420/head:
ceph-daemon: do not screw up line splitting
mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: do not screw up line splitting

When we log stdout and stderr, we were artificially adding line splits
due to reading 1024 bytes at a time and assuming it was aligned with a
newline.

Fix by keeping partial lines buffered and only log them once they are
complete (or the call finishes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: fix open condition fix

Fix fc60989bf7a72c35b8f6b8fec2407b3080ad9bbd, which reversed this
condition.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: disable pg_autoscale for new pools in e2e tests (#31396)

mgr/dashboard: disable pg_autoscale for new pools in e2e tests

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31365 into master

* refs/pull/31365/head:
ceph-daemon: make rm-daemon reset-failed
ceph-daemon: make rm-cluster handle failed unit cleanup

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Fix error on ceph-iscsi version pre controls_limits

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Set iSCSI disk WWN and LUN number from the UI

Fixes: https://tracker.ceph.com/issues/41749
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #31303 into master

* refs/pull/31303/head:
ceph-daemon: make systemd unit descriptions Capitalized
ceph-daemon: handle daemon names with multiple .'s
mgr/ssh: rgw: ensure map rgws are mapped to the zone
mgr/ssh: use . as delimiter between service group and id
mgr/ssh: implement 'rgw update'
mgr/orchestrator_cli: add 'rgw update' CLI command
mgr/ssh: add 'rgw rm'
mgr/ssh: implement 'rgw add'
ceph-daemon: handle 'rgw' type daemons
mgr/orchestrator_cli: rgw add: optionally take count and node names

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: make systemd unit descriptions Capitalized

Same style as other units.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: handle daemon names with multiple .'s

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: rgw: ensure map rgws are mapped to the zone

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: use . as delimiter between service group and id

This aligns with . as an entity delimiter in the ceph config.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'rgw update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: add 'rgw update' CLI command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'rgw rm'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'rgw add'

Note that this doesn't correctly confine a daemon to a zone.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: handle 'rgw' type daemons

- weird data dir: /var/lib/ceph/radosgw/ceph-rgw.$id (not
  /var/lib/ceph/rgw/ceph-$id)
- weird auth name: client.rgw.$id (not rgw.$id)
- weird entrypoint: /usr/bin/radosgw (not /usr/bin/ceph-rgw)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rgw add: optionally take count and node names

This avoids the need for JSON in the common case.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31327 into master

* refs/pull/31327/head:
mon/ConfigMap: search nested sections

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31412 into master

* refs/pull/31412/head:
osd: send smart asok result to stdout, not stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31363 from cbodley/wip-qa-rgw-ec-is-fast-again

qa/rgw: remove whitelist for SLOW_OPS against ec pools

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

osd: send smart asok result to stdout, not stderr

Uncross the streams!

Reported-by: Patrick Seidensal <pseidensal@suse.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31321 into master

* refs/pull/31321/head:
ceph-daemon: do not relabel system directories

Merge PR #31342 into master

* refs/pull/31342/head:
qa/standalone/test_ceph_daemon: fix multi-version python test

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/orchestrator: Improve ceph CLI for blink lights

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mds: Fix MDCache.h reorder compiler warnings

Fixes: https://tracker.ceph.com/issues/42642
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #30598 from smithfarm/wip-doc-releases-toc

doc/releases: access main releases page from top-level TOC

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #31291 from liewegas/bug-42578

mgr/ActivePyModules: behave if a module queries a devid that does not exist

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31230 from aclamk/fix-rados-put-truncate

tools/rados: prevent 'put' operation from recreating object when --offset=0

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: check for DEVICE_{IDENT|FAULT}_ON

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Add interface and tests to device lights

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: 'ceph device {fault,ident}-{on,off}', 'ceph device ls-lights'

Add generic CLI commands to turn device lights on and off.

These will call out to orchestrator stubs, once it has the appropriate hook
implemented, which will in turn call out to salt or rook or ssh or whatever
to actually turn a light on or off.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31366 from smithfarm/wip-cbs-4

ceph-backport.sh: implement interactive setup routine and new options

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix iSCSI Target's validation (#30894)

mgr/dashboard: Fix iSCSI Target's validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Fix OSD's safe_to_destroy method (#30499)

mgr/dashboard: Fix OSD's safe_to_destroy method

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: disable pg_autoscale for new pools in e2e tests

The pg_autoscaler might interfere the editing pool e2e test, disable the
global config to avoid this problem.

Fixes: https://tracker.ceph.com/issues/42608
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mds: clarify comment

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30835 into master

* refs/pull/30835/head:
mds: no assert on frozen dir when scrub path

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30915 into master

* refs/pull/30915/head:
mds: Reorganize class members in MDSAuthCaps header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30745 into master

* refs/pull/30745/head:
mds: Reorganize class members in MDCache header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31330 into master

* refs/pull/31330/head:
messages: protect CephFS messages get/put methods
messages: protect MClientCapRelease cons
messages: simplify MMDSCacheRejoin default cons
mds: simplify type declarations

Reviewed-by: Kefu Chai <kchai@redhat.com>

cls/rbd: add snapshot limit UINT64_MAX test case

Signed-off-by: Chen Pan <chenpan@cmss.chinamobile.com>

Merge pull request #31374 from cbodley/wip-qa-rgw-multisite-sse

qa/rgw: use 'testing' kms backend for multisite tests

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

ceph-backport.sh: increment the version number

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: delete deprecated config file after setup complete

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: implement interactive setup routine and new options

This commit implements several new features:

* a --cherry-pick-only option
* a --force option
* an --existing-pr option
* an interactive setup routine

The --cherry-pick-only option can be used to test whether a backport
cherry-picks cleanly, for example. This is the same as the --prepare
functionality that was provided by an earlier version of the script, and
--prepare is re-introduced as a synonym for --cherry-pick-only.

The --force option can be used to make the script less careful (less
"cowardly"). For example, if the script refuses to do a backport because the
backport tracker issue is assigned to someone else, the script will "cowardly"
refuse to continue. Use --force to override. Be aware that --force will also
blow away an existing wip branch - the script asks for user confirmation in this
case.

The new --existing-pr option can be used to specify the number (ID) of an
existing backport PR that addresses the backport tracker issue given via the
positional argument.

The new "interactive setup routine" should make the setup process much simpler
for the user. If there is a setup issue, the script produces a report and starts
the interactive setup routine, which prompts the user for the needed
information.

Also, the script no longer requires the user to explicitly provide values for
github_user and redmine_user_id. Instead, it divines the correct values from the
GitHub token and the Redmine key, respectively.

Finally, the existing ~/bin/backport_common.sh file is deprecated in favor of
two files:

    ~/.github_token
    ~/.redmine_key

(The latter is already used by Sage's build-integration-branch tool and it
didn't make sense to have two different configuration files for a single
purpose.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/rgw: use 'testing' kms backend for multisite tests

a missing piece from https://github.com/ceph/ceph/pull/30940

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31192 from sebastian-philipp/run_mypy-0.740

script/run_mypy: Support mypy 0.740

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #31341 into master

* refs/pull/31341/head:
ceph-daemon: only set up /var/run/ceph/$fsid if it exists

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31331 into master

* refs/pull/31331/head:
ceph-daemon: Use `shutil` to move/copy files

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31353 into master

* refs/pull/31353/head:
doc/orchestrator: Fix various issues in Orchestrator CLI documentation

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31329 into master

* refs/pull/31329/head:
ceph-daemon: Fix `ls` cmd for legacy confs

Reviewed-by: Sage Weil <sage@redhat.com>