ceph-backport.sh/interactive_setup: abort if Step 3 fails

Fixes confusing error message reported by a user:

Setup step 3 of 4 - remote repos
---------------------------------------------------------------------
Searching "git remote -v" for remote repos

ceph-backport.sh: ERROR: Cannot auto-determine fork remote
ceph-backport.sh: (Could not find GitHub user callithea's fork of ceph/ceph in
"git remote -v")
ceph-backport.sh: ERROR: (internal error) fork_remote not set, even after
completing Steps 1-3 of interactive setup
ceph-backport.sh: This could be reported as a bug!

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: increment version number

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: fix interactive setup brokenness

A user did "ceph-backport.sh --setup" and entered the interactive setup routine.
After entering a valid GitHub personal access token, she got the following
message:

ceph-backport.sh: my GitHub username is callithea
ceph-backport.sh: You must provide a valid GitHub personal access token
ceph-backport.sh: ERROR: problem detected in your setup
ceph-backport.sh: Run "ceph-backport.sh --setup" to fix

and the script aborted. This commit fixes the issue.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: allow cherry-picking from non-master (via --force)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh --existing-pr: do not clobber existing labels

Fixes: https://tracker.ceph.com/issues/42657
Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: --existing-pr ... --force should update PR body

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-backport.sh: guess component with --existing-pr

Fixes: https://tracker.ceph.com/issues/42656
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31398 into master

* refs/pull/31398/head:
pybind/mgr/pg_autoscaler: implement shutdown method

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #31064 into master

* refs/pull/31064/head:
test: Test balancer module commands
mgr: Improve balancer module status
mgr: Release GIL before calling OSDMap::calc_pg_upmaps()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30976 into master

* refs/pull/30976/head:
cmake: bump libceph-common SO version for compliance

Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #31438 into master

* refs/pull/31438/head:
mgr/ssh: implement blink_device_light

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31462 into master

* refs/pull/31462/head:
mgr/ssh: remove superfluous parameters

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31445 into master

* refs/pull/31445/head:
ceph-daemon: set container_image during bootstrap

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/ssh: implement blink_device_light

On my test box I get as far as

  NO_SUPPORT(153): SCSI VPD page 0x83 is not supported

because my hardware doesn't support this.  The same command works
on another box that does.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31443 into master

* refs/pull/31443/head:
ceph-daemon: fix 'version' field for legacy `ls`

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31458 into master

* refs/pull/31458/head:
ceph-daemon: Only run in the __main__ scope

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: set container_image during bootstrap

Set the new cluster's container_image to match the image we bootstrapped
with.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31463 into master

* refs/pull/31463/head:
ceph-daemon: switch default image

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31448 into master

* refs/pull/31448/head:
ceph-daemon: make 'ls' log less noisy

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: switch default image

Use this until octopus is ready.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: remove superfluous parameters

in _set_ssh_config and _clear_ssh_config

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/{dashboard,prometheus}: return FQDN instead of '0.0.0.0' (#31430)

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Use htmllint (#30998)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Set iSCSI disk WWN and LUN number from the UI (#31056)

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-daemon: Only run in the __main__ scope

That makes unit testing easier to setup because the code is not loaded
when ceph-daemon gets imported. Instead it is only loaded when
executed.
For that, the parser also moved to a function instead of being on
module level.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #31070 from sebastian-philipp/dashbaord-run-backend-zsh

mgr/dashboard: Fix zsh support in run-backend-api-tests.sh

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: fix e2e test failure for OSDs (#31451)

mgr/dashboard: fix e2e test failure for OSDs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: fix e2e test failure for OSDs

When clicking a selected row, the row becomes unselected and the detail
component is hidden. Rearrange related tests by selecting the row only
once.

Fixes: https://tracker.ceph.com/issues/42671
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #31350 from chenpan1012/rbd_testcase

cls/rbd: add snapshot limit UINT64_MAX test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-daemon: make 'ls' log less noisy

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31208 from weiyingze-git/master

test:add test_remove_snap2 interface to remove  snap when it's protected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-daemon: fix 'version' field for legacy `ls`

The `ls` command was reporting the git commit id instead of the ceph
version.

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31441 into master

* refs/pull/31441/head:
github/codeowners: Add orchestrator team

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31437 into master

* refs/pull/31437/head:
ceph-daemon: Remove data dir during adopt

Reviewed-by: Sage Weil <sage@redhat.com>

github/codeowners: Add orchestrator team

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #31440 into master

* refs/pull/31440/head:
qa/standalone/test_ceph_daemon: Make container images configurable
qa/standalone/test_ceph_daemon: Allow running from root dir
qa/standalone/test_ceph_daemon: Fix hang when CEPH_DAEMON is not set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31439 into master

* refs/pull/31439/head:
ceph-daemon: fix traceback during `ls` command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31385 into master

* refs/pull/31385/head:
mgr/ssh: add 'osd rm'
mgr/ssh: keep inventory and service cache keys in sync
mgr/orchestrator_cli: fix 'service ls' table format
mgr/ssh: invalidate service state on create, remove, and service action
mgr/orchestrator: add invalidate() to OutdatableDictMixin
mgr/ssh: cache services
mgr/ssh: drop specialized _remove_{mds,mgr,rgw}
mgr/ssh: change inventory_cache_timeout to seconds (not minutes)
mgr/orchestrator: make wait poll every 1s (not 5s)

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/standalone/test_ceph_daemon: Make container images configurable

Instead of hardcoding the images, make them configureable via
environment variables.
That way, downstream can use the script with custom images.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: fix traceback during `ls` command

fixup for 3a1ac3ff6ac2ef6e3e8c361c0c55562b732b9aac

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/standalone/test_ceph_daemon: Allow running from root dir

Allow running the script from root directory via:

./qa/standalone/test_ceph_daemon.sh

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

qa/standalone/test_ceph_daemon: Fix hang when CEPH_DAEMON is not set

When running test_ceph_daemon.sh from the root dir and not setting
$CEPH_DAEMON manually, the call hangs at:

$ ./qa/standalone/test_ceph_daemon.sh
[...]
+ for p in $PYTHONS
+ echo '=== re-running with python3 ==='
=== re-running with python3 ===
++ which python3
+ ln -s /usr/bin/python3 /tmp/tmp.6hneCsNMio/python
+ echo '#!/tmp/tmp.6hneCsNMio/python'
+ cat

Check that there is a ceph-daemon found before continue.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: Remove data dir during adopt

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: add missing test_orchestrator suite (#31198)

mgr/dashboard: add missing test_orchestrator suite

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31422 into master

* refs/pull/31422/head:
mgr/MgrClient: fix open condition fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/ssh: add 'osd rm'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: keep inventory and service cache keys in sync

Our host list is implicitly defined as the set of keys for the inventory
(and service) caches.  Ensure they are in sync by unioning and inserting
any missing items.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: fix 'service ls' table format

Make this match the rest of Ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: invalidate service state on create, remove, and service action

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add invalidate() to OutdatableDictMixin

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: cache services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: drop specialized _remove_{mds,mgr,rgw}

_remove_daemon is sufficient

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: change inventory_cache_timeout to seconds (not minutes)

The descriptio didn't match the implementation (which took minutes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: make wait poll every 1s (not 5s)

More responsive!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/prometheus: return FQDN for default server_addr

Instead of returning '0.0.0.0' the prometheus module returns the FQDN of
the host, as it is the case for '::'.

Fixes: https://tracker.ceph.com/issues/42664
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: return FQDN for default server_addr

Instead of returning '0.0.0.0' the dashboard returns the FQDN of the
host, as it is the case for '::'.

Fixes: https://tracker.ceph.com/issues/42664
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: Use htmllint

This package will replace htmlhint.

Some rules from the previous installed packages were not working that well.
This new package has more rules and seems to work better.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add support for device management (#30759)

mgr/dashboard: Add support for device management

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Searchable objects for table (#31199)

mgr/dashboard: Searchable objects for table

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26768 from sebastian-philipp/upstream-pull-26684

mgr/orchestrator: device lights

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31409 into master

* refs/pull/31409/head:
mds: Fix MDCache.h reorder compiler warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31401 into master

* refs/pull/31401/head:
mds: clarify comment

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #31420 into master

* refs/pull/31420/head:
ceph-daemon: do not screw up line splitting
mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: do not screw up line splitting

When we log stdout and stderr, we were artificially adding line splits
due to reading 1024 bytes at a time and assuming it was aligned with a
newline.

Fix by keeping partial lines buffered and only log them once they are
complete (or the call finishes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: fix open condition fix

Fix fc60989bf7a72c35b8f6b8fec2407b3080ad9bbd, which reversed this
condition.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: disable pg_autoscale for new pools in e2e tests (#31396)

mgr/dashboard: disable pg_autoscale for new pools in e2e tests

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31365 into master

* refs/pull/31365/head:
ceph-daemon: make rm-daemon reset-failed
ceph-daemon: make rm-cluster handle failed unit cleanup

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Fix error on ceph-iscsi version pre controls_limits

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Set iSCSI disk WWN and LUN number from the UI

Fixes: https://tracker.ceph.com/issues/41749
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #31303 into master

* refs/pull/31303/head:
ceph-daemon: make systemd unit descriptions Capitalized
ceph-daemon: handle daemon names with multiple .'s
mgr/ssh: rgw: ensure map rgws are mapped to the zone
mgr/ssh: use . as delimiter between service group and id
mgr/ssh: implement 'rgw update'
mgr/orchestrator_cli: add 'rgw update' CLI command
mgr/ssh: add 'rgw rm'
mgr/ssh: implement 'rgw add'
ceph-daemon: handle 'rgw' type daemons
mgr/orchestrator_cli: rgw add: optionally take count and node names

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: make systemd unit descriptions Capitalized

Same style as other units.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: handle daemon names with multiple .'s

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: rgw: ensure map rgws are mapped to the zone

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: use . as delimiter between service group and id

This aligns with . as an entity delimiter in the ceph config.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'rgw update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: add 'rgw update' CLI command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'rgw rm'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'rgw add'

Note that this doesn't correctly confine a daemon to a zone.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: handle 'rgw' type daemons

- weird data dir: /var/lib/ceph/radosgw/ceph-rgw.$id (not
  /var/lib/ceph/rgw/ceph-$id)
- weird auth name: client.rgw.$id (not rgw.$id)
- weird entrypoint: /usr/bin/radosgw (not /usr/bin/ceph-rgw)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rgw add: optionally take count and node names

This avoids the need for JSON in the common case.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31327 into master

* refs/pull/31327/head:
mon/ConfigMap: search nested sections

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31412 into master

* refs/pull/31412/head:
osd: send smart asok result to stdout, not stderr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31363 from cbodley/wip-qa-rgw-ec-is-fast-again

qa/rgw: remove whitelist for SLOW_OPS against ec pools

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

osd: send smart asok result to stdout, not stderr

Uncross the streams!

Reported-by: Patrick Seidensal <pseidensal@suse.com>
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31321 into master

* refs/pull/31321/head:
ceph-daemon: do not relabel system directories

Merge PR #31342 into master

* refs/pull/31342/head:
qa/standalone/test_ceph_daemon: fix multi-version python test

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/orchestrator: Improve ceph CLI for blink lights

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mds: Fix MDCache.h reorder compiler warnings

Fixes: https://tracker.ceph.com/issues/42642
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #30598 from smithfarm/wip-doc-releases-toc

doc/releases: access main releases page from top-level TOC

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #31291 from liewegas/bug-42578

mgr/ActivePyModules: behave if a module queries a devid that does not exist

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31230 from aclamk/fix-rados-put-truncate

tools/rados: prevent 'put' operation from recreating object when --offset=0

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: check for DEVICE_{IDENT|FAULT}_ON

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Add interface and tests to device lights

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: 'ceph device {fault,ident}-{on,off}', 'ceph device ls-lights'

Add generic CLI commands to turn device lights on and off.

These will call out to orchestrator stubs, once it has the appropriate hook
implemented, which will in turn call out to salt or rook or ssh or whatever
to actually turn a light on or off.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31366 from smithfarm/wip-cbs-4

ceph-backport.sh: implement interactive setup routine and new options

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix iSCSI Target's validation (#30894)

mgr/dashboard: Fix iSCSI Target's validation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Fix OSD's safe_to_destroy method (#30499)

mgr/dashboard: Fix OSD's safe_to_destroy method

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: disable pg_autoscale for new pools in e2e tests

The pg_autoscaler might interfere the editing pool e2e test, disable the
global config to avoid this problem.

Fixes: https://tracker.ceph.com/issues/42608
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mds: clarify comment

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>