mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Fixes: https://tracker.ceph.com/issues/44201
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #33394 into master

* refs/pull/33394/head:
mgr/cephadm: trigger device refresh after osd creation
mgr/orch,cephadm: implement 'device zap'
mgr/cephadm: remove inventory_cache
mgr/cephadm: reimplement 'device ls' in terms of cache
mgr/cephadm: fix invalidate helpers
mgr/cephadm: scrape and cache devices, too
mgr/cephadm: rename DaemonCache methods etc to generalize

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33400 into master

* refs/pull/33400/head:
mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33402 into master

* refs/pull/33402/head:
mgr/pg_autoscaler: fix division by zero

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33352 into master

* refs/pull/33352/head:
ceph.in: catch KeyboardInterrupt from outer code
ceph.in: fix second exception on ^C

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33374 into master

* refs/pull/33374/head:
cephadm: remove distutils dependency
mgr/cephadm: add prepare-host command
cephadm: prepare-host during bootstrap
cephadm: add prepare-host command
cephadm: include docker.io in image name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: trigger device refresh after osd creation

Presumably the new OSD(s) consumed one or more devices.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch,cephadm: implement 'device zap'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove inventory_cache

No longer used.

Clean up _get_hosts now that there is only one user.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: reimplement 'device ls' in terms of cache

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix invalidate helpers

These should wake up the serve() thread.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: scrape and cache devices, too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: rename DaemonCache methods etc to generalize

The HostCache is going to store both daemon and device state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32677 into master

* refs/pull/32677/head:
build/ops: make/install osd_support package
mgr/osd_support: move docs to docs/mgr/osd_support.rst
mgr/osd_support: new module for osd utility calls

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33123 into master

* refs/pull/33123/head:
qa/tasks/cephadm: deploy node-exporter
mgr/orch: deploy node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33359 into master

* refs/pull/33359/head:
mgr/dashboard/tests/test_orchestrator: disable service test
mgr/cephadm: fix race between check_hosts and host rm
mgr/volumes: fix remove_mds call
mgr/rook: hackly describe_service -> list_daemons
mgr/tests/test_orchestrator: fix ServiceDescription test
mgr/test_orchestrator: describe_service -> list_daemons
mgr/orch: resurrect ServiceDescription, 'orch ls'
qa/tasks/mgr/dashboard/test_orchestrator: disable services test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard/tests/test_orchestrator: disable service test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix race between check_hosts and host rm

Fixes: https://tracker.ceph.com/issues/44188
Signed-off-by: Sage Weil <sage@redhat.com>

ceph.in: catch KeyboardInterrupt from outer code

This avoids getting random tracebacks when you control-C early in the
process.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: fix remove_mds call

Fixes: https://tracker.ceph.com/issues/44176
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: hackly describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: fix ServiceDescription test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: resurrect ServiceDescription, 'orch ls'

This is a meta-service description that maps to multiple daemons.  Remove
the old useless properties, and add in size and running counts.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_orchestrator: disable services test

We need to either fill out the list_daemons APIs in the dashboard and test
that, or redesign and reimplement the services abstractions in the
orchestration layer.  Until then, disable this test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33333 from sebastian-philipp/cephadm-mon-apply-host-no-network

mgr/cephadm: mon: Don't show traceback for user errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33310 from sebastian-philipp/cephadm-add-host-exception

mgr/cephadm: Convert HostNotFound to OrchestratorError

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33387 into master

* refs/pull/33387/head:
spec: ceph-mgr: add missing python3-pyyaml

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: deploy node-exporter

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: deploy node-exporter

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: remove distutils dependency

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: add prepare-host command

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: prepare-host during bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add prepare-host command

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include docker.io in image name

This removes a configuration dependency on the container environment.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32895 from tspmelo/wip-sidebar-notification-v2

mgr/dashboard: Improve Notification sidebar

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33410 from capri1989/docs-fix

doc/mgr/orchestrator.rst: updated current implementation status

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33249 into master

* refs/pull/33249/head:
mgr/cephadm: move cutoff calc inside helper
mgr/orch: clean up service_action, remove_service
mgr/cephadm: move into DaemonCache class
mgr/cephadm: fix tests
mgr/cephadm: implete hacky 'refresh=True' path
mgr/cephadm: raise health alert when scrape fails
mgr/cephadm: persist cached daemon state
mgr/orch: serialize DaemonDescription last_refresh
mgr/cephadm: move _get_daemons() impl into list_daemons
mgr/cephadm: replace remaining _get_daemons() with daemon cache
mgr/cephadm: use daemon map for service|daemon removal
mgr/cephadm: avoid _get_daemons for service|daemon actions
mgr/cephadm: replace daemon_cache with an explicit set of dicts
mgr/cephadm: move DaemonDescription construction into helper

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33408 from tchaikov/wip-crimson-editor-prop

crimson: add editor properties header

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/mgr/orchestrator.rst updated current implementation status

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #33060 from votdev/issue_39252_rgw_beast

mgr/dashboard: RGW port autodetection does not support "Beast" RGW frontend

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

build/ops: make/install osd_support package

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: move docs to docs/mgr/osd_support.rst

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: new module for osd utility calls

it lets you start/stop and monitor osd draining processes

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33395 into master

* refs/pull/33395/head:
cephadm: error out on filestore OSDs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33396 from liewegas/fix-cephadm-upgrade-test

qa/suites/rados/cephadm/upgrade: fix initial version

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32949 into master

* refs/pull/32949/head:
cephadm: separate out require files in config-json
cephadm: add alertmanager deployment feature

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

spec: ceph-mgr: add missing python3-pyyaml

Remove stray req from the -k8events package, since the base ceph-mgr now
requires it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33370 into master

* refs/pull/33370/head:
mgr/orchestrator_cli: sort host list

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/pg_autoscaler: fix division by zero

Fixes: https://tracker.ceph.com/issues/44186
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move cutoff calc inside helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: clean up service_action, remove_service

Standardize on service_name argument that looks like 'mgr', 'mds.fsname',
'mds', or some other daemon name prefix.  Avoid the ambiguously-named
service_type+service_name combinations.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move into DaemonCache class

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implete hacky 'refresh=True' path

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: raise health alert when scrape fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: persist cached daemon state

- load cached state on startup
- persist state after a scrape only
- scrape everything after startup

Note that we modify our in-memory cache when we add or remove a service
and then immediately trigger a scrape, but we do not invalidate the
persisted state, since it's simpler (and presumably a good idea) to
simply re-scrape everything after a mgr restart.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: serialize DaemonDescription last_refresh

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move _get_daemons() impl into list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace remaining _get_daemons() with daemon cache

New function _get_daemons_by_type() returns immediately with a result
from the cache.

Remove the with_daemons decorator.

Push daemon list fetch into _add_new_daemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use daemon map for service|daemon removal

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: avoid _get_daemons for service|daemon actions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace daemon_cache with an explicit set of dicts

- Cache DaemonDescription explicitly
- explicit timestamp for the host
- serve() scrapes inventory based on that timestamp

For the moment, persistence is broken, and --refresh is broken.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move DaemonDescription construction into helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33390 into master

* refs/pull/33390/head:
mds: remove dead get_commands code

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #33336 into master

* refs/pull/33336/head:
osd: fix racy accesses to OSD::osdmap.

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/ardos/cephadm/upgrade: fix initial version

Switch initial version to one with the renamed orchestrator module.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: error out on filestore OSDs

We do not currently support filestore OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: RGW port autodetection does not support "Beast" RGW frontend

* Improve regular expressions to support more configuration variations.
* Modify error message. It includes the config line to be parsed. This should help to debug errors much easier.
* If there are multiple (ssl_)ports/(ssl_)endpoints options, then the first found option will be returned.

Fixes: https://tracker.ceph.com/issues/39252
Signed-off-by: Volker Theile <vtheile@suse.com>

mds: remove dead get_commands code

Left over from #31255, d8c0bde04b88cb3ad96855bd0948ae10072c9da7.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33369 into master

* refs/pull/33369/head:
cephadm: check for both chrony service names

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33360 from liewegas/fix-prom

qa/suites/rados/cephadm/smoke: disable rgw role for now

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: sort host list

Signed-off-by: Sage Weil <sage@redhat.com>

crimson: add editor properties header

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add_host: Convert HostNotFound to OrchestratorError

Don't bother users with Python Tracebacks.

Fixes: https://tracker.ceph.com/issues/44026
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: mon: Don't show traceback for user errors

E.g. when complaining about a missing netowork.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: exception pickle: don't raise

Otherwiese the pickle exception hides the underlying root cause.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Fix logging of exceptions

`multiprocessing.pool.Pool#map_async` no longer has the exception
context, therefore calling `logging.exception` did not print the exception anymore.

Now, let's call `fail` while we have the exception context.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32817 from sebastian-philipp/rename-orchestrator_cli-orchestrator

pybind/mgr: Rename orchestrator_cli to orchestrator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33167 from tchaikov/wip-formatter-string_view

common,mgr,osd: pass string_view as "name"

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #33227 into master

* refs/pull/33227/head:
mds: remove unused CDir members

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33197 into master

* refs/pull/33197/head:
mount.ceph: fix incorrect options parsing

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #33180 into master

* refs/pull/33180/head:
mds: add scrub_info_t into mempool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33104 into master

* refs/pull/33104/head:
client: Fixes for missing consts SEEK_DATA and SEEK_HOLE on alpine linux

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33005 into master

* refs/pull/33005/head:
mds: fix 'can wrlock' check in Locker::acquire_locks()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32435 into master

* refs/pull/32435/head:
mds: Reorganize structure and class members in mdstypes header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33344 from athanatos/sjust/wip-errorator-handlers

errorator: improve general error handlers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31001 from rosinL/wip-set-radosgw-cpu-affinity

rgw/rgw_main: auto set radosgw's cpu affinity according to numa_node configuration

qa/suites/rados/cephadm/smoke: remove rgw

Fixes: https://tracker.ceph.com/issues/44168
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33206 from rhcs-dashboard/44075-rgw-user-system-field

mgr/dashboard: show correct RGW user 'system' info

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33125 from yaarith/telemetry-add-last-report-to-status

mgr/telemetry: add 'last_upload' to status

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33013 from s0nea/wip-dashboard-43912

mgr/dashboard: wait for PG unknown state to be cleared

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #33107 from votdev/cleanup_code

mgr/dashboard: Cleanup code

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cephadm: check for both chrony service names

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31901 from matthewoliver/rgw-admin-user-add-modify

 rgw: make radosgw-admin user create and modify distinct

Merge pull request #32632 from cyx1231st/rfc-seastar-test-socket-nevermove

crimson/net: configure seastar to accept on a fixed core

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add pyyaml to requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: rename orchestrator_cli -> orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: add prettytable to admin/doc-requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>