rpm,deb: package always-enabled plugins in a separated package

ceph-mgr depends on it in the sense that these plugins are
a hard dependency of ceph-mgr. like cmake depends on cmake-data,
even if the later cannot function on its own without the former.

the reason why we need a separated package is that mgr modules
are written in python, and should be arch independent. this change
helps the downstream to understand this and help to minimize the
divergence between upstream and downstream, because distros like
debian and its derivatives encourage splitting architecture independent
data into its own package.
see
https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#architecture-independent-data

also, the python related runtime dependencies are also moved into
ceph-mgr-modules-core.

if a python dependency is shared by another ceph-mgr-* package and
ceph-mgr-modules-core. it's only added to ceph-mgr-modules-core for
simplicity and consistency. as the name of dependency might vary
on different distros, so it's tedious and error-prone to repeat
these conditons and checks.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: update to adapt to el7

make sure ceph-mgr installs on el7

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33394 into master

* refs/pull/33394/head:
mgr/cephadm: trigger device refresh after osd creation
mgr/orch,cephadm: implement 'device zap'
mgr/cephadm: remove inventory_cache
mgr/cephadm: reimplement 'device ls' in terms of cache
mgr/cephadm: fix invalidate helpers
mgr/cephadm: scrape and cache devices, too
mgr/cephadm: rename DaemonCache methods etc to generalize

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33400 into master

* refs/pull/33400/head:
mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33402 into master

* refs/pull/33402/head:
mgr/pg_autoscaler: fix division by zero

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #33352 into master

* refs/pull/33352/head:
ceph.in: catch KeyboardInterrupt from outer code
ceph.in: fix second exception on ^C

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33374 into master

* refs/pull/33374/head:
cephadm: remove distutils dependency
mgr/cephadm: add prepare-host command
cephadm: prepare-host during bootstrap
cephadm: add prepare-host command
cephadm: include docker.io in image name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: trigger device refresh after osd creation

Presumably the new OSD(s) consumed one or more devices.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch,cephadm: implement 'device zap'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove inventory_cache

No longer used.

Clean up _get_hosts now that there is only one user.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: reimplement 'device ls' in terms of cache

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix invalidate helpers

These should wake up the serve() thread.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: scrape and cache devices, too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: rename DaemonCache methods etc to generalize

The HostCache is going to store both daemon and device state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32677 into master

* refs/pull/32677/head:
build/ops: make/install osd_support package
mgr/osd_support: move docs to docs/mgr/osd_support.rst
mgr/osd_support: new module for osd utility calls

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33123 into master

* refs/pull/33123/head:
qa/tasks/cephadm: deploy node-exporter
mgr/orch: deploy node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33359 into master

* refs/pull/33359/head:
mgr/dashboard/tests/test_orchestrator: disable service test
mgr/cephadm: fix race between check_hosts and host rm
mgr/volumes: fix remove_mds call
mgr/rook: hackly describe_service -> list_daemons
mgr/tests/test_orchestrator: fix ServiceDescription test
mgr/test_orchestrator: describe_service -> list_daemons
mgr/orch: resurrect ServiceDescription, 'orch ls'
qa/tasks/mgr/dashboard/test_orchestrator: disable services test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard/tests/test_orchestrator: disable service test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix race between check_hosts and host rm

Fixes: https://tracker.ceph.com/issues/44188
Signed-off-by: Sage Weil <sage@redhat.com>

ceph.in: catch KeyboardInterrupt from outer code

This avoids getting random tracebacks when you control-C early in the
process.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: fix remove_mds call

Fixes: https://tracker.ceph.com/issues/44176
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: hackly describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: fix ServiceDescription test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: resurrect ServiceDescription, 'orch ls'

This is a meta-service description that maps to multiple daemons.  Remove
the old useless properties, and add in size and running counts.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_orchestrator: disable services test

We need to either fill out the list_daemons APIs in the dashboard and test
that, or redesign and reimplement the services abstractions in the
orchestration layer.  Until then, disable this test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33333 from sebastian-philipp/cephadm-mon-apply-host-no-network

mgr/cephadm: mon: Don't show traceback for user errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33310 from sebastian-philipp/cephadm-add-host-exception

mgr/cephadm: Convert HostNotFound to OrchestratorError

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33387 into master

* refs/pull/33387/head:
spec: ceph-mgr: add missing python3-pyyaml

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: deploy node-exporter

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: deploy node-exporter

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: remove distutils dependency

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: add prepare-host command

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: prepare-host during bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: add prepare-host command

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include docker.io in image name

This removes a configuration dependency on the container environment.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32895 from tspmelo/wip-sidebar-notification-v2

mgr/dashboard: Improve Notification sidebar

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33410 from capri1989/docs-fix

doc/mgr/orchestrator.rst: updated current implementation status

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33249 into master

* refs/pull/33249/head:
mgr/cephadm: move cutoff calc inside helper
mgr/orch: clean up service_action, remove_service
mgr/cephadm: move into DaemonCache class
mgr/cephadm: fix tests
mgr/cephadm: implete hacky 'refresh=True' path
mgr/cephadm: raise health alert when scrape fails
mgr/cephadm: persist cached daemon state
mgr/orch: serialize DaemonDescription last_refresh
mgr/cephadm: move _get_daemons() impl into list_daemons
mgr/cephadm: replace remaining _get_daemons() with daemon cache
mgr/cephadm: use daemon map for service|daemon removal
mgr/cephadm: avoid _get_daemons for service|daemon actions
mgr/cephadm: replace daemon_cache with an explicit set of dicts
mgr/cephadm: move DaemonDescription construction into helper

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33408 from tchaikov/wip-crimson-editor-prop

crimson: add editor properties header

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/mgr/orchestrator.rst updated current implementation status

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #33060 from votdev/issue_39252_rgw_beast

mgr/dashboard: RGW port autodetection does not support "Beast" RGW frontend

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

build/ops: make/install osd_support package

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: move docs to docs/mgr/osd_support.rst

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: new module for osd utility calls

it lets you start/stop and monitor osd draining processes

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33395 into master

* refs/pull/33395/head:
cephadm: error out on filestore OSDs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33396 from liewegas/fix-cephadm-upgrade-test

qa/suites/rados/cephadm/upgrade: fix initial version

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32949 into master

* refs/pull/32949/head:
cephadm: separate out require files in config-json
cephadm: add alertmanager deployment feature

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

spec: ceph-mgr: add missing python3-pyyaml

Remove stray req from the -k8events package, since the base ceph-mgr now
requires it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33370 into master

* refs/pull/33370/head:
mgr/orchestrator_cli: sort host list

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/pg_autoscaler: fix division by zero

Fixes: https://tracker.ceph.com/issues/44186
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move cutoff calc inside helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: clean up service_action, remove_service

Standardize on service_name argument that looks like 'mgr', 'mds.fsname',
'mds', or some other daemon name prefix.  Avoid the ambiguously-named
service_type+service_name combinations.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move into DaemonCache class

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implete hacky 'refresh=True' path

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: raise health alert when scrape fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: persist cached daemon state

- load cached state on startup
- persist state after a scrape only
- scrape everything after startup

Note that we modify our in-memory cache when we add or remove a service
and then immediately trigger a scrape, but we do not invalidate the
persisted state, since it's simpler (and presumably a good idea) to
simply re-scrape everything after a mgr restart.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: serialize DaemonDescription last_refresh

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move _get_daemons() impl into list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace remaining _get_daemons() with daemon cache

New function _get_daemons_by_type() returns immediately with a result
from the cache.

Remove the with_daemons decorator.

Push daemon list fetch into _add_new_daemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use daemon map for service|daemon removal

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: avoid _get_daemons for service|daemon actions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace daemon_cache with an explicit set of dicts

- Cache DaemonDescription explicitly
- explicit timestamp for the host
- serve() scrapes inventory based on that timestamp

For the moment, persistence is broken, and --refresh is broken.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move DaemonDescription construction into helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33390 into master

* refs/pull/33390/head:
mds: remove dead get_commands code

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #33336 into master

* refs/pull/33336/head:
osd: fix racy accesses to OSD::osdmap.

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/ardos/cephadm/upgrade: fix initial version

Switch initial version to one with the renamed orchestrator module.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: error out on filestore OSDs

We do not currently support filestore OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: RGW port autodetection does not support "Beast" RGW frontend

* Improve regular expressions to support more configuration variations.
* Modify error message. It includes the config line to be parsed. This should help to debug errors much easier.
* If there are multiple (ssl_)ports/(ssl_)endpoints options, then the first found option will be returned.

Fixes: https://tracker.ceph.com/issues/39252
Signed-off-by: Volker Theile <vtheile@suse.com>

mds: remove dead get_commands code

Left over from #31255, d8c0bde04b88cb3ad96855bd0948ae10072c9da7.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33369 into master

* refs/pull/33369/head:
cephadm: check for both chrony service names

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33360 from liewegas/fix-prom

qa/suites/rados/cephadm/smoke: disable rgw role for now

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: sort host list

Signed-off-by: Sage Weil <sage@redhat.com>

crimson: add editor properties header

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add_host: Convert HostNotFound to OrchestratorError

Don't bother users with Python Tracebacks.

Fixes: https://tracker.ceph.com/issues/44026
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: mon: Don't show traceback for user errors

E.g. when complaining about a missing netowork.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: exception pickle: don't raise

Otherwiese the pickle exception hides the underlying root cause.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Fix logging of exceptions

`multiprocessing.pool.Pool#map_async` no longer has the exception
context, therefore calling `logging.exception` did not print the exception anymore.

Now, let's call `fail` while we have the exception context.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32817 from sebastian-philipp/rename-orchestrator_cli-orchestrator

pybind/mgr: Rename orchestrator_cli to orchestrator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33167 from tchaikov/wip-formatter-string_view

common,mgr,osd: pass string_view as "name"

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #33227 into master

* refs/pull/33227/head:
mds: remove unused CDir members

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33197 into master

* refs/pull/33197/head:
mount.ceph: fix incorrect options parsing

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #33180 into master

* refs/pull/33180/head:
mds: add scrub_info_t into mempool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33104 into master

* refs/pull/33104/head:
client: Fixes for missing consts SEEK_DATA and SEEK_HOLE on alpine linux

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33005 into master

* refs/pull/33005/head:
mds: fix 'can wrlock' check in Locker::acquire_locks()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32435 into master

* refs/pull/32435/head:
mds: Reorganize structure and class members in mdstypes header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33344 from athanatos/sjust/wip-errorator-handlers

errorator: improve general error handlers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31001 from rosinL/wip-set-radosgw-cpu-affinity

rgw/rgw_main: auto set radosgw's cpu affinity according to numa_node configuration

qa/suites/rados/cephadm/smoke: remove rgw

Fixes: https://tracker.ceph.com/issues/44168
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33206 from rhcs-dashboard/44075-rgw-user-system-field

mgr/dashboard: show correct RGW user 'system' info

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33125 from yaarith/telemetry-add-last-report-to-status

mgr/telemetry: add 'last_upload' to status

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33013 from s0nea/wip-dashboard-43912

mgr/dashboard: wait for PG unknown state to be cleared

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #33107 from votdev/cleanup_code

mgr/dashboard: Cleanup code

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cephadm: check for both chrony service names

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31901 from matthewoliver/rgw-admin-user-add-modify

 rgw: make radosgw-admin user create and modify distinct

Merge pull request #32632 from cyx1231st/rfc-seastar-test-socket-nevermove

crimson/net: configure seastar to accept on a fixed core

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add pyyaml to requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: rename orchestrator_cli -> orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>