mgr/cephadm: trigger device refresh after osd creation

Presumably the new OSD(s) consumed one or more devices.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch,cephadm: implement 'device zap'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove inventory_cache

No longer used.

Clean up _get_hosts now that there is only one user.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: reimplement 'device ls' in terms of cache

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix invalidate helpers

These should wake up the serve() thread.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: scrape and cache devices, too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: rename DaemonCache methods etc to generalize

The HostCache is going to store both daemon and device state.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32677 into master

* refs/pull/32677/head:
build/ops: make/install osd_support package
mgr/osd_support: move docs to docs/mgr/osd_support.rst
mgr/osd_support: new module for osd utility calls

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33123 into master

* refs/pull/33123/head:
qa/tasks/cephadm: deploy node-exporter
mgr/orch: deploy node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33359 into master

* refs/pull/33359/head:
mgr/dashboard/tests/test_orchestrator: disable service test
mgr/cephadm: fix race between check_hosts and host rm
mgr/volumes: fix remove_mds call
mgr/rook: hackly describe_service -> list_daemons
mgr/tests/test_orchestrator: fix ServiceDescription test
mgr/test_orchestrator: describe_service -> list_daemons
mgr/orch: resurrect ServiceDescription, 'orch ls'
qa/tasks/mgr/dashboard/test_orchestrator: disable services test

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard/tests/test_orchestrator: disable service test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix race between check_hosts and host rm

Fixes: https://tracker.ceph.com/issues/44188
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: fix remove_mds call

Fixes: https://tracker.ceph.com/issues/44176
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: hackly describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: fix ServiceDescription test

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: describe_service -> list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: resurrect ServiceDescription, 'orch ls'

This is a meta-service description that maps to multiple daemons.  Remove
the old useless properties, and add in size and running counts.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/dashboard/test_orchestrator: disable services test

We need to either fill out the list_daemons APIs in the dashboard and test
that, or redesign and reimplement the services abstractions in the
orchestration layer.  Until then, disable this test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33333 from sebastian-philipp/cephadm-mon-apply-host-no-network

mgr/cephadm: mon: Don't show traceback for user errors

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33310 from sebastian-philipp/cephadm-add-host-exception

mgr/cephadm: Convert HostNotFound to OrchestratorError

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33387 into master

* refs/pull/33387/head:
spec: ceph-mgr: add missing python3-pyyaml

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: deploy node-exporter

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: deploy node-exporter

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #32895 from tspmelo/wip-sidebar-notification-v2

mgr/dashboard: Improve Notification sidebar

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33410 from capri1989/docs-fix

doc/mgr/orchestrator.rst: updated current implementation status

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33249 into master

* refs/pull/33249/head:
mgr/cephadm: move cutoff calc inside helper
mgr/orch: clean up service_action, remove_service
mgr/cephadm: move into DaemonCache class
mgr/cephadm: fix tests
mgr/cephadm: implete hacky 'refresh=True' path
mgr/cephadm: raise health alert when scrape fails
mgr/cephadm: persist cached daemon state
mgr/orch: serialize DaemonDescription last_refresh
mgr/cephadm: move _get_daemons() impl into list_daemons
mgr/cephadm: replace remaining _get_daemons() with daemon cache
mgr/cephadm: use daemon map for service|daemon removal
mgr/cephadm: avoid _get_daemons for service|daemon actions
mgr/cephadm: replace daemon_cache with an explicit set of dicts
mgr/cephadm: move DaemonDescription construction into helper

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33408 from tchaikov/wip-crimson-editor-prop

crimson: add editor properties header

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/mgr/orchestrator.rst updated current implementation status

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #33060 from votdev/issue_39252_rgw_beast

mgr/dashboard: RGW port autodetection does not support "Beast" RGW frontend

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

build/ops: make/install osd_support package

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: move docs to docs/mgr/osd_support.rst

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/osd_support: new module for osd utility calls

it lets you start/stop and monitor osd draining processes

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33395 into master

* refs/pull/33395/head:
cephadm: error out on filestore OSDs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33396 from liewegas/fix-cephadm-upgrade-test

qa/suites/rados/cephadm/upgrade: fix initial version

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32949 into master

* refs/pull/32949/head:
cephadm: separate out require files in config-json
cephadm: add alertmanager deployment feature

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

spec: ceph-mgr: add missing python3-pyyaml

Remove stray req from the -k8events package, since the base ceph-mgr now
requires it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33370 into master

* refs/pull/33370/head:
mgr/orchestrator_cli: sort host list

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: move cutoff calc inside helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: clean up service_action, remove_service

Standardize on service_name argument that looks like 'mgr', 'mds.fsname',
'mds', or some other daemon name prefix.  Avoid the ambiguously-named
service_type+service_name combinations.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move into DaemonCache class

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implete hacky 'refresh=True' path

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: raise health alert when scrape fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: persist cached daemon state

- load cached state on startup
- persist state after a scrape only
- scrape everything after startup

Note that we modify our in-memory cache when we add or remove a service
and then immediately trigger a scrape, but we do not invalidate the
persisted state, since it's simpler (and presumably a good idea) to
simply re-scrape everything after a mgr restart.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: serialize DaemonDescription last_refresh

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move _get_daemons() impl into list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace remaining _get_daemons() with daemon cache

New function _get_daemons_by_type() returns immediately with a result
from the cache.

Remove the with_daemons decorator.

Push daemon list fetch into _add_new_daemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use daemon map for service|daemon removal

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: avoid _get_daemons for service|daemon actions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace daemon_cache with an explicit set of dicts

- Cache DaemonDescription explicitly
- explicit timestamp for the host
- serve() scrapes inventory based on that timestamp

For the moment, persistence is broken, and --refresh is broken.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move DaemonDescription construction into helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33390 into master

* refs/pull/33390/head:
mds: remove dead get_commands code

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #33336 into master

* refs/pull/33336/head:
osd: fix racy accesses to OSD::osdmap.

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/ardos/cephadm/upgrade: fix initial version

Switch initial version to one with the renamed orchestrator module.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: error out on filestore OSDs

We do not currently support filestore OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: RGW port autodetection does not support "Beast" RGW frontend

* Improve regular expressions to support more configuration variations.
* Modify error message. It includes the config line to be parsed. This should help to debug errors much easier.
* If there are multiple (ssl_)ports/(ssl_)endpoints options, then the first found option will be returned.

Fixes: https://tracker.ceph.com/issues/39252
Signed-off-by: Volker Theile <vtheile@suse.com>

mds: remove dead get_commands code

Left over from #31255, d8c0bde04b88cb3ad96855bd0948ae10072c9da7.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33369 into master

* refs/pull/33369/head:
cephadm: check for both chrony service names

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33360 from liewegas/fix-prom

qa/suites/rados/cephadm/smoke: disable rgw role for now

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: sort host list

Signed-off-by: Sage Weil <sage@redhat.com>

crimson: add editor properties header

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: add_host: Convert HostNotFound to OrchestratorError

Don't bother users with Python Tracebacks.

Fixes: https://tracker.ceph.com/issues/44026
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: mon: Don't show traceback for user errors

E.g. when complaining about a missing netowork.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: exception pickle: don't raise

Otherwiese the pickle exception hides the underlying root cause.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Fix logging of exceptions

`multiprocessing.pool.Pool#map_async` no longer has the exception
context, therefore calling `logging.exception` did not print the exception anymore.

Now, let's call `fail` while we have the exception context.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32817 from sebastian-philipp/rename-orchestrator_cli-orchestrator

pybind/mgr: Rename orchestrator_cli to orchestrator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33167 from tchaikov/wip-formatter-string_view

common,mgr,osd: pass string_view as "name"

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #33227 into master

* refs/pull/33227/head:
mds: remove unused CDir members

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33197 into master

* refs/pull/33197/head:
mount.ceph: fix incorrect options parsing

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #33180 into master

* refs/pull/33180/head:
mds: add scrub_info_t into mempool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33104 into master

* refs/pull/33104/head:
client: Fixes for missing consts SEEK_DATA and SEEK_HOLE on alpine linux

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33005 into master

* refs/pull/33005/head:
mds: fix 'can wrlock' check in Locker::acquire_locks()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32435 into master

* refs/pull/32435/head:
mds: Reorganize structure and class members in mdstypes header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33344 from athanatos/sjust/wip-errorator-handlers

errorator: improve general error handlers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31001 from rosinL/wip-set-radosgw-cpu-affinity

rgw/rgw_main: auto set radosgw's cpu affinity according to numa_node configuration

qa/suites/rados/cephadm/smoke: remove rgw

Fixes: https://tracker.ceph.com/issues/44168
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33206 from rhcs-dashboard/44075-rgw-user-system-field

mgr/dashboard: show correct RGW user 'system' info

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33125 from yaarith/telemetry-add-last-report-to-status

mgr/telemetry: add 'last_upload' to status

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33013 from s0nea/wip-dashboard-43912

mgr/dashboard: wait for PG unknown state to be cleared

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #33107 from votdev/cleanup_code

mgr/dashboard: Cleanup code

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cephadm: check for both chrony service names

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31901 from matthewoliver/rgw-admin-user-add-modify

 rgw: make radosgw-admin user create and modify distinct

Merge pull request #32632 from cyx1231st/rfc-seastar-test-socket-nevermove

crimson/net: configure seastar to accept on a fixed core

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add pyyaml to requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: rename orchestrator_cli -> orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: add prettytable to admin/doc-requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mon: always_on_modules: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

debian,spec: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

.github: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: add prettytable to requirmenets.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix doc urls to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Use CLICommand, except it's global variable

`CLICommand.COMMANDS` is a global varialbe that prevents
anyone from importing other modules, as the `COMMANS` are then
merged together. Let's use a meta class instead of a global variable.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: orchestrator_cli rename: fix imports

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestsrator: make parse_host_specs a classmethod

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: rename to mgr/orchestrator

* Move `mgr/orchestrator.py` to `orchestrator/_interface.py`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33361 from bk201/wip-44164

ceph.spec.in: fix python coverage dependency for non-rhel distros

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: remove duplicated error codes and conditions

The duplicated error codes and conditions were originally introduced to
match connection errors with both system category (thrown by seastar)
and generic category (thrown by standard library).  Since error_code
with system category can be matched by error_condition with generic
category (see std::errc and
system_error_category::default_error_condition(int)), our duplicated
counterparts are not needed actually.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #33358 from howard0su/nvme1

os/bluestore: allocate Task on stack

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix python coverage dependency for non-rhel distros

The coverage package under openSUSE (and other distros) are named as
python{major_version}-coverage (without minor version).

Fixes: https://tracker.ceph.com/issues/44164
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

rgw: make radosgw-admin user create and modify distinct

Currently if you run 'radosgw-admin user create ..' when the user
already exists and you happen to specify, at least,  '--uid' and
'--display-name' that match the existing user, radowgw-admin will
actaully go modify the existing user.

This behaviour is a little confusing, hence the bug this patch is
fixing. This patch instead simplifies the tool to make
'create' create and 'modify' modify.
Meaning when you go 'create' a user that already exists, you'll get an
error, as expected. If you want to modify a user, you actually have to
use 'modify'.

For exapmle, now:

  $ radosgw-admin user create --uid="test-user"  --display-name="test user"
  could not create user: unable to create user, user: test-user exists

Signed-off-by: Matthew Oliver <moliver@suse.com>
Fixes: https://tracker.ceph.com/issues/38619