mgr/cephadm: clean up log messages a bit

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: force remove daemons when removing service

The rm CLI checks for --force based on type, so it isn't passed down
through the implementations.  So, pass it through.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not error out on service rm|action if no daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: allow 'orch rm mon|mgr --force'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33704 into master

* refs/pull/33704/head:
mgr/cephadm: avoid racing updates to spec dict
mgr/cephadm: make _do_upgrade work synchronously
mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test
mgr/cephadm: make _apply_service move services
mgr/cephadm: log when we create, destroy daemons
mgr/cephadm: drop async_map_completion decorators from create funcs
mgr/cephadm: _create_osd -> _create_osds
mgr/cephadm: _remove_daemon -> _remove_daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: avoid racing updates to spec dict

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _do_upgrade work synchronously

It mostly was, but it was futzing with completions when it didn't need to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test

This was trying to show that we won't put 2 RGWs from different zones
on the same host.  In reality, the test was only passing because it was
specifying an invalid placement spec (count=3 in a 2-host cluster).

It's true that we shouldn't do that because they won't be able to bind,
but the HostAssignment code doesn't handle this case currently.

See https://tracker.ceph.com/issues/44397

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _apply_service move services

This transitions us to a synchronous mode of operation.

Fixes: https://tracker.ceph.com/issues/44167
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: log when we create, destroy daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop async_map_completion decorators from create funcs

So we can call them synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _create_osd -> _create_osds

So we could do a single daemon synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _remove_daemon -> _remove_daemons

So we can remove a single daemon synchronously, if we choose.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33565 into master

* refs/pull/33565/head:
os/bluestore: log allocation stats on a daily basis.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33539 from cbodley/wip-rgw-doc-reshard

doc/rgw: add design doc for multisite resharding

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33698 from rhcs-dashboard/vstart-set-prometheus-port-for-each-mgr

vstart.sh: set prometheus port for each mgr.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33047 into master

* refs/pull/33047/head:
qa/rgw: exercise DeleteRange in test_bucket_index_log_trim

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge PR #33287 into master

* refs/pull/33287/head:
rgw: move frontends initial init to after global_init()
rgw: ssl: don't try to init certificate if not needed
rgw: frontend: add rgw_frontend_defaults configurable
rgw: beast ssl: enable use of meta variable for cert config
rgw: beast ssl: improve output
rgw: beast frontend: handle default ssl configurables
rgw: update docs about ssl config through config-key
rgw: allow beast ssl frontend cert config via mon config-key
rgw: create config-key svc
rgw: svc/rados: new mon_command call

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33612 from votdev/issue_44322_rgw_user_max_buckets

mgr/dashboard: Not able to restrict bucket creation for new user

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33721 from smithfarm/wip-nautilus_14_2_8

doc/releases: add 14.2.8 to release timeline

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc/rgw: add design doc for multisite resharding

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33705 into master

* refs/pull/33705/head:
qa/suites/upgrade/nautilus-x/parallel: restart mgr.x before mons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33723 from tchaikov/wip-seastar-on-demand

run-make-check.sh: WITH_SEASTAR on demand

Reviewed-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: WITH_SEASTAR on demand

don't compile WITH_SEASTAR=ON unless required

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: add 14.2.8 to release timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33670 from theanalyst/doc/releases/v14.2.8

doc: releases: v14.2.8 release notes

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>

Merge pull request #33709 from tchaikov/wip-ceph-py-py3-rados

qa/tasks: py3 compat (tasks exercised by rados suites)

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: set prometheus port for each mgr.

Fixes: https://tracker.ceph.com/issues/44417
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #33690 from rhcs-dashboard/44237-remove-config-opt-read-from-system-roles

mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33699 from liewegas/fix-44078

common/ceph_time: tolerate mono time going backwards

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33701 from tchaikov/wip-mgr-orch-exception

mgr/orch: try harder when pickle fails to marshal an exception

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33682 from thinkercui/bugfix_monsession_map_lock

mon: Get session_map_lock before remove_session

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33703 from smithfarm/wip-doc-mon_status

doc: adjust for mon_status changes in octopus

Reviewed-by: Sage Weil <sage@redhat.com>

doc: releases: v14.2.8 release notes

Co-Authored-By: Neha Ojha <nojha@redhat.com>
Co-Authored-By: Patrick Donelly <pdonell@redhat.com>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mon: Get session_map_lock before remove_session

Get session_map_lock to prevent race.

Fixes: https://tracker.ceph.com/issues/44407
Signed-off-by: Xiaofei Cui <cuixiaofei@sangfor.com.cn>

Merge pull request #33482 from rosinL/wip-fix-unitest-seastar-socket

test/test_socket: fix dispatch_sockets() unexpected exception

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/tasks/ceph_deploy: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/manypools: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/vstart_runner: use io.BytesIO for py3 compat

Use io.BytesIO instead of StringIO for py3 compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: ensure str for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_objectstore_tool: get rid of cStringIO for py3

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/divergent_priors2: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbenchsweep: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/reg11184: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/netem: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/scrub_test: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/systemd: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/admin_socket: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/cephadm: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/omapbench: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/mgr: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/watch_notify_stress: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/osd_max_pg_per_osd: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbench.py: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/rados: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/daemonwatchdog: py3 compat

Address error:
        RuntimeError: dictionary changed size during iteration

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/workunit: py3 compat

Use six.ensure_str for `get_file()` output in order to
make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix import module path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/thrashosds: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: py3 compatibility

Addresses:
  TypeError: 'dict_values' object is not subscriptable

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix imports for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: get rid of CStringIO for py3

Use io.BytesIO instead cStringIO.StringIO for py3 compatibility

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: get rid of cStringIO for py3 compat

Use io.BytesIO instead of cStringIO.StringIO
Use six.ensure_str whenever it needs to convert binary to str.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #33584 from liewegas/fix-39525

compressor/lz4: work around bug in liblz4 versions <1.8.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32444 from majianpeng/simple-debug-message

osd/PrimaryLogPG: simple debug message.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33659 from rzarzynski/wip-osd-ps-plco-via-setter

osd: peering updates peer_last_complete_ondisk via setter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33660 from rzarzynski/wip-osd-clean-pgtrim-includes

osd: drop unnecessary includes of messages/MOSDPGTrim.h.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33673 from cbodley/wip-denc-enum

denc: support enums wider than 8 bits

Reviewed-By: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33689 from Songweibin/wip-fix-typos

doc/rbd: fix small typos

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #33667 into master

* refs/pull/33667/head:
mgr/orch: show placement in 'orch ls'
mgr/orch: fix SPEC alignment in 'orch ls'
mgr/orch: include spec ref in ServiceDescription

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33669 into master

* refs/pull/33669/head:
mgr/cephadm: reduce log level for NodeAssignment
mgr/cephadm: make NodeAssignment return a simple host list

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc: adjust for mon_status changes in octopus

In octopus, we no longer have "ceph mon_status", but we do have
"ceph tell mon.<id> mon_status", so reflect this in the docs.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #33678 into master

* refs/pull/33678/head:
mgr/cephadm: fix upgrade when daemon is stopped

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33671 into master

* refs/pull/33671/head:
mgr/cephadm: refresh configs when dependencies change
mgr/cephadm: track last config time and dependencies

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33695 into master

* refs/pull/33695/head:
mgr/orch: add apply alertmanager to CLI
mgr/orch: add apply grafana to CLI

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33672 into master

* refs/pull/33672/head:
Refactor, prepare for other adoptions

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/nautilus-x/parallel: restart mgr.x before mons

Fixes: https://tracker.ceph.com/issues/44387
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/test_orchestrator_cli: enable test_osd_create exception check

this reverts 9639acfefe09c87adc821bb5c5cc41974685331d, as the test does
make sense. what fails this test is the machinary to marshal/unmarshal
exception fails to handle un-picklable exceptions. the previous commit
is supposed to use a fallback to handle them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: try harder when pickle fails to marshal an exception

pickle cannot marshal instances of class not defined in the top level of
a module. for instance, `DriveGroupValidationError` is defined in a
submodule of `ceph` python module, that's why we cannot capture it.

this prevent the `ceph` command line from getting a proper error when
the command fails if the command is implemented using cross python
module call(s).

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33664 into master

* refs/pull/33664/head:
doc: change case from `apis` to `APIs`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33655 from p-se/pse-fix-prom-alert-pool-full

monitoring: restore lost fix for `pool full` alert

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33636 into master

* refs/pull/33636/head:
qa: add upgrade test for volume upgrade from legacy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

common/ceph_time: tolerate mono time going backwards

Some kernels (and possibly some hardware?) can trigger a monotonic clock
that goes back in time.  That, in turn, can lead to a negative monotonic
time span.  This would trigger an assert.

This this problem seems to be widespread, tolerate the case and interpret
it as a 0-length interval (vs something negative).

Fixes: https://tracker.ceph.com/issues/44078
Fixes: https://tracker.ceph.com/issues/43365
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33694 into master

* refs/pull/33694/head:
cephadm: fix `alertmanager not implemented yet`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33679 into master

* refs/pull/33679/head:
cephadm: fix failure when getting keyring for deploying daemons

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply alertmanager to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply grafana to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refresh configs when dependencies change

If a daemon config (e.g., prometheus) depends on other daemons'
existence, refresh the config if that list changes (e.g., new
node-exporter, mgr removed, etc.).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33588 from dillaman/wip-rbd-snapshot-mirror

rbd-mirror: initial end-to-end test and associated bug fixes

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/orch: show placement in 'orch ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade when daemon is stopped

Tried to fix this in 5649d5571c14 but didn't remove the early image_id
check.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix `alertmanager not implemented yet`

```
Traceback (most recent call last):
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 391, in do_work
    res = self._on_complete_(*args, **kwargs)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 455, in call_self
    return f(self, *inner_args)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2576, in _create_alertmanager
    custom_config=self._generate_alertmanager_config)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2051, in _create_daemon
    stdin=json.dumps(cephadm_config))
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 1459, in _run_cephadm
    code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:ERROR: alertmanager not implemented yet
```

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Fixes: https://tracker.ceph.com/issues/44237
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

doc/rbd: fix small typos

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #33120 from vshankar/wip-mgr-volumes-clone-sync-attrs

mgr/volumes: sync inode attributes for cloned subvolumes

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32423 from bk201/wip-43420

mgr/dashboard: allow refreshing inventory page

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: change case from `apis` to `APIs`

use APIs instead of apis to be consistent throughout.

fixes: https://tracker.ceph.com/issues/44374
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #33680 from tchaikov/wip-crimson-cleanups

crimson: clang related cleanups

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>