mgr: update metadata if an osd just joins

instead of using "front_address" for checking if we have a new OSD
reusing existing a known identify shows up in the osdmap, it'd be
simpler to compare the up_from epoch with the osdmap's epoch. as
objecter will subscribe **every** osdmap after mgr boots. so mgr
should be able to see the osdmap when the osd joins the cluster
where the up_from epoch is identical to osdmap's epoch.

this way is simpler than existing approach. but it will involve
more overhead if osd reboots frequently without changing their
metadata.

before this change, the metadata is requested/updated only if
the public (front) address is changed.

after this change, the metadata is requested/updated whenever
an osd reboots.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: update "hostname" when we already have the daemon state from the same entity

there is chance that we reuse the identity of a daemon and deploy the
daemon on a different host. in that case, the existing daemon state
should be updated with the new hostname.

Fixes: https://tracker.ceph.com/issues/44446
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33730 into master

* refs/pull/33730/head:
cephadm/vstart-smoke.sh: a smoke test to run against vstart
mgr/cephadm: fix prom reconfig loop when dashboard is disabled
mgr/cephadm: clean up log messages a bit
mgr/cephadm: force remove daemons when removing service
mgr/cephadm: do not error out on service rm|action if no daemons
mgr/orchestrator: allow 'orch rm mon|mgr --force'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33731 from wjwithagen/wjw-fix-test-compressor-gtest

src/test/compressor: Add missing gtest

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33663 into master

* refs/pull/33663/head:
mgr/cephadm: reduce boilerplate for unittests

Reviewed-by: Sage Weil <sage@redhat.com>

src/test/compressor: Add missing gtest

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/cephadm: reduce boilerplate for unittests

Signed-off-by: Joshua Schmid <jschmid@suse.de>

cephadm/vstart-smoke.sh: a smoke test to run against vstart

This provides decent coverage of cephadm in practice, and can run against
a vstart cluster without having to go build containers.

vstart-cleanup.sh will clean up an aborted test reasonably well without
requiring sudo.

Caveats:
- there is no coverage for osds at this point.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix prom reconfig loop when dashboard is disabled

We need to consider the mgrs a dep even if the prometheus module isn't
enabled, so that we are consistent with _calc_daemon_deps() and don't
reconfig prom in a loop.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: clean up log messages a bit

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: force remove daemons when removing service

The rm CLI checks for --force based on type, so it isn't passed down
through the implementations.  So, pass it through.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not error out on service rm|action if no daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: allow 'orch rm mon|mgr --force'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33704 into master

* refs/pull/33704/head:
mgr/cephadm: avoid racing updates to spec dict
mgr/cephadm: make _do_upgrade work synchronously
mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test
mgr/cephadm: make _apply_service move services
mgr/cephadm: log when we create, destroy daemons
mgr/cephadm: drop async_map_completion decorators from create funcs
mgr/cephadm: _create_osd -> _create_osds
mgr/cephadm: _remove_daemon -> _remove_daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: avoid racing updates to spec dict

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _do_upgrade work synchronously

It mostly was, but it was futzing with completions when it didn't need to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test

This was trying to show that we won't put 2 RGWs from different zones
on the same host.  In reality, the test was only passing because it was
specifying an invalid placement spec (count=3 in a 2-host cluster).

It's true that we shouldn't do that because they won't be able to bind,
but the HostAssignment code doesn't handle this case currently.

See https://tracker.ceph.com/issues/44397

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _apply_service move services

This transitions us to a synchronous mode of operation.

Fixes: https://tracker.ceph.com/issues/44167
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: log when we create, destroy daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop async_map_completion decorators from create funcs

So we can call them synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _create_osd -> _create_osds

So we could do a single daemon synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _remove_daemon -> _remove_daemons

So we can remove a single daemon synchronously, if we choose.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33565 into master

* refs/pull/33565/head:
os/bluestore: log allocation stats on a daily basis.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33539 from cbodley/wip-rgw-doc-reshard

doc/rgw: add design doc for multisite resharding

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33698 from rhcs-dashboard/vstart-set-prometheus-port-for-each-mgr

vstart.sh: set prometheus port for each mgr.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33047 into master

* refs/pull/33047/head:
qa/rgw: exercise DeleteRange in test_bucket_index_log_trim

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge PR #33287 into master

* refs/pull/33287/head:
rgw: move frontends initial init to after global_init()
rgw: ssl: don't try to init certificate if not needed
rgw: frontend: add rgw_frontend_defaults configurable
rgw: beast ssl: enable use of meta variable for cert config
rgw: beast ssl: improve output
rgw: beast frontend: handle default ssl configurables
rgw: update docs about ssl config through config-key
rgw: allow beast ssl frontend cert config via mon config-key
rgw: create config-key svc
rgw: svc/rados: new mon_command call

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33612 from votdev/issue_44322_rgw_user_max_buckets

mgr/dashboard: Not able to restrict bucket creation for new user

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33721 from smithfarm/wip-nautilus_14_2_8

doc/releases: add 14.2.8 to release timeline

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc/rgw: add design doc for multisite resharding

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33705 into master

* refs/pull/33705/head:
qa/suites/upgrade/nautilus-x/parallel: restart mgr.x before mons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33723 from tchaikov/wip-seastar-on-demand

run-make-check.sh: WITH_SEASTAR on demand

Reviewed-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: WITH_SEASTAR on demand

don't compile WITH_SEASTAR=ON unless required

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: add 14.2.8 to release timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33670 from theanalyst/doc/releases/v14.2.8

doc: releases: v14.2.8 release notes

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>

Merge pull request #33709 from tchaikov/wip-ceph-py-py3-rados

qa/tasks: py3 compat (tasks exercised by rados suites)

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: set prometheus port for each mgr.

Fixes: https://tracker.ceph.com/issues/44417
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #33690 from rhcs-dashboard/44237-remove-config-opt-read-from-system-roles

mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33699 from liewegas/fix-44078

common/ceph_time: tolerate mono time going backwards

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33701 from tchaikov/wip-mgr-orch-exception

mgr/orch: try harder when pickle fails to marshal an exception

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33682 from thinkercui/bugfix_monsession_map_lock

mon: Get session_map_lock before remove_session

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33703 from smithfarm/wip-doc-mon_status

doc: adjust for mon_status changes in octopus

Reviewed-by: Sage Weil <sage@redhat.com>

doc: releases: v14.2.8 release notes

Co-Authored-By: Neha Ojha <nojha@redhat.com>
Co-Authored-By: Patrick Donelly <pdonell@redhat.com>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mon: Get session_map_lock before remove_session

Get session_map_lock to prevent race.

Fixes: https://tracker.ceph.com/issues/44407
Signed-off-by: Xiaofei Cui <cuixiaofei@sangfor.com.cn>

Merge pull request #33482 from rosinL/wip-fix-unitest-seastar-socket

test/test_socket: fix dispatch_sockets() unexpected exception

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/tasks/ceph_deploy: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/manypools: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/vstart_runner: use io.BytesIO for py3 compat

Use io.BytesIO instead of StringIO for py3 compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: ensure str for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_objectstore_tool: get rid of cStringIO for py3

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/divergent_priors2: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbenchsweep: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/reg11184: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/netem: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/scrub_test: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/systemd: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/admin_socket: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/cephadm: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/omapbench: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/mgr: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/watch_notify_stress: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/osd_max_pg_per_osd: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbench.py: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/rados: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/daemonwatchdog: py3 compat

Address error:
        RuntimeError: dictionary changed size during iteration

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/workunit: py3 compat

Use six.ensure_str for `get_file()` output in order to
make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix import module path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/thrashosds: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: py3 compatibility

Addresses:
  TypeError: 'dict_values' object is not subscriptable

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix imports for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: get rid of CStringIO for py3

Use io.BytesIO instead cStringIO.StringIO for py3 compatibility

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: get rid of cStringIO for py3 compat

Use io.BytesIO instead of cStringIO.StringIO
Use six.ensure_str whenever it needs to convert binary to str.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #33584 from liewegas/fix-39525

compressor/lz4: work around bug in liblz4 versions <1.8.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32444 from majianpeng/simple-debug-message

osd/PrimaryLogPG: simple debug message.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33659 from rzarzynski/wip-osd-ps-plco-via-setter

osd: peering updates peer_last_complete_ondisk via setter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33660 from rzarzynski/wip-osd-clean-pgtrim-includes

osd: drop unnecessary includes of messages/MOSDPGTrim.h.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33673 from cbodley/wip-denc-enum

denc: support enums wider than 8 bits

Reviewed-By: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33689 from Songweibin/wip-fix-typos

doc/rbd: fix small typos

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #33667 into master

* refs/pull/33667/head:
mgr/orch: show placement in 'orch ls'
mgr/orch: fix SPEC alignment in 'orch ls'
mgr/orch: include spec ref in ServiceDescription

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33669 into master

* refs/pull/33669/head:
mgr/cephadm: reduce log level for NodeAssignment
mgr/cephadm: make NodeAssignment return a simple host list

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc: adjust for mon_status changes in octopus

In octopus, we no longer have "ceph mon_status", but we do have
"ceph tell mon.<id> mon_status", so reflect this in the docs.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #33678 into master

* refs/pull/33678/head:
mgr/cephadm: fix upgrade when daemon is stopped

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33671 into master

* refs/pull/33671/head:
mgr/cephadm: refresh configs when dependencies change
mgr/cephadm: track last config time and dependencies

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33695 into master

* refs/pull/33695/head:
mgr/orch: add apply alertmanager to CLI
mgr/orch: add apply grafana to CLI

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33672 into master

* refs/pull/33672/head:
Refactor, prepare for other adoptions

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/nautilus-x/parallel: restart mgr.x before mons

Fixes: https://tracker.ceph.com/issues/44387
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/test_orchestrator_cli: enable test_osd_create exception check

this reverts 9639acfefe09c87adc821bb5c5cc41974685331d, as the test does
make sense. what fails this test is the machinary to marshal/unmarshal
exception fails to handle un-picklable exceptions. the previous commit
is supposed to use a fallback to handle them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: try harder when pickle fails to marshal an exception

pickle cannot marshal instances of class not defined in the top level of
a module. for instance, `DriveGroupValidationError` is defined in a
submodule of `ceph` python module, that's why we cannot capture it.

this prevent the `ceph` command line from getting a proper error when
the command fails if the command is implemented using cross python
module call(s).

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33664 into master

* refs/pull/33664/head:
doc: change case from `apis` to `APIs`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33655 from p-se/pse-fix-prom-alert-pool-full

monitoring: restore lost fix for `pool full` alert

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33636 into master

* refs/pull/33636/head:
qa: add upgrade test for volume upgrade from legacy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

common/ceph_time: tolerate mono time going backwards

Some kernels (and possibly some hardware?) can trigger a monotonic clock
that goes back in time.  That, in turn, can lead to a negative monotonic
time span.  This would trigger an assert.

This this problem seems to be widespread, tolerate the case and interpret
it as a 0-length interval (vs something negative).

Fixes: https://tracker.ceph.com/issues/44078
Fixes: https://tracker.ceph.com/issues/43365
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33694 into master

* refs/pull/33694/head:
cephadm: fix `alertmanager not implemented yet`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33679 into master

* refs/pull/33679/head:
cephadm: fix failure when getting keyring for deploying daemons

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply alertmanager to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply grafana to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refresh configs when dependencies change

If a daemon config (e.g., prometheus) depends on other daemons'
existence, refresh the config if that list changes (e.g., new
node-exporter, mgr removed, etc.).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33588 from dillaman/wip-rbd-snapshot-mirror

rbd-mirror: initial end-to-end test and associated bug fixes

Reviewed-by: Mykola Golub <mgolub@suse.com>