mgr/orch: show spec age in 'orch ls'

Also rearrange the columns a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: store timestamp with specs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add created timestamp to ServiceDescription

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include uptime in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include AGE column in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: populate new DaemonDescription timestamps

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add new timestamps in DaemonDescription

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include timestamps for configured, created

created = when daemon was first created
configured = when the daemonw as last (re)configured

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include timestamps for started, deploy

started == when the container started
deployed == when the container was (most recently) (re)deployed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33733 into master

* refs/pull/33733/head:
mgr/cephadm: if we had no record of deps, and deps are [], do not reconfig

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33734 into master

* refs/pull/33734/head:
mgr/cephadm: do not refresh daemon and device inventory as often

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33736 into master

* refs/pull/33736/head:
mgr/cephadm: upgrade crash collectors too
mgr/cephadm: fix service rm when there are no daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33549 from rhcs-dashboard/44307-e2e-custom-credentials

mgr/dashboard: ability to provide custom credentials for E2E tests.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30224 from chardan/jfw-wip-random-deduction-guide

common: random: added a deduction guide to make using the function ob…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32495 from toabctl/qa-mypy

qa: Add basic mypy support for the qa directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: Enable basic mypy support for qa/ directory

A first step to do more automatic code checks on the qa/
directory. This is useful while transitioning to python3.

Also use log_exc to top-level to not run into:

error: Argument 1 to "log_exc" has incompatible type
  "Callable[[OSDThrasher], Any]"; expected "OSDThrasher"

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

qa: Fix problems detected by mypy

This is a first step to enable mypy on the qa/ directory.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #33657 from tchaikov/wip-clang

client,common,mgr,rbd: clang related cleanups

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>

Merge PR #33662 into master

* refs/pull/33662/head:
mds: handle ceph_assert on blacklisting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/cephadm: upgrade crash collectors too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix service rm when there are no daemons

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33707 from guits/guits-quick_fix_doc

doc: update erasure-code-profile.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33730 into master

* refs/pull/33730/head:
cephadm/vstart-smoke.sh: a smoke test to run against vstart
mgr/cephadm: fix prom reconfig loop when dashboard is disabled
mgr/cephadm: clean up log messages a bit
mgr/cephadm: force remove daemons when removing service
mgr/cephadm: do not error out on service rm|action if no daemons
mgr/orchestrator: allow 'orch rm mon|mgr --force'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33731 from wjwithagen/wjw-fix-test-compressor-gtest

src/test/compressor: Add missing gtest

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: do not refresh daemon and device inventory as often

The device inventory rarely changes.

A daemon inventory of 60s may trigger every time around the loop when
doing upgrades, which is pretty slow (esp since it's currently done
synchronously).  We're doing a good job (I think?) of queueing refreshes,
so the only reason to time this data out is to catch daemon stops that
are caused by external factors.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: if we had no record of deps, and deps are [], do not reconfig

None != []

This avoids a reconfig of every daemon on upgrade, among other things.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: update erasure-code-profile.rst

Update this part of the documentation to reflect the change made by
3e86be7d50d485dbcce50d99d2f52891ca7bd399

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge PR #33663 into master

* refs/pull/33663/head:
mgr/cephadm: reduce boilerplate for unittests

Reviewed-by: Sage Weil <sage@redhat.com>

src/test/compressor: Add missing gtest

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/cephadm: reduce boilerplate for unittests

Signed-off-by: Joshua Schmid <jschmid@suse.de>

cephadm/vstart-smoke.sh: a smoke test to run against vstart

This provides decent coverage of cephadm in practice, and can run against
a vstart cluster without having to go build containers.

vstart-cleanup.sh will clean up an aborted test reasonably well without
requiring sudo.

Caveats:
- there is no coverage for osds at this point.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix prom reconfig loop when dashboard is disabled

We need to consider the mgrs a dep even if the prometheus module isn't
enabled, so that we are consistent with _calc_daemon_deps() and don't
reconfig prom in a loop.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: clean up log messages a bit

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: force remove daemons when removing service

The rm CLI checks for --force based on type, so it isn't passed down
through the implementations.  So, pass it through.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not error out on service rm|action if no daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: allow 'orch rm mon|mgr --force'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33704 into master

* refs/pull/33704/head:
mgr/cephadm: avoid racing updates to spec dict
mgr/cephadm: make _do_upgrade work synchronously
mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test
mgr/cephadm: make _apply_service move services
mgr/cephadm: log when we create, destroy daemons
mgr/cephadm: drop async_map_completion decorators from create funcs
mgr/cephadm: _create_osd -> _create_osds
mgr/cephadm: _remove_daemon -> _remove_daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: avoid racing updates to spec dict

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _do_upgrade work synchronously

It mostly was, but it was futzing with completions when it didn't need to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test

This was trying to show that we won't put 2 RGWs from different zones
on the same host.  In reality, the test was only passing because it was
specifying an invalid placement spec (count=3 in a 2-host cluster).

It's true that we shouldn't do that because they won't be able to bind,
but the HostAssignment code doesn't handle this case currently.

See https://tracker.ceph.com/issues/44397

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _apply_service move services

This transitions us to a synchronous mode of operation.

Fixes: https://tracker.ceph.com/issues/44167
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: log when we create, destroy daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop async_map_completion decorators from create funcs

So we can call them synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _create_osd -> _create_osds

So we could do a single daemon synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _remove_daemon -> _remove_daemons

So we can remove a single daemon synchronously, if we choose.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33565 into master

* refs/pull/33565/head:
os/bluestore: log allocation stats on a daily basis.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33539 from cbodley/wip-rgw-doc-reshard

doc/rgw: add design doc for multisite resharding

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33698 from rhcs-dashboard/vstart-set-prometheus-port-for-each-mgr

vstart.sh: set prometheus port for each mgr.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33047 into master

* refs/pull/33047/head:
qa/rgw: exercise DeleteRange in test_bucket_index_log_trim

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge PR #33287 into master

* refs/pull/33287/head:
rgw: move frontends initial init to after global_init()
rgw: ssl: don't try to init certificate if not needed
rgw: frontend: add rgw_frontend_defaults configurable
rgw: beast ssl: enable use of meta variable for cert config
rgw: beast ssl: improve output
rgw: beast frontend: handle default ssl configurables
rgw: update docs about ssl config through config-key
rgw: allow beast ssl frontend cert config via mon config-key
rgw: create config-key svc
rgw: svc/rados: new mon_command call

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33612 from votdev/issue_44322_rgw_user_max_buckets

mgr/dashboard: Not able to restrict bucket creation for new user

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33721 from smithfarm/wip-nautilus_14_2_8

doc/releases: add 14.2.8 to release timeline

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc/rgw: add design doc for multisite resharding

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33705 into master

* refs/pull/33705/head:
qa/suites/upgrade/nautilus-x/parallel: restart mgr.x before mons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33723 from tchaikov/wip-seastar-on-demand

run-make-check.sh: WITH_SEASTAR on demand

Reviewed-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: WITH_SEASTAR on demand

don't compile WITH_SEASTAR=ON unless required

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: add 14.2.8 to release timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33670 from theanalyst/doc/releases/v14.2.8

doc: releases: v14.2.8 release notes

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>

Merge pull request #33709 from tchaikov/wip-ceph-py-py3-rados

qa/tasks: py3 compat (tasks exercised by rados suites)

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: set prometheus port for each mgr.

Fixes: https://tracker.ceph.com/issues/44417
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #33690 from rhcs-dashboard/44237-remove-config-opt-read-from-system-roles

mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33699 from liewegas/fix-44078

common/ceph_time: tolerate mono time going backwards

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33701 from tchaikov/wip-mgr-orch-exception

mgr/orch: try harder when pickle fails to marshal an exception

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33682 from thinkercui/bugfix_monsession_map_lock

mon: Get session_map_lock before remove_session

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33703 from smithfarm/wip-doc-mon_status

doc: adjust for mon_status changes in octopus

Reviewed-by: Sage Weil <sage@redhat.com>

doc: releases: v14.2.8 release notes

Co-Authored-By: Neha Ojha <nojha@redhat.com>
Co-Authored-By: Patrick Donelly <pdonell@redhat.com>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mon: Get session_map_lock before remove_session

Get session_map_lock to prevent race.

Fixes: https://tracker.ceph.com/issues/44407
Signed-off-by: Xiaofei Cui <cuixiaofei@sangfor.com.cn>

Merge pull request #33482 from rosinL/wip-fix-unitest-seastar-socket

test/test_socket: fix dispatch_sockets() unexpected exception

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/tasks/ceph_deploy: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/manypools: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/vstart_runner: use io.BytesIO for py3 compat

Use io.BytesIO instead of StringIO for py3 compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: ensure str for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_objectstore_tool: get rid of cStringIO for py3

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/divergent_priors2: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbenchsweep: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/reg11184: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/netem: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/scrub_test: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/systemd: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/admin_socket: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/cephadm: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/omapbench: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/mgr: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/watch_notify_stress: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/osd_max_pg_per_osd: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbench.py: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/rados: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/daemonwatchdog: py3 compat

Address error:
        RuntimeError: dictionary changed size during iteration

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/workunit: py3 compat

Use six.ensure_str for `get_file()` output in order to
make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix import module path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/thrashosds: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: py3 compatibility

Addresses:
  TypeError: 'dict_values' object is not subscriptable

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix imports for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: get rid of CStringIO for py3

Use io.BytesIO instead cStringIO.StringIO for py3 compatibility

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: get rid of cStringIO for py3 compat

Use io.BytesIO instead of cStringIO.StringIO
Use six.ensure_str whenever it needs to convert binary to str.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #33584 from liewegas/fix-39525

compressor/lz4: work around bug in liblz4 versions <1.8.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32444 from majianpeng/simple-debug-message

osd/PrimaryLogPG: simple debug message.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33659 from rzarzynski/wip-osd-ps-plco-via-setter

osd: peering updates peer_last_complete_ondisk via setter.

Reviewed-by: Kefu Chai <kchai@redhat.com>