qa/tasks/ceph.py: quote "<kind>" in command line

otherwise bash will intepret "kind" as a file when handling command like
```
sudo zgrep <kind> /var/log/ceph/valgrind/* /dev/null | sort | uniq
```
and try to feed its content to zgrep, and write the output of zgrep
to /var/log/ceph/valgrind/*. this is not the intended behavior. what we
what to do is to pass "<kind>" as an argument to zgrep, along with
the globbed files names which matches "/var/log/ceph/valgrind/*".

in this change, "<kind>" is quoted as in the command line. it's also
what `pipes.quote()` does before the change of
35cf5131e7152ce20d916aa99c124751d6a97f5c.

this addresses the regression introduced by
35cf5131e7152ce20d916aa99c124751d6a97f5c.

Fixes: https://tracker.ceph.com/issues/44454
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33766 into master

* refs/pull/33766/head:
mgr/cephadm: fix placement when existing + specified don't overlap

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33744 into master

* refs/pull/33744/head:
qa/tasks/radosbench: fix usage of -O

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: fix placement when existing + specified don't overlap

If we have a daemon on A, and our spec is (count=2, hosts=[B]), we should
always return [A,B], but we sometimes were returning [B,B].

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33531 into master

* refs/pull/33531/head:
qa/tasks/mgr/test_orchestrator_cli: fix test_ps* tests
mgr/test_orchestrator: refactor listing services/daemons
mgr/dashboard: list services and daemons

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #33718 into master

* refs/pull/33718/head:
mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #33745 into master

* refs/pull/33745/head:
mgr/cephadm: upgrade: fix daemons missing image_id

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33702 from ivancich/wip-bucket-listing-clear-results

rgw: cls_bucket_list_(un)ordered should clear results collection

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

rgw: cls_bucket_list_(un)ordered should clear results collection

Each call to cls_bucket_list_(un)ordered should have an empty
collection to populate with results. Rather than rely on the caller to
insure this, it's more reliable to have these functions do the clear.

Additionally in some cases, a reserve call was added to the collection
to pre-allocate the space needed for the expected number of
results. This will potentially result in fewer re-allocations plus
copies.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33693 from yaozongyou/fix-rgw-check-bucket-empty-bug

rgw: clear ent_list for each loop of bucket list

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33398 from clwluvw/subuser-policy

rgw: Fix bug on subuser policy identity checker

Reviewed-by: Adam Emerson <aemerson@redhat.com>

mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Fixes: https://tracker.ceph.com/issues/44405
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #33687 into master

* refs/pull/33687/head:
mgr/volume: adapt arg passing to ServiceSpec

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #33728 into master

* refs/pull/33728/head:
mgr/orch: factor out nice_delta
mgr/orch: show spec age in 'orch ls'
mgr/cephadm: store timestamp with specs
mgr/orch: add created timestamp to ServiceDescription
mgr/orch: include uptime in 'orch ps'
mgr/orch: include AGE column in 'orch ps'
mgr/cephadm: populate new DaemonDescription timestamps
mgr/orch: add new timestamps in DaemonDescription
cephadm: include timestamps for configured, created
cephadm: include timestamps for started, deploy

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33708 into master

* refs/pull/33708/head:
ceph_test_cls_hello: set RETURNVEC on the expected EINVAL request

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix test_ps* tests

The ps output names daemons like 'type.foo', e.g., 'mgr.x'.  Now that
the test_orchestrator impl is less bonkers this needs to be adjusted to
match reality.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: fix daemons missing image_id

If a daemon isn't running, we don't know the image_id (hash), so we skip.
But it's also possible to have a running daemon that doesn't report an
image_id... like right after we deploy it when the container hasn't
started up yet.  Skip those too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33729 into master

* refs/pull/33729/head:
doc/releases: Octopus is not stable yet

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/radosbench: fix usage of -O

I tried to fix this in fb991fa5f62c4aaa6c564a6d15aeee4bfaa33b39 but
was comparing an array to an int.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: factor out nice_delta

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: show spec age in 'orch ls'

Also rearrange the columns a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: store timestamp with specs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add created timestamp to ServiceDescription

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include uptime in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include AGE column in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: populate new DaemonDescription timestamps

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add new timestamps in DaemonDescription

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include timestamps for configured, created

created = when daemon was first created
configured = when the daemonw as last (re)configured

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include timestamps for started, deploy

started == when the container started
deployed == when the container was (most recently) (re)deployed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33733 into master

* refs/pull/33733/head:
mgr/cephadm: if we had no record of deps, and deps are [], do not reconfig

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33734 into master

* refs/pull/33734/head:
mgr/cephadm: do not refresh daemon and device inventory as often

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33736 into master

* refs/pull/33736/head:
mgr/cephadm: upgrade crash collectors too
mgr/cephadm: fix service rm when there are no daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33549 from rhcs-dashboard/44307-e2e-custom-credentials

mgr/dashboard: ability to provide custom credentials for E2E tests.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30224 from chardan/jfw-wip-random-deduction-guide

common: random: added a deduction guide to make using the function ob…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32495 from toabctl/qa-mypy

qa: Add basic mypy support for the qa directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: Enable basic mypy support for qa/ directory

A first step to do more automatic code checks on the qa/
directory. This is useful while transitioning to python3.

Also use log_exc to top-level to not run into:

error: Argument 1 to "log_exc" has incompatible type
  "Callable[[OSDThrasher], Any]"; expected "OSDThrasher"

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

qa: Fix problems detected by mypy

This is a first step to enable mypy on the qa/ directory.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #33657 from tchaikov/wip-clang

client,common,mgr,rbd: clang related cleanups

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>

Merge PR #33662 into master

* refs/pull/33662/head:
mds: handle ceph_assert on blacklisting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/cephadm: upgrade crash collectors too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix service rm when there are no daemons

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33707 from guits/guits-quick_fix_doc

doc: update erasure-code-profile.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33730 into master

* refs/pull/33730/head:
cephadm/vstart-smoke.sh: a smoke test to run against vstart
mgr/cephadm: fix prom reconfig loop when dashboard is disabled
mgr/cephadm: clean up log messages a bit
mgr/cephadm: force remove daemons when removing service
mgr/cephadm: do not error out on service rm|action if no daemons
mgr/orchestrator: allow 'orch rm mon|mgr --force'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33731 from wjwithagen/wjw-fix-test-compressor-gtest

src/test/compressor: Add missing gtest

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: do not refresh daemon and device inventory as often

The device inventory rarely changes.

A daemon inventory of 60s may trigger every time around the loop when
doing upgrades, which is pretty slow (esp since it's currently done
synchronously).  We're doing a good job (I think?) of queueing refreshes,
so the only reason to time this data out is to catch daemon stops that
are caused by external factors.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: if we had no record of deps, and deps are [], do not reconfig

None != []

This avoids a reconfig of every daemon on upgrade, among other things.

Signed-off-by: Sage Weil <sage@redhat.com>

doc: update erasure-code-profile.rst

Update this part of the documentation to reflect the change made by
3e86be7d50d485dbcce50d99d2f52891ca7bd399

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge PR #33663 into master

* refs/pull/33663/head:
mgr/cephadm: reduce boilerplate for unittests

Reviewed-by: Sage Weil <sage@redhat.com>

src/test/compressor: Add missing gtest

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/cephadm: reduce boilerplate for unittests

Signed-off-by: Joshua Schmid <jschmid@suse.de>

cephadm/vstart-smoke.sh: a smoke test to run against vstart

This provides decent coverage of cephadm in practice, and can run against
a vstart cluster without having to go build containers.

vstart-cleanup.sh will clean up an aborted test reasonably well without
requiring sudo.

Caveats:
- there is no coverage for osds at this point.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix prom reconfig loop when dashboard is disabled

We need to consider the mgrs a dep even if the prometheus module isn't
enabled, so that we are consistent with _calc_daemon_deps() and don't
reconfig prom in a loop.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: clean up log messages a bit

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: force remove daemons when removing service

The rm CLI checks for --force based on type, so it isn't passed down
through the implementations.  So, pass it through.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not error out on service rm|action if no daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: allow 'orch rm mon|mgr --force'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33704 into master

* refs/pull/33704/head:
mgr/cephadm: avoid racing updates to spec dict
mgr/cephadm: make _do_upgrade work synchronously
mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test
mgr/cephadm: make _apply_service move services
mgr/cephadm: log when we create, destroy daemons
mgr/cephadm: drop async_map_completion decorators from create funcs
mgr/cephadm: _create_osd -> _create_osds
mgr/cephadm: _remove_daemon -> _remove_daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

doc/releases: Octopus is not stable yet

Remove Octopus from list of active stable releases (it's not stable yet)
and put the 15.1.0 link in the Development column.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/cephadm: avoid racing updates to spec dict

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _do_upgrade work synchronously

It mostly was, but it was futzing with completions when it didn't need to.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/tests/test_cephadm: remove broken rgw_update_fail test

This was trying to show that we won't put 2 RGWs from different zones
on the same host.  In reality, the test was only passing because it was
specifying an invalid placement spec (count=3 in a 2-host cluster).

It's true that we shouldn't do that because they won't be able to bind,
but the HostAssignment code doesn't handle this case currently.

See https://tracker.ceph.com/issues/44397

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make _apply_service move services

This transitions us to a synchronous mode of operation.

Fixes: https://tracker.ceph.com/issues/44167
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: log when we create, destroy daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop async_map_completion decorators from create funcs

So we can call them synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _create_osd -> _create_osds

So we could do a single daemon synchronously, if we chose.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: _remove_daemon -> _remove_daemons

So we can remove a single daemon synchronously, if we choose.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33565 into master

* refs/pull/33565/head:
os/bluestore: log allocation stats on a daily basis.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33539 from cbodley/wip-rgw-doc-reshard

doc/rgw: add design doc for multisite resharding

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33698 from rhcs-dashboard/vstart-set-prometheus-port-for-each-mgr

vstart.sh: set prometheus port for each mgr.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33047 into master

* refs/pull/33047/head:
qa/rgw: exercise DeleteRange in test_bucket_index_log_trim

Reviewed-by: Vikhyat Umrao <vikhyat@redhat.com>

Merge PR #33287 into master

* refs/pull/33287/head:
rgw: move frontends initial init to after global_init()
rgw: ssl: don't try to init certificate if not needed
rgw: frontend: add rgw_frontend_defaults configurable
rgw: beast ssl: enable use of meta variable for cert config
rgw: beast ssl: improve output
rgw: beast frontend: handle default ssl configurables
rgw: update docs about ssl config through config-key
rgw: allow beast ssl frontend cert config via mon config-key
rgw: create config-key svc
rgw: svc/rados: new mon_command call

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph_test_cls_hello: set RETURNVEC on the expected EINVAL request

We are sending a request that returns an error *and* a data payload.  In order
to reliably get the payload, we need to set the RETURNVEC flag so that it is
recorded in the pg log.  Otherwise, we might get unlucky and resend the request
and end up in the dup handler and get the error code but no payload.

Fixes: https://tracker.ceph.com/issues/44385
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33612 from votdev/issue_44322_rgw_user_max_buckets

mgr/dashboard: Not able to restrict bucket creation for new user

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33721 from smithfarm/wip-nautilus_14_2_8

doc/releases: add 14.2.8 to release timeline

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

doc/rgw: add design doc for multisite resharding

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #33705 into master

* refs/pull/33705/head:
qa/suites/upgrade/nautilus-x/parallel: restart mgr.x before mons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33723 from tchaikov/wip-seastar-on-demand

run-make-check.sh: WITH_SEASTAR on demand

Reviewed-by: Nathan Cutler <ncutler@suse.com>

run-make-check.sh: WITH_SEASTAR on demand

don't compile WITH_SEASTAR=ON unless required

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/releases: add 14.2.8 to release timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33670 from theanalyst/doc/releases/v14.2.8

doc: releases: v14.2.8 release notes

Reviewed-By: Nathan Cutler <ncutler@suse.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Patrick Donelly <pdonell@redhat.com>

Merge pull request #33709 from tchaikov/wip-ceph-py-py3-rados

qa/tasks: py3 compat (tasks exercised by rados suites)

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: set prometheus port for each mgr.

Fixes: https://tracker.ceph.com/issues/44417
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #33690 from rhcs-dashboard/44237-remove-config-opt-read-from-system-roles

mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33699 from liewegas/fix-44078

common/ceph_time: tolerate mono time going backwards

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33701 from tchaikov/wip-mgr-orch-exception

mgr/orch: try harder when pickle fails to marshal an exception

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33682 from thinkercui/bugfix_monsession_map_lock

mon: Get session_map_lock before remove_session

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33703 from smithfarm/wip-doc-mon_status

doc: adjust for mon_status changes in octopus

Reviewed-by: Sage Weil <sage@redhat.com>

doc: releases: v14.2.8 release notes

Co-Authored-By: Neha Ojha <nojha@redhat.com>
Co-Authored-By: Patrick Donelly <pdonell@redhat.com>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mon: Get session_map_lock before remove_session

Get session_map_lock to prevent race.

Fixes: https://tracker.ceph.com/issues/44407
Signed-off-by: Xiaofei Cui <cuixiaofei@sangfor.com.cn>

Merge pull request #33482 from rosinL/wip-fix-unitest-seastar-socket

test/test_socket: fix dispatch_sockets() unexpected exception

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/tasks/ceph_deploy: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/manypools: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/vstart_runner: use io.BytesIO for py3 compat

Use io.BytesIO instead of StringIO for py3 compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: ensure str for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_objectstore_tool: get rid of cStringIO for py3

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/divergent_priors2: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbenchsweep: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/reg11184: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>