qa/tasks/ceph_manager: use StringIO for capturing COT output

there are couple factors we should consider when choosing between
BytesIO and StringIO:

- if the producer is producing binary
- if we are expecting binary
- if the layers in between them are doing the decoding/encoding
  automatically.

in our case, the producer is either the ChannelFile instances returned
by paramiko.SSHClient or subprocess.CompletedProcess insances returned
by subprocess.run(). the former are file-like objects opened in "r" mode,
but their contents are decoded with utf-8 when reading if
ChannelFile.FLAG_BINARY is not specified. that's why we always try to
add this flag in orchestra/run.py when collecting the stdout and stderr
from paramiko.SSHClient after executing a command.

back in python2, this works just fine. as we don't differentiate bytes
from str by then.

but in python3, we have to make a decision. in the case of
ceph-objectstore-tool (COT for short), it does not produce binary and
we don't check its output with binary, so, if neither Remote.run() nor
LocalRemote.run() decodes/encodes for us, it's fine.

so it boils down to `copy_to_log()`:

i think we we should respect the consumer's expectation, and only decode
the output if a StringIO is passed in as stdout or stderr.

as we always log the output with logging we could either set
`ChannelFile.FLAG_BINARY` depending on the type of `capture` or not.
if it's not set, paramiko will return str (bytes) on python2, and str on
python3. if it's not set paramiko will return str (bytes) on python2,
and bytes on python3.

if there is non-ASCII in the output, logging will bail fail with
`UnicodeDecodeError` exception. and paramiko throws the same exception
when trying to decode for us if `ChannelFile.FLAG_BINARY` is not
specified.

so to ensure that we always have logging messages no matter if the
producer follows the rule of "use StringIO if you only emit text" or
not, we have to use `ChannelFile.FLAG_BINARY`, and force paramiko
to send us the bytes. but we still have the luxury to use StringIO
and do the decode when the caller asks for str explicitly. that'd save
the pain of using `str.decode()` or `six.ensure_str()` everywhere
even if we can assure that the program does not write binary.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/ceph_manager: capture stderr for COT

as we are expecting the error message written to stderr, and we need to
check for the error messages in it.

this change addresses the regression introduced by
204ceee156cbb8a20bdf56efb0cd0610ee4c107e

Fixes: https://tracker.ceph.com/issues/44500
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33780 from xxhdx1985126/wip-crimson-osd-pg-creation-issue

crimson: solve the problem that crimson-osd's created pgs stuck in "unknown" state

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix the problem that crimson-osd can't establish stable connection with mgr

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: decouple mgr client reconnect and connect reset handling

As of now, the following invocation sequence triggers deadlock when
closing crimson-osd's connection with mgr:
  ProtocolV2::dispatch_reset() --> crimson::mgr::Client::ms_handle_reset
--> crimson::mgr::Client::reconnect --> crimson::net::SocketConnection::close
--> crimson::net::Protocol::close()

In the above invocation sequence, ProtocalV2::dispatch_reset() enters the gate
"pending_dispatch" the leaving of which would wait for the complete of crimson::\
net::Protocal::close() which further wait for the complete of the gate's close().

This commit decouples this waiting chain.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

Merge PR #33700 into master

* refs/pull/33700/head:
mgr/cephadm: point dashboard at grafana automatically
doc/cephadm/monitoring: document process to set up monitoring with cephadm

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33785 into master

* refs/pull/33785/head:
qa/suites/rados/cephadm: deploy all monitoring components
qa/tasks/cephadm: deploy other monitoring components

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33602 into master

* refs/pull/33602/head:
mgr/cephadm/osd: call synchronous _remove_daemon()
mgr/cephadm/osd: Add pytest for osd removal
mgr/cephadm/osd: delete transitional attributes
mgr/cephadm: Move remove osd code to osd.py

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

qa/suites/rados/cephadm: deploy all monitoring components

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/osd: call synchronous _remove_daemon()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/osd: Add pytest for osd removal

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/osd: delete transitional attributes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Move remove osd code to osd.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: point dashboard at grafana automatically

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/monitoring: document process to set up monitoring with cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33772 into master

* refs/pull/33772/head:
mgr/cephadm: upgrade all ceph daemon types
mgr/cephadm: do not specify --image arg for non-ceph daemons; fix upgrade
mgr/cephadm: fix error-path return values for _run_cephadm
mgr/orch: fix RGWSpec loading
mgr/cephadm: fix 'orch rm'

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33779 into master

* refs/pull/33779/head:
mgr/orch: fix ServiceSpec deserialization error

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33614 into master

* refs/pull/33614/head:
mgr/cephadm: enable custom TLS certificates for grafana
mgr: enable verification of TLS certs without files
mgr/cephadm: dump config to JSON only once when creating daemons

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33696 from cbodley/wip-common-spliterator

common: add iterator-based string splitter

Reviewed-by: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33754 from cbodley/wip-rgw-formatter-plain-segfault

rgw: fix string_view formatting in RGWFormatter_Plain

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge PR #33784 into master

* refs/pull/33784/head:
mgr/cephadm: update type annotation

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: update type annotation

the method signature was changed in
ea987a0e56db106f7c76d11f86b3e602257f365e

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephadm: deploy other monitoring components

Grafana and alertmanager.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33757 into master

* refs/pull/33757/head:
qa/valgrind.supp: abstract from ceph::buffer's symbol versioning.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33755 into master

* refs/pull/33755/head:
mgr/cephadm: make osd create on an existing LV idempotent

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/orch: fix ServiceSpec deserialization error

cephadm fails to load Service spec data when deserializing
HostPlacementSpec resource. Fix the issue by loading it properly.

Renaming the call from_dict to from_json for consistency.

This change also makes `spec` filed visible in JSON output of
ServiceSpec list.

Fixes: https://tracker.ceph.com/issues/44491
Fixes: https://tracker.ceph.com/issues/44392
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #33081 from bk201/wip-43166

mgr/dashboard: improve device selection modal for creating OSDs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33775 from tchaikov/wip-44454

qa/tasks/ceph.py: quote "<kind>" in command line

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #33674 into master

* refs/pull/33674/head:
mgr/volumes: OpSmException.error -> OpSmException.errno
mgr/volumes: remove unneeded assignment to `NoneType`
mgr/volumes: add missing OpSmException import
mgr/volumes: add missing error code
mgr/volumes: remove stray return
mgr/volumes: fixup format string args
mgr/volumes: add missing `mgr` param
mgr/volumes: assert self.fs
mgr/volumes: skip type checking on RTimer class
mgr/volumes: fixup threading.Timer import
mgr/volumes: fix positional str formatting
mgr/volumes: place getters and setters next to each other
mgr/volumes: reverse params passed to `isinstace()`
mgr/volumes: fix arg typo '.' -> ','
mgr/volumes: add `deque` type
mgr/volumes: `threading.Event.isSet` no longer exists in py3
mgr/volumes: assert JobThread class
mgr/volumes: import VolumeException
mgr/volumes: use `SafeConfigParser` with py2
mgr/volumes: use python version during configparser import
mgr/volumes: add `Dict` type
mgr/volumes: import errno
mgr/volumes: add `mgr/volumes` to the mypy checks

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/tasks/ceph.py: quote "<kind>" in command line

otherwise bash will intepret "kind" as a file when handling command like
```
sudo zgrep <kind> /var/log/ceph/valgrind/* /dev/null | sort | uniq
```
and try to feed its content to zgrep, and write the output of zgrep
to /var/log/ceph/valgrind/*. this is not the intended behavior. what we
what to do is to pass "<kind>" as an argument to zgrep, along with
the globbed files names which matches "/var/log/ceph/valgrind/*".

in this change, "<kind>" is quoted as in the command line. it's also
what `pipes.quote()` does before the change of
35cf5131e7152ce20d916aa99c124751d6a97f5c.

this addresses the regression introduced by
35cf5131e7152ce20d916aa99c124751d6a97f5c.

Fixes: https://tracker.ceph.com/issues/44454
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: upgrade all ceph daemon types

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not specify --image arg for non-ceph daemons; fix upgrade

If we are calling the cephadm script for a non-ceph daemon (prometheus,
etc), do not specify the --image argument, and do not pull it out of
the config db from sections that don't exist.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix error-path return values for _run_cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix RGWSpec loading

Fixes ServiceSpec.from_json

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix 'orch rm'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33766 into master

* refs/pull/33766/head:
mgr/cephadm: fix placement when existing + specified don't overlap

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: make osd create on an existing LV idempotent

If we try to prepare an LV that was already prepared, ceph-volume will
return an error message and code.  We want our osd create command to be
idempotent, though, so recognize the error string and continue.

This is an ugly hack, but quicker than changing ceph-volume behavior, and
it is sufficient to stop all of the teuthology failures.

The second part of this is that we have to deploy the daemon on OSDs that
are already prepared and already exist in our osdmap beforehand, but have
never started.

Works-around: https://tracker.ceph.com/issues/44313
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33744 into master

* refs/pull/33744/head:
qa/tasks/radosbench: fix usage of -O

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: fix placement when existing + specified don't overlap

If we have a daemon on A, and our spec is (count=2, hosts=[B]), we should
always return [A,B], but we sometimes were returning [B,B].

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: OpSmException.error -> OpSmException.errno

fixes mypy errors:

volumes/fs/async_cloner.py: note: In function "handle_clone_pending":
volumes/fs/async_cloner.py:71: error: "OpSmException" has no attribute "error"; maybe "errno"?

volumes/fs/async_cloner.py: note: In function "handle_clone_in_progress":
volumes/fs/async_cloner.py:139: error: "OpSmException" has no attribute "error"; maybe "errno"?

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: remove unneeded assignment to `NoneType`

fixes mypy error:

volumes/fs/operations/versions/__init__.py: note: In member "get_subvolume_object" of class "SubvolumeLoader":
volumes/fs/operations/versions/__init__.py:70: error: Incompatible types in assignment (expression has type "None", variable has type "SubvolumeBase")

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: add missing OpSmException import

fixes mypy error:

volumes/fs/operations/versions/__init__.py: note: In member "upgrade_legacy_subvolume" of class "SubvolumeLoader":
volumes/fs/operations/versions/__init__.py:56: error: Name 'OpSmException' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: add missing error code

fixes mypy error:

volumes/fs/operations/versions/__init__.py: note: In member "_load_supported_versions" of class "SubvolumeLoader":
volumes/fs/operations/versions/__init__.py:35: error: Too few arguments for "VolumeException"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: remove stray return

fixes mypy errors:

volumes/fs/purge_queue.py: note: In function "get_trash_entry_for_volume":
volumes/fs/purge_queue.py:28: error: Name 'ret' is not defined

volumes/fs/async_cloner.py: note: In function "get_next_clone_entry":
volumes/fs/async_cloner.py:39: error: Name 'ret' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: fixup format string args

fixes mypy errors:

volumes/fs/purge_queue.py:26: error: Cannot find replacement for positional format specifier 1
volumes/fs/async_cloner.py:37: error: Cannot find replacement for positional format specifier 1

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: add missing `mgr` param

fixes mypy errors:

volumes/fs/operations/volume.py: note: In function "create_volume":
volumes/fs/operations/volume.py:216: error: Too few arguments for "remove_pool"
volumes/fs/operations/volume.py:223: error: Too few arguments for "remove_pool"
volumes/fs/operations/volume.py:224: error: Too few arguments for "remove_pool"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: assert self.fs

fixes mypy errors:

volumes/fs/operations/volume.py: note: In member "disconnect" of class "Connection":
volumes/fs/operations/volume.py:94: error: Item "None" of "Optional[Any]" has no attribute "get_addrs"
volumes/fs/operations/volume.py:95: error: Item "None" of "Optional[Any]" has no attribute "shutdown"

volumes/fs/operations/volume.py: note: In member "abort" of class "Connection":
volumes/fs/operations/volume.py:105: error: Item "None" of "Optional[Any]" has no attribute "abort_conn"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: skip type checking on RTimer class

Unclear why mypy does not like this:

volumes/fs/operations/volume.py: note: In member "run" of class "RTimer":
volumes/fs/operations/volume.py:118: error: "RTimer" has no attribute "finished"
volumes/fs/operations/volume.py:119: error: "RTimer" has no attribute "finished"
volumes/fs/operations/volume.py:119: error: "RTimer" has no attribute "interval"
volumes/fs/operations/volume.py:120: error: "RTimer" has no attribute "function"
volumes/fs/operations/volume.py:120: error: "RTimer" has no attribute "args"
volumes/fs/operations/volume.py:120: error: "RTimer" has no attribute "kwargs"
volumes/fs/operations/volume.py:121: error: "RTimer" has no attribute "finished"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: fixup threading.Timer import

Python 3.3 changed `Timer` from a factory function to a class
https://docs.python.org/3.8/library/threading.html#timer-objects

fixes mypy error:

volumes/fs/operations/volume.py:9: error: Module 'threading' has no attribute '_Timer'; maybe "Timer"?

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: fix positional str formatting

fixes mypy error:

volumes/fs/operations/group.py: note: In function "create_group":
volumes/fs/operations/group.py:135: error: Not all arguments converted during string formatting

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: place getters and setters next to each other

workaround for mypy issue:
https://github.com/python/mypy/issues/1465

fixes mypy errors:

volumes/fs/operations/group.py: note: In class "Group":
volumes/fs/operations/group.py:44: error: Name 'uid' already defined on line 36
volumes/fs/operations/group.py:44: error: "Callable[[Group], Any]" has no attribute "setter"
volumes/fs/operations/group.py:48: error: Name 'gid' already defined on line 40
volumes/fs/operations/group.py:48: error: "Callable[[Group], Any]" has no attribute "setter"

volumes/fs/operations/group.py: note: In function "open_group":
volumes/fs/operations/group.py:170: error: Property "uid" defined in "Group" is read-only
volumes/fs/operations/group.py:171: error: Property "gid" defined in "Group" is read-only

volumes/fs/operations/versions/subvolume_base.py: note: In class "SubvolumeBase":
volumes/fs/operations/versions/subvolume_base.py:45: error: Name 'uid' already defined on line 33
volumes/fs/operations/versions/subvolume_base.py:45: error: "Callable[[SubvolumeBase], Any]" has no attribute "setter"
volumes/fs/operations/versions/subvolume_base.py:49: error: Name 'gid' already defined on line 37
volumes/fs/operations/versions/subvolume_base.py:49: error: "Callable[[SubvolumeBase], Any]" has no attribute "setter"
volumes/fs/operations/versions/subvolume_base.py:53: error: Name 'mode' already defined on line 41
volumes/fs/operations/versions/subvolume_base.py:53: error: "Callable[[SubvolumeBase], Any]" has no attribute "setter"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: reverse params passed to `isinstace()`

fixes mypy error:

volumes/fs/operations/clone_index.py: note: In member "track" of class "CloneIndex":
volumes/fs/operations/clone_index.py:38: error: Argument 2 to "isinstance" has incompatible type "Union[VolumeException, Any]"; expected "Union[type, Tuple[Union[type, Tuple[Any, ...]], ...]]"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: fix arg typo '.' -> ','

fixes mypy error:

volumes/fs/fs_util.py: note: In function "get_ancestor_xattr":
volumes/fs/fs_util.py:129: error: Too few arguments for "VolumeException"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: add `deque` type

fixes mypy error:

volumes/fs/async_job.py: note: In member "__init__" of class "AsyncJobs":
volumes/fs/async_job.py:97: error: Need type annotation for 'q'

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: `threading.Event.isSet` no longer exists in py3

fixes mypy errors:

volumes/fs/async_job.py: note: In member "should_cancel" of class "JobThread":
volumes/fs/async_job.py:70: error: "Event" has no attribute "isSet"

volumes/fs/volume.py: note: In member "is_stopping" of class "VolumeClient":
volumes/fs/volume.py:59: error: "Event" has no attribute "isSet"

volumes/fs/volume.py: note: In member "list_fs_volumes" of class "VolumeClient":
volumes/fs/volume.py:106: error: "Event" has no attribute "isSet"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: assert JobThread class

fixes mypy error:

volumes/fs/async_job.py: note: In member "run" of class "JobThread":
volumes/fs/async_job.py:43: error: "Thread" has no attribute "should_cancel"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: import VolumeException

fixes mypy errors:

volumes/fs/operations/index.py: note: In member "track" of class "Index":
volumes/fs/operations/index.py:19: error: Name 'VolumeException' is not defined

volumes/fs/operations/index.py: note: In member "untrack" of class "Index":
volumes/fs/operations/index.py:22: error: Name 'VolumeException' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: use `SafeConfigParser` with py2

Python3.2 renamed `SafeConfigParser` -> `ConfigParser`

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: use python version during configparser import

fixes mypy error:

volumes/fs/operations/versions/metadata_manager.py:8: error: Name 'configparser' already defined (by an import)

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: add `Dict` type

fixes mypy errors:

volumes/fs/operations/op_sm.py:39: error: "object" has no attribute "get"
volumes/fs/operations/op_sm.py:49: error: "object" has no attribute "get"

volumes/fs/operations/lock.py: note: In member "__init__" of class "GlobalLock":
volumes/fs/operations/lock.py:27: error: "object" has no attribute "__enter__"
volumes/fs/operations/lock.py:27: error: "object" has no attribute "__exit__"

volumes/fs/operations/lock.py: note: In member "lock_op" of class "GlobalLock":
volumes/fs/operations/lock.py:35: error: "object" has no attribute "__enter__"
volumes/fs/operations/lock.py:35: error: "object" has no attribute "__exit__"

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: import errno

fixes mypy errors:

volumes/fs/operations/op_sm.py:36: error: Name 'errno' is not defined
volumes/fs/operations/op_sm.py:39: error: Name 'errno' is not defined
volumes/fs/operations/op_sm.py:46: error: Name 'errno' is not defined
volumes/fs/operations/op_sm.py:49: error: Name 'errno' is not defined
volumes/fs/operations/template.py:5: error: Name 'errno' is not defined
volumes/fs/operations/template.py:14: error: Name 'errno' is not defined
volumes/fs/operations/template.py:23: error: Name 'errno' is not defined
volumes/fs/operations/template.py:32: error: Name 'errno' is not defined
volumes/fs/operations/template.py:42: error: Name 'errno' is not defined
volumes/fs/operations/template.py:45: error: Name 'errno' is not defined
volumes/fs/operations/template.py:62: error: Name 'errno' is not defined
volumes/fs/operations/template.py:74: error: Name 'errno' is not defined
volumes/fs/operations/template.py:85: error: Name 'errno' is not defined
volumes/fs/operations/template.py:94: error: Name 'errno' is not defined
volumes/fs/operations/template.py:103: error: Name 'errno' is not defined
volumes/fs/operations/template.py:112: error: Name 'errno' is not defined
volumes/fs/operations/template.py:121: error: Name 'errno' is not defined
volumes/fs/operations/template.py:130: error: Name 'errno' is not defined
volumes/fs/operations/template.py:139: error: Name 'errno' is not defined
volumes/fs/operations/template.py:148: error: Name 'errno' is not defined
volumes/fs/operations/template.py:158: error: Name 'errno' is not defined
volumes/fs/operations/template.py:169: error: Name 'errno' is not defined
volumes/fs/operations/template.py:180: error: Name 'errno' is not defined
volumes/fs/operations/index.py:18: error: Name 'errno' is not defined
volumes/fs/operations/index.py:21: error: Name 'errno' is not defined

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/volumes: add `mgr/volumes` to the mypy checks

Fixes: https://tracker.ceph.com/issues/44393
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #33531 into master

* refs/pull/33531/head:
qa/tasks/mgr/test_orchestrator_cli: fix test_ps* tests
mgr/test_orchestrator: refactor listing services/daemons
mgr/dashboard: list services and daemons

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #33718 into master

* refs/pull/33718/head:
mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #33745 into master

* refs/pull/33745/head:
mgr/cephadm: upgrade: fix daemons missing image_id

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/valgrind.supp: abstract from ceph::buffer's symbol versioning.

Analysis: https://tracker.ceph.com/issues/44430#note-1
Fixes: https://tracker.ceph.com/issues/44430
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33702 from ivancich/wip-bucket-listing-clear-results

rgw: cls_bucket_list_(un)ordered should clear results collection

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

rgw: cls_bucket_list_(un)ordered should clear results collection

Each call to cls_bucket_list_(un)ordered should have an empty
collection to populate with results. Rather than rely on the caller to
insure this, it's more reliable to have these functions do the clear.

Additionally in some cases, a reserve call was added to the collection
to pre-allocate the space needed for the expected number of
results. This will potentially result in fewer re-allocations plus
copies.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: fix string_view formatting in RGWFormatter_Plain

two string_views were being passed directly to vsnprintf where it
expected null-terminated strings. the compiler didn't catch this, and
resulted in segfaults

string_views aren't guaranteed to be null-terminated, so printf formats
have to specify a length as well

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33693 from yaozongyou/fix-rgw-check-bucket-empty-bug

rgw: clear ent_list for each loop of bucket list

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33398 from clwluvw/subuser-policy

rgw: Fix bug on subuser policy identity checker

Reviewed-by: Adam Emerson <aemerson@redhat.com>

mgr/dashboard: fix tasks.mgr.dashboard.test_rgw suite

Fixes: https://tracker.ceph.com/issues/44405
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #33687 into master

* refs/pull/33687/head:
mgr/volume: adapt arg passing to ServiceSpec

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #33728 into master

* refs/pull/33728/head:
mgr/orch: factor out nice_delta
mgr/orch: show spec age in 'orch ls'
mgr/cephadm: store timestamp with specs
mgr/orch: add created timestamp to ServiceDescription
mgr/orch: include uptime in 'orch ps'
mgr/orch: include AGE column in 'orch ps'
mgr/cephadm: populate new DaemonDescription timestamps
mgr/orch: add new timestamps in DaemonDescription
cephadm: include timestamps for configured, created
cephadm: include timestamps for started, deploy

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33708 into master

* refs/pull/33708/head:
ceph_test_cls_hello: set RETURNVEC on the expected EINVAL request

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix test_ps* tests

The ps output names daemons like 'type.foo', e.g., 'mgr.x'.  Now that
the test_orchestrator impl is less bonkers this needs to be adjusted to
match reality.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: fix daemons missing image_id

If a daemon isn't running, we don't know the image_id (hash), so we skip.
But it's also possible to have a running daemon that doesn't report an
image_id... like right after we deploy it when the container hasn't
started up yet.  Skip those too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33729 into master

* refs/pull/33729/head:
doc/releases: Octopus is not stable yet

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/radosbench: fix usage of -O

I tried to fix this in fb991fa5f62c4aaa6c564a6d15aeee4bfaa33b39 but
was comparing an array to an int.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: factor out nice_delta

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: show spec age in 'orch ls'

Also rearrange the columns a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: store timestamp with specs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add created timestamp to ServiceDescription

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include uptime in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include AGE column in 'orch ps'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: populate new DaemonDescription timestamps

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add new timestamps in DaemonDescription

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include timestamps for configured, created

created = when daemon was first created
configured = when the daemonw as last (re)configured

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: include timestamps for started, deploy

started == when the container started
deployed == when the container was (most recently) (re)deployed

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33733 into master

* refs/pull/33733/head:
mgr/cephadm: if we had no record of deps, and deps are [], do not reconfig

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33734 into master

* refs/pull/33734/head:
mgr/cephadm: do not refresh daemon and device inventory as often

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33736 into master

* refs/pull/33736/head:
mgr/cephadm: upgrade crash collectors too
mgr/cephadm: fix service rm when there are no daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: enable custom TLS certificates for grafana

using `ceph config-key`.

E.g.:

    ceph config-key set mgr/cephadm/grafana_crt -i cert.pem
    ceph config-key set mgr/cephadm/grafana_key -i key.pem

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr: enable verification of TLS certs without files

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: dump config to JSON only once when creating daemons

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #33549 from rhcs-dashboard/44307-e2e-custom-credentials

mgr/dashboard: ability to provide custom credentials for E2E tests.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30224 from chardan/jfw-wip-random-deduction-guide

common: random: added a deduction guide to make using the function ob…

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32495 from toabctl/qa-mypy

qa: Add basic mypy support for the qa directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: Enable basic mypy support for qa/ directory

A first step to do more automatic code checks on the qa/
directory. This is useful while transitioning to python3.

Also use log_exc to top-level to not run into:

error: Argument 1 to "log_exc" has incompatible type
  "Callable[[OSDThrasher], Any]"; expected "OSDThrasher"

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

qa: Fix problems detected by mypy

This is a first step to enable mypy on the qa/ directory.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>