pybind/mgr/mgr_module: annotate variable type

to silence following mypy error:

cephadm/module.py:1155: error: Incompatible types in assignment
(expression has type "None", variable has type "str")

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: rename variables to avoid name reuse

to applease mypy, otherwise, for instance, "err" would be reused for
holding the return value of self.mgr.check_mon_command(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: annotate OSDMap and friends

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()

it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: define HandleCommandResult using NamedTuple

simpler this way, so we don't need to define __new__

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: use numeric values of logging level for index

less magic values in code this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: use dict instead of defaultdict to silence mypy warning

otherwise we have:

1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: handle the case where given osd is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: pass str as the 2nd param of set_store()

MgrModule.set_store() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass a str as the value of option to set_module_option()

MgrModule.set_module_option() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass id to constructor of Event

to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:

progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: be PEP8 compliant

to applease flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: use CLICommand to define command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: minor cleanups

* do not pass typename and self to super()
* do not add parentheses around condition expression

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: use correct types

for better readability and for the sake of correctness.

* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/iostat: use CLICommand to define command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: annotate more methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: define CLUSTER_LOG_PRIO* using enum.IntEnum

so the clog type / prio can be typed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: add poll param to CLICommand

and add it to the output of dump_cmd() call, if "poll" is missing
in the output of dump_cmd() call, ceph CLI consider the it as False.

but it'd be better to be explicit. so this field is always set.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr,pybind/mgr: use bool for poll in COMMANDS

use correctly typed variable helps with readability and less
error-prone.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: flake8 fixes

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39042 from tchaikov/wip-cram

test/run-cli-tests: use cram from github

Reviewed-by: Sage Weil <sage@redhat.com>

test/downloads: drop cram-0.5 tar ball

since we install cram from github, drop the tarball from repo

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/run-cli-tests: use cram from github

to address the failure of

  WARNING: Built wheel for cram is invalid: Metadata 1.2 mandates PEP 440 version, but '0.5.0ceph.2011-01-14' is not
Failed to build cram

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39035 from tchaikov/wip-cephadm-call

cephadm: rewrite call() with asyncio

Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: rewrite call() with asyncio

for better readability, also return 124 when subprocess times out

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #38974 into master

* refs/pull/38974/head:
qa/suites/rados/cephadm/upgrade: avoid allow_ptrace with old cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #39032 from adamemerson/wip-layout-initialize

rgw: Initialize Layout Logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39029 from dillaman/wip-48964

ocf: add support for mapping images within an RBD namespace

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39024 from dillaman/wip-librbd-q-release

librbd: bump minor version to match Q release

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39034 from liewegas/fix-vstart-init-ceph

init-ceph: fix python and library paths for vstart

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38957 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific, changed octopus number of runs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

init-ceph: fix python and library paths for vstart

The current version does not result in a functional mgr after restarting
from a vstart cluster.

Signed-off-by: Sage Weil <sage@newdream.net>

rgw: Initialize logs on layout initialization

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #39013 from MahatiC/wip-ssd-doc

doc: update rwl doc to include ssd config

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38942 from smithfarm/wip-bci-pacific

script: make backport scripts know about pacific

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #39003 into master

* refs/pull/39003/head:
cephadm: fix 2> syntax in unit.run

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

qa/tests: moved 'master' runs to 'gibba' machines

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38990 into master

* refs/pull/38990/head:
mgr/devicehealth: fix 'device monitoring on|off' return value

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

ocf: add support for mapping images within an RBD namespace

Support for RBD namespaces was not added to the OCF resource agent.
This commit fixes this oversight and also fixes a bug in handling
of the "rbd device list" output.

Fixes: https://tracker.ceph.com/issues/48964
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

script: make backport scripts know about pacific

Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: bump minor version to match Q release

This will assist layered projects like QEMU and libvirt
determine which features are supported.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38938 from rhcs-dashboard/rgw-user-validation

mgr/dashboard: Fix for incorrect validation in rgw user form

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #39016 from sebastian-philipp/cephadm-mypy-status-none

mgr/cephadm: mypy says: dd.status can be None

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38999 from tchaikov/wip-crimson-bootstrap

crimson/osd: fetch_config() before mkfs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cephadm: fix 2> syntax in unit.run

We need a space between the command (which ends with a container name)
and the 2> or else the 2 is considered part of the command.  E.g.,

! /usr/bin/podman rm -f ceph-a9a8c7ee-5b72-11eb-8f93-001a4aab830c-mon.a2> /dev/null

Fixes: 1bed46e4b0094863a119df59c6ae5f254c2e211d
Signed-off-by: Sage Weil <sage@newdream.net>

vstart.sh: print out osd mkfs command

for better understanding the progress of vstart, and also allows
developer to repeat the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: do not include unused header

MGetConfig.h is not used anywhere in this source file, so no need to
include it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not use __func__ in lambda

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use switch case for checking return code

also, since seastar supports returning plain value instead a ready
future, let's return plain value. simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32027 into master

* refs/pull/32027/head:
librados: avoid symbol versioning on Windows

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: Minor pep8 fix

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: mypy says: dd.status can be None

`DaemonDescription.status` is annotated
as `Optional[int]` and thus can be `None`

This is a conflict between c95ba878c66aae44816b1451049449685d444f0a
and 01f60cf4e0a751c314120c02956d4ff941eb71b4

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38978 from liewegas/cephadm-daemon-health

mgr/cephadm: raise HEALTH_WARN when cephadm daemon in 'error' state

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38967 from liewegas/wip-cephadm-pacific

cephadm: latest stable release is now pacific

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38739 from jecluis/wip-cephadm-splits

cephadm: splits bootstrap function, add context, drop global variables

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: update rwl doc to include ssd config

... options.

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

crimson/osd: fetch_config() before mkfs

* fetch_config() before mkfs and starting osd
  for populating settings related to booting and transport layer
  before it starts.
* set fsid read from monitor before mkfs
  it's crucial to mkfs if osd is supposed to retrieve the fsid
  from monitor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: move implementation of dtor into .cc file

ceph_assert() is expanded into 5 lines of code. it'd help to speed up
the compiling a little bit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: add mon::Client::wait_for_config()

just for waiting for monmap and config from mon. crimson-osd needs this
for populating settings related to booting and transport layer before it
starts.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: fallback to msgr v2 for unbound msgr

so, for instance, if we want to connect to monitor without bind to any
address, we can try to use the v2 addresses advertised in monmap or
local settings, instead of being unable to connect to mon because we are
using an `entity_addr_t::TYPE_NONE` address which is returned by
`entity_addrvec_t::front()` if the addrvec is empty.

see also AsyncMessenger::should_use_msgr2().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: print out entity addr type when non peer address matches

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39010 from mdw-at-linuxbox/wip-master-barbicany2021

qa/tasks/barbican.py: fix year2021 problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39008 from mdw-at-linuxbox/wip-master-vaultunzip

qa/tasks/vault.py: unzip: try harder to find a working unzip.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Since we aren't using these anyway

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38988 from orozery/librbd-crypto-empty-copyup

librbd: don't restart empty copyups in crypto layer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38985 from orozery/rbd-nbd-luks2-blksize

tools/rbd-nbd: change block size to 4K for LUKS2 images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38983 from orozery/qa-qemu-rbd-encryption

qa/tasks/rbd: increase image size of encrypted disks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Fix for incorrect validation in rgw user form

The rgw users create form doesnt validate the username correctly if the username is a tenated one. For eg. Consider there is a user called tenate$sample. Now I am trying to create another user and I entered tenate$sample as username. But it doesn't async validate the username as existing. Instead it just shows the green tick and once the submit button is clicked it'll show the user as existing.

Fixes: https://tracker.ceph.com/issues/48907
Signed-off-by: Nizamudeen A <nia@redhat.com>

qa/tasks/barbican.py: fix year2021 problem

The expiration timestamp was hard-coded as
2020-12-31T19:14:44.180394
which is now in the past.  Instead, use a timestamp
90 minutes in the future.

Fixes: https://tracker.ceph.com/issues/48919
Signed-off-by: Marcus Watts <mwatts@redhat.com>

qa/tasks/vault.py: unzip: try harder to find a working unzip.

The existing logic uses "python -m zipfile" to unzip files.
This will (most likely) fail on CentOS 8-Stream , where python
defaults to 'unset' (see man unversioned-python).

So: try harder: try unzip, python3, and python in that order,
to find something that can unzip files.

Fixes: https://tracker.ceph.com/issues/48921
Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #38691 from soumyakoduri/lc_fixes

rgw/lc: Fix use-after-free in RGWLC::process

Merge pull request #38465 from IlsooByun/fix_lc_infinite_loop

rgw: lc: fix infinite loop in bucket_lc_prepare

Merge pull request #36698 from vshankar/wip-cephfs-mirror-tests

cephfs-mirror: teuthology task and tests

Merge pull request #35993 from vshankar/wip-fs-mirror-interface

pybind/mgr/mirroring: directory snapshot mirror support

qa/suites/rados/cephadm/upgrade: avoid allow_ptrace with old cephadm

Older cephadm is not smart enough to not combine --cap-add=SYS_PTRACE
and --privileged, which some version of podman cannot handle.

For upgrades, leave off the allow_ptrace behavior since we may be starting
on one of those old versions.

See also https://tracker.ceph.com/issues/46429

Fixes: https://tracker.ceph.com/issues/48142
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38991 from cbodley/wip-rgw-perm-state-valgrind

rgw: fix reference to temporary in perm_state ctor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38817 from ideepika/fix-interactive-error

qa/tasks/ceph: do not update info.yaml if ctx.archive is not set

Merge pull request #38941 from sseshasa/wip-48906-async-rec-min-cost-fix

osd: Remove override for osd_async_recovery_min_cost for mclock profiles

Reviewed-by: Sunny Kumar <sunkumar@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: avoid (rgw_file) illegal access to stack memory

Fixes: https://tracker.ceph.com/issues/48941
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/devicehealth: fix 'device monitoring on|off' return value

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38987 from cbodley/wip-qa-rgw-valgrind-supp-tls

valgrind: update suppression for _dl_allocate_tls

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: don't restart empty copyups in crypto layer

This commit fixes a bug where an empty parent copyup is restarted indefinitely.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38966 from rzarzynski/wip-qa-rgw-no-valgrind-early-exit

qa/task/rgw: instruct Valgrind to don't exit on first error.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

valgrind: update suppression for _dl_allocate_tls

in the rgw suite, there's an extra allocate_dtv() call in this stack.
add a ... to match it

  <stack>
    <frame>
      <ip>0x4C3321A</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>calloc</fn>
      <dir>/builddir/build/BUILD/valgrind-3.16.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>760</line>
    </frame>
    <frame>
      <ip>0x4012341</ip>
      <obj>/usr/lib64/ld-2.28.so</obj>
      <fn>allocate_dtv</fn>
    </frame>
    <frame>
      <ip>0x4012CD1</ip>
      <obj>/usr/lib64/ld-2.28.so</obj>
      <fn>_dl_allocate_tls</fn>
    </frame>
    <frame>
      <ip>0x101C7F32</ip>
      <obj>/usr/lib64/libpthread-2.28.so</obj>
      <fn>pthread_create@@GLIBC_2.2.5</fn>
    </frame>
    <frame>
      <ip>0x6F14E64</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>Thread::try_create(unsigned long)</fn>
    </frame>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cephadm: fix stuff

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: annotate variable type for mypy

mypy was complaining about the type of the variable being None, later on
on assignment with an int.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/test: support context class

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: have context handle config options

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: drop ctx class from where we don't need it

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/bootstrap: move mon net discovery to function

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: don't forget ipv6 behind

Fixes a bug introduced earlier in the patchset. It's just too hard to go
back and fix it up with the right commit.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off config work on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off dashboard setup on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off ssh preparations on bootstrap

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split-off mgr creation

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split mon prepare and create

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: move configuration out of the way

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm/bootstrap: spin-off waiting for mon

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

cephadm: split some of bootstrap to other functions

Signed-off-by: Joao Eduardo Luis <joao@suse.com>