qa/suites: some clean up for quincy

- remove upgrades from nautilus
- stubs for completing upgrade to quincy

Still missing the pacific-x upgrade tests.

Signed-off-by: Sage Weil <sage@newdream.net>

mon: updates for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

mon: update for quincy ondisk features

Signed-off-by: Sage Weil <sage@newdream.net>

script: add pacific

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklist: we tagged v17.0.0

Signed-off-by: Sage Weil <sage@newdream.net>

ceph-volume: change to quincy

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: retire MON_SINGLE_PAXOS

X-3 = Q-3 = Nautilus

Signed-off-by: Sage Weil <sage@newdream.net>

include/ceph_features: define FEATURE_SERVER_QUINCY

Signed-off-by: Sage Weil <sage@newdream.net>

mon/MgrMonitor: add always_on_modules for quincy

Signed-off-by: Sage Weil <sage@newdream.net>

add feature/release name quincy

Signed-off-by: Sage Weil <sage@newdream.net>

kickoff v17 quincy

Signed-off-by: Sage Weil <sage@newdream.net>

doc/dev/release-checklists: uncheck everything!

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38861 from alimaredia/wip-rgw-create-bucket-dpp

rgw: Thread dpp through Create Bucket logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39105 from dillaman/wip-48989

librbd: tweaks to support Boost 1.75.0

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add DoutPrefixProvider to create_bucket logging

This commit hits all log messages for a create bucket
request and ensures each has a DoutPrefixProvider.

This commit gets dpp's all the way down to rgw/services
up way though to rgw_op.cc, rgw_rest.cc files,
rgw_admin.cc, etc.

This commit also removes store as dpp in rgw_sal.h

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #39068 from dillaman/wip-librbd-migration-qcow-2

librbd/migration: tweaks and unit tests for QCOW handler

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39107 from yuriw/wip-yuriw-crontab-master

qa/tests: reduced octopus, nautilus, master frequency

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39043 from tchaikov/wip-cephadm-call

cephadm: refactor call() using asyncio.asyncio.StreamReader

Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: changed priority - master 100, pacific 80

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tests: reduced octopus, nautilus, master frequency

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #39015 from lixiaoy1/fix_lib

librbd/cmake: add library dependency for pwl_cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39101 from sebastian-philipp/cephadm-fix-ok-to-stop-return-value

mgr/cephadm: Fix wrong argument type to HandleCommandResult

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39098 from tchaikov/wip-zabbix-ann

mgr: add annotations to zabbix and devicehealth

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39065 from cbodley/wip-48988

cmake: boost>=1.74 adds BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT to radosgw

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38786 from kotreshhr/evict-clients-of-deauthorized-authids

mgr/volumes: Evict clients based on auth-IDs and subvolume path

Reviewed-by: Victoria Martinez de la Cruz <victoria@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

librbd: tweaks to support Boost 1.75.0

Fixes: https://tracker.ceph.com/issues/48989
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: minor cleanup to QCOWFormat code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37259 from soumyakoduri/head_req

rgw: Set CURLOPT_NOBODY for HEAD request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38982 from jmolmo/fix_iscsi_client_caps

cephadm: Fix iscsi client caps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39099 from ZhenLiu94/liuz

doc: Correct spelling mistakes.

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Fix wrong argument type to HandleCommandResult

the argument must be a string

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38854 from adk3798/ots-monitoring

mgr/cephadm: force flag for ok-to-stop and ok-to-stop for monitoring stack

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38998 from sebastian-philipp/fix-serve-handle-json-error

mgr/cephadm: Properly handle JSON Decode error

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38764 from rhcs-dashboard/fix-48748-master

mgr/dashboard: trigger alert if some nodes have a MTU different than the median value

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/devicehealth: return -1 if prediction plugin is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: remove unused variable

silences flake8 warning F841.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: remove handle_command()

as commands are collected by COMMANDS already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use CLICommand for command handling

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: do not catch exception with name

silences flake8 warning of F841:
https://www.flake8rules.com/rules/F841.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: cast port to int

to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: talk to zabbix_sender in text not bytes

as we expect read from / write to it in text mode.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use "uint" for type of discovery_interval

we only support following types:

'uint', 'int', 'str', 'float', 'bool', 'addr', 'addrvec', 'uuid',
'size', 'secs',

see also str_to_type()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: add annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use Option for adding options

more type safety

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: add more annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: correct annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Correct spelling mistakes.

Signed-off-by: ZhenLiu94 <zhenliu94@163.com>

Merge pull request #39093 from rzarzynski/wip-crimson-guard-op-lifetime

crimson: fix dangling ClientRequest::this pointer.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39087 into master

* refs/pull/39087/head:
qa/suites/rados/cephadm: drop 20.04 podman
qa/suites/rados/cephadm: stop testing 18.04 + podman

Reviewed-by: Sebastian Wagner <swagner@suse.com>

crimson: fix dangling ClientRequest::this pointer.

`ShardServices::start_operation<T>()` spawns an operation,
calls `start()` on it and returns a future plus a smart
pointer controlling its life-time. Callers are responsible
to ensure the pointer doesn't go out-of-scope before
the entire execution is finished. This is error-prone.

`OSD::handle_osd_op()` forgets about its responsibility
which results in dangling `this` pointer of `ClientRequest`.
I believe the problem is much wider spread and the class
is just the tip of the iceberg.

In this commit `start_operation<T>()` is altered to extend
the life-time a bit. However, this isn't an ultimate solution
as callers are still able to e.g. put extra `this`-catching
lambdas on the returned future. A new `with_operation<T>()`-
like interface is expected as a follow-up.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39086 from orozery/qa-rbd-delete-passphrase-file

qa/tasks/rbd: cleanup passphrase file

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39089 from orozery/librbd-luks-cli-test

test/librbd: fix luks encryption cli test on unsupported runs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39091 from tchaikov/wip-crimson-build

cmake: let alienstore link against zoned allocator

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: let alienstore link against zoned allocator

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #39028 into master

* refs/pull/39028/head:
cephadm: retry for 30s instead of 10s
cephadm: try 'mgr stat' before 'mgr dump' to get the epoch
mon: add 'mgr stat' command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

test/librbd: fix luks encryption cli test on unsupported runs

This commit changes the luks encryption cli test to return success instead of failure when not supported.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: initial unit tests for migration/QCOWFormat

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: compute QCOW snapshot deltas via L2 table deltas

Since each entry in a L2 table represents a cluster block within the
QCOW image, we can compute deltas by comparing L2 tables. This fixes
an issue where snapshots were not being properly thin-provisioned
during the live-migration execute process.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39077 from tchaikov/wip-libzbd

install-deps.sh: set codename when installing libzbd

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #38935 from tchaikov/wip-mgr-ann

pybind/mgr/mgr_module: annotate more methods and add poll to CLICommand

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kamoltat <ksirivad@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/cephadm: drop 20.04 podman

This version is also affected

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rbd: cleanup passphrase file

This commit adds a cleanup for the passphrase file used to create/mount encrypted rbd images.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39022 from petrutlucian94/rbd_fixes

Windows RBD fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38984 from petrutlucian94/dll

Windows DLL support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39073 from tchaikov/wip-rbd-bench

rbd/bench: include used headers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39049 from lixiaoy1/fix_spec

spec: enable RWL/SSD caches

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39044 from orozery/librbd-crypto-close-bug

librbd: do not share crypto image layers with ancestors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39006 from orozery/librbd-luks-cli-test

test/librbd: add luks encryption cli test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39005 from orozery/librbd-plain64-iv-fix

librbd: use 512 sector size for crypto IV

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39064 from idryomov/wip-doc-osdtimeout

doc: don't mention osdtimeout option in mount.ceph man page

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #39060 from idryomov/wip-krbd-msgr2

krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: set codename when installing libzbd

this was an oversight in e7e3b86762ff3fccdae21e713c64215ab3bb7bb4

also use updated build with sha1 of
1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.

and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: return -ENOENT if self.ssh_user is None

there is chance that self.ssh_user is None when handling "cephadm
get-user" command, so return -ENOENT and error messages in that case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: use _get_localized_key for composing store key

Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: add assert to appease mypy

self.get_osd_perf_counters() might return None, add assert() to assure
mypy that it does not return None here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/serve: add assert to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/module: assure self.ssh_user is not None

to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/inventory: cast variable to expected type

to silence mypy warnings like:

cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: do not pass --use-feature=2020-resolver to pip

pip use the new resolver by default now. so no need to pass this option
to it anymore.

This reverts commit fa9e2bfd4b3648f08ed3a88ce737d432ab97cce1.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: annotate variable type

to silence following mypy error:

cephadm/module.py:1155: error: Incompatible types in assignment
(expression has type "None", variable has type "str")

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: rename variables to avoid name reuse

to applease mypy, otherwise, for instance, "err" would be reused for
holding the return value of self.mgr.check_mon_command(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: annotate OSDMap and friends

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()

it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: define HandleCommandResult using NamedTuple

simpler this way, so we don't need to define __new__

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: use numeric values of logging level for index

less magic values in code this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: use dict instead of defaultdict to silence mypy warning

otherwise we have:

1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: handle the case where given osd is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: pass str as the 2nd param of set_store()

MgrModule.set_store() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass a str as the value of option to set_module_option()

MgrModule.set_module_option() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass id to constructor of Event

to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:

progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: be PEP8 compliant

to applease flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: use CLICommand to define command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: minor cleanups

* do not pass typename and self to super()
* do not add parentheses around condition expression

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: use correct types

for better readability and for the sake of correctness.

* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/iostat: use CLICommand to define command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: annotate more methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: refactor call() using asyncio.asyncio.StreamReader

simpler this way, also fix a couple issues:

* create a child watcher explicitly, see
  https://bugs.python.org/issue35621
* use StringIO for collecting outputs for better performance,
  instead of appending the lines to an existing str
* catch ValueError when reading from the stream reader,
  because StreamReader.readline() could raise ValueError when
  it reaches the buffer limit while looking for a separator.
  in this case, we should try again, in hope that the spawned
  process can feed the reader with more data which contains
  the separator (i.e., b'\n').
* backport ThreadedChildWatcher from Python 3.8 so we can
  run create_subprocess_exec() in non-main threads.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38929 from jdurgin/wip-cot-log

qa/tasks/ceph_manager: let c-o-t log errors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38931 from trociny/wip-48821

osd: fix potential null pointer dereference when sending ping

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38901 from gmayyyha/io_uring-48872

blk/kernel: fix io_uring got (4) Interrupted system call

Reviewed-by: Kefu Chai <kchai@redhat.com>