cephadm: fix get_data_dir calls for ha-rgw daemons

calls need to use the context's data dir rather than daemon type
puts these calls in line with the other daemons

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #39113 from liewegas/fix-datetime-test

python-common: fix test_datetime_to_str_2 on non-UTC hosts

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #39100 from xxhdx1985126/wip-crimson-bug-fix-2

crimson/osd: fix recovery related bugs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use do_with to capture MOSDPGPull::take_pulls()'s return

MOSDPGPull::take_pulls() returns a temporary object, so we need to use seastar::do_with
to create an object whose lives until seastar::do_with's inner continuation chain ends

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: objects without omap header may not necessarily be a error

There are circumstances in which objects don't have omap headers, so this should
not be treated as an error

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: should capture seastar::do_with params by reference

The lambda within seastar::do_with need to use lvalue reference as its parameter,
otherwise it would be referencing a variable whose lifecycle ends when lambda finishes
runing

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #39115 from wjwithagen/wjw-fix-QCOWFormat.cc

librbd: Fix variable declaration in capture list in lambda

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39108 from zdover23/wip-doc-leads-update-epuertat-28-Jan-2021

doc/dev: add Ernesto Puerta as dashboard lead

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #39114 from liewegas/drop-rhel-8.0

qa/suites/rados/cephadm: drop rhel_8.0 tests

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #37949 from ideepika/wip-docs-update

doc/dev/developer_guide:  teuthology update developer guide

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: Fix variable declaration in capture list in lambda

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

qa/suites/rados/cephadm/*/distro: drop rhel_8.0 targets

We still have centos 8.0.  We're missing an 8.0 image for the gibba.

Signed-off-by: Sage Weil <sage@newdream.net>

python-common: fix test_datetime_to_str_2 on non-UTC hosts

The old test parsed to a datetime without a tz, which was interpreted as
the local time zone when rendering back to a string.  Specify that it's a
UTC datetime so that behavior is consistent regardless of the test host
timezone.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38861 from alimaredia/wip-rgw-create-bucket-dpp

rgw: Thread dpp through Create Bucket logging

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #39105 from dillaman/wip-48989

librbd: tweaks to support Boost 1.75.0

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add DoutPrefixProvider to create_bucket logging

This commit hits all log messages for a create bucket
request and ensures each has a DoutPrefixProvider.

This commit gets dpp's all the way down to rgw/services
up way though to rgw_op.cc, rgw_rest.cc files,
rgw_admin.cc, etc.

This commit also removes store as dpp in rgw_sal.h

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #39068 from dillaman/wip-librbd-migration-qcow-2

librbd/migration: tweaks and unit tests for QCOW handler

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #39107 from yuriw/wip-yuriw-crontab-master

qa/tests: reduced octopus, nautilus, master frequency

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #39043 from tchaikov/wip-cephadm-call

cephadm: refactor call() using asyncio.asyncio.StreamReader

Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/tests: changed priority - master 100, pacific 80

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tests: reduced octopus, nautilus, master frequency

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #39015 from lixiaoy1/fix_lib

librbd/cmake: add library dependency for pwl_cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/dev: add Ernesto Puerta as dashboard lead

This commit removes Lenz Grimmer's name and github
handle from the list of component leads and replaces
it with Ernesto Puerta's name and github handle.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #39101 from sebastian-philipp/cephadm-fix-ok-to-stop-return-value

mgr/cephadm: Fix wrong argument type to HandleCommandResult

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #39098 from tchaikov/wip-zabbix-ann

mgr: add annotations to zabbix and devicehealth

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39065 from cbodley/wip-48988

cmake: boost>=1.74 adds BOOST_ASIO_USE_TS_EXECUTOR_AS_DEFAULT to radosgw

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38786 from kotreshhr/evict-clients-of-deauthorized-authids

mgr/volumes: Evict clients based on auth-IDs and subvolume path

Reviewed-by: Victoria Martinez de la Cruz <victoria@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

librbd: tweaks to support Boost 1.75.0

Fixes: https://tracker.ceph.com/issues/48989
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: minor cleanup to QCOWFormat code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37259 from soumyakoduri/head_req

rgw: Set CURLOPT_NOBODY for HEAD request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38982 from jmolmo/fix_iscsi_client_caps

cephadm: Fix iscsi client caps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #39099 from ZhenLiu94/liuz

doc: Correct spelling mistakes.

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: Fix wrong argument type to HandleCommandResult

the argument must be a string

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38854 from adk3798/ots-monitoring

mgr/cephadm: force flag for ok-to-stop and ok-to-stop for monitoring stack

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38998 from sebastian-philipp/fix-serve-handle-json-error

mgr/cephadm: Properly handle JSON Decode error

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38764 from rhcs-dashboard/fix-48748-master

mgr/dashboard: trigger alert if some nodes have a MTU different than the median value

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/devicehealth: return -1 if prediction plugin is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: remove unused variable

silences flake8 warning F841.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: remove handle_command()

as commands are collected by COMMANDS already.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use CLICommand for command handling

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: do not catch exception with name

silences flake8 warning of F841:
https://www.flake8rules.com/rules/F841.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: cast port to int

to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: talk to zabbix_sender in text not bytes

as we expect read from / write to it in text mode.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use "uint" for type of discovery_interval

we only support following types:

'uint', 'int', 'str', 'float', 'bool', 'addr', 'addrvec', 'uuid',
'size', 'secs',

see also str_to_type()

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: add annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/zabbix: use Option for adding options

more type safety

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: add more annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: correct annotation

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: Correct spelling mistakes.

Signed-off-by: ZhenLiu94 <zhenliu94@163.com>

Merge pull request #39093 from rzarzynski/wip-crimson-guard-op-lifetime

crimson: fix dangling ClientRequest::this pointer.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39087 into master

* refs/pull/39087/head:
qa/suites/rados/cephadm: drop 20.04 podman
qa/suites/rados/cephadm: stop testing 18.04 + podman

Reviewed-by: Sebastian Wagner <swagner@suse.com>

crimson: fix dangling ClientRequest::this pointer.

`ShardServices::start_operation<T>()` spawns an operation,
calls `start()` on it and returns a future plus a smart
pointer controlling its life-time. Callers are responsible
to ensure the pointer doesn't go out-of-scope before
the entire execution is finished. This is error-prone.

`OSD::handle_osd_op()` forgets about its responsibility
which results in dangling `this` pointer of `ClientRequest`.
I believe the problem is much wider spread and the class
is just the tip of the iceberg.

In this commit `start_operation<T>()` is altered to extend
the life-time a bit. However, this isn't an ultimate solution
as callers are still able to e.g. put extra `this`-catching
lambdas on the returned future. A new `with_operation<T>()`-
like interface is expected as a follow-up.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #39086 from orozery/qa-rbd-delete-passphrase-file

qa/tasks/rbd: cleanup passphrase file

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39089 from orozery/librbd-luks-cli-test

test/librbd: fix luks encryption cli test on unsupported runs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39091 from tchaikov/wip-crimson-build

cmake: let alienstore link against zoned allocator

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: let alienstore link against zoned allocator

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #39028 into master

* refs/pull/39028/head:
cephadm: retry for 30s instead of 10s
cephadm: try 'mgr stat' before 'mgr dump' to get the epoch
mon: add 'mgr stat' command

Reviewed-by: Sebastian Wagner <swagner@suse.com>

test/librbd: fix luks encryption cli test on unsupported runs

This commit changes the luks encryption cli test to return success instead of failure when not supported.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

test/librbd: initial unit tests for migration/QCOWFormat

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: compute QCOW snapshot deltas via L2 table deltas

Since each entry in a L2 table represents a cluster block within the
QCOW image, we can compute deltas by comparing L2 tables. This fixes
an issue where snapshots were not being properly thin-provisioned
during the live-migration execute process.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39077 from tchaikov/wip-libzbd

install-deps.sh: set codename when installing libzbd

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #38935 from tchaikov/wip-mgr-ann

pybind/mgr/mgr_module: annotate more methods and add poll to CLICommand

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kamoltat <ksirivad@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/cephadm: drop 20.04 podman

This version is also affected

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rbd: cleanup passphrase file

This commit adds a cleanup for the passphrase file used to create/mount encrypted rbd images.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #39022 from petrutlucian94/rbd_fixes

Windows RBD fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38984 from petrutlucian94/dll

Windows DLL support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39073 from tchaikov/wip-rbd-bench

rbd/bench: include used headers

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39049 from lixiaoy1/fix_spec

spec: enable RWL/SSD caches

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39044 from orozery/librbd-crypto-close-bug

librbd: do not share crypto image layers with ancestors

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39006 from orozery/librbd-luks-cli-test

test/librbd: add luks encryption cli test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39005 from orozery/librbd-plain64-iv-fix

librbd: use 512 sector size for crypto IV

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #39064 from idryomov/wip-doc-osdtimeout

doc: don't mention osdtimeout option in mount.ceph man page

Reviewed-by: Jeff Layton <jlayton@redhat.com>

doc/dev/developer_guide/basic-workflow: fix integration tests link

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

doc/dev: improving teuthology docs

This commit is a proof of concept and a repair of
links to the "Integration Tests" section of the
Developer Guide.

This fixes the "reading a standalone test" link
from the "Running Tests in the Cloud" page
to the "Reading a Standalone Test" section in
the "Tests: Integration Tests" page.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/dev/developer_guide/testing_integration_tests: fix broken links and
update teuthology guide

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

doc/dev/developer_guide: rearrange and improve docs

* move running-tests-using-teuth.rst to
doc/dev/developer_guide/tests-integration-testing-teuthology-workflow.rst

* introduce developer's guide for Sentry and improve teuthology docs

* add teuthology debugging guide * create testing_integration_tests
subfolder for teuthology

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #39060 from idryomov/wip-krbd-msgr2

krbd: add support for msgr2 (kernel 5.11)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps.sh: set codename when installing libzbd

this was an oversight in e7e3b86762ff3fccdae21e713c64215ab3bb7bb4

also use updated build with sha1 of
1fadde94b08fab574b17637c2bebd2b1e7f9127b, the older version has an issue
where both libzbd and libzbd-dev packaged .pc file with the same path,
so they conflicted with each other. the new version addressed this
issue.

and install libzbd-dev instead of libzbd and libzbd-dev, as the package
name of the former could change if its so version is bumped, so use the
*-dev package is a safer choice from the perspective of testing the
dev build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: return -ENOENT if self.ssh_user is None

there is chance that self.ssh_user is None when handling "cephadm
get-user" command, so return -ENOENT and error messages in that case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/dashboard: use _get_localized_key for composing store key

Just for keeping this 'encapsulated', there's _get_localized_key helper
in mgr_module (there's also a set_localized_store but it only deals with
the active manager)

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: add assert to appease mypy

self.get_osd_perf_counters() might return None, add assert() to assure
mypy that it does not return None here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/serve: add assert to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/module: assure self.ssh_user is not None

to appease mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm/inventory: cast variable to expected type

to silence mypy warnings like:

cephadm/module.py:55: note: In module imported here,
cephadm/__init__.py:6: note: ... from here:
cephadm/inventory.py: note: In member "load" of class "SpecStore":
cephadm/inventory.py:130: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"
cephadm/inventory.py:131: error: Invalid index type "str" for "str"; expected type "Union[int, slice]"

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: do not pass --use-feature=2020-resolver to pip

pip use the new resolver by default now. so no need to pass this option
to it anymore.

This reverts commit fa9e2bfd4b3648f08ed3a88ce737d432ab97cce1.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: annotate variable type

to silence following mypy error:

cephadm/module.py:1155: error: Incompatible types in assignment
(expression has type "None", variable has type "str")

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: rename variables to avoid name reuse

to applease mypy, otherwise, for instance, "err" would be reused for
holding the return value of self.mgr.check_mon_command(...)

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: annotate OSDMap and friends

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: correct annotation for BasePyOSDMap._apply_incremental()

it's incorrect. as the underlying C implementation,
"osdmap_apply_incremental()", actually expects an instance of
`BasePyOSDMapIncrementalType`. which is mapped to
`BasePyOSDMapIncremental` in Python. and this class is wrapped using
`OSDMapIncremental` in mgr_module.py.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: define HandleCommandResult using NamedTuple

simpler this way, so we don't need to define __new__

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: use numeric values of logging level for index

less magic values in code this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: use dict instead of defaultdict to silence mypy warning

otherwise we have:

1: cephadm/module.py:30: note: In module imported here:
1: cephadm/serve.py: note: In member "_check_for_strays" of class "CephadmServe":
1: cephadm/serve.py:395: error: Argument "default" to "get_metadata" of "MgrModule" has incompatible type "defaultdict[str, None]"; expected "Optional[Dict[str, str]]"

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: handle the case where given osd is not found

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: pass str as the 2nd param of set_store()

MgrModule.set_store() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass a str as the value of option to set_module_option()

MgrModule.set_module_option() only accepts str or None as its second
parameter.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/progress: pass id to constructor of Event

to assure that the self.id is always valid, it helps to silence the
warnings from mypy like:

progress/module.py: note: In member "_refresh" of class "Event":
progress/module.py:46: error: Argument 1 to "update_progress_event" of "MgrModule" has incompatible type "Optional[str]"; expected "str"

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: be PEP8 compliant

to applease flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/insights: use CLICommand to define command

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: minor cleanups

* do not pass typename and self to super()
* do not add parentheses around condition expression

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/iostat: use correct types

for better readability and for the sake of correctness.

* do not cast denominator to float before diving by it
* use '//' for divding an int
* cast int to str as it will be used as an element in a list of str

Signed-off-by: Kefu Chai <kchai@redhat.com>