pybind/mgr/mgr_module: annotate some methods

as a vehicle to understand the interface

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard/awsauth: drop py2 support

silences the warning from mypy:

dashboard/awsauth.py:42: error: Module 'base64' has no attribute
'encodestring'

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: use iterator of tuple to build dict

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: disable unsubscriptable-object warning from pylint

see https://github.com/PyCQA/pylint/issues/3882

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: PEP8 compliance

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: fix typo in comment

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: do not mock ceph_argparse

this module is used for convert argspec to argdesc by
CLICommand.load_func_metadata()

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: add python_path option to ceph-mgr-commands

so we can add more python paths in addition to the one used for finding
mgr modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: parse prefix and args for command signature

unlike the commands defined by C++, the commands defined by python
now uses "prefix" and "args" properties of elements in COMMAND class
attribute to define their command and arguments. the "cmd" property is
still available for the ceph CLI.

but the ceph_command sphinx extension should now use "prefix" and "args"
for printing out the usage of commands implemented using python. in
this change, the extension is updated to read "prefix" and "args"
properties of command defined by python modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: do not pass cmd_args or desc to CLICommand anymore

they will be attached to the handler function as annotations and docstring

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: use enum parameter when appropriate

and move desc as docstring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: cast string to enum when collecting kwargs

as the parameters of handlers are properly typed, they are expecting
enum parameter, let's cast string parameter to enum if the callee claims
that it wants an enum.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: preserve the signature of wrapped func

before this change, the annotations of the func wrapped by
CLICheckNonemptyFileInput and returns_command_result are lost after being
wrapped with this decorator. after this change, the parameter annotations
are preserved. these annotations are used for generating the argdesc in a
later commit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: extract load_func_metadata()

so it can be reused by Command class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: annotate command handler

and use enum as the argument

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: use desc of command for docstring

in the new command mechinary, the docstring of command handler
is used as the description of the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: drop arg parameter from CLICommand constructor

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap_schedule: use param annotation for argsdesc

use parameter annotation for CLICommand args desc, less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: use param annotation for args desc

use parameter annotation for CLICommand arg desc

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: use 'not in' for checking membership

silences the warning from flake8 like

test for membership should be 'not in' [E713]

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: add argdesc() method to type classes

so they can be used to build argdesc from method annotations, these
argdesc strings will be consumed by ceph CLI to do client-side
validation of command arguments.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: add arg annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: PEP8 compliance cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: translate argspec to argdesc

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: use annotations when collecting args

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: do not check for kwargs if already collecting kwargs

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: make CLICommand._parse_args() a staticmethod

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: extract method for collecting kwargs

the existing method is using argdesc for collecting kwargs, by
extracting into a separated method, we can use the annotation as an
alternative.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: extract kwargs handling into separated method

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38850 from liewegas/mgr-handle-old-host-inventory

mgr/cephadm: tolerate old host inventory without 'hostname' key

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38866 from sebastian-philipp/cephadm-pyright

cephadm: Fix type errors reported by pyright

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38851 from liewegas/fix-mailmap-sage

.mailmap: map myself to sweil@redhat.com

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38838 from orozery/rbd-encryption-cli

librbd: rbd encryption cli

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38864 from MahatiC/wip-ssd-fix

librbd/cache/pwl: Rename config options

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38751 from rhcs-dashboard/fix-48591-master

mgr/dashboard: Monitoring alert badge includes suppressed alerts

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38879 from Rethan/remove-osd-heartbeat-option

doc/rados: remove unused osd heartbeat option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38564 from ronen-fr/wip-ronenf-clangcrims-1

rgw: fixing compilation errors

Reviewed-by: Willem Jan Withagen (@wjwithagen)
Reviewed-by: Matt Benjamin mbenjamin@redhat.com
Reviewed-by: Daniel Gryniewicz dang@redhat.com dang@fprintf.net
Reviewed-by: Adam C. Emerson aemerson@redhat.com
Reviewed-by: Yuval Lifshitz ylifshit@redhat.com

Merge PR #37095 into master

* refs/pull/37095/head:
mds: make purge_queue delete objects asynchronously and keep accepting pushes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37876 into master

* refs/pull/37876/head:
systemd: cephfs-mirror systemd unit files
doc, man: man page for `cephfs-mirror` tool
doc: document mirror daemon internals
cephfs-mirror: switch to using PeerReplayer class
cephfs-mirror: cancel ongoing snapshot syncs on dir removal
cephfs-mirror: display peer snapshot sync stats
cephfs-mirror: carve out (and implement) mirroring snapshots to peers
cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option
cephfs-mirror: include helper routines to separate source
cephfs-mirror: remove peer only when peer is tracked
cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer
cephfs-mirror: enclose json dump in object section
cephfs-mirror: note current peer set
cephfs-mirror: fix option typo and document certain options
cephfs-mirror: remove unnecessary command line options
cephfs-mirror: default log level 0/5

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38820 into master

* refs/pull/38820/head:
include: use narrower ostream instead of iostream
mds: remove unused fstream header
mds: use narrower ostream header instead of sstream
mds: remove unused iostream header
mds: remove unused unix headers

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38835 into master

* refs/pull/38835/head:
qa: add test for fs rm idempotency
mon/FSCommands: restore idempotent behavior of fs rm
qa: add ceph cmd helper
qa: allow kwargs for raw_cluster_cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38848 into master

* refs/pull/38848/head:
qa: ignore MDS_SLOW_METADATA_IO with osd thrasher

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38886 from cbodley/wip-rgw-pubsub-warning

rgw: fix initialization of s3_event in pubsub

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38882 from tchaikov/wip-no-print-nanoseconds

tests/test_bluestore_types: cast duration to ceph::timespan

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: add encryption cli

This commit adds rbd encryption cli (encryption format and nbd map)

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge PR #38652 into master

* refs/pull/38652/head:
mgr/volumes: Filter inherited snapshots while listing snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38785 from sebastian-philipp/cephadm-move-create-daemon-to-serve

mgr/cephadm: move a bunch of methods from module.py to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: fix initialization of s3_event in pubsub

resolves compiler warning:

rgw_sync_module_pubsub.cc: In constructor ‘RGWPSHandleObjEventCR::RGWPSHandleObjEventCR(RGWDataSyncCtx*, PSEnvRef, const rgw_user&, EventRef<rgw_pubsub_event>&, EventRef<rgw_pubsub_s3_event>&, const TopicsRef&)’:
rgw_sync_module_pubsub.cc:1028:3: warning: ‘RGWPSHandleObjEventCR::s3_event’ is initialized with itself [-Winit-self]
 1028 |   RGWPSHandleObjEventCR(RGWDataSyncCtx* const _sc,
      |   ^~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tests/test_bluestore_types: cast duration to ceph::timespan

on FreeBSD + Clang + libc++ + amd64, std::chrono::nanoseconds::rep is `long
long` instead of `long`, so the explictly instantiated template
operator<< operator instances in `src/common/ceph_time.cc` are not able to
cover this use case. so in this change, the duration is casted to
ceph::timespan instead of nanoseconds. so we can ensure that `operator<<` is
always available in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38860 from tchaikov/wip-48777

osd/pg_scrubber: do not print log in dtors

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38724 from tchaikov/wip-install-deps-remove-mirror

install-deps.sh: do not use deb.rug.nl as a mirror anymore

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rados: remove unused osd heartbeat option

Since commit a5f9ca, option osd_mon_ack_timeout was removed, change
doc accordingly.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

mgr/volumes: Filter inherited snapshots while listing snapshots

Filter inherited snapshots resulted as part of a snapshot
at ancestor level while listing snapshots of a subvolume
and subvolumegroup

Also, fail the snapshot info on inherited snapshot.

Fixes: https://tracker.ceph.com/issues/48501
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #33692 from majianpeng/osd-remove-useless-code

osd: remove usless check condition.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38765 from tchaikov/wip-mutex-debug-atomic

common/mutex_debug: add memory barrier before load/store nlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd/cache/pwl: Rename config options

Signed-off-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #38696 from cyx1231st/wip-seastar-onode-tree-denc-delta

crimson/onode-staged-tree: implement delta encode, decode, replay and tests

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: set indent-tabs-mode to nil

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement get_type() for test extents

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: implement seastore replay test for onode-staged-tree

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38806 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added rbd_import_export workload

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tests: added stress-split-no-cephadm tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38849 into master

* refs/pull/38849/head:
doc: document ms_mode option in the mount.ceph manpage

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38793 into master

* refs/pull/38793/head:
qa: check for rank hole without racy joinable flag gymnastics
qa: skip client mounts for TestResize

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa: add test for fs rm idempotency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/FSCommands: restore idempotent behavior of fs rm

Fixes: https://tracker.ceph.com/issues/48808
Fixes: 811e48d6ffd841ecd93a57eb00c6aceae476f4f8
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add ceph cmd helper

A more programmer friendly command to use.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: allow kwargs for raw_cluster_cmd

And refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/pg_scrubber: do not print log in dtors

PG::recovery_state is defined after PG::m_scrubber, while
PG::gen_prefix() retrieves the osdmap from recovery_state. in
PgScrubber::~PgScrubber(), we print out the osdmap by dereferencing
recovery_state for the prefix of the logging message. in other words,
PgScrubber's destructor is referencing "recovery_state" which is
already destroyed.

in this change, the logging messages are removed from dtors. we can
print out the pgid in a follow-up PR if it helps.

Fixes: https://tracker.ceph.com/issues/48777
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38718 from tchaikov/wip-mgr-health-cleanup

mgr/devicehealth: use helpers to define options and commands

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

cephadm: Fix type errors reported by pyright

pyright is picky about `except ImportError` constructs
and also checks un-annotated code by default.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

systemd: cephfs-mirror systemd unit files

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc, man: man page for `cephfs-mirror` tool

Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document mirror daemon internals

... fix some typos and reorganize sections to be reader
friendly. Also include a requirements section.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: switch to using PeerReplayer class

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: cancel ongoing snapshot syncs on dir removal

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: display peer snapshot sync stats

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: carve out (and implement) mirroring snapshots to peers

Remote file synchronization by @mchangir.

Signed-off-by: Milind Changire <mchangir@redhat.com>
Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option

Basically, the mirror daemon would process directory list in in a
round-robin fashion. Randomly choosing directories probably does not
make much sense right now.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: include helper routines to separate source

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove peer only when peer is tracked

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: enclose json dump in object section

So that mirroring module can decode the JSON object.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: note current peer set

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: fix option typo and document certain options

s/cephfs_mirror_mirror_action_update_interval/cephfs_mirror_action_update_interval/

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: remove unnecessary command line options

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-mirror: default log level 0/5

Signed-off-by: Venky Shankar <vshankar@redhat.com>

.mailmap: map myself to sweil@redhat.com

Signed-off-by: Sage Weil <sweil@redhat.com>

Merge pull request #38863 from tchaikov/wip-osd-scrubber-tab

osd/pg_scrubber: use tab-width:8

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #34859 into master

* refs/pull/34859/head:
test: Initialize timer
test: Avoid named semaphores on Windows
test: Update cmake files, allowing tests to run on Windows
test: port Ceph tests to Windows

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/pg_scrubber: use tab-width:8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38781 from tchaikov/wip-googletest-gcc11

googletest: pick up change to silence error=maybe-uninitialized warning

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38809 from Rethan/fs-status-style

pybind/mgr: change format_bytes unit color from black to yellow

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38814 from tchaikov/wip-fix-throttle-test

common/Throttle: return new count in put() right away

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38811 from majianpeng/optimize-is_blackhole

msg: using switch-case in place of if-else.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38780 from majianpeng/fix-wrong-num_ios

os/bluestore: fix wrong num_ios for read op.

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38837 from vbashkirtsev/master

common/options.h: use uint64_t for TYPE_SIZE options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: document ms_mode option in the mount.ceph manpage

Fixes: https://tracker.ceph.com/issues/48838
Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/cephadm: tolerate old host inventory without 'hostname' key

Older cephadm clusters lack the 'hostname' key in the host spec.  e.g.,

 "cpach": {"addr": "cpach", "labels": ["mon"]}, "eutow": {"addr": "eutow", "labels": ["mon"]}, "stud": {"addr": "stud", "labels": ["mon"]}}

Populate hostname from the dict key if necessary for compatibility.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tests: added rbd_import_export tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>