Merge pull request #38722 from tchaikov/wip-mgr-args-ann

mgr: use annotations for argdesc

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38899 from ronen-fr/wip-ronenf-blue-dangling

os/bluestore: fixing a dangling pointer

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38902 from ifed01/wip-ifed-fix-deleting

osd/PG: fix no exit() call for PeeringState::Deleting state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38857 from dzafman/wip-48842

test: osd-recovery-prio.sh replace sleep with wait for both PGs recovering

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38762 from peng-jiaqi/master

mgr: fix deadlock in ActivePyModules::get_osdmap()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38878 from tchaikov/wip-ceph-time-print

common/ceph_time: print {milli,nano}seconds as float

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38903 from yuvalif/wip-yuval-fix-48869

rgw/amqp: fix race condition in amqp manager initialization

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/mgr: document CLICommand and friends

the annotation and docstring are critical after we use them for
describing the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38815 from sebastian-philipp/cephadm-osd-queue-race

mgr/cephadm: lock multithreaded access to OSDRemovalQueue

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38865 from s0nea/wip-dashboard-48847

mgr/dashboard: add `--ssl` to `ng serve`

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38919 from tchaikov/wip-cephadm-yield_fixture

mgr/cephadm: s/yield_fixture/fixture/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr/mgr_module: annotate some methods

as a vehicle to understand the interface

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard/awsauth: drop py2 support

silences the warning from mypy:

dashboard/awsauth.py:42: error: Module 'base64' has no attribute
'encodestring'

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: use iterator of tuple to build dict

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: disable unsubscriptable-object warning from pylint

see https://github.com/PyCQA/pylint/issues/3882

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: PEP8 compliance

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: fix typo in comment

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: do not mock ceph_argparse

this module is used for convert argspec to argdesc by
CLICommand.load_func_metadata()

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: add python_path option to ceph-mgr-commands

so we can add more python paths in addition to the one used for finding
mgr modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: parse prefix and args for command signature

unlike the commands defined by C++, the commands defined by python
now uses "prefix" and "args" properties of elements in COMMAND class
attribute to define their command and arguments. the "cmd" property is
still available for the ceph CLI.

but the ceph_command sphinx extension should now use "prefix" and "args"
for printing out the usage of commands implemented using python. in
this change, the extension is updated to read "prefix" and "args"
properties of command defined by python modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: do not pass cmd_args or desc to CLICommand anymore

they will be attached to the handler function as annotations and docstring

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: use enum parameter when appropriate

and move desc as docstring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: fix deadlock in ActivePyModules::get_osdmap()

In function "ActivePyModules::get_osdmap()", We do not read or write to
object "ActivePyModules", so it is safe to delete lock
"ActivePyModules::lock", and it can avoid other thread waiting for lock
"ActivePyModules::lock"

Fixes: https://tracker.ceph.com/issues/48852
Signed-off-by: peng jiaqi <peng.jiaqi@zte.com.cn>

pybind/mgr/mgr_module: cast string to enum when collecting kwargs

as the parameters of handlers are properly typed, they are expecting
enum parameter, let's cast string parameter to enum if the callee claims
that it wants an enum.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: preserve the signature of wrapped func

before this change, the annotations of the func wrapped by
CLICheckNonemptyFileInput and returns_command_result are lost after being
wrapped with this decorator. after this change, the parameter annotations
are preserved. these annotations are used for generating the argdesc in a
later commit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: extract load_func_metadata()

so it can be reused by Command class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: annotate command handler

and use enum as the argument

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: s/yield_fixture/fixture/

silences pytest warning. it complained:`

1: cephadm/tests/fixtures.py:68
1:   /var/ssd/ceph/src/pybind/mgr/cephadm/tests/fixtures.py:68: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
1:   Use @pytest.fixture instead; they are the same.
1:     @pytest.yield_fixture()

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: use desc of command for docstring

in the new command mechinary, the docstring of command handler
is used as the description of the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: drop arg parameter from CLICommand constructor

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap_schedule: use param annotation for argsdesc

use parameter annotation for CLICommand args desc, less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: use param annotation for args desc

use parameter annotation for CLICommand arg desc

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #38896 into master

* refs/pull/38896/head:
cephfs: Remove explicit forward declare CephContext

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38912 from xenago/patch-1

common/options: Fix typos for osd_recovery_max_active

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38891 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added stress-split-erasure-code-no-cephadm tests

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38721 from changchengx/pwl

librbd/cache/pwl: fix parameter type to update argument

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/options: Fix typos for osd_recovery_max_active

This commit fixes a minor typo in the descriptions of osd_recovery_max_active_hdd and osd_recovery_max_active_ssd.

Signed-off-by: Noah Kruiper <nkruiper@gmail.com>

Merge pull request #38907 from rzarzynski/wip-crimson-errorator-handle_exception_type

crimson/common: errorated future exposes handle_exception_type().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38708 into master

* refs/pull/38708/head:
doc: update multiple mds scrub docs

Reviewed-by: Jeff Layton <jlayton@redhat.com>

qa: remove invalid characters from octopus-x/parallel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38870 from orozery/rbd-encryption-pybind

librbd: python bindings for encryption api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: fixed yaml format

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38846 into master

* refs/pull/38846/head:
*: remove legacy ceph_volume_client.py library

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #34858 into master

* refs/pull/34858/head:
mds: do not allow the service to run on Windows
win32*.sh: fix SKIP_TESTS flag
include/win32/win32_errno:h Add missing include
client: Port CephFS client to Windows

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38574 into master

* refs/pull/38574/head:
mds: update defaults for recall configs

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38889 into master

* refs/pull/38889/head:
qa: add delays only for osd/mds

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38858 into master

* refs/pull/38858/head:
client/libcephfs: Don't let one unmount screw up another mount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/common: errorated future exposes handle_exception_type().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: python bindings for encryption api

This commit adds the python bindings for the new librbd encryption api

Signed-off-by: Or Ozeri <oro@il.ibm.com>

pybind/mgr/mgr_module: use 'not in' for checking membership

silences the warning from flake8 like

test for membership should be 'not in' [E713]

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: add argdesc() method to type classes

so they can be used to build argdesc from method annotations, these
argdesc strings will be consumed by ceph CLI to do client-side
validation of command arguments.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: add arg annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: PEP8 compliance cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: translate argspec to argdesc

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: use annotations when collecting args

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: fix no exit() call for PeeringState::Deleting state

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/cephadm: lock multithreaded access to OSDRemovalQueue

Since the set can be changed also from the CLI thread

Fixes: https://tracker.ceph.com/issues/47700
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: OSDRemovalQueue: remove some trivial wrappers

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move RemoveUtil into OSDRemovalQueue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move process_removal_queue into OSDRemovalQueue

`process_removal_queue` belongs to OSDRemovalQueue
instead of RemoveUtil

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make OSDRemovalQueue not inherent from set

Instead, let's make it a member.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: rename OSDQueue -> OSDRemovalQueue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: do not check for kwargs if already collecting kwargs

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: make CLICommand._parse_args() a staticmethod

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: extract method for collecting kwargs

the existing method is using argdesc for collecting kwargs, by
extracting into a separated method, we can use the annotation as an
alternative.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: extract kwargs handling into separated method

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38850 from liewegas/mgr-handle-old-host-inventory

mgr/cephadm: tolerate old host inventory without 'hostname' key

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38866 from sebastian-philipp/cephadm-pyright

cephadm: Fix type errors reported by pyright

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

rgw/amqp: fix race condition in amqp manager initialization

Fixes: https://tracker.ceph.com/issues/48869
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

cephfs: Remove explicit forward declare CephContext

It is already correctly declared in:
    src/include/common_fwd.h

fixes: https://github.com/ceph/ceph/pull/37876
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: fixing a dangling pointer

... and silencing compiler warnings:

Building CXX object src/os/CMakeFiles/os.dir/bluestore/BlueStore.cc.o
../src/os/bluestore/BlueStore.cc:8603:25: warning: object backing the pointer will be destroyed at the end of the full-expression [-Wdangling-gsl]
        const char *c = it->key().c_str();
                        ^~~~~~~~~
../src/os/bluestore/BlueStore.cc:8623:25: warning: object backing the pointer will be destroyed at the end of the full-expression [-Wdangling-gsl]
        const char* c = it->key().c_str();
                        ^~~~~~~~~
(reverting a minor part of ifed01)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38851 from liewegas/fix-mailmap-sage

.mailmap: map myself to sweil@redhat.com

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd/cache/pwl: fix parameter type to update argument

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #38838 from orozery/rbd-encryption-cli

librbd: rbd encryption cli

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

client/libcephfs: Don't let one unmount screw up another mount

Share threadpools between mounts. Don't shut them down until the last
user goes away, and make sure to start it back up when needed.

Fixes: https://tracker.ceph.com/issues/47294
For real this time.
Maybe.
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38864 from MahatiC/wip-ssd-fix

librbd/cache/pwl: Rename config options

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: added stress-split-erasure-code-no-cephadm tests (not using cephadm)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38751 from rhcs-dashboard/fix-48591-master

mgr/dashboard: Monitoring alert badge includes suppressed alerts

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mds: do not allow the service to run on Windows

Some file related flags and types are stubbed on Windows.
In order to avoid incorrect behavior, we're going to prevent
the service from running on Windows for now.

This does not apply to MDS clients.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

Merge pull request #38879 from Rethan/remove-osd-heartbeat-option

doc/rados: remove unused osd heartbeat option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38564 from ronen-fr/wip-ronenf-clangcrims-1

rgw: fixing compilation errors

Reviewed-by: Willem Jan Withagen (@wjwithagen)
Reviewed-by: Matt Benjamin mbenjamin@redhat.com
Reviewed-by: Daniel Gryniewicz dang@redhat.com dang@fprintf.net
Reviewed-by: Adam C. Emerson aemerson@redhat.com
Reviewed-by: Yuval Lifshitz ylifshit@redhat.com

Merge PR #37095 into master

* refs/pull/37095/head:
mds: make purge_queue delete objects asynchronously and keep accepting pushes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37876 into master

* refs/pull/37876/head:
systemd: cephfs-mirror systemd unit files
doc, man: man page for `cephfs-mirror` tool
doc: document mirror daemon internals
cephfs-mirror: switch to using PeerReplayer class
cephfs-mirror: cancel ongoing snapshot syncs on dir removal
cephfs-mirror: display peer snapshot sync stats
cephfs-mirror: carve out (and implement) mirroring snapshots to peers
cephfs-mirror: remove `cephfs_mirror_directory_choose_policy` option
cephfs-mirror: include helper routines to separate source
cephfs-mirror: remove peer only when peer is tracked
cephfs-mirror: typedef ceph_mount_info as MountRef shared pointer
cephfs-mirror: enclose json dump in object section
cephfs-mirror: note current peer set
cephfs-mirror: fix option typo and document certain options
cephfs-mirror: remove unnecessary command line options
cephfs-mirror: default log level 0/5

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38820 into master

* refs/pull/38820/head:
include: use narrower ostream instead of iostream
mds: remove unused fstream header
mds: use narrower ostream header instead of sstream
mds: remove unused iostream header
mds: remove unused unix headers

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38835 into master

* refs/pull/38835/head:
qa: add test for fs rm idempotency
mon/FSCommands: restore idempotent behavior of fs rm
qa: add ceph cmd helper
qa: allow kwargs for raw_cluster_cmd

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38848 into master

* refs/pull/38848/head:
qa: ignore MDS_SLOW_METADATA_IO with osd thrasher

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38886 from cbodley/wip-rgw-pubsub-warning

rgw: fix initialization of s3_event in pubsub

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38882 from tchaikov/wip-no-print-nanoseconds

tests/test_bluestore_types: cast duration to ceph::timespan

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

librbd: add encryption cli

This commit adds rbd encryption cli (encryption format and nbd map)

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa: add delays only for osd/mds

The delays were applied everywhere and needlessly interfere with test
commands sent to mons from the ceph admin command. Furthermore, the
delays would not affect the kernel client. Now the delays are performed
by the MDS on clients.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38652 into master

* refs/pull/38652/head:
mgr/volumes: Filter inherited snapshots while listing snapshots

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #38785 from sebastian-philipp/cephadm-move-create-daemon-to-serve

mgr/cephadm: move a bunch of methods from module.py to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw: fix initialization of s3_event in pubsub

resolves compiler warning:

rgw_sync_module_pubsub.cc: In constructor ‘RGWPSHandleObjEventCR::RGWPSHandleObjEventCR(RGWDataSyncCtx*, PSEnvRef, const rgw_user&, EventRef<rgw_pubsub_event>&, EventRef<rgw_pubsub_s3_event>&, const TopicsRef&)’:
rgw_sync_module_pubsub.cc:1028:3: warning: ‘RGWPSHandleObjEventCR::s3_event’ is initialized with itself [-Winit-self]
 1028 |   RGWPSHandleObjEventCR(RGWDataSyncCtx* const _sc,
      |   ^~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tests/test_bluestore_types: cast duration to ceph::timespan

on FreeBSD + Clang + libc++ + amd64, std::chrono::nanoseconds::rep is `long
long` instead of `long`, so the explictly instantiated template
operator<< operator instances in `src/common/ceph_time.cc` are not able to
cover this use case. so in this change, the duration is casted to
ceph::timespan instead of nanoseconds. so we can ensure that `operator<<` is
always available in this case.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38860 from tchaikov/wip-48777

osd/pg_scrubber: do not print log in dtors

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38724 from tchaikov/wip-install-deps-remove-mirror

install-deps.sh: do not use deb.rug.nl as a mirror anymore

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

win32*.sh: fix SKIP_TESTS flag

The "SKIP_TESTS" flag used by the Windows build script is
currently ignored.

This patch will skip some of the make targets when "SKIP_TESTS"
is set.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

include/win32/win32_errno:h Add missing include

We'll need to include "include/int_types.h" for the "__s32" type
definition.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

client: Port CephFS client to Windows

This patch provides a few changes required in order to use the
CephFS client on Windows.

A subsequent patch will make use of it along with Dokan, an
interface similar to Fuse.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

doc/rados: remove unused osd heartbeat option

Since commit a5f9ca, option osd_mon_ack_timeout was removed, change
doc accordingly.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>