librbd: remove inclusion of endian.h

The file is not available on FreeBSD.

Fixes: https://github.com/ceph/ceph/pull/38862
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #38880 from xxhdx1985126/wip-crimson-bug-fix

crimson/osd: make PGRecovery::start_primary_recovery_ops take objs recovered by UrgentRecovery into account

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38245 from batrick/ceph-status-pg-state-sort

osd: use more efficient CachedStackStringStream

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37391 from aclamk/wip-kv-onode-cache

Add ability to control rocksdb block cache

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38925 from thomasgoirand/master

common/ipaddr: Allow binding on lo

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38730 from ronen-fr/wip-ronenf-scrub-interval

osd: improved handling of new-interval events by the scrubber

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #38856 from dzafman/wip-48789

test: Fix osd-scrub-scaps.sh to handle DB format change

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38726 from ronen-fr/wip-ronenf-48720

qa/standalone/scrub/osd-recovery-scrub: handle primary change when waiting for scrub

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #38160 into master

* refs/pull/38160/head:
doc/mgr/volumes: Document 'fs subvolume authorized_list' cli
qa: Add tests for list auth-ids of a subvolume
mgr/volumes: Update the 'volumes' key to 'subvolumes' in auth metadata file
mgr/volumes: Optionally authorize existing auth-ids
mgr/volumes: Preserve existing caps while authorize/deauthorize auth-id
mgr/volumes: Disallow authorize existing auth_id
mgr/volumes: Add subvolume authorized_list command
mgr/volumes: Add tenant_id option to subvolume authorize
mgr/volumes: Persist auth and subvolume metadata

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38862 from dillaman/wip-librbd-migration-qcow

librbd/migration: basic QCOW format handler

Reviewed-by: Mykola Golub <mgolub@suse.com>

test: Fix osd-scrub-scaps.sh to handle DB format change

Caused by: f9c95fa7fc7b0ee992c0249ff090fa7f751e9719

Fixes: https://tracker.ceph.com/issues/48789
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #38900 from Exotelis/update-i18n

mgr/dashboard: Update locale en to en-US to match transifex

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38722 from tchaikov/wip-mgr-args-ann

mgr: use annotations for argdesc

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38899 from ronen-fr/wip-ronenf-blue-dangling

os/bluestore: fixing a dangling pointer

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38902 from ifed01/wip-ifed-fix-deleting

osd/PG: fix no exit() call for PeeringState::Deleting state

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38857 from dzafman/wip-48842

test: osd-recovery-prio.sh replace sleep with wait for both PGs recovering

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38762 from peng-jiaqi/master

mgr: fix deadlock in ActivePyModules::get_osdmap()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38878 from tchaikov/wip-ceph-time-print

common/ceph_time: print {milli,nano}seconds as float

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

common/options: Set pacific+ rocksdb column family options.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

BlueStore: Add block_cache logic to column family definition

Modified original onode column family cache into generic feature.
Now 2 options are possible for each column family:
1) use generic block cache but apply different BlockBasedTableOptions
2) create separate block cache. it will be applied for all shards of column family

It is done by specifying special option "block_cache" to CF definition:
Example: O(3,0-13)=block_cache={high_ratio=1.000}

"block_cache" accepts all BlockBasedTableOptions options with additions:
-"type" - binned_lru/lru/clock (default: ceph_options.rocksdb_cache_type)
-"size" - e.g.: 100M           (default: ceph_options.rocksdb_cache_size)
-"high_ratio" - e.g.: 0.75     (default: 0.0)
If any of above is set, new block cache is created, otherwise default is used.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38903 from yuvalif/wip-yuval-fix-48869

rgw/amqp: fix race condition in amqp manager initialization

Reviewed-by: Casey Bodley <cbodley@redhat.com>

BlueStore: Add onode column family KV block_cache.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

doc/mgr: document CLICommand and friends

the annotation and docstring are critical after we use them for
describing the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/ipaddr: Allow binding on lo

Commmit 5cf0fa872231f4eaf8ce6565a04ed675ba5b689b, solves the issue that
the osd can't restart after seting a virtual local loopback IP. However,
this commit also prevents a bgp-to-the-host over unumbered Ipv6
local-link is setup, where OSD typically are bound to the lo interface.

To solve this, this single char patch simply checks against "lo:" to
match only virtual interfaces instead of anything that starts with "lo".

Fixes: https://tracker.ceph.com/issues/48893
Signed-off-by: Thomas Goirand <zigo@debian.org>

librbd/migration: address code review comments

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38815 from sebastian-philipp/cephadm-osd-queue-race

mgr/cephadm: lock multithreaded access to OSDRemovalQueue

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38865 from s0nea/wip-dashboard-48847

mgr/dashboard: add `--ssl` to `ng serve`

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38919 from tchaikov/wip-cephadm-yield_fixture

mgr/cephadm: s/yield_fixture/fixture/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr/mgr_module: annotate some methods

as a vehicle to understand the interface

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard/awsauth: drop py2 support

silences the warning from mypy:

dashboard/awsauth.py:42: error: Module 'base64' has no attribute
'encodestring'

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: use iterator of tuple to build dict

As suggested by Ernesto Puerta <epuertat@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: disable unsubscriptable-object warning from pylint

see https://github.com/PyCQA/pylint/issues/3882

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: PEP8 compliance

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: fix typo in comment

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: do not mock ceph_argparse

this module is used for convert argspec to argdesc by
CLICommand.load_func_metadata()

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: add python_path option to ceph-mgr-commands

so we can add more python paths in addition to the one used for finding
mgr modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/_ext: parse prefix and args for command signature

unlike the commands defined by C++, the commands defined by python
now uses "prefix" and "args" properties of elements in COMMAND class
attribute to define their command and arguments. the "cmd" property is
still available for the ceph CLI.

but the ceph_command sphinx extension should now use "prefix" and "args"
for printing out the usage of commands implemented using python. in
this change, the extension is updated to read "prefix" and "args"
properties of command defined by python modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: do not pass cmd_args or desc to CLICommand anymore

they will be attached to the handler function as annotations and docstring

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/orchestrator: use enum parameter when appropriate

and move desc as docstring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: fix deadlock in ActivePyModules::get_osdmap()

In function "ActivePyModules::get_osdmap()", We do not read or write to
object "ActivePyModules", so it is safe to delete lock
"ActivePyModules::lock", and it can avoid other thread waiting for lock
"ActivePyModules::lock"

Fixes: https://tracker.ceph.com/issues/48852
Signed-off-by: peng jiaqi <peng.jiaqi@zte.com.cn>

doc/mgr/volumes: Document 'fs subvolume authorized_list' cli

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>

qa: Add tests for list auth-ids of a subvolume

Fixes: https://tracker.ceph.com/issues/44931
Signed-off-by: Kotresh HR <khiremat@redhat.com>

crimson/osd: make PGRecovery::start_primary_recovery_ops take objs recovered by UrgentRecovery into account

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

pybind/mgr/mgr_module: cast string to enum when collecting kwargs

as the parameters of handlers are properly typed, they are expecting
enum parameter, let's cast string parameter to enum if the callee claims
that it wants an enum.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: preserve the signature of wrapped func

before this change, the annotations of the func wrapped by
CLICheckNonemptyFileInput and returns_command_result are lost after being
wrapped with this decorator. after this change, the parameter annotations
are preserved. these annotations are used for generating the argdesc in a
later commit.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: extract load_func_metadata()

so it can be reused by Command class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: annotate command handler

and use enum as the argument

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: s/yield_fixture/fixture/

silences pytest warning. it complained:`

1: cephadm/tests/fixtures.py:68
1:   /var/ssd/ceph/src/pybind/mgr/cephadm/tests/fixtures.py:68: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
1:   Use @pytest.fixture instead; they are the same.
1:     @pytest.yield_fixture()

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/cephadm: use desc of command for docstring

in the new command mechinary, the docstring of command handler
is used as the description of the command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/mgr_module: drop arg parameter from CLICommand constructor

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/snap_schedule: use param annotation for argsdesc

use parameter annotation for CLICommand args desc, less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: use param annotation for args desc

use parameter annotation for CLICommand arg desc

less repeating this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #38896 into master

* refs/pull/38896/head:
cephfs: Remove explicit forward declare CephContext

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38912 from xenago/patch-1

common/options: Fix typos for osd_recovery_max_active

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #38891 from yuriw/wip-yuriw-octopus-x-master

qa/tests: added stress-split-erasure-code-no-cephadm tests

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38721 from changchengx/pwl

librbd/cache/pwl: fix parameter type to update argument

Reviewed-by: Xiaoyan Li <xiaoyan.li@intel.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/options: Fix typos for osd_recovery_max_active

This commit fixes a minor typo in the descriptions of osd_recovery_max_active_hdd and osd_recovery_max_active_ssd.

Signed-off-by: Noah Kruiper <nkruiper@gmail.com>

Merge pull request #38907 from rzarzynski/wip-crimson-errorator-handle_exception_type

crimson/common: errorated future exposes handle_exception_type().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38708 into master

* refs/pull/38708/head:
doc: update multiple mds scrub docs

Reviewed-by: Jeff Layton <jlayton@redhat.com>

qa: remove invalid characters from octopus-x/parallel

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: simplify QCOW migration distro check

If the qemu-img command fails when attempting to use the QCOW (v1) format,
assume that the distro does not have support for the format enabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38870 from orozery/rbd-encryption-pybind

librbd: python bindings for encryption api

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tests: fixed yaml format

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge PR #38846 into master

* refs/pull/38846/head:
*: remove legacy ceph_volume_client.py library

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #34858 into master

* refs/pull/34858/head:
mds: do not allow the service to run on Windows
win32*.sh: fix SKIP_TESTS flag
include/win32/win32_errno:h Add missing include
client: Port CephFS client to Windows

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38574 into master

* refs/pull/38574/head:
mds: update defaults for recall configs

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38889 into master

* refs/pull/38889/head:
qa: add delays only for osd/mds

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38858 into master

* refs/pull/38858/head:
client/libcephfs: Don't let one unmount screw up another mount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/common: errorated future exposes handle_exception_type().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: python bindings for encryption api

This commit adds the python bindings for the new librbd encryption api

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd/migration: compute deltas between QCOW2 snapshots

Utilize the COPIED bit within the cluster offset to quickly determine
if the corresponding cluster block has been copied.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: move snapshot delta helpers to utils

The RawFormat class has logic for computing snapshot deltas that
can be re-used by the new QCOWFormat class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: add support for reading from QCOW2 snapshots

Tweak the IO read path to now utilize the L1 table associated with
the specified snapshot id. This will cause the IO to properly read
from the specific snapshot.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: decode QCOW2 snapshot metadata during open

This will allow the migration state machine to build matching RBD
snapshots to mimic the QCOW2 snapshots. The next commit will add the
ability to read from a snapshot.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: basic QCOW2 format handler

The initial implementation does not support backing files, compression,
snapshots, (deprecated) encryption, external data files, nor L2 subcluster
allocation. The former three features will be added in a future commit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: basic QCOW v1 format handler

The initial implementation does not support backing files, compression, nor
(deprecated) encryption. The former two features will be added in a future commit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/mgr/mgr_module: use 'not in' for checking membership

silences the warning from flake8 like

test for membership should be 'not in' [E713]

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: add argdesc() method to type classes

so they can be used to build argdesc from method annotations, these
argdesc strings will be consumed by ceph CLI to do client-side
validation of command arguments.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: add arg annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/devicehealth: PEP8 compliance cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: translate argspec to argdesc

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: use annotations when collecting args

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PG: fix no exit() call for PeeringState::Deleting state

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/cephadm: lock multithreaded access to OSDRemovalQueue

Since the set can be changed also from the CLI thread

Fixes: https://tracker.ceph.com/issues/47700
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: OSDRemovalQueue: remove some trivial wrappers

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move RemoveUtil into OSDRemovalQueue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: move process_removal_queue into OSDRemovalQueue

`process_removal_queue` belongs to OSDRemovalQueue
instead of RemoveUtil

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: make OSDRemovalQueue not inherent from set

Instead, let's make it a member.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: rename OSDQueue -> OSDRemovalQueue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: do not check for kwargs if already collecting kwargs

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: make CLICommand._parse_args() a staticmethod

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: extract method for collecting kwargs

the existing method is using argdesc for collecting kwargs, by
extracting into a separated method, we can use the annotation as an
alternative.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybing/mgr/mgr_module: extract kwargs handling into separated method

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38850 from liewegas/mgr-handle-old-host-inventory

mgr/cephadm: tolerate old host inventory without 'hostname' key

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38866 from sebastian-philipp/cephadm-pyright

cephadm: Fix type errors reported by pyright

Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

rgw/amqp: fix race condition in amqp manager initialization

Fixes: https://tracker.ceph.com/issues/48869
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

cephfs: Remove explicit forward declare CephContext

It is already correctly declared in:
    src/include/common_fwd.h

fixes: https://github.com/ceph/ceph/pull/37876
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

os/bluestore: fixing a dangling pointer

... and silencing compiler warnings:

Building CXX object src/os/CMakeFiles/os.dir/bluestore/BlueStore.cc.o
../src/os/bluestore/BlueStore.cc:8603:25: warning: object backing the pointer will be destroyed at the end of the full-expression [-Wdangling-gsl]
        const char *c = it->key().c_str();
                        ^~~~~~~~~
../src/os/bluestore/BlueStore.cc:8623:25: warning: object backing the pointer will be destroyed at the end of the full-expression [-Wdangling-gsl]
        const char* c = it->key().c_str();
                        ^~~~~~~~~
(reverting a minor part of ifed01)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #38851 from liewegas/fix-mailmap-sage

.mailmap: map myself to sweil@redhat.com

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd/cache/pwl: fix parameter type to update argument

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>