ceph-daemon: Add basic mypy support

Add mypy support and some type hints to be able to do static code
analysis.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #31583 into master

* refs/pull/31583/head:
ceph-daemon: -v|--verbose, not -d|--debug

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31585 into master

* refs/pull/31585/head:
ceph-daemon: add `--legacy-dir` arg to `ls` command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31533 from tspmelo/wip-bc-redmine-key

backport-create-issue: read redmine key from file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #31544 from ifed01/wip-ifed-gettid2

test/fio: bump to fio-3.15

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31426 from cyx1231st/rfc-seastar-fix-build-dpdk

crimson: fix build seastar with dpdk

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30662 from sebastian-philipp/orchestrator-c-v-inventory

python-common, mgr/orchestrator, mgr/dashboard: Use common "Devices"

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

crimson: build seastar dpdk from src/seastar/dpdk

src/spdk/dpdk and src/seastar/dpdk are both at their private branches
with project-specific modifications, so select proper dpdk source
directory according to flags WITH_SPDK and Seastar_DPDK.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: link required dpdk components for seastar

see https://github.com/scylladb/seastar/blob/
809b3ef875e7edb7c56643812dc8347ff9a82cee/cmake/Finddpdk.cmake#L60-L85

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #31571 from rhcs-dashboard/42775-fix-typo

common: fix typo in rgw_user_max_buckets option long description.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31510 from yuvalif/wip-yuval-remove-flaky-amqp-test

rgw/amqp: remove flaky amqp test

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: add `--legacy-dir` arg to `ls` command

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: -v|--verbose, not -d|--debug

-v is more natural IMO

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31575 into master

* refs/pull/31575/head:
ceph-daemon: 'logs' command

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: 'logs' command

This just invokes podman logs with the appropriate argument(s).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31577 into master

* refs/pull/31577/head:
pybind/mgr/CMakeLists: exclude tox.ini, requirements.txt from install

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31542 from neha-ojha/wip-42756

osd/OSDMap.cc: don't output over/underfull messages to lderr

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/CMakeLists: exclude tox.ini, requirements.txt from install

Signed-off-by: Sage Weil <sage@redhat.com>

backport-create-issue: read redmine key from file

Now you don't need to provide redmine key or user/password as a flag, as the
script will read it from "~/.redmine_key".

This makes it more consistent with other scripts.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #31157 into master

* refs/pull/31157/head:
doc: update inf/infinite option for subvolume resize
qa/tasks: fs subvolume resize inf command
mgr/volumes: fs subvolume resize inf/infinite command

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31503 into master

* refs/pull/31503/head:
Revert "os/bluestore: add kv_drain_preceding_waiters indicate drain_preceding."

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31537 into master

* refs/pull/31537/head:
mgr/orchestrator: move split_host into shared helpers
mgr/ssh: allow names to be specified for stateless services
mgr/ssh: optionally specify names for mon daemons
mgr/ssh: optionally specify names for mgr daemons

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31280 from zy751713126/snap_get_name_and_id

librbd: add snap_get_name and snap_get_id method API

Merge pull request #31491 from zy751713126/rbd_bench

rbd/bench: outputs bytes/s format dynamically

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31497 from trociny/wip-mock-warnings

test/librbd: fix mock warnings in TestMockIoImageRequest

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31487 into master

* refs/pull/31487/head:
ceph-daemon: add `--base-dir` arg to `adopt` command

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix typo in rgw_user_max_buckets option long description.

https://tracker.ceph.com/issues/42775
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/orchestrator: move split_host into shared helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: allow names to be specified for stateless services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: optionally specify names for mon daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: optionally specify names for mgr daemons

Use '=' as a separator.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Manage PG autoscaling (#31417)

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #31559 into master

* refs/pull/31559/head:
mgr: Python cleanup and type check

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31539 into master

* refs/pull/31539/head:
mgr/orchestrator_cli: sort by service name
mgr/orchestrator_cli: collapse TYPE and ID columns

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31394 into master

* refs/pull/31394/head:
mgr/crash: Calculate and add stack_sig to metadata

Reviewed-by: Sage Weil <sage@redhat.com>

mgr: Python cleanup and type check

- Add/remove missing/obsolete empty lines
- Add type check doc
- Rename inner function args to make pylint happy

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Support RBD namespaces (#30935)

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31357 from votdev/improve_timediff_unittest

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

cmake: detect gettid() presense

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30534 from jiahuizeng/osd_dev_node

mgr/prometheus: assign a value to osd_dev_node when obj_store is not filestore or bluestore

Fix backend testing error due to attribute rename

`id` in Device is renamed to `path`.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #31431 from jschmid1/fix_cherrypy_imports

mgr/dashboard: cheroot moved into a separate project

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr: Run python unit tests with tox in the mgr (#30364)

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Remove compression mode unset in pool from (#31376)

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29944 from xxhdx1985126/wip-fix-blustore-fadvice_flags-leak

osd: copy ObjectOperation::BufferUpdate::Write::fadvise_flag to ceph::os::Transaction

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31333 from adamemerson/wip-who-warns-the-warnings

src: use un-deprecated version of aio_create_completion

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31416 from sseshasa/wip-41666-replicaSizeWarn

osd/OSDMap: Show health warning if a pool is configured with size 1

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31475 from zdover23/wip-doc-community-calendar

doc: Added a link to Ceph Community Calendar.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: Added a link to Ceph Community Calendar.

Co-Authored by: Neha Ojha <nojha@redhat.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge PR #31501 into master

* refs/pull/31501/head:
ceph-mon: bind to public_addrv if set (before public_addr or public_network)
common/options: add public_addrv
common/options: fix ADDRVEC parsing
ceph-daemon: support --mon-addrv for deploy subcommand

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/crash: Calculate and add stack_sig to metadata

Signed-off-by: Dan Mick <dan.mick@redhat.com>

test/fio: bump to fio 3.15

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #30914 into master

* refs/pull/30914/head:
ceph: Add doc for deploying cephfs-nfs cluster using rook

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #31540 into master

* refs/pull/31540/head:
ceph-daemon: make ps1 a raw string

Reviewed-by: Sage Weil <sage@redhat.com>

osd/OSDMap.cc: don't output over/underfull messages to lderr

There can be cases where overfull and underfull(see example in
https://tracker.ceph.com/issues/42756) will be empty, which is not
necessarily an error. These error messages can end up spamming
the ceph-mgr log.

Fixes: https://tracker.ceph.com/issues/42756
Signed-off-by: Neha Ojha <nojha@redhat.com>

ceph-daemon: make ps1 a raw string

python3 interprets '\u' as a unicode escape

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31024 into master

* refs/pull/31024/head:
doc: improve in mount.ceph man page
doc: describe mds_namespace option in mount.ceph man page

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30911 into master

* refs/pull/30911/head:
doc/ceph-fuse: describe -n option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Douglas Fuller <dfuller@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #30657 into master

* refs/pull/30657/head:
test_cephfs_shell: pass arg to "help" in test_help
cephfs-shell: set exit_code when an error occurs
cephfs-shell: print traceback only when debug switch is on
cephfs-shell: catch libcephfs.Error only in onecmd()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31452 into master

* refs/pull/31452/head:
Add chown ceph:ceph for /tmp/ceph.mon.keyring

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/orchestrator_cli: sort by service name

This is more consistent when applied across hosts.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: collapse TYPE and ID columns

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31538 into master

* refs/pull/31538/head:
ceph-daemon: do not fetch daemon detail when removing a cluster
ceph-daemon: ls: add --no-detail

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: do not fetch daemon detail when removing a cluster

It's slow and not needed.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: ls: add --no-detail

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: add `--base-dir` arg to `adopt` command

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31387 into master

* refs/pull/31387/head:
src/CMakeLists: remove leading 'v' from 'git describe' version

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31524 into master

* refs/pull/31524/head:
mgr/ssh: Fix various Python issues.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

mgr/dashboard: Improve position of MDS chart tooltip (#31368)

mgr/dashboard: Improve position of MDS chart tooltip

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31512 into master

* refs/pull/31512/head:
mgr/ssh: allow passing LV to 'orchestrator osd create'

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Support RBD namespaces

Fixes: https://tracker.ceph.com/issues/25125
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Fix data point alignment in MDS counters chart (#31288)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31519 from votdev/fix_ssh_vagrant

mgr/ssh: Install SSH public key in Vagrantfile box fails

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Fix various Python issues.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: RBD tests must use pools with power-of-two pg_n… (#31490)

mgr/dashboard: RBD tests must use pools with power-of-two pg_num

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/orchestrator: move InventoryDevice to python-common

Adapted:
* `mgr/dashboard`
* `mgr/ansible`
* `mgr/rook`
* `mgr/ssh`
* `mgr/orchestrator_cli`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Co-authored--by: Kiefer Chang <kiefer.chang@suse.com>

mgr/ssh: Install SSH public key in Vagrantfile box fails

The current implementation to copy the host users SSH public key into the Vagrant box does not work with Vagrant 2.2.6.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #31419 from smithfarm/wip-cbs-5

ceph-backport.sh: guess component with --existing-pr

Reviewed-by: Laura Paduano <lpaduano@suse.com>

osd/OSDMap: Show health warning if a pool is configured with size 1

Introduce a config option called 'mon_warn_on_pool_no_redundancy' that is
used to show a health warning if any pool in the ceph cluster is
configured with a size of 1. The user can mute/unmute the warning using
'ceph health mute/unmute POOL_NO_REDUNDANCY'.

Add standalone test to verify warning on setting pool size=1. Set the
associated warning to 'false' in ceph.conf.template under qa/tasks so
that existing tests do not break.

Fixes: https://tracker.ceph.com/issues/41666
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Add chown ceph:ceph for /tmp/ceph.mon.keyring

The keyring will be 600 by default with the owner of the current user who issues the command, which will make command "sudo -u ceph ceph-mon --mkfs" fail since the user "ceph" don't have access to the keyring file.
Because of that, the mon data directory won't have keyring and when "sudo systemctl start ceph-mon" it will show error "unable to read magic from mon data"

Signed-off-by: Jeffrey Chu <peihuachu1112@gmail.com>

Merge pull request #31271 from ivancich/wip-fix-cancel-bucket-reshard

rgw: allow reshard log entries for non-existent buckets to be cancelled

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #31492 from jan--f/c-v-add-part-size

ceph-volume: add proper size attribute to partitions

mgr/ssh: allow passing LV to 'orchestrator osd create'

In this case,

 ceph orchestrator osd create smithi009:/dev/vg_nvme/lv_1

produced

    "0": [
        {
            "devices": [
                "/dev/nvme0n1"
            ],
            "lv_name": "lv_1",
            "lv_path": "/dev/vg_nvme/lv_1",
            "lv_size": "89.40g",
            "lv_tags": "ceph.block_device=/dev/vg_nvme/lv_1,ceph.block_uuid=a1vfDi-0jor-VEPK-DZVd-K90y-cqG1-QoXr9l,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=5d6d2c1c-0362-11ea-825a-001a4aab830c,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=0,ceph.osd_fsid=04175a6e-55e7-4244-b683-79da76f723af,ceph.osd_id=0,ceph.type=block,ceph.vdo=0",
            "lv_uuid": "a1vfDi-0jor-VEPK-DZVd-K90y-cqG1-QoXr9l",
            "name": "lv_1",
            "path": "/dev/vg_nvme/lv_1",
            "tags": {
                "ceph.block_device": "/dev/vg_nvme/lv_1",
                "ceph.block_uuid": "a1vfDi-0jor-VEPK-DZVd-K90y-cqG1-QoXr9l",
                "ceph.cephx_lockbox_secret": "",
                "ceph.cluster_fsid": "5d6d2c1c-0362-11ea-825a-001a4aab830c",
                "ceph.cluster_name": "ceph",
                "ceph.crush_device_class": "None",
                "ceph.encrypted": "0",
                "ceph.osd_fsid": "04175a6e-55e7-4244-b683-79da76f723af",
                "ceph.osd_id": "0",
                "ceph.type": "block",
                "ceph.vdo": "0"
            },
            "type": "block",
            "vg_name": "vg_nvme"
        }
    ],

Signed-off-by: Sage Weil <sage@redhat.com>

rgw/amqp: remove a test that requires syncronization
betwen the amqp manager internal thread and the test itself
amqp reconnect should be tested by system test instead

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

rbd/bench: outputs bytes/s format dynamically

Fixes: https://tracker.ceph.com/issues/40256
Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

Merge pull request #30853 from ivancich/wip-tune-sharded-bucket-listing

rgw: reduce per-shard entry count during ordered bucket listing

Merge pull request #31323 from ivancich/wip-autoclean-reshard-long-entries

rgw: auto-clean reshard queue entries for non-existent buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #31505 into master

* refs/pull/31505/head:
mgr/ssh: take IP, CIDR, or addrvec for new mon(s)

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31507 into master

* refs/pull/31507/head:
ceph-daemon: ceph/daemon-base:latest-master-devel

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-mon: bind to public_addrv if set (before public_addr or public_network)

This lets you specify a precise addrvec for the monitor to bind to,
instead of inferring a v1 or v2 address from the port number, or assuming
both with default ports, or v2 only with an unrecognized port.

I unfortunately don't quite remember why I didn't make public_addr a
ADDRVEC option way back in nautilus--I only remmeber that it was going to
cause problems.  Adding a separate addrv option here is safe.

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: add public_addrv

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31474 into master

* refs/pull/31474/head:
mgr/devicehealth: ensure we don't store empty objects

Reviewed-by: Neha Ojha <nojha@redhat.com>

ceph-daemon: ceph/daemon-base:latest-master-devel

The -devel one rebuilds daily, while latest-master rebuilds when
ceph-container.git changes.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: reduce per-shard entry count during ordered bucket listing

Currently, if a client requests the 1000 next entries from a bucket,
each bucket index shard will receive a request for the 1000 next
entries. When there are hundreds, thousands, or tens of thousands of
bucket index shards, this results in a huge amplification of the
request, even though only 1000 entries will be returned.

These changes reduce the per-bucket index shard requests. These also
allow re-requests in edge cases where all of one shard's returned
entries are consumed. Finally these changes improve the determination
of whether the resulting list is truncated.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #31498 into master

* refs/pull/31498/head:
ceph-daemon: customize the bash prompt for shell + enter

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/ssh: take IP, CIDR, or addrvec for new mon(s)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31499 into master

* refs/pull/31499/head:
ceph-daemon: the 'ceph' command may not be installed

Reviewed-by: Michael Fritch <mfritch@suse.com>

librbd: add snap_get_name and snap_get_id method API

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

ceph-daemon: the 'ceph' command may not be installed

...even though there are legacy data dirs in /var/lib/ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31500 into master

* refs/pull/31500/head:
ceph-daemon: set ssh public identity

Reviewed-by: Michael Fritch <mfritch@suse.com>