mgr/dashboard: Use new ImageSpec class

This class should be used when dealing with RBD image specs.
It allows the creation of an ImageSpec given the names of the pool, image and
namespace. Alternatively you can also create one with an already existing
image spec string.

With it you keep the access to each individual component and can also convert it
to a well formated string.

Fixes: https://tracker.ceph.com/issues/42787
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: fix restored RBD image naming issue (#31590)

mgr/dashboard: fix restored RBD image naming issue

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #31758 from votdev/fix_typo

python-common: Fix typo in device type

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31410 from jan--f/c-v-create-vgs-on-demand

ceph-volume: allow raw block devices everywhere

mgr/dashboard: Fixes removal of custom tags during pool edit (#31373)

mgr/dashboard: Fixes removal of custom tags during pool edit

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

python-common: Fix typo in device type

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #31751 from tchaikov/wip-run-tox.sh-python3

script/run_tox.sh: do not use python2 if we have python3

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31607 from tspmelo/wip-pool-pattern

mgr/dashboard: Add invalid pattern message for Pool name

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: show "Rename" in header & button when renaming… (#31551)

mgr/dashboard: show "Rename" in header & button when renaming RBD snapshot

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: CephFS snapshot management UI (#30996)

mgr/dashboard: CephFS snapshot management UI

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31563 from tspmelo/wip-default-i18n

mgr/dashboard: Use default language when running "npm run build"

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: update ceph-volume lvm prepare

Add option to pass raw physical devices everywhere, restructure a little
(bluestore section before filestore) and reword a few things.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: make lvm report fields into constants

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: api/lvm create or reuse a vg

This changes create_lv so one can pass the desired device and either a
VG with a name starting with ceph is re-used or a new one is created.
This commit also adds two new lvm primitives, making use of lvm's select
feature. The goal is to eventually avoid keeping a full list of lv's (or
vg's) around and query the lvm system as needed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31755 from tchaikov/wip-mgr-orch

mgr/orch: pass unicode string to ipaddress.ip_network()

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/orch: pass unicode string to ipaddress.ip_network()

otherwise it complains like

>               raise e
E               AddressValueError: '10.1.1.10' does not appear to be an
IPv4 or IPv6 address. Did you pass in a bytes (str in Python 2) instead
of a unicode object?

orchestrator.py:94: AddressValueError

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: add ipaddress to requirements.txt

ipaddress dependency was introduced in 16f281d2. because the tox testenv
is isolated from the host python env, the test env has no access to the
ipaddress python module installed by install-deps.sh. we have to install
ipaddress for python<3.3 for testing which uses `orchestrator.py`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

script/run_tox.sh: do not use python2 if we have python3

always use python3 for bootstrapping tox environment unless the tox-env
contains "py2" but not "py3"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31574 from jan--f/c-v-refactor-get_devices

ceph-volume: make get_devices fs location independent

Merge PR #30698 into master

* refs/pull/30698/head:
common/options: Set bluestore min_alloc size to 4K

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31638 into master

* refs/pull/31638/head:
mgr/pg_autoscaler: only generate target_* health warnings if targets set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31572 into master

* refs/pull/31572/head:
mgr/diskprediction_local: some debug output during predict (and self-test)

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31721 into master

* refs/pull/31721/head:
mgr/ssh: set UserKnownHostsFile

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31660 from ivancich/wip-add-some-const

rgw: add const correctness to some rest functions

Merge pull request #16715 from adamemerson/wip-I-Object!

osdc/Objecter: Boost.Asio (I object!)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31672 into master

* refs/pull/31672/head:
mgr/orch: adapt update_mgr/mon funcs to new parsing
mgr/orch: improve commandline parsing

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31679 into master

* refs/pull/31679/head:
ceph-daemon: add osd create test(s)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31731 from varshar16/wip-doc-update-orchestrator_cli-url

mgr/orchestrator_cli: Update doc link in README

mgr/ssh: set UserKnownHostsFile

Set this to /dev/null, since we have StrictHostKeyChecking
off anyway.  This gets rid of messages to stderr every time we connect
to a new host.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: refactor tests for refactored get_devices

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: refactor get_devices, don't use os.path.realpath

Fixes: https://tracker.ceph.com/issues/42777
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #29399 from ofriedma/master

rgw: Fix expiration header does not return the earliest rule

Merge pull request #29937 from ofriedma/wip-ofriedma-days0

Add days0 to rgw lc

Merge pull request #31642 from trociny/wip-rbd-mirror-service-daemon-ns-support

rbd-mirror: add namespace support to service daemon

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31627 from trociny/wip-pybind-rbd-snap_remove

pybind/rbd: provide snap remove flags

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31608 from trociny/wip-mock-warnings

test/rbd_mirror: fix mock warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31606 from trociny/wip-test_rbd.py-tabs

test/pybind: inconsistent use of tabs and spaces in indentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: CephFS snapshot management UI

CephFS snapshots can now be created on a directory basis. Multiple
snapshots can be deleted at once.

Fixes: https://tracker.ceph.com/issues/41791
Signed-off-by: Volker Theile <vtheile@suse.com>
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/orchestrator_cli: Update doc link in README

Update the orchestrator cli doc link to fix page not found error. Also remove
some unnecessary blank lines.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #31594 from smithfarm/wip-bri-regex-fix

backport-resolve-issue: narrow regular expression and read key/token from files

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30336 from joscollin/wip-F40959-subvolume-uid-gid

mgr/volumes: uid, gid for subvolume create and subvolumegroup create commands

mgr/dashboard: show Rename in modal header & button when renaming RBD snapshot

Fixes: https://tracker.ceph.com/issues/42762
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #31546 from zjcmszh/execute

pybind/rados: add WriteOp::execute()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30968 from majianpeng/osd-optimze-send_message_osd_cluster

osd: optimize send_message to peers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31550 from wido/issue-42761

systemd: Wait 5 seconds before attempting a restart of an OSD

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #31623 from badone/wip-tracker-42387-extend-admin-socket-timeout

common/admin_socket: Increase socket timeouts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31241 from zjcmszh/add_rados_create_pool2

pybind/rados/rados.pyx: improve Rados.create_pool()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31617 from Devp00l/wip-42776

mgr/dashboard: Fix pool renaming functionality

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

ceph-daemon: add osd create test(s)

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31515 from drunkard/master

doc: typo fix in doc/dev/dev_cluster_deployement.rst: s/hostanme/hostname/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31541 from BlackLotus/master

added a remark to always use powers of two for pg_num

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31685 from liewegas/fix-mgr-shutdown

mgr: _exit(0) from signal handler even if we are standby

Merge pull request #31695 from liewegas/wip-cd-simpler-bootstrap

ceph-daemon: bootstrap: make --output-* args optional

mgr/orch: adapt update_mgr/mon funcs to new parsing

update_mgr and update_mon (and their correspinding orch-backend
implementation) are adapted to the new parsing parsing method
introduced in ff9093873.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/dashboard: Remove title from sparkline tooltips (#31645)

mgr/dashboard: Remove title from sparkline tooltips

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31689 from tchaikov/wip/workunits/cephtool/test.sh

qa/workunits/cephtool/test.sh: a handful fixes

Reviewed-by: Sage Weil <sweil@redhat.com>

Merge pull request #30397 from MahatiC/wip-ref-counting

librbd: add reference counting

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados: whitelist health warnings

in cephtool/test.sh, we

ceph fs set cephfs inline_data {1,0}

so the health check fails when the test ends, like

mon.a (mon.0) 3498 : cluster [WRN] Health check failed: 1 filesystem
with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED)" in
cluster log

so, before we remove the test, we need to whitelist this warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31688 from smithfarm/wip-bci-clean-print

backport-create-issue: flush line before overprinting

Reviewed-by: Tiago Melo <tmelo@suse.com>

backport-resolve-issue: read Redmine key and GitHub token from file

Thanks to Tiago Melo for inspiring me to do this.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/workunits/cephtool/test.sh: set min_{bytes,pgs}_per_osd for reweight

otherwise ceph-mgr will refuse to perform reweight

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: create pool with power-of-2 pg num

otherwise wait_for_health() fails like:

wait_for_health:  ceph health detail
HEALTH_WARN 1 pool(s) have non-power-of-two pg_num
[WRN] POOL_PG_NUM_NOT_POWER_OF_TWO: 1 pool(s) have non-power-of-two pg_num
    pool 'rbd' pg_num 10 is not a power of two
../qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1613: wait_for_health:  return 1

the failure was found when testing test_mon_pg().

this behavior was introduced by 6e46b1c0e50ad377d5db058e0452b0d956b3fad6

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use "ceph compact" in test_mon_deprecated_commands()

both "compact" and "scrub" are now cli command, so we cannot issue them
using "tell" anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

backport-resolve-issue: narrow regular expression

The regular expression used to extract tracker URLs from the PR body
was too generous. When there are two URLs on the same line and the
second is a tracker URl, the regex picked up both of them. For example,
the following string matched the regex:

"https://github.com/ceph/ceph/pull/26538, https://tracker.ceph.com/issues/41452"

This commit fixes the issue.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #31710 from votdev/ignore_ssh_config

mgr/ssh: Ignore ssh-config file

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Ignore ssh-config file

Ignore the ssh-config file that is created when working with the Vagrant box.

Signed-off-by: Volker Theile <vtheile@suse.com>

qa/workunits/cephtool/test.sh: use "sessions" in test_mon_tell()

"version" is not an asok command anymore in the sense that it's served
by registered asock hook. so in this change, we replace "version" with
"sessions", so we can verify that audit channel does not the dispatched
"sessions" command sent from ceph cli.

also, restructure the test as a loop for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: use tell instead of --admin-daemon for asok commands

ceph cli will be able to route the command to the target daemon when
handling "tell" commands.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: remove heap cli tests

"heap" cli commands are removed in the big asok/tell cleanup, they are now
asock commands.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/ceph: "heap" is now an asok command

after the big asok/tell cleanup, "heap" is now an asok command, so
update the manpage accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/cephtool/test.sh: fix iec/si tests

tests all IEC and SI units related test with a tier pool. as
`target_max_objects` and `target_size_bytes` only apply to tier
pools. so, for the sake of simplicity, tests all of them using
a tier pool.

introduced by 9095f67e

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: bootstrap: fix example shell command

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: default output file names

- These are reasonable names that a user would probably want
- If they run bootstrap in /etc/ceph then they are already installed in
  the right location
- This means that the bootstrap command has only 1 (!) required argument
  (--mon-ip or equivalent)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31637 from carletes/patch-1

src/msg/async/net_handler.cc: Fix compilation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31543 from tchaikov/wip/cmake/librt

cmake: detect librt for POSIX time functions

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #31532 from tchaikov/wip-42742

msg/async/rdma: do not init mutex before lockdeps is ready

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #31506 from Yan-waller/wip-walle-191109-fixomapentries

common/options: decrease the default max_omap_entries_per_request

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31310 from badone/wip-tracker-42477-reinstate-o-outfile

tools/rados: Unmask '-o' to restore original behaviour

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31469 from p-se/wip-pse-summary-reload-traceback

mgr/dashboard: KeyError on dashboard reload

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31399 from zy751713126/osd_stat

mon/OSDMonitor: fix format error ceph osd stat --format json

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31427 from zy751713126/osd_status

pybind/mgr: Cancel output color control

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31560 from rosinL/wip-fix-incorrect-pg-pool-sum

mon/PGMap: fix incorrect pg_pool_sum when delete pool

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31158 from zjcmszh/test_remove

test/pybind/test_rados.py: test test_operate_aio_write_op()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30717 from rzarzynski/wip-bug-42026

msg/async: perform the v2 resets in proper EventCenter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31517 from changchengx/rdma_gid

msg/async/rdma: add an option for choosing different RoCE protocol

Reviewed-by: Kefu Chai <kchai@redhat.com>

backport-create-issue: flush line before overprinting

When iterating over "Pending Backport" issues, the script displays
a message for each issue, like so:

    Examining issue#53432 (345/543)

Since a newline is not printed, each subsequent message overwrites
the previous one. But the last one remained on the screen and was
being overwritten by a shorter log message, leaving behind an
unwanted artifact:

    INFO:root:Processed 334 issues)

Fixes: 8495d37695d43ea38e1b70851e4a6127ebb4fc76
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31620 into master

* refs/pull/31620/head:
mgr: handle race with finisher after shutdown

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr: _exit(0) from signal handler even if we are standby

In 3d360b97ed7dead8e9a5f602cfaf61ab7b41e531 a signal handler was added to
shut down modules and associated clients cleanly, but the standby case
was left ignoring the signal completely.

Fixes: https://tracker.ceph.com/issues/42744
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31681 from ifed01/wio-ifed-fix-fio

test/fio: handle nullptr when parsing throttle params

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #31025 from scarvalhojr/ssevault2

rgw: improvements to SSE-KMS with Vault

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/fio: handle nullptr when throttle params parsing

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #31671 from sebastian-philipp/pybind-mgr-exclude-tests

pybind/mgr: Exclude tests/

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #31486 into master

* refs/pull/31486/head:
ceph-daemon: add standalone `adopt` tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31618 from dsavineau/ceph_daemon_rename_podman_vars

ceph-daemon: replace podman variables by container

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: Exclude tests/

Fixes: d6c10192c61a59f6a77d04a9d601c9df2f72d07f
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31666 from hrchu/origin/patch-2

doc: Update mailing lists

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orch: improve commandline parsing

We have to parse options like hostname, network and optional name

Examples:

"myhost"
"myhost=name"
"myhost:1.2.3.4"
"myhost:1.2.3.4=name"
"myhost:1.2.3.0/24"
"myhost:1.2.3.4/24=name"
"myhost:v2:1.2.3.4:3000/24=name"
"myhost:v2:1.2.3.4:3000,v1:1.2.3.4:6789/24=name"

This patch improves the durability of the code by using
regexes and adds validations for ip_addresses and subnets.

This will now also act as the single parser for all inputs.

Signed-off-by: Joshua Schmid <jschmid@suse.de>
(cherry picked from commit 4812167228813adc2fa45ff502257ca38ed53530)

Merge pull request #31667 from tchaikov/wip/run-cbt-sh/fs.aio-max-nr

script/run-cbt.sh: set fs.aio-max-nr for seastar

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

script/run-cbt.sh: set fs.aio-max-nr for seastar

seastar requires 11027 for each reactor, so we need at least

fs.aio-max-nr = 11027 * n_osd

but to make it simple and to take the needs of other applications using
aio on the machine, we increase the number of io context to 32768,
and always set this setting no matter if we are starting a vstart
cluster with classic osd or crimson osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: add namespace support to service daemon

Signed-off-by: Mykola Golub <mgolub@suse.com>