mgr/balancer: avoid pulling pg_dump twice

For large cluster with thousands of osds/pgs, it may take
seconds even mitutes to refresh the relevant stats.
Since we already have pulling them down at initialization,
make a direct use of them whenever possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #32142 from changchengx/doc_libvirt

doc/rbd: simplify libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32243 from trociny/wip-43289

rbd-mirror: fix race on namespace replayer initialization failure

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32184 from trociny/wip-43004

rbd-mirror: rename per-image replication perf counters

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32113 from lixiaoy1/master

librbd: finish write request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32251 into master

* refs/pull/32251/head:
mds/MDSDaemon: fix asok 'exit' and 'respawn' commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32260 into master

* refs/pull/32260/head:
.organizationmap: Wido 42on -> 42on

Reviewed-by: Kefu Chai <kchai@redhat.com>

.organizationmap: Wido 42on -> 42on

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32158 into master

* refs/pull/32158/head:
cephadm: fix exception when no time sync is running
cephadm: properly extract osd fsid during adoption
cephadm: do ceph-volume activate+deactivate as part of systemd unit
ceph-daemon: behave with no /etc/ceph/ceph.conf

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: fix exception when no time sync is running

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: properly extract osd fsid during adoption

This needs to be passed to the deploy_daemon_units method so that we can
set up the pre and post c-v calls properly.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do ceph-volume activate+deactivate as part of systemd unit

1- Do the activate as part of the unit on start, since it may include
setup that disappears on a reboot, like starting up the dmcrypted volume.

2- Do the matching deactivate step when the container stops, like shutting
down the dmcrypt volume.

Right now the OSD is the only thing that needs this special behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: behave with no /etc/ceph/ceph.conf

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32244 into master

* refs/pull/32244/head:
qa/workunits/cephadm/test_cephadm.sh: re-enable `adopt` tests

Reviewed-by: Sage Weil <sage@redhat.com>

mds/MDSDaemon: fix asok 'exit' and 'respawn' commands

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32253 into master

* refs/pull/32253/head:
mgr/orchestrator: make hosts and label args consistent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32209 into master

* refs/pull/32209/head:
mgr/cephadm: Add unittest for service_action

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: make hosts and label args consistent

- hosts is always n=N; labels is not
- hosts always before labels

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32192 into master

* refs/pull/32192/head:
common: handle return value from read(2)

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32195 into master

* refs/pull/32195/head:
osd/PeeringState.cc: skip peer_purged when discovering all missing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #32203 into master

* refs/pull/32203/head:
Revert "mgr/DaemonServer: safe-to-destroy - do not consider irrelevant pgs"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32219 into master

* refs/pull/32219/head:
common: move xattr -> os/filestore/os_xattr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32250 into master

* refs/pull/32250/head:
mgr/cephadm: fix temp file prefixes
qa: fix lingering ceph-mgr-ssh -> ceph-mgr-cephadm refs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32144 into master

* refs/pull/32144/head:
doc: mention multimds suite in dev guide
doc: clarify difference between fs and kcephfs suite

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32024 into master

* refs/pull/32024/head:
stop: Unmount cephFS volumes

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: fix temp file prefixes

Signed-off-by: Sage Weil <sage@redhat.com>

qa: fix lingering ceph-mgr-ssh -> ceph-mgr-cephadm refs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32226 from neha-ojha/wip-four-percent

doc/rados: Better block.db size recommendations for bluestore

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc/rados: Better block.db size recommendations for bluestore

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32040 from smithfarm/wip-rn-13-2-8-mimic

doc/releases: add release notes for mimic v13.2.8

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/releases: add Mimic v13.2.8 to releases timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32236 into master

* refs/pull/32236/head:
pybind/mgr: add_tox_test: Add mypy to TOX_ENVS

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: re-enable `adopt` tests

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32005 from s0nea/wip-dashboard-reactivate-suites

mgr/dashboard: reactivate dashboard test suites

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

pybind/mgr: add_tox_test: Add mypy to TOX_ENVS

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

rbd-mirror: fix race on namespace replayer initialization failure

Fixes: https://tracker.ceph.com/issues/43289
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #31851 from votdev/issue_42609_device_lights

mgr/dashboard: Add support for blinking enclosure LEDs

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

stop: Unmount cephFS volumes

When stop.sh is run while the cephFS is mounted,
mount processes are retained and can't be killed.
And also the mount and df commands hang after this.
This patch unmounts the mounted volumes cleanly.

Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #32186 from rhcs-dashboard/43263-fix-rgw-subuser-key

mgr/dashboard: fix RGW subuser auto-generate key.

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31753 from smithfarm/wip-bci-resolve-fix

backport-create-issue: resolve parent only if parent has backport issues

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Simplify PrometheusService::getRules implementa… (#32130)

mgr/dashboard: Simplify PrometheusService::getRules implementation

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: Add unittest for service_action

(That was missing previously)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32177 from Takios/ceph_volume_fix

ceph-volume: use correct extents if using db-devices and >1 osds_per_device

mgr/dashboard: reactivate dashboard test suites

Reactivate the dashboard test suites that were commented
out in https://github.com/ceph/ceph/pull/30864 because
https://tracker.ceph.com/issues/41538 has been resolved
in the meanwhile.

Fixes: https://tracker.ceph.com/issues/42652
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #32224 from tspmelo/wip-e2e-insecure

mgr/dashboard: Fix e2e chromedriver problem

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31255 into master

* refs/pull/31255/head:
mds/MDSDaemon: make con disposable before passing message to asok
mds/SessionMap: interpret bare id as a valid filter
mds: get rid of unused send_command_reply helpers
mds: remove old handle_command machinery
mds/MDSRank: remove C_ExecAndReply helpers
mds/MDSRank: too many words about op_tracker
mds: convert 'cache drop'
mds: convert 'damage ls' and 'damage rm'
mds: convert '{session,client} config'
mds: convert 'scrub {start,abort,pause,resume,status}'
mds: convert 'session ls' and 'client ls'
mds: convert 'session evict' and 'session kill'
mds: convert 'heap' and 'cpu_profiler'
mds: convert 'exit', 'respawn'
mds/MDSDaemon: remove old duplicated tell commands
mds: send tell commands to asok infrastructure
common/admin_socket: switch tell queue to cref_t (from ref_t)
mds: switch asok calls over to call_async interface

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32230 from shellwedance/ceph-volume_typo_fix

doc: fix a typo in a command

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32170 from rosinL/wip-rss

msg/async/DPDK: refactor set_rss_table to support DPDK 19.05

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix a typo in a command

Co-authored-by: donggyu_park <donggyu_park@tmax.co.kr>
Signed-off-by: taeuk_kim <taeuk_kim@tmax.co.kr>

doc/rbd: align with other part in the context

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge PR #32179 into master

* refs/pull/32179/head:
lvm: add deactivate subcommand
util/system: add unmount_tmpfs helper
api/lvm: add get_lv_by_osd_id method
api/lvm: add deactivate method to Volume class

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rbd: correct targe dev name and existence check

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge PR #32225 into master

* refs/pull/32225/head:
qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32003 into master

* refs/pull/32003/head:
mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror
Allow List[HostSpec] in PlacementSpec
mgr/rook: adapt to arg passing change in orchestrator_cli.py
mgr/ssh: Add SimpleScheduler and streamline arg passing
orch_cli: adapt to changes in PlacementSpec and ServiceSpec
orch: extend PlacementSpec and add StatefulServiceSpec

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32218 into master

* refs/pull/32218/head:
cephadm: use raw string during `re.sub`
cephadm: fix ConfigParser DeprecationWarning in py3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32187 into master

* refs/pull/32187/head:
ceph.in: (py3) use binary stdout
pybind/mgr: index=4, sort_keys=True for json.dumps throughput

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32013 from liewegas/doc-foundation

doc/foundation: add cloudbase and vexxhost

Merge pull request #32197 from neha-ojha/wip-report-doc-bug

doc/_templates/page.html: redirect to etherpad

Reviewed-by: Zac Dover zac.dover@gmail.com
Reviewed-by: Sage Weil sage@redhat.com

qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32220 into master

* refs/pull/32220/head:
qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Reviewed-by: Michael Fritch <mfritch@suse.com>

mds/MDSDaemon: make con disposable before passing message to asok

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: interpret bare id as a valid filter

This works around the fact that the old asok 'session evict' took a bare
id and the tell version took a filter.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: get rid of unused send_command_reply helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mds: remove old handle_command machinery

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSRank: remove C_ExecAndReply helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSRank: too many words about op_tracker

Signed-off-by: Sage Weil <sage@redhat.com>

mds: convert 'cache drop'

Signed-off-by: Sage Weil <sage@redhat.com>

mds: convert 'damage ls' and 'damage rm'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31629 into master

* refs/pull/31629/head:
cephfs-shell: rearrange code for convenience

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Fix e2e chromedriver problem

New version of Chrome/ChromeDriver seems to have a problem with insecure
certificates.

Since we use a vStart cluster with no real certificate,
we always face this problem when running e2e tests.

This will fix the current problem and hopefully any future problems related to
certificates.

Fixes: https://tracker.ceph.com/issues/43254
Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Otherwise the test breaks if there are other cluster(s) on the host (like
there normally are in my dev environment).

Signed-off-by: Sage Weil <sage@redhat.com>

backport-create-issue: resolve parent only if parent has backport issues

This fixes an edge case. If a parent issue is in "Pending Backport" status
but without any backport issues, the script (when running with --resolve-parent)
was wrongly changing the status to "Resolved".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32129 into master

* refs/pull/32129/head:
qa: Enable flake8 tox and fix failures

ceph.in: (py3) use binary stdout

This fixes an error like

$ ceph -s -o -
Traceback (most recent call last):
  File "bin/ceph", line 1267, in <module>
    retval = main()
  File "bin/ceph", line 1229, in main
    outf.write(outbuf)
TypeError: write() argument must be str, not bytes

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr: index=4, sort_keys=True for json.dumps throughput

- fixes balancer test on py3 by making the pool ls order deterministic
- makes everybody else behave consistently

I left internal json storage alone--no need to sort/normalize.  I also
left out some CLI commands returning non-indented json.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32094 from cbodley/wip-40585

test/rgw: raise timer durations for unittest_rgw_reshard_wait

Reviewed-by: Kefu Chai <kchai@redhat.com>

common: move xattr -> os/filestore/os_xattr

Aside from unittest_chain_xattr, these functions are only used by
filestore, so it makes sense to move them out of common.

Signed-off-by: David Disseldorp <ddiss@suse.de>

Merge pull request #32153 from smithfarm/wip-23981

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

cephadm: use raw string during `re.sub`

cephadm:198: DeprecationWarning: invalid escape sequence \s
  nice_conf = re.sub('\n(\s)+', '\n', raw_conf)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: fix ConfigParser DeprecationWarning in py3

DeprecationWarning: This method will be removed in future versions.
Use 'parser.read_file()' instead.

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32151 from liewegas/fix-scipy

install-deps.sh: preload wheel for all mgr requirements.txt files

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

lvm: add deactivate subcommand

This new subcommand unmounts and OSDs tmpfs mount and closes crypt
devices if there are any.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

util/system: add unmount_tmpfs helper

Thsi unmounts a path if and only if it's a tmpfs mount.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

msg/async/dpdk: refactor set_rss_table to support DPDK 19.05

The filter API was deprecated in DPDK 19.05, using the flow
API to set rss table.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

mgr/dashboard: Add support for blinking enclosure LEDs

Fixes: https://tracker.ceph.com/issues/42609
Signed-off-by: Volker Theile <vtheile@suse.com>

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Recent versions of openSUSE are fully Python 3 systems, but Python 2 can still
be installed on them. When Python 2 *and* Python 3 are installed, tox will try
to run unit tests ("make check" tests) on both. But the spec-file-based
mechanism used by install-deps.sh for determining "make check" dependencies only
installs python3-virtualenv and python3-devel.

Fixes: https://tracker.ceph.com/issues/23981
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32172 from wjwithagen/wjw-fix-test_cls_rbd-compat

test/cls_rbd: include compat.h for ERESTART

Reviewed-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: allow to have different priority levels

for aggregated and per-image perf counters

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32182 from sebastian-philipp/upstream-pull-31757

mgr/test_orchestrator: Add dummy data

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31920 from ricardoasmarques/fix-sso-certs

mgr/dashboard: Invalid SSO configuration when certificate path does not exist

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Use badges for counters in tabs (#31938)

mgr/dashboard: Use badges for counters in tabs

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31787 from shyukri/wip-36728-c_v-util-which-path

ceph-volume: util: look for executable in $PATH

mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Allow List[HostSpec] in PlacementSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa: Enable flake8 tox and fix failures

There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #32080 from smithfarm/wip-43171

ceph.spec.in: move distro-conditional deps to dedicated section

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/rook: adapt to arg passing change in orchestrator_cli.py

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/ssh: Add SimpleScheduler and streamline arg passing

* adding a mechanism to augment specs.placement.nodes
  even when no nodes are provided via dynamic determination
  from host_pools. (NodeAssignment)

* added SimpleScheduler
  implements most simple placment algorithm possible.
  Choose from a shuffled list of hosts.

* streamlined args passed to the public update_$service
  functions. They all take `spec` now.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch_cli: adapt to changes in PlacementSpec and ServiceSpec

* adapt to changes in PlacementSpec (count moved to PlacementSpec)
* added label param to update_& functions

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch: extend PlacementSpec and add StatefulServiceSpec

* add public set_nodes()
* add StatefulServiceSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>