mgr/dashboard: Make form modal more flexible

Now any input type is supported and the special binary type which will
use the cd-binary input and automatically format the binary size into
bytes when submitting.

Now any field can have custom validators and error messages.
The form will fallback on predefined error messages if not configured.
It will use the error messages provided by the binary min and max
validators. It still provides the error message for a required field.

Fixes: https://tracker.ceph.com/issues/38287
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: CephFS quota management

Now both CephFS quotas can be changed with a validation against the next
tree maximum in the file tree, that prevents setting the quotas in a way
that would not be usable.

Fixes: https://tracker.ceph.com/issues/38287
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #31851 from votdev/issue_42609_device_lights

mgr/dashboard: Add support for blinking enclosure LEDs

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32186 from rhcs-dashboard/43263-fix-rgw-subuser-key

mgr/dashboard: fix RGW subuser auto-generate key.

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31753 from smithfarm/wip-bci-resolve-fix

backport-create-issue: resolve parent only if parent has backport issues

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Simplify PrometheusService::getRules implementa… (#32130)

mgr/dashboard: Simplify PrometheusService::getRules implementation

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #32177 from Takios/ceph_volume_fix

ceph-volume: use correct extents if using db-devices and >1 osds_per_device

Merge pull request #32224 from tspmelo/wip-e2e-insecure

mgr/dashboard: Fix e2e chromedriver problem

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31255 into master

* refs/pull/31255/head:
mds/MDSDaemon: make con disposable before passing message to asok
mds/SessionMap: interpret bare id as a valid filter
mds: get rid of unused send_command_reply helpers
mds: remove old handle_command machinery
mds/MDSRank: remove C_ExecAndReply helpers
mds/MDSRank: too many words about op_tracker
mds: convert 'cache drop'
mds: convert 'damage ls' and 'damage rm'
mds: convert '{session,client} config'
mds: convert 'scrub {start,abort,pause,resume,status}'
mds: convert 'session ls' and 'client ls'
mds: convert 'session evict' and 'session kill'
mds: convert 'heap' and 'cpu_profiler'
mds: convert 'exit', 'respawn'
mds/MDSDaemon: remove old duplicated tell commands
mds: send tell commands to asok infrastructure
common/admin_socket: switch tell queue to cref_t (from ref_t)
mds: switch asok calls over to call_async interface

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32230 from shellwedance/ceph-volume_typo_fix

doc: fix a typo in a command

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32170 from rosinL/wip-rss

msg/async/DPDK: refactor set_rss_table to support DPDK 19.05

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix a typo in a command

Co-authored-by: donggyu_park <donggyu_park@tmax.co.kr>
Signed-off-by: taeuk_kim <taeuk_kim@tmax.co.kr>

Merge PR #32179 into master

* refs/pull/32179/head:
lvm: add deactivate subcommand
util/system: add unmount_tmpfs helper
api/lvm: add get_lv_by_osd_id method
api/lvm: add deactivate method to Volume class

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32225 into master

* refs/pull/32225/head:
qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32003 into master

* refs/pull/32003/head:
mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror
Allow List[HostSpec] in PlacementSpec
mgr/rook: adapt to arg passing change in orchestrator_cli.py
mgr/ssh: Add SimpleScheduler and streamline arg passing
orch_cli: adapt to changes in PlacementSpec and ServiceSpec
orch: extend PlacementSpec and add StatefulServiceSpec

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32218 into master

* refs/pull/32218/head:
cephadm: use raw string during `re.sub`
cephadm: fix ConfigParser DeprecationWarning in py3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32187 into master

* refs/pull/32187/head:
ceph.in: (py3) use binary stdout
pybind/mgr: index=4, sort_keys=True for json.dumps throughput

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32013 from liewegas/doc-foundation

doc/foundation: add cloudbase and vexxhost

Merge pull request #32197 from neha-ojha/wip-report-doc-bug

doc/_templates/page.html: redirect to etherpad

Reviewed-by: Zac Dover zac.dover@gmail.com
Reviewed-by: Sage Weil sage@redhat.com

qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32220 into master

* refs/pull/32220/head:
qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Reviewed-by: Michael Fritch <mfritch@suse.com>

mds/MDSDaemon: make con disposable before passing message to asok

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: interpret bare id as a valid filter

This works around the fact that the old asok 'session evict' took a bare
id and the tell version took a filter.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: get rid of unused send_command_reply helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mds: remove old handle_command machinery

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSRank: remove C_ExecAndReply helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSRank: too many words about op_tracker

Signed-off-by: Sage Weil <sage@redhat.com>

mds: convert 'cache drop'

Signed-off-by: Sage Weil <sage@redhat.com>

mds: convert 'damage ls' and 'damage rm'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31629 into master

* refs/pull/31629/head:
cephfs-shell: rearrange code for convenience

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Fix e2e chromedriver problem

New version of Chrome/ChromeDriver seems to have a problem with insecure
certificates.

Since we use a vStart cluster with no real certificate,
we always face this problem when running e2e tests.

This will fix the current problem and hopefully any future problems related to
certificates.

Fixes: https://tracker.ceph.com/issues/43254
Signed-off-by: Tiago Melo <tmelo@suse.com>

qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Otherwise the test breaks if there are other cluster(s) on the host (like
there normally are in my dev environment).

Signed-off-by: Sage Weil <sage@redhat.com>

backport-create-issue: resolve parent only if parent has backport issues

This fixes an edge case. If a parent issue is in "Pending Backport" status
but without any backport issues, the script (when running with --resolve-parent)
was wrongly changing the status to "Resolved".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32129 into master

* refs/pull/32129/head:
qa: Enable flake8 tox and fix failures

ceph.in: (py3) use binary stdout

This fixes an error like

$ ceph -s -o -
Traceback (most recent call last):
  File "bin/ceph", line 1267, in <module>
    retval = main()
  File "bin/ceph", line 1229, in main
    outf.write(outbuf)
TypeError: write() argument must be str, not bytes

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr: index=4, sort_keys=True for json.dumps throughput

- fixes balancer test on py3 by making the pool ls order deterministic
- makes everybody else behave consistently

I left internal json storage alone--no need to sort/normalize.  I also
left out some CLI commands returning non-indented json.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32094 from cbodley/wip-40585

test/rgw: raise timer durations for unittest_rgw_reshard_wait

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32153 from smithfarm/wip-23981

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

cephadm: use raw string during `re.sub`

cephadm:198: DeprecationWarning: invalid escape sequence \s
  nice_conf = re.sub('\n(\s)+', '\n', raw_conf)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: fix ConfigParser DeprecationWarning in py3

DeprecationWarning: This method will be removed in future versions.
Use 'parser.read_file()' instead.

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32151 from liewegas/fix-scipy

install-deps.sh: preload wheel for all mgr requirements.txt files

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

lvm: add deactivate subcommand

This new subcommand unmounts and OSDs tmpfs mount and closes crypt
devices if there are any.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

util/system: add unmount_tmpfs helper

Thsi unmounts a path if and only if it's a tmpfs mount.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

msg/async/dpdk: refactor set_rss_table to support DPDK 19.05

The filter API was deprecated in DPDK 19.05, using the flow
API to set rss table.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

mgr/dashboard: Add support for blinking enclosure LEDs

Fixes: https://tracker.ceph.com/issues/42609
Signed-off-by: Volker Theile <vtheile@suse.com>

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Recent versions of openSUSE are fully Python 3 systems, but Python 2 can still
be installed on them. When Python 2 *and* Python 3 are installed, tox will try
to run unit tests ("make check" tests) on both. But the spec-file-based
mechanism used by install-deps.sh for determining "make check" dependencies only
installs python3-virtualenv and python3-devel.

Fixes: https://tracker.ceph.com/issues/23981
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32172 from wjwithagen/wjw-fix-test_cls_rbd-compat

test/cls_rbd: include compat.h for ERESTART

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32182 from sebastian-philipp/upstream-pull-31757

mgr/test_orchestrator: Add dummy data

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31920 from ricardoasmarques/fix-sso-certs

mgr/dashboard: Invalid SSO configuration when certificate path does not exist

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Use badges for counters in tabs (#31938)

mgr/dashboard: Use badges for counters in tabs

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31787 from shyukri/wip-36728-c_v-util-which-path

ceph-volume: util: look for executable in $PATH

mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Allow List[HostSpec] in PlacementSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa: Enable flake8 tox and fix failures

There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #32080 from smithfarm/wip-43171

ceph.spec.in: move distro-conditional deps to dedicated section

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/rook: adapt to arg passing change in orchestrator_cli.py

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/ssh: Add SimpleScheduler and streamline arg passing

* adding a mechanism to augment specs.placement.nodes
  even when no nodes are provided via dynamic determination
  from host_pools. (NodeAssignment)

* added SimpleScheduler
  implements most simple placment algorithm possible.
  Choose from a shuffled list of hosts.

* streamlined args passed to the public update_$service
  functions. They all take `spec` now.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch_cli: adapt to changes in PlacementSpec and ServiceSpec

* adapt to changes in PlacementSpec (count moved to PlacementSpec)
* added label param to update_& functions

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch: extend PlacementSpec and add StatefulServiceSpec

* add public set_nodes()
* add StatefulServiceSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #32175 from rzarzynski/wip-crimson-errorator-do_with-friendship

crimson: make seastar::do_with() a friend of errorated futures.

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: install scipy for build too

Needed for make check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32143 from tchaikov/wip-qa-rocksdb-test

qa/workunits/rados/test_envlibrados_for_rocksdb: accomodate rocksdb c…

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephfs-shell: rearrange code for convenience

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/_templates/page.html: redirect to etherpad

This simplifies things for users who are not able to add cards to
https://github.com/ceph/ceph/projects/4 due to insufficient permissions.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32099 from hjwsm1989/wip-cancel-invalid-upmap-2

crush: remove invalid upmap items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #32193 into master

* refs/pull/32193/head:
qa/tasks/ceph2 -> cephadm
mgr/ssh -> mgr/cephadm
mgr/ssh,cephadm: ceph-daemon -> cephadm
doc/bootstrap: ceph-daemon -> cephadm
ceph-daemon-adoption-corpus -> cephadm-adoption-corpus
qa: rename ceph-daemon tests -> cephadm
qa/tasks: ceph-daemon -> cephadm throughput var names and comments
mgr/ssh,qa/tasks/ceph2: fix mode to be cephadm-package (vs root)
mgr/ssh: rename option ceph_daemon_path -> cephadm_path
files,rpm,deb: rename ceph-daemon -> cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/tasks/ceph2 -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh -> mgr/cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,cephadm: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

doc/bootstrap: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon-adoption-corpus -> cephadm-adoption-corpus

Signed-off-by: Sage Weil <sage@redhat.com>

qa: rename ceph-daemon tests -> cephadm

Also move the workunit to a better location.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks: ceph-daemon -> cephadm throughput var names and comments

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,qa/tasks/ceph2: fix mode to be cephadm-package (vs root)

At the same time align the option names with ceph2.py, yay.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: rename option ceph_daemon_path -> cephadm_path

Signed-off-by: Sage Weil <sage@redhat.com>

files,rpm,deb: rename ceph-daemon -> cephadm

This is just renaming the files and adjusting the packages.  Lots of
cleanup to do still.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32189 into master

* refs/pull/32189/head:
mgr/orchestrator_cli: set type for 'orchestrator' option

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32157 into master

* refs/pull/32157/head:
ceph-daemon: fix pod stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

python-common: Also test mgr/test_orchestrator/dummy_data.json

Make sure, `pybind/mgr/test_orchestrator/dummy_data.json` is not getting out of date.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: set type for 'orchestrator' option

If we don't specify the type, the mgr_module.py will str() the default
value, turning it into "None", breaking the self-test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31729 into master

* refs/pull/31729/head:
qa: reduce cache size further
mds: obsoleting 'mds_cache_size'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31718 into master

* refs/pull/31718/head:
mds: Reorganize class members in ScrubStack header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31717 into master

* refs/pull/31717/head:
mds: Reorganize class members in ScrubHeader header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31716 into master

* refs/pull/31716/head:
mds: Reorganize class members in ScatterLock header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31635 into master

* refs/pull/31635/head:
mds: Reorganize class members in RecoveryQueue header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31597 into master

* refs/pull/31597/head:
mds: Reorganize class members in OpenFileTable header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30201 into master

* refs/pull/30201/head:
qa/tasks: remove subvolume, subvolumegroup and their snapshots with --force
qa/tasks: Fix the commands success
qa/tasks: Fix raises that doesn't re-raise

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32178 into master

* refs/pull/32178/head:
qa/standalone/test_ceph_daemon.sh: disable adoption for the moment

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: fix RGW subuser auto-generate key.

Fixes: https://tracker.ceph.com/issues/43263
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #32176 from idryomov/wip-doc-fancy-striping

doc: clarify support for rbd fancy striping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32163 into master

* refs/pull/32163/head:
qa: fix output check to not be sensitive to debugging

Reviewed-by: Rishabh Dave <ridave@redhat.com>

ceph.spec.in: move distro-conditional deps to dedicated section

Post e92cb7a0336406a2981e9241031497b1749b26aa cleanup. Restore previous
structure of "distro-conditional make check dependencies" section.

Fixes: https://tracker.ceph.com/issues/43171
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32083 from smithfarm/wip-42612

rpm: add rpm-build to SUSE-specific make check deps

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/test_orchestrator: Add dummy data

Add a JSON file containing dummy data for testing the orchestrator.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #32160 into master

* refs/pull/32160/head:
doc/releases/nautilus,PendingReleaseNotes: consolidate telemetry note

Reviewed-by: Nathan Cutler <ncutler@suse.com>

api/lvm: add get_lv_by_osd_id method

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/standalone/test_ceph_daemon.sh: disable adoption for the moment

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: use correct extents when using db-devices and >1 osds_per_device

Actual data size depending on osds_per_device needs to be calculated here. Otherwise, if osds_per_device is greater than 1, ceph-volume will allocate 100% of the device to the first osd and then fail to create the LV for the second because the volume group is already full.

Fixes: https://tracker.ceph.com/issues/39442
Signed-off-by: Fabian Niepelt <f.niepelt@mittwald.de>

Merge pull request #32161 from mgfritch/wip-cd-tox-warn

ceph-daemon: `imp` module  DeprecationWarning

mgr/dashboard: Disable cache for static files (#31018)

mgr/dashboard: Disable cache for static files

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>