qa: Fix flake8 for qa/tasks/cephfs/test_volumes.py

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

qa: Run flake8 on python2 and python3

To be able to catch problems with python2 *and* python3, run flake8
with both versions. From the flake8 homepage:

It is very important to install Flake8 on the correct version of
Python for your needs. If you want Flake8 to properly parse new
language features in Python 3.5 (for example), you need it to be
installed on 3.5 for Flake8 to understand those features. In many
ways, Flake8 is tied to the version of Python on which it runs.

Also fix the problems with python3 on the way.
Note: This requires now the six module for teuthology. But this is
already an install_require in teuthology itself.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #32230 from shellwedance/ceph-volume_typo_fix

doc: fix a typo in a command

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32170 from rosinL/wip-rss

msg/async/DPDK: refactor set_rss_table to support DPDK 19.05

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix a typo in a command

Co-authored-by: donggyu_park <donggyu_park@tmax.co.kr>
Signed-off-by: taeuk_kim <taeuk_kim@tmax.co.kr>

Merge PR #32179 into master

* refs/pull/32179/head:
lvm: add deactivate subcommand
util/system: add unmount_tmpfs helper
api/lvm: add get_lv_by_osd_id method
api/lvm: add deactivate method to Volume class

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32225 into master

* refs/pull/32225/head:
qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32003 into master

* refs/pull/32003/head:
mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror
Allow List[HostSpec] in PlacementSpec
mgr/rook: adapt to arg passing change in orchestrator_cli.py
mgr/ssh: Add SimpleScheduler and streamline arg passing
orch_cli: adapt to changes in PlacementSpec and ServiceSpec
orch: extend PlacementSpec and add StatefulServiceSpec

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32218 into master

* refs/pull/32218/head:
cephadm: use raw string during `re.sub`
cephadm: fix ConfigParser DeprecationWarning in py3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32187 into master

* refs/pull/32187/head:
ceph.in: (py3) use binary stdout
pybind/mgr: index=4, sort_keys=True for json.dumps throughput

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32013 from liewegas/doc-foundation

doc/foundation: add cloudbase and vexxhost

Merge pull request #32197 from neha-ojha/wip-report-doc-bug

doc/_templates/page.html: redirect to etherpad

Reviewed-by: Zac Dover zac.dover@gmail.com
Reviewed-by: Sage Weil sage@redhat.com

qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32220 into master

* refs/pull/32220/head:
qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31629 into master

* refs/pull/31629/head:
cephfs-shell: rearrange code for convenience

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Otherwise the test breaks if there are other cluster(s) on the host (like
there normally are in my dev environment).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32129 into master

* refs/pull/32129/head:
qa: Enable flake8 tox and fix failures

ceph.in: (py3) use binary stdout

This fixes an error like

$ ceph -s -o -
Traceback (most recent call last):
  File "bin/ceph", line 1267, in <module>
    retval = main()
  File "bin/ceph", line 1229, in main
    outf.write(outbuf)
TypeError: write() argument must be str, not bytes

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr: index=4, sort_keys=True for json.dumps throughput

- fixes balancer test on py3 by making the pool ls order deterministic
- makes everybody else behave consistently

I left internal json storage alone--no need to sort/normalize.  I also
left out some CLI commands returning non-indented json.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32094 from cbodley/wip-40585

test/rgw: raise timer durations for unittest_rgw_reshard_wait

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32153 from smithfarm/wip-23981

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

cephadm: use raw string during `re.sub`

cephadm:198: DeprecationWarning: invalid escape sequence \s
  nice_conf = re.sub('\n(\s)+', '\n', raw_conf)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: fix ConfigParser DeprecationWarning in py3

DeprecationWarning: This method will be removed in future versions.
Use 'parser.read_file()' instead.

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32151 from liewegas/fix-scipy

install-deps.sh: preload wheel for all mgr requirements.txt files

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

lvm: add deactivate subcommand

This new subcommand unmounts and OSDs tmpfs mount and closes crypt
devices if there are any.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

util/system: add unmount_tmpfs helper

Thsi unmounts a path if and only if it's a tmpfs mount.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

msg/async/dpdk: refactor set_rss_table to support DPDK 19.05

The filter API was deprecated in DPDK 19.05, using the flow
API to set rss table.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Recent versions of openSUSE are fully Python 3 systems, but Python 2 can still
be installed on them. When Python 2 *and* Python 3 are installed, tox will try
to run unit tests ("make check" tests) on both. But the spec-file-based
mechanism used by install-deps.sh for determining "make check" dependencies only
installs python3-virtualenv and python3-devel.

Fixes: https://tracker.ceph.com/issues/23981
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32172 from wjwithagen/wjw-fix-test_cls_rbd-compat

test/cls_rbd: include compat.h for ERESTART

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32182 from sebastian-philipp/upstream-pull-31757

mgr/test_orchestrator: Add dummy data

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31920 from ricardoasmarques/fix-sso-certs

mgr/dashboard: Invalid SSO configuration when certificate path does not exist

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Use badges for counters in tabs (#31938)

mgr/dashboard: Use badges for counters in tabs

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31787 from shyukri/wip-36728-c_v-util-which-path

ceph-volume: util: look for executable in $PATH

mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Allow List[HostSpec] in PlacementSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa: Enable flake8 tox and fix failures

There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #32080 from smithfarm/wip-43171

ceph.spec.in: move distro-conditional deps to dedicated section

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/rook: adapt to arg passing change in orchestrator_cli.py

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/ssh: Add SimpleScheduler and streamline arg passing

* adding a mechanism to augment specs.placement.nodes
  even when no nodes are provided via dynamic determination
  from host_pools. (NodeAssignment)

* added SimpleScheduler
  implements most simple placment algorithm possible.
  Choose from a shuffled list of hosts.

* streamlined args passed to the public update_$service
  functions. They all take `spec` now.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch_cli: adapt to changes in PlacementSpec and ServiceSpec

* adapt to changes in PlacementSpec (count moved to PlacementSpec)
* added label param to update_& functions

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch: extend PlacementSpec and add StatefulServiceSpec

* add public set_nodes()
* add StatefulServiceSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #32175 from rzarzynski/wip-crimson-errorator-do_with-friendship

crimson: make seastar::do_with() a friend of errorated futures.

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: install scipy for build too

Needed for make check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32143 from tchaikov/wip-qa-rocksdb-test

qa/workunits/rados/test_envlibrados_for_rocksdb: accomodate rocksdb c…

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephfs-shell: rearrange code for convenience

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/_templates/page.html: redirect to etherpad

This simplifies things for users who are not able to add cards to
https://github.com/ceph/ceph/projects/4 due to insufficient permissions.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32099 from hjwsm1989/wip-cancel-invalid-upmap-2

crush: remove invalid upmap items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #32193 into master

* refs/pull/32193/head:
qa/tasks/ceph2 -> cephadm
mgr/ssh -> mgr/cephadm
mgr/ssh,cephadm: ceph-daemon -> cephadm
doc/bootstrap: ceph-daemon -> cephadm
ceph-daemon-adoption-corpus -> cephadm-adoption-corpus
qa: rename ceph-daemon tests -> cephadm
qa/tasks: ceph-daemon -> cephadm throughput var names and comments
mgr/ssh,qa/tasks/ceph2: fix mode to be cephadm-package (vs root)
mgr/ssh: rename option ceph_daemon_path -> cephadm_path
files,rpm,deb: rename ceph-daemon -> cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/tasks/ceph2 -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh -> mgr/cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,cephadm: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

doc/bootstrap: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon-adoption-corpus -> cephadm-adoption-corpus

Signed-off-by: Sage Weil <sage@redhat.com>

qa: rename ceph-daemon tests -> cephadm

Also move the workunit to a better location.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks: ceph-daemon -> cephadm throughput var names and comments

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,qa/tasks/ceph2: fix mode to be cephadm-package (vs root)

At the same time align the option names with ceph2.py, yay.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: rename option ceph_daemon_path -> cephadm_path

Signed-off-by: Sage Weil <sage@redhat.com>

files,rpm,deb: rename ceph-daemon -> cephadm

This is just renaming the files and adjusting the packages.  Lots of
cleanup to do still.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32189 into master

* refs/pull/32189/head:
mgr/orchestrator_cli: set type for 'orchestrator' option

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32157 into master

* refs/pull/32157/head:
ceph-daemon: fix pod stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

python-common: Also test mgr/test_orchestrator/dummy_data.json

Make sure, `pybind/mgr/test_orchestrator/dummy_data.json` is not getting out of date.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: set type for 'orchestrator' option

If we don't specify the type, the mgr_module.py will str() the default
value, turning it into "None", breaking the self-test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31729 into master

* refs/pull/31729/head:
qa: reduce cache size further
mds: obsoleting 'mds_cache_size'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31718 into master

* refs/pull/31718/head:
mds: Reorganize class members in ScrubStack header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31717 into master

* refs/pull/31717/head:
mds: Reorganize class members in ScrubHeader header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31716 into master

* refs/pull/31716/head:
mds: Reorganize class members in ScatterLock header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31635 into master

* refs/pull/31635/head:
mds: Reorganize class members in RecoveryQueue header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31597 into master

* refs/pull/31597/head:
mds: Reorganize class members in OpenFileTable header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30201 into master

* refs/pull/30201/head:
qa/tasks: remove subvolume, subvolumegroup and their snapshots with --force
qa/tasks: Fix the commands success
qa/tasks: Fix raises that doesn't re-raise

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32178 into master

* refs/pull/32178/head:
qa/standalone/test_ceph_daemon.sh: disable adoption for the moment

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32176 from idryomov/wip-doc-fancy-striping

doc: clarify support for rbd fancy striping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32163 into master

* refs/pull/32163/head:
qa: fix output check to not be sensitive to debugging

Reviewed-by: Rishabh Dave <ridave@redhat.com>

ceph.spec.in: move distro-conditional deps to dedicated section

Post e92cb7a0336406a2981e9241031497b1749b26aa cleanup. Restore previous
structure of "distro-conditional make check dependencies" section.

Fixes: https://tracker.ceph.com/issues/43171
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32083 from smithfarm/wip-42612

rpm: add rpm-build to SUSE-specific make check deps

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/test_orchestrator: Add dummy data

Add a JSON file containing dummy data for testing the orchestrator.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #32160 into master

* refs/pull/32160/head:
doc/releases/nautilus,PendingReleaseNotes: consolidate telemetry note

Reviewed-by: Nathan Cutler <ncutler@suse.com>

api/lvm: add get_lv_by_osd_id method

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

qa/standalone/test_ceph_daemon.sh: disable adoption for the moment

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32161 from mgfritch/wip-cd-tox-warn

ceph-daemon: `imp` module  DeprecationWarning

mgr/dashboard: Disable cache for static files (#31018)

mgr/dashboard: Disable cache for static files

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

api/lvm: add deactivate method to Volume class

Simply calls lvchange -an to deactivate a logical volume.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

doc: clarify support for rbd fancy striping

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson: make seastar::do_with() a friend of errorated futures.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/cls_rbd: include compat.h for ERESTART

Fixes:9328b402c2d471c7cfc34fa9414825bf2915e7a2
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #32025 from tchaikov/wip-pick-address-silence-warning

common/pick_address.cc: silence GCC warning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32091 into master

* refs/pull/32091/head:
common/config: update values when they are removed via mon

Reviewed-by: Neha Ojha <nojha@redhat.com>

common/pick_address.cc: silence GCC warning

* use `string_view` instead of string
* use `auto` instead of `int` to silence the warning
* use `enum class` instead of plain `enum`
* do not specify the value of enum, as
  we don't care about it

this change silences following warning

```
../src/common/pick_address.cc: In function ‘int get_iface_numa_node(const string&, int*)’:
../src/common/pick_address.cc:517:11: warning: comparison of integer expressions of different signedness: ‘int’ and ‘const size_type’ {aka ‘const long unsigned int’} [-Wsign-com\
pare]
  517 |   if (pos != string::npos ) {
      |       ~~~~^~~~~~~~~~~~~~~
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31839 from alimaredia/wip-s3-test-branch-refactor

qa: update s3-test download code for s3-test tasks

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa: fix output check to not be sensitive to debugging

The traceback would screw up the equality checks.

Also, use unittest asserts so we can easily see the assert values.

Fixes: https://tracker.ceph.com/issues/43247
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31772 into master

* refs/pull/31772/head:
qa/standalone/test_ceph_daemon.sh: clone corpus explicitly
ceph-daemon: re-enable the OSD standalone test
ceph-daemon: move standalone test tgz to corpus
ceph-daemon: add ceph-daemon-adoption-corpus.git submodule

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30548 from trociny/wip-41557

rbd-mirror: snapshot mirror mode

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31122 into master

* refs/pull/31122/head:
mds: Reorganize class members in MDSTable header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31253 into master

* refs/pull/31253/head:
mds: Reorganize class members in Migrator header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31596 into master

* refs/pull/31596/head:
mds: Reorganize class members in PurgeQueue header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: reduce cache size further

1M isn't low enough to trigger recall/trimming.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: `imp` module  DeprecationWarning

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc/releases/nautilus,PendingReleaseNotes: consolidate telemetry note

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32148 into master

* refs/pull/32148/head:
doc/releases/nautilus: fix release notes (crash->device)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

ceph-daemon: fix pod stop

- rm -f, not stop, so we don't leave things lingering in 'podman ps -a'
  output
- this also seems to fix/work around a problem where stopping the container
  leaves the daemon running?

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32041 from dzafman/wip-balancer3-pending

release note: Add pending release notes for already merged code

Reviewed-by: Neha Ojha <nojha@redhat.com>