Merge PR #32225 into master

* refs/pull/32225/head:
qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32003 into master

* refs/pull/32003/head:
mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror
Allow List[HostSpec] in PlacementSpec
mgr/rook: adapt to arg passing change in orchestrator_cli.py
mgr/ssh: Add SimpleScheduler and streamline arg passing
orch_cli: adapt to changes in PlacementSpec and ServiceSpec
orch: extend PlacementSpec and add StatefulServiceSpec

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32218 into master

* refs/pull/32218/head:
cephadm: use raw string during `re.sub`
cephadm: fix ConfigParser DeprecationWarning in py3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32187 into master

* refs/pull/32187/head:
ceph.in: (py3) use binary stdout
pybind/mgr: index=4, sort_keys=True for json.dumps throughput

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32013 from liewegas/doc-foundation

doc/foundation: add cloudbase and vexxhost

Merge pull request #32197 from neha-ojha/wip-report-doc-bug

doc/_templates/page.html: redirect to etherpad

Reviewed-by: Zac Dover zac.dover@gmail.com
Reviewed-by: Sage Weil sage@redhat.com

qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32220 into master

* refs/pull/32220/head:
qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31629 into master

* refs/pull/31629/head:
cephfs-shell: rearrange code for convenience

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Otherwise the test breaks if there are other cluster(s) on the host (like
there normally are in my dev environment).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32129 into master

* refs/pull/32129/head:
qa: Enable flake8 tox and fix failures

ceph.in: (py3) use binary stdout

This fixes an error like

$ ceph -s -o -
Traceback (most recent call last):
  File "bin/ceph", line 1267, in <module>
    retval = main()
  File "bin/ceph", line 1229, in main
    outf.write(outbuf)
TypeError: write() argument must be str, not bytes

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr: index=4, sort_keys=True for json.dumps throughput

- fixes balancer test on py3 by making the pool ls order deterministic
- makes everybody else behave consistently

I left internal json storage alone--no need to sort/normalize.  I also
left out some CLI commands returning non-indented json.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32094 from cbodley/wip-40585

test/rgw: raise timer durations for unittest_rgw_reshard_wait

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32153 from smithfarm/wip-23981

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

cephadm: use raw string during `re.sub`

cephadm:198: DeprecationWarning: invalid escape sequence \s
  nice_conf = re.sub('\n(\s)+', '\n', raw_conf)

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: fix ConfigParser DeprecationWarning in py3

DeprecationWarning: This method will be removed in future versions.
Use 'parser.read_file()' instead.

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32151 from liewegas/fix-scipy

install-deps.sh: preload wheel for all mgr requirements.txt files

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install python2-{virtualenv,devel} on SUSE if needed

Recent versions of openSUSE are fully Python 3 systems, but Python 2 can still
be installed on them. When Python 2 *and* Python 3 are installed, tox will try
to run unit tests ("make check" tests) on both. But the spec-file-based
mechanism used by install-deps.sh for determining "make check" dependencies only
installs python3-virtualenv and python3-devel.

Fixes: https://tracker.ceph.com/issues/23981
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32172 from wjwithagen/wjw-fix-test_cls_rbd-compat

test/cls_rbd: include compat.h for ERESTART

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32182 from sebastian-philipp/upstream-pull-31757

mgr/test_orchestrator: Add dummy data

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31920 from ricardoasmarques/fix-sso-certs

mgr/dashboard: Invalid SSO configuration when certificate path does not exist

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Use badges for counters in tabs (#31938)

mgr/dashboard: Use badges for counters in tabs

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31787 from shyukri/wip-36728-c_v-util-which-path

ceph-volume: util: look for executable in $PATH

mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Allow List[HostSpec] in PlacementSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa: Enable flake8 tox and fix failures

There were a couple of problems found by flake8 in the qa/
directory (most of them fixed now). Enabling flake8 during the usual
check runs hopefully avoids adding new issues in the future.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #32080 from smithfarm/wip-43171

ceph.spec.in: move distro-conditional deps to dedicated section

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

mgr/rook: adapt to arg passing change in orchestrator_cli.py

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/ssh: Add SimpleScheduler and streamline arg passing

* adding a mechanism to augment specs.placement.nodes
  even when no nodes are provided via dynamic determination
  from host_pools. (NodeAssignment)

* added SimpleScheduler
  implements most simple placment algorithm possible.
  Choose from a shuffled list of hosts.

* streamlined args passed to the public update_$service
  functions. They all take `spec` now.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch_cli: adapt to changes in PlacementSpec and ServiceSpec

* adapt to changes in PlacementSpec (count moved to PlacementSpec)
* added label param to update_& functions

Signed-off-by: Joshua Schmid <jschmid@suse.de>

orch: extend PlacementSpec and add StatefulServiceSpec

* add public set_nodes()
* add StatefulServiceSpec

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #32175 from rzarzynski/wip-crimson-errorator-do_with-friendship

crimson: make seastar::do_with() a friend of errorated futures.

Reviewed-by: Liu-Chunmei <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: install scipy for build too

Needed for make check

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32143 from tchaikov/wip-qa-rocksdb-test

qa/workunits/rados/test_envlibrados_for_rocksdb: accomodate rocksdb c…

Reviewed-by: Neha Ojha <nojha@redhat.com>

cephfs-shell: rearrange code for convenience

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/_templates/page.html: redirect to etherpad

This simplifies things for users who are not able to add cards to
https://github.com/ceph/ceph/projects/4 due to insufficient permissions.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32099 from hjwsm1989/wip-cancel-invalid-upmap-2

crush: remove invalid upmap items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #32193 into master

* refs/pull/32193/head:
qa/tasks/ceph2 -> cephadm
mgr/ssh -> mgr/cephadm
mgr/ssh,cephadm: ceph-daemon -> cephadm
doc/bootstrap: ceph-daemon -> cephadm
ceph-daemon-adoption-corpus -> cephadm-adoption-corpus
qa: rename ceph-daemon tests -> cephadm
qa/tasks: ceph-daemon -> cephadm throughput var names and comments
mgr/ssh,qa/tasks/ceph2: fix mode to be cephadm-package (vs root)
mgr/ssh: rename option ceph_daemon_path -> cephadm_path
files,rpm,deb: rename ceph-daemon -> cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/tasks/ceph2 -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh -> mgr/cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,cephadm: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

doc/bootstrap: ceph-daemon -> cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon-adoption-corpus -> cephadm-adoption-corpus

Signed-off-by: Sage Weil <sage@redhat.com>

qa: rename ceph-daemon tests -> cephadm

Also move the workunit to a better location.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks: ceph-daemon -> cephadm throughput var names and comments

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh,qa/tasks/ceph2: fix mode to be cephadm-package (vs root)

At the same time align the option names with ceph2.py, yay.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: rename option ceph_daemon_path -> cephadm_path

Signed-off-by: Sage Weil <sage@redhat.com>

files,rpm,deb: rename ceph-daemon -> cephadm

This is just renaming the files and adjusting the packages.  Lots of
cleanup to do still.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32189 into master

* refs/pull/32189/head:
mgr/orchestrator_cli: set type for 'orchestrator' option

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32157 into master

* refs/pull/32157/head:
ceph-daemon: fix pod stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

python-common: Also test mgr/test_orchestrator/dummy_data.json

Make sure, `pybind/mgr/test_orchestrator/dummy_data.json` is not getting out of date.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: set type for 'orchestrator' option

If we don't specify the type, the mgr_module.py will str() the default
value, turning it into "None", breaking the self-test.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31729 into master

* refs/pull/31729/head:
qa: reduce cache size further
mds: obsoleting 'mds_cache_size'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31718 into master

* refs/pull/31718/head:
mds: Reorganize class members in ScrubStack header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31717 into master

* refs/pull/31717/head:
mds: Reorganize class members in ScrubHeader header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31716 into master

* refs/pull/31716/head:
mds: Reorganize class members in ScatterLock header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31635 into master

* refs/pull/31635/head:
mds: Reorganize class members in RecoveryQueue header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31597 into master

* refs/pull/31597/head:
mds: Reorganize class members in OpenFileTable header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30201 into master

* refs/pull/30201/head:
qa/tasks: remove subvolume, subvolumegroup and their snapshots with --force
qa/tasks: Fix the commands success
qa/tasks: Fix raises that doesn't re-raise

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32178 into master

* refs/pull/32178/head:
qa/standalone/test_ceph_daemon.sh: disable adoption for the moment

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32176 from idryomov/wip-doc-fancy-striping

doc: clarify support for rbd fancy striping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32163 into master

* refs/pull/32163/head:
qa: fix output check to not be sensitive to debugging

Reviewed-by: Rishabh Dave <ridave@redhat.com>

ceph.spec.in: move distro-conditional deps to dedicated section

Post e92cb7a0336406a2981e9241031497b1749b26aa cleanup. Restore previous
structure of "distro-conditional make check dependencies" section.

Fixes: https://tracker.ceph.com/issues/43171
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32083 from smithfarm/wip-42612

rpm: add rpm-build to SUSE-specific make check deps

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/test_orchestrator: Add dummy data

Add a JSON file containing dummy data for testing the orchestrator.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #32160 into master

* refs/pull/32160/head:
doc/releases/nautilus,PendingReleaseNotes: consolidate telemetry note

Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa/standalone/test_ceph_daemon.sh: disable adoption for the moment

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32161 from mgfritch/wip-cd-tox-warn

ceph-daemon: `imp` module  DeprecationWarning

mgr/dashboard: Disable cache for static files (#31018)

mgr/dashboard: Disable cache for static files

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc: clarify support for rbd fancy striping

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson: make seastar::do_with() a friend of errorated futures.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/cls_rbd: include compat.h for ERESTART

Fixes:9328b402c2d471c7cfc34fa9414825bf2915e7a2
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #32025 from tchaikov/wip-pick-address-silence-warning

common/pick_address.cc: silence GCC warning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32091 into master

* refs/pull/32091/head:
common/config: update values when they are removed via mon

Reviewed-by: Neha Ojha <nojha@redhat.com>

common/pick_address.cc: silence GCC warning

* use `string_view` instead of string
* use `auto` instead of `int` to silence the warning
* use `enum class` instead of plain `enum`
* do not specify the value of enum, as
  we don't care about it

this change silences following warning

```
../src/common/pick_address.cc: In function ‘int get_iface_numa_node(const string&, int*)’:
../src/common/pick_address.cc:517:11: warning: comparison of integer expressions of different signedness: ‘int’ and ‘const size_type’ {aka ‘const long unsigned int’} [-Wsign-com\
pare]
  517 |   if (pos != string::npos ) {
      |       ~~~~^~~~~~~~~~~~~~~
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31839 from alimaredia/wip-s3-test-branch-refactor

qa: update s3-test download code for s3-test tasks

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa: fix output check to not be sensitive to debugging

The traceback would screw up the equality checks.

Also, use unittest asserts so we can easily see the assert values.

Fixes: https://tracker.ceph.com/issues/43247
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31772 into master

* refs/pull/31772/head:
qa/standalone/test_ceph_daemon.sh: clone corpus explicitly
ceph-daemon: re-enable the OSD standalone test
ceph-daemon: move standalone test tgz to corpus
ceph-daemon: add ceph-daemon-adoption-corpus.git submodule

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30548 from trociny/wip-41557

rbd-mirror: snapshot mirror mode

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31122 into master

* refs/pull/31122/head:
mds: Reorganize class members in MDSTable header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31253 into master

* refs/pull/31253/head:
mds: Reorganize class members in Migrator header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31596 into master

* refs/pull/31596/head:
mds: Reorganize class members in PurgeQueue header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: reduce cache size further

1M isn't low enough to trigger recall/trimming.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: `imp` module  DeprecationWarning

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc/releases/nautilus,PendingReleaseNotes: consolidate telemetry note

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32148 into master

* refs/pull/32148/head:
doc/releases/nautilus: fix release notes (crash->device)

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

ceph-daemon: fix pod stop

- rm -f, not stop, so we don't leave things lingering in 'podman ps -a'
  output
- this also seems to fix/work around a problem where stopping the container
  leaves the daemon running?

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32041 from dzafman/wip-balancer3-pending

release note: Add pending release notes for already merged code

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/standalone/test_ceph_daemon.sh: clone corpus explicitly

When this is run by teuthology we don't have a full ceph source tree
checkout with submodules.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: re-enable the OSD standalone test

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: move standalone test tgz to corpus

Fixes: https://tracker.ceph.com/issues/42876
Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: add ceph-daemon-adoption-corpus.git submodule

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #32127 into master

* refs/pull/32127/head:
mgr/rook: Added Mypy static type checking

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32138 into master

* refs/pull/32138/head:
ceph-daemon: combine SUDO and ARGS into a single var

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32137 into master

* refs/pull/32137/head:
ceph-daemon: don't deref symlinks during chown

Reviewed-by: Sage Weil <sage@redhat.com>

qa: update s3-test download code for s3-test tasks

- Ensure the download code for all tasks running
s3-tests is consistent.
- Simplify download code to only use the config
variable 'force-branch' for the branch being
cloned.
- make ceph-master the force-branch for all
suites using s3-tests.

Fixes: https://tracker.ceph.com/issues/43077
Signed-off-by: Ali Maredia <amaredia@redhat.com>

rpm: add rpm-build to SUSE-specific make check deps

The binary /usr/bin/rpmspec was recently moved to rpm-build, breaking
install-deps.sh on openSUSE Tumbleweed. The package is not strictly
needed for SLE-15-SP* and openSUSE Leap 15.*, but it doesn't hurt to
have it, and will future-proof these distros from this regression.

Putting the dependency in the spec file does not address the issue,
because /usr/bin/rpmspec must be available before install-deps.sh runs it to
determine the dependencies, but it's nice to have it explicitly listed there,
since it *is* a dependency of "make check" on SUSE distros.

SUSE versions < 15 are of no interest in master/octopus+.

Fixes: https://tracker.ceph.com/issues/42612
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32145 into master

* refs/pull/32145/head:
doc: show how to count jobs before triggering them

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32140 from tchaikov/wip-cmake-cleanup

cmake: silence messages when cppcheck/IWYU is not found

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>