cephadm: shell --mount shouldnt enforce ':z' option

cephadm shouldn't enforce this option.
For instance, it can be an issue when you try to bindmount a file in /usr

Fixes: https://tracker.ceph.com/issues/52853
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit aeae360e2595f348f10ddc36cdb270b018f7eb02)

src/cephadm: Add mariner as a supported distro.

Using cephadm install commands on Mariner will result in an error
"Distro not supported" though the packages are supported and can be
installed manually. This patch adds mariner as a supported distro.
Fix validated on mariner.

Signed-off-by: Roaa Sakr <romoh@microsoft.com>
(cherry picked from commit 1612427ea549d3fc259c8eb0e37953a9323dcb31)

cephadm: add daemon_name in daemon description

This adds the daemon_name in json output when asking for daemon
description.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 19ee16a46b90d92f1a49ace5ba1a3c47eb8c1403)

doc/cephadm/services: unindent note section

before this change the note on "apply" command is embedded in the note
on "_no_schedule". and they are not related. so let's move the former
out. also, highlight the yaml file sample in YAML.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 48f5cd5b9bb1ccf89e1eaad163cf11874ad3e176)

mgr/cephadm: populate trusted_ip_list in iscsi-gateway.cfg with mgr ips

Iscsi gateways do not show "UP" in dashboard without this

fixes: https://tracker.ceph.com/issues/52692
Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>
(cherry picked from commit cda82c98a32f51cb392fc51ba854bcae409567f8)

Conflicts:
src/pybind/mgr/cephadm/module.py

cephadm/iscsi: fix a typo

This commit fixes a typo in the comment of unit.run of iscsi service.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 5a4f29de7a6080b0476ad134426faf94c01fb316)

cephadm: fix iscsi test

the current test is wrong because it generates the tcmu-runner part two
times.
given the function `deploy_daemon_units()` in cephadm already writes a
first time the tcmu-runner command, calling a second time
`get_tcmu_runner_container()` from the test makes `deploy_daemon_units()`
write the same command again.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 7e9762e0ce36d773c34fc9353d2eacc694554f39)

cephadm: modify iscsi containers bindmount

tcmu-runner logs in `/var/log/tcmu-runner.log`, there's no option to
make it log to stdout/stderr so the log is only available from the
container.
Modifying the bindmount from `-v /var/log/ceph/<fsid>/:/var/log/rbd-target-api:z`
to `-v /var/log/ceph/<fsid>/:/var/log:z` makes it at least available
from the host.

Fixes: https://tracker.ceph.com/issues/47480
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit b130214195ff84af57f447cf3c3410f0918becaf)

doc/cephadm: move services into services/

This is going to clean up the toctree a bit.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 8c70398949f773d8b992ad8ae6c71383460d2932)

doc/cephadm: move service-management to services/index.rst

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 5e25b19cc7bd5845eccdd0bbb3d6ee8a5852563d)

doc/cephadm: Add `networks` to spec documentation

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit fd32c1651b95ac14662bb5048e42f92fc1f45e56)

doc/cephadm: add mgr.rst

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 6773cb01b8f42a4a37b4360d7a032524df39a6f8)

doc/cephadm: Add `Specifying Networks` to rgw.rst

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 4243a7936a9972a1d9c677207ad6a4e56908e7af)

mgr/dashboard: More robust handling of timedelta() representation

The representation of timedelta() appears to have changed recently so
we should try and be more accommodating in that regard.

Fixes: https://tracker.ceph.com/issues/52696
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit ad1a4f2410393cab6ac728c4ddb79d111e252559)

cephadm: Fix TestMaintenance trying to list /var/lib/ceph

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 767564864f15ea8afe9b925345cdf59db1815cb0)

cephadm: TestCheckHost: also mock `check_time_sync`

Fixes: https://tracker.ceph.com/issues/52722
```
TestCheckHost.test_container_engine fails at cephadm:5834: Error cephadm.Error: No time synchronization is active
```

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit fc9d9b39144fd16f2b2a9621f80dcd1162459000)

mgr/cephadm: base maintenance enter/exit success off of return message

rather than on whether there was any stdout from the command

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 393330bae6b259102bba83b0d3c6213970838c02)

mgr/cephadm: unit tests for maintenance enter/exit properly handling success/failure messages

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 5350d58b9ad45359b7885d19592c2b4826e9aed7)

Conflicts:
src/pybind/mgr/cephadm/tests/test_cephadm.py

cephadm: unit tests for maitenance mode return values

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 73e6aa8d9ca69c96361fae6998dd71d252808aa4)

cephadm: fix mypy complaints for ThreadedChildWatcher class

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 8bdddfa02eea7331971c16a9f8806b8451f09c5b)

cephadm: fix exiting maintenance when systemd target doesn't exist

If the systemd target doesn't exist we need to just bypass enabling
it and return success or the host will just be stuck in maitnenance
mode.

Signed-off-by: Adam King <adking@redhat.com>
(cherry picked from commit 3a15f1dc81dfd5f963e1bf36316fa5b289a898b0)

cephadm: raise error during `pull` failure

instead of a traceback to the console

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit d07e72c4b0bdc4d74cf1a0d8826061942f757d08)

mgr/cephadm: osd should not be zap when it is running

When osd is running, we exec the command of "ceph orch device zap".
Then ceph-volume will exec dd command which cause the osd of lvs to be destroyed.
If the osd of systemctl unit restarts, it will be unable to boot.

Signed-off-by: jianglong01 <jianglong01@qianxin.com>
(cherry picked from commit aed472f7b6663c487a5ae89fcdd788b32910b621)

doc: from v16.2.6 you have to point to the quay.io image if using docker

New images are no longer uploaded to the Docker Hub registry, so Docker users should point the upgrade
command to the quay.io image.

Fixes: https://tracker.ceph.com/issues/52683
Signed-off-by: Javier Cacheiro <javier.cacheiro.lopez@cesga.es>
(cherry picked from commit 961a4fcffa9bae348810a8542c08ca38947ed111)

doc/ceph-volume/activate: Users should call cephadm instead

Fixes: https://tracker.ceph.com/issues/52575
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 925820e73cfeae290900d2077cc792e9085b5767)

doc/cephadm/service-management: Move Deployment of Daemons

.. into "Daemon Placement"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit b68c99cd739d8f92784a3398e238e3ed6903046f)

doc/cephadm/service-management: Move "Updating Service Specifications"

into "Service Specification"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 80f33a4e7df949759f9ac02c50b3bf99d03678e1)

cephadm: skip podman check during `rm-repo`

allow the `rm-repo` command to succeed when podman is not installed

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit fd977773a57e12003fb02bdc762bf6bc89d785a1)

doc/cephadm: Removing a service

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 7af138e089bf0972a2067f84fe9dd6cd4588e7f8)

doc/cephadm: Add lots of links to other chapters

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit d9ec8eb7a8da3c7dff40d7ed89feaebf7cadd37d)

cephadm: show podman version during `check-host`

Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 44aee33945f285ed4366b960e9526ed9d1984382)

cephadm: avoid unhandled `AttributeError`

when docker/podman are not present

Fixes: https://tracker.ceph.com/issues/51818
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 4d5694a9f0977a22c2a6dac680d594ab3feb070b)

mgr/cephadm: show unhandled exceptions during `host add`

138700e59bcd assumes stderr will always have a line containing the
prefix 'ERROR', which leads to an empty error reason when `check-host`
fails with an unhandled exception

Fixes: https://tracker.ceph.com/issues/51818
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit dac9225085a1f6d2eeaf209fc3d77c54208db2e8)

mgr/cephadm: Add OSDService.post_remove()

Do not remove the osd.N keyring, if we failed to deploy the OSD, because
we cannot recover from it. The OSD keys are created by ceph-volume and not by
us.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit d7a4c5794034e60e94dd12951f7dbf4685647686)

mgr/cephadm: Add MonService.post_remove()

We should never remove the mon keyring. Let's move
this piece of code into the MonService class

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 71eaf35aa755735574f8bc53b38fa1bac550792c)

cephadm: (re)add command argv logging

introduced by 81a7df0498d and inadvertently removed by 3afec2ab30c

Fixes: https://tracker.ceph.com/issues/52484
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 6d18759bcb75c68c3a2d421e5d39c6cee8c18526)

cephadm: add thread ident to log messages

can be used to filter msgs from a specific cephadm command

Fixes: https://tracker.ceph.com/issues/52484
Signed-off-by: Michael Fritch <mfritch@suse.com>
(cherry picked from commit 2f1482890bad99cf29623585c6e4f8abf15cecc5)

qa/distros: Remove stale kubic distros

Cause they're broken

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 65e2cc084073ea7e05ecbc06f5773617676708ff)

qa/distros/podman: Add rhel_8.rhel_8.4_container_tools_3.0.yaml

mainly for the cephfs suite

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 0293580b502da3dc874430861c6cfac976403a67)

doc/cephadm: monitoring: Further Reading

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit dc1180c485f91f86df0194b0769234d51a450816)

.github/labeler: Add monitoring

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 5624a62024ef64209d16d619e4a610870a244f37)

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 90f4cc017a49859224f0817fefc31ae459f5deec)

doc/cephadm: monitoring: Add "Setting up Grafana"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit d17613086697252f31b57cddef1813b3dc2625d8)

doc/cephadm: monitoring: move "deploying w/o" up

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit 5aa5fea8ee3e57e7636627031f651113a1d31cb4)

doc/cephadm: monitoring: default placements

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>
(cherry picked from commit efd79a4adcfefc887bb42f1dc6882eec18576c1d)

Merge pull request #43748 from tchaikov/pacific-doc-build

pacific: admin/doc-requirements.txt: pin Sphinx at 3.5.4

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

admin/doc-requirements.txt: pin Sphinx at 3.5.4

* pin Sphinx at 3.5.4
* pin docutils at 0.18

at least the combination of these two versions
is known to compile.

to address the bug reported at
https://sourceforge.net/p/docutils/bugs/431/

the backtrace looks like:

/home/jenkins-build/build/workspace/ceph-pr-docs/build-doc/virtualenv/lib/python3.8/site-packages/sphinx/util/docutils.py:285:
RemovedInSphinx30Warning: function based directive support is now
deprecated. Use class based directive instead.
  warnings.warn('function based directive support is now deprecated. '

Exception occurred:
  File
"/home/jenkins-build/build/workspace/ceph-pr-docs/build-doc/virtualenv/lib/python3.8/site-packages/docutils/writers/html5_polyglot/__init__.py",
line 445, in section_title_tags
    if (ids and self.settings.section_self_link
AttributeError: 'Values' object has no attribute 'section_self_link'

please note this change is not cherry-picked from
master, because master already bumped Sphinx to 3.5.4
in 4968baa2523bd2a5ca6be147b26bc28906a864c9.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43543 from rhcs-dashboard/wip-52870-pacific

pacific: mgr/dashboard: clean-up controllers and API backward versioning compatibility

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #43417 from trociny/wip-51646-pacific

pacific: osd/OSD: mkfs need wait for transcation completely finish

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43562 from lxbsz/vino_fix

Pacific: test/libcephfs: put inodes after lookup

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43559 from batrick/i52654-pacific

pacific: pybind/mgr/cephadm: set allow_standby_replay during CephFS upgrade

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #43475 from lxbsz/tracker_52876

pacific: test: shutdown the mounter after test finishes

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #43644 from aaSharma14/wip-52965-pacific

pacific: mgr/dashboard: monitoring: grafonnet refactoring for radosgw dashboards

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43619 from smithfarm/wip-53005-pacific

pacific: rgw/tracing: unify SO version numbers within librgw2 package

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43512 from neha-ojha/wip-52770-pacific

pacific: os/bluestore: list obj which equals to pend

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #43513 from neha-ojha/wip-52620-pacific

pacific: osd: fix partial recovery become whole object recovery after restart osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #43511 from neha-ojha/wip-52843-pacific

pacific: msg/async/ProtocolV2: Set the recv_stamp at the beginning of receiving a message

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #43445 from k0ste/wip-52848-pacific

pacific: mgr: Add check to prevent mgr from crashing

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #43437 from trociny/wip-52831-pacific

pacific: osd: re-cache peer_bytes on every peering state activate

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43421 from callithea/wip-52289-pacific

pacific: qa/tasks/mgr: skip test_diskprediction_local on python>=3.8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43353 from kamoltat/wip-ksirivad-backport-pacific-37544

pacific: mgr/progress: optimize global recovery && introduce 5 seconds interval

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: monitoring: grafonnet refactoring for hosts dashboards

This PR intends to refactor hosts dashboards using grafonnet

Fixes:https://tracker.ceph.com/issues/52777
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit f7714de294dd7376a9a8ae5131aa429322b459c3)

Conflicts:
monitoring/grafana/dashboards/jsonnet/grafana_dashboards.jsonnet(merging all the jsonnet dashboards in one PR)

Merge pull request #43646 from rhcs-dashboard/wip-53026-pacific

pacific: mgr/dashboard: pin a version for autopep8 and pyfakefs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: pin a version for autopep8 and pyfakefs

Fixes: https://tracker.ceph.com/issues/53024
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit 946dab4f608ec47e0a3cfefdf8e7d1afda69117f)

mgr/dashboard: monitoring: grafonnet refactoring for cephfs dashboards

This PR intends to refactor cephfs dashboards using grafonnet

Fixes:https://tracker.ceph.com/issues/52777
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit ed954b0e6ce24fbae66f78f7e4f90416b9ed7749)

mgr/dashboard: monitoring: grafonnet refactoring for osds dashboards

This PR intends to refactor osds dashboards using grafonnet

Fixes:https://tracker.ceph.com/issues/52777
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit e490e2f3abe707a2e891171f3c230d44e282c601)

mgr/dashboard: monitoring: grafonnet refactoring for pools dashboards

This PR intends to refactor pools dashboards using grafonnet

Fixes:https://tracker.ceph.com/issues/52777
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit 8c48821c21f7a6b248de10ff6750a63bab1e4948)

mgr/dashboard: monitoring: grafonnet refactoring for rbd dashboards

This PR intends to refactor rbd dashboards using grafonnet

Fixes:https://tracker.ceph.com/issues/52777
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit e737aaa000a31e2f37ca90eb813f031a42edef3b)

mgr/dashboard: monitoring: grafonnet refactoring for radosgw dashboards

This PR intends to refactor radosgw dashboards using grafonnet

Fixes:https://tracker.ceph.com/issues/52777
Signed-off-by: Aashish Sharma <aasharma@redhat.com>
(cherry picked from commit eb01954cd999430417555628e0099f645d371746)

rgw/tracing: unify SO version numbers within librgw2 package

The librgw2 package contains several SO files. Two of those - librgw_op_tp.so
and librgw_rados_tp.so - had a different version number than the main librgw.

This was a violation of the openSUSE Shared Library Packaging Policy [1] but it
also seems like a "violation" of common sense.

[1] https://en.opensuse.org/openSUSE:Shared_library_packaging_policy#Package_naming

Fixes: https://tracker.ceph.com/issues/52979
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 172d6e01d5079f445044da9fe0823ceb353bdc86)

Merge pull request #43548 from rzarzynski/pacific-50483

pacific: msgr/async: fix unsafe access in unregister_conn()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43610 from rhcs-dashboard/wip-pr_triage_dashboard-pacific

.github: add dashboard PRs to Dashboard project

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43440 from rhcs-dashboard/wip-52835-pacific

pacific: qa/mgr/dashboard/test_pool: don't check HEALTH_OK

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

.github/pr-triage: rename GH token

Repo projects use GITHUB_TOKEN instead of MY_GITHUB_TOKEN:
https://github.com/srggrs/assign-one-project-github-action/blob/master/entrypoint.sh#L19

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 2220646c2085f6967e61d21ff19145666f5a1285)

.github: add dashboard PRs to Dashboard project

This action automatically adds PRs with 'dashboard' label to the
'Dashboard' project (https://github.com/ceph/ceph/projects/6).

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit ed55c527f10237c0ab48038639a971e85f8e1377)

Merge pull request #43200 from batrick/i52639

pacific: MDSMonitor: handle damaged state from standby-replay

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/tasks/backfill_toofull: make test work when compression on

The osd backfill reservation does not take compression into account so
we need to operate with "uncompressed" bytes when calculating nearfull
ratio.

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 429ac06cbb44b8a8263beb0d0780a01cedb517ba)

Merge pull request #43267 from cfsnyder/wip-52588-pacific

pacific: ceph-volume: fix lvm activate --all --no-systemd

Merge pull request #43523 from rhcs-dashboard/wip-52911-pacific

pacific:  mgr/dashboard: replace "Ceph-cluster" Client connections with active-standby MGRs

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43541 from rhcs-dashboard/wip-52931-pacific

pacific: mgr/dashboard: Fix orchestrator/01-hosts.e2e-spec.ts failure

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43240 from callithea/wip-52292-pacific

pacific: mgr/dashboard: visual tests: Add more ignore regions for dashboard component

Reviewed-by: aaryanporwal <NOT@FOUND>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

mgr/dashboard: replace string version with class

* APIVersion:
  * Moved to a separate file
  * Added doctests
  * Added sentinel values:
    * DEFAULT = 1.0
    * EXPERIMENTAL = 0.1
    * NONE = 0.0
  * Added to_mime_type() helper method
* Controllers.__init__:
  * Added type hints
  * Replaced string versions with APIVersions
* Feedback controller:
  * Replaced with EXPERIMENTAL (probably it should be NONE)

Fixes: https://tracker.ceph.com/issues/52480
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
 Conflicts:
src/pybind/mgr/dashboard/controllers/__init__.py
   - Remove the current changes and keep the incoming new changes
src/pybind/mgr/dashboard/controllers/crush_rule.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/controllers/docs.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/controllers/feedback.py
   - Deleted the file since feedback module isn't backported to pacific
src/pybind/mgr/dashboard/controllers/host.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/openapi.yaml
   - Generated a new openapi yaml file
src/pybind/mgr/dashboard/tests/__init__.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/tests/test_docs.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/tests/test_host.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/tests/test_tools.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/tests/test_versioning.py
   - Changes related to the versioning like importing the APIVersion
src/pybind/mgr/dashboard/controllers/crush_rule.py
   - Removed the MethodMap decorator which updates the version of the
     enpoint to 2.0 because those changes which caused that version
     updating were not backported to pacific

test: shutdown the mounter after test finishes

In the previous backport commit (5772641cb9bde083), when resolving
the conflicts, this has been missed.

Fixes: https://tracker.ceph.com/issues/52876
Signed-off-by: Xiubo Li <xiubli@redhat.com>

test/libcephfs: put inodes after lookup

Otherwise, the client umount will hang due to inability to trim the
inodes looked up using the low-level interface. This results in slow-op
warnings and an eviction:

2021-09-11T17:23:31.097+0000 7f99c3522700  0 log_channel(cluster) log [WRN] : evicting unresponsive client smithi176 (9756), after 303.924 seconds
2021-09-11T17:23:31.097+0000 7f99c3522700 10 mds.0.server autoclosing stale session client.9756 172.21.15.176:0/3891214934 last renewed caps 303.924s ago

From: /ceph/teuthology-archive/yuriw-2021-09-11_16:21:09-smoke-pacific-distro-basic-smithi/6385038/remote/smithi175/log/ceph-mds.b.log.gz

Fixes: https://tracker.ceph.com/issues/52572
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c0252063b94d811dc7863058999856ac5614d1eb)

 Conflicts:
src/test/libcephfs/test.cc

qa: add test for cephfs upgrade sequence

This also checks max_mds>1 and allow_standby_replay are restored to
previous values.

Future work can add tests for multiple file systems (or volumes).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit b1420e5771927f5c659e0e5edbc5714035f3df09)

qa: add tasks to check mds upgrade state

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5a7382214fe4dbd4b79773c6e732512ade22793a)

qa: add note about where caps are generated

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit dbe5573ed4781cb4b214e701c77be7bc2cddabf3)

qa: move CephManager cluster instantiation to subtask

This needs to be available for the cephfs_setup task so administration
mounts can run ceph commands, potentially through `cephadm shell`.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7812cfb6744fc3bce50e26aa7dd6a4e47a43bb23)

pybind/mgr/cephadm: disable allow_standby_replay during CephFS upgrade

Following procedure in [1].

Also: harden checks for active. Ensure "up" and "in" are both [0]. There
should be no standby-replay daemon.

[1] https://docs.ceph.com/en/pacific/cephfs/upgrading/

Fixes: https://tracker.ceph.com/issues/52654
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit bca21f01ce3bb32e0951f0fe15da88a81750a191)

pybind/mgr/cephadm: always do mds upgrade sequence

Minor versions also require this sequence.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4affb5c7029f6b83d640aa7b7206d9cf61e75f1d)

mgr/dashboard: make modified API endpoints backward compatible

Fixes: https://tracker.ceph.com/issues/52480
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Introducing APIVersion class to handle versioning for API-endpints and making
them backward compatible.

mgr/dashboard: clean-up controllers

Fixes: https://tracker.ceph.com/issues/52589
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
 Conflicts:
src/pybind/mgr/dashboard/CMakeLists.txt
   - Added some testts in the CephTest section

msgr/async: fix unsafe access in unregister_conn()

We were looking at anon_conns and accepting_conns without holding
the lock (deleted_lock is not sufficient).

Drop this test, and move the decrements:

- inc when we add to conns or anon_conns (no changes there)
- dec when we remove from deleted_conns (several different paths!)

Fixes: https://tracker.ceph.com/issues/49237
Signed-off-by: Sage Weil <sage@newdream.net>
(cherry picked from commit d51d80b3234e17690061f65dc7e1515f4244a5a3)
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Fix orchestrator/01-hosts.e2e-spec.ts failure

The test is failing on deleting a host because the agent daemon is
present in that host. Its not possible to simply delete a host. We need
to drain it first and then delete it.

Fixes: https://tracker.ceph.com/issues/52764
Signed-off-by: Nizamudeen A <nia@redhat.com>
(cherry picked from commit db5cfb15e55dadf7bd5c381f53a4ea548fcea152)

mgr/dashboard: replace Client connections with active-stdby mgrs

Fixes: https://tracker.ceph.com/issues/52121
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
(cherry picked from commit d388c5e958ddf5447c78db50ca2061bb443d2227)

osd: fix partial recovery become whole object recovery after restart osd

support SERVER_OCTOPUS feature for pg_missing_item::encode()

Fixes: https://tracker.ceph.com/issues/52583
Signed-off-by: Jianwei Zhang <jianwei1216@qq.com>
(cherry picked from commit dcdb188b6f577551fb377ba34145419f81322b03)

os/bluestore: list obj which equals to pend

otherwise we could have failures like

scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes."

where the numbers of scrubbed object, clones, dirty and omap are always
less than the total number of corresponding numbers, if the PG contains
object(s) whose hash happens to be 0xffffffff.

in this change, if the calculated hash of the upper bound is greater
than the maximum possible number represented by uint32_t, in addition to
setting the hash of the upper bound hobj to 0xffffffff, we also set the
nspace of hobj of the upper bound to "\xff", so that the upper bound
is greater than an hobj whose hash happens to be 0xfffffff. please note,
the nspace of "\xff" is not an ascii string, so it's not likely to be
less than a real-world nspace of an hobj.

with this new *greater* upper bound, we are able to include the previous
missing hobj when listing the objects in a PG. so the scrub won't be
annoyed when the number of objects does not match.

Fixes: https://tracker.ceph.com/issues/52705
Signed-off-by: Mykola Golub <mykola.golub@clyso.com>
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit ffab13bcd9006c1f961a24b8016df9d1fe06ba1d)

os/bluestore: use scope_guard to log latency

simpler this way, and avoid using `goto`.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>
(cherry picked from commit 715a83822ebc1a3d102d1ec13323b69db0600719)

msg/async/ProtocolV2: replace ltt_recv_stamp with recv_stamp

Fixes: https://tracker.ceph.com/issues/52739
Signed-off-by: dongdong tao <dongdong.tao@canonical.com>
(cherry picked from commit 1b1a91c31ba6078caff045c499b8737e0068460f)

msg/async/ProtocolV2: Set the recv_stamp at the beginning of receiving a message instead of after receiving.

Fixes: https://tracker.ceph.com/issues/52739
Signed-off-by: dongdong tao <dongdong.tao@canonical.com>
(cherry picked from commit 5ca30f396bface2a8e95a0efb1b97f8c1b64de1c)