Merge pull request #33889 from ifed01/wip-ifed-flexible-bluefs-nautilus

nautilus: os/bluestore: more flexible DB volume space usage

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

test/store_test: test coverage for anti-spillover framework.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 57c6a79136a390c74a800d03bc595f3363718c2f)

 Conflicts:
src/os/bluestore/BlueFS.h
Future stuff tried to get into.

os/bluestore: introduce bluestore_volume_selection_reserved cfg option.

This is an alternative (and straightforward) way to specify barred space not allowed for 'use
some extra' BlueFS volume selector policy.
Disabled by default since it should depend on RocksDB settings and actual volume size.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 185e6a1adb2add91209d5f937c61c309528518f9)

os/bluestore: beautify RocksDBBlueFSVolumeSelector::dump output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fc84588536b4757bbe75065951a7215ffbcc0f60)

os/bluestore: introduce 'bluestore bluefs stats' admin socket command

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f5568eb7209e43e6b916ea8450b2bd0c9484919a)

 Conflicts:
src/os/bluestore/BlueFS.cc
 Future stuff, legacy admin command iface
        src/os/bluestore/BlueStore.cc
 Need to use legacy bluefs_shared_bdev

os/bluestore: implement more intelligent DB volume space managment for BlueFS.

It allows excessive space usage for higher DB levels.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 5f72c376deb64562e5e88be2f22339135ac7372b)

 Conflicts:
src/common/legacy_config_opts.h
src/common/options.cc
Future options tried to get into.

Merge pull request #34373 from votdev/wip-44690-nautilus

nautilus: mgr/dashboard: list configured prometheus alerts

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34433 from badone/wip-44260-nautilus

nautilus: selinux: Allow ceph to setsched

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34434 from badone/wip-nautilus-ceph-mgr-httpd-selinux-denial

nautilus: selinux: Allow ceph-mgr access to httpd dir

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #31842 from sseshasa/wip-42662-nautilus

nautilus: osd/OSDMap: Show health warning if a pool is configured with size 1

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34120 from tspmelo/wip-rbd-error

nautilus: mgr/dashboard: Fix error when listing RBD while deleting or moving

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34099 from shyukri/wip-44472-nautilus

nautilus: rgw: clear ent_list for each loop of bucket list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33641 from tchaikov/nautilus-42477

nautilus: tools/rados: Unmask '-o' to restore original behaviour

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #33160 from shyukri/wip-43995-nautilus

nautilus: ceph_argparse: increment matchcnt on kwargs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34417 from smithfarm/wip-44952-nautilus

nautilus: mgr/dashboard: Repair broken grafana panels

Reviewed-by: Laura Paduano <lpaduano@suse.com>

selinux: Allow ceph to setsched

In several places, such as common/numa.cc we call sched_setaffinity
which requires this permission.

Fixes: https://tracker.ceph.com/issues/44196
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 43103e0207bfacf02f7f9533b36443d65f95d718)

selinux: Allow ceph-mgr access to httpd dir

ceph-mgr loads modules which require read access and this causes a
denial on el7.

Fixes: https://tracker.ceph.com/issues/44216
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

dashboard: Convert FQDN to hostname in grafana panels

The $ceph_hosts variable contained the FQDN for hosts
while the instance label created by ceph only has
the hostname.

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit b7abaab5bd2b72f48f41b3d2a9732743afdcecfc)

dashboard: Resolve FQDN / hostname mismatch in hosts overview panel

In the AVG Disk Utilization panel, the result is calculated
by combining the output of node_disk_io_time_seconds_total
with the output of ceph_disk_occupation. However, the
first vector encodes the instance label with the full FQDN
while the ceph label only contains the hostname:port. In
order for these to match correctly, the domain name and port
has to be stripped from the labels.

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit 136d21e21dc3c05ec8c586a47eed1904ffbda578)

dashboard: Use exported_instance to identify OSDs

When moving to LVM-based ceph-volume setups, several
grafana dashboards stopped working. The problem is that
(device, instance) no longer results in unique labels
which causes errors like:

"many-to-many matching not allowed: matching labels must be unique on one side"

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit 8b61b8d3d781b9d762098cf45b41b596a32509ab)

dashboard: AVG RAM Utilization panel always showed "N/A"

The references to `$osd_hosts` etc. were encoded as
`[[osd_hosts]]` in the PromQL expression divisor, and
the panel always displayed N/A as the result of the
query.

Replacing the `[[...]]` with `$...` makes the expression
work again.

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit 4444333243aaa81de1d7347bf0c57e039b657d43)

Merge pull request #34222 from ceph/wip-yuriw-clients-upgrades-nautilus-octopus

qa/tests: initial check in for client-upgrade-nautilus-octopus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: initial check in for client-upgrade-nautilus-octopus

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34129 from rhcs-dashboard/nautilus-44666-fix-sso-certificate-file

nautilus: mgr/dashboard: fix error when enabling SSO with cert. file

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: show alert panel if prometheus/alertmanager is unconfigured

If the tabs under the "Monitoring" page aren't properly configured, a
notification is shown which explains the user which setting needs to be
enabled and also provides a link to the corresponding documentation.

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 460f7bb3272c6536c9a5fc0919071d7c17e9aa5a)

Conflicts:
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/monitoring-list/monitoring-list.component.html
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/monitoring-list/monitoring-list.component.ts
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.ts

mgr/dashboard: extend dashboard documentation

by adding the previously added monitoring related features as well as
the newest feature addition. Extends the documentation where necessary
to describe the Prometheus' alert configuration.

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 36421284c315baf7e79a8c0586ca98ac0126037e)

mgr/dashboard: move monitoring tabs to a single page

with a tab for 'active alerts', 'all alerts' and 'silences'. Due to
ambiguity with existing names, `AlertListComponent` has been renamed to
`ActiveAlertListComponent`. Introduces `MonitoringListComponent` as
first page for monitoring concerns, using path `/monitoring`.

Keeps the activated tab open, independent of the way that's used to go
back to the previous page, be it the cancel button or submit button or
the link on the breadcrumb. Also keeps the active tab open even when the
page is reloaded.

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 855f214b29c8ed935c8f4ba0b8a8396692f946a1)

Conflicts:
  src/pybind/mgr/dashboard/frontend/e2e/cluster/alerts.e2e-spec.ts
  src/pybind/mgr/dashboard/frontend/e2e/cluster/alerts.po.ts
  src/pybind/mgr/dashboard/frontend/src/app/app-routing.module.ts
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/cluster.module.ts
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/active-alert-list/active-alert-list.component.spec.ts
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/active-alert-list/active-alert-list.component.ts
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs/prometheus-tabs.component.html
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/prometheus/prometheus-tabs/prometheus-tabs.component.spec.ts
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.ts
  src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.html

mgr/dashboard: Unify Tasks and Notifications into a sidebar

Fixes: https://tracker.ceph.com/issues/37402
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 7fdb3c1504a96a991041934c8abd6453d762857d)

Conflicts:
  src/pybind/mgr/dashboard/frontend/e2e/cluster/crush-map.po.ts
  src/pybind/mgr/dashboard/frontend/package-lock.json
  src/pybind/mgr/dashboard/frontend/package.json
  src/pybind/mgr/dashboard/frontend/src/app/app.component.html
  src/pybind/mgr/dashboard/frontend/src/app/app.module.ts
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.html
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.scss
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.ts
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/task-manager/task-manager.component.html
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/task-manager/task-manager.component.spec.ts
  src/pybind/mgr/dashboard/frontend/src/app/core/navigation/task-manager/task-manager.component.ts
  src/pybind/mgr/dashboard/frontend/src/app/shared/components/components.module.ts
  src/pybind/mgr/dashboard/frontend/src/styles/popover.scss

mgr/dashboard: refactor test of Prometheus alert service

Mocking the test the way it was removed the asynchronous nature of the
test. By using an Observable the test can stay asynchronous and be
tested as well.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit b85b24b2d085fd775298625b546b9fd5abe4d741)

mgr/dashboard: list configured Prometheus alerts

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 2a50e277d4ae764aa50f9b726f79c84f335ef15a)

Conflicts:
  src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/cluster.module.ts

mgr/dashboard: return Prometheus alert groups, too

Fixes: https://tracker.ceph.com/issues/42877
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 4917279f9f5ed321cada297706a96e7ce16f01eb)

Merge pull request #34316 from tchaikov/nautilus-six

nautilus: pybind/mgr: use six==1.14.0

Reviewed-by: Laura Paduano <lpaduano@suse.com>

pybind/mgr: use six==1.14.0

this change is not cherry-picked from master. because in master, we
are using different ways to install six:

see
- 68527ab83a77d20b779759581ce0b8844991231b
- 1e07237d92da3bb5eba82b49f6276d17696e6985

because in teuthology we are using six.ensure_str, which was added in
six 1.12.0, see https://github.com/benjaminp/six/blob/1.12.0/CHANGES ,
we cannot continue using six 1.11.0, as a result, we need switch over to
six>1.12.0. since the latest stable version of six is now 1.14.0, let's
just use it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34162 from k0ste/wip-44735-nautilus

nautilus: mgr/prometheus: ceph_pg_* metrics contains last value instead of sum across all reported states

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32849 from smithfarm/wip-43469-nautilus

nautilus: osd/PeeringState: transit async_recovery_targets back into acting before backfilling

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

tools/rados: Unmask '-o' to restore original behaviour

0b369e1aff1 masked the original behaviour of '-o' which was to indicate
'outfile' as documented in the man page. Changing object-size to capital
o will restore the original behaviour.

Fixes: https://tracker.ceph.com/issues/42477
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit d2b806070de5f56474283035a7ffc91fe96afbab)

Conflicts:
PendingReleaseNotes: trivial resolution

Merge pull request #34116 from smithfarm/wip-43820-nautilus

nautilus: pybind/mgr/*: fix config_notify handling of default values

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33665 from rhcs-dashboard/wip-44375-nautilus

nautilus: mgr/dashboard: do not show RGW API keys if only read-only privileges

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34039 from rhcs-dashboard/wip-44639-nautilus

nautilus: mgr/dashboard: Pool read/write OPS shows too many decimal places

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33141 from shyukri/wip-44046-nautilus

nautilus: mgr/telemetry: catch exception during requests.put

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #33024 from shyukri/wip-39230-nautilus

nautilus: mgr/dashboard: Updated existing E2E tests to match new format

Reviewed-by: Laura Paduano <lpaduano@suse.com>

osd/OSDMap: Show health warning if a pool is configured with size 1

Introduce a config option called 'mon_warn_on_pool_no_redundancy' that is
used to show a health warning if any pool in the ceph cluster is
configured with a size of 1. The user can mute/unmute the warning using
'ceph health mute/unmute POOL_NO_REDUNDANCY'.

Add standalone test to verify warning on setting pool size=1. Set the
associated warning to 'false' in ceph.conf.template under qa/tasks so
that existing tests do not break.

Fixes: https://tracker.ceph.com/issues/41666
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 33c647e8114b37404d8d62a08c85664cea709118)

 Conflicts:
PendingReleaseNotes
- Added release notes under 14.2.9
qa/standalone/mon/health-mute.sh
- Deleted the script as 'health mute/unmute' cmd is unavailable in nautilus
qa/tasks/ceph.conf.template
- Removed a flag not available in nautilus
src/common/options.cc
- Removed a flag not available in nautilus
src/osd/OSDMap.cc

Merge pull request #30704 from vshankar/wip-41508

nautilus: mds: display scrub status in ceph status

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test: query using mds id, not rank

Fixes: http://tracker.ceph.com/issues/42835
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit fb4cd42cbcdaa4812838b846ae33ee26356ce733)

mgr: re-enable mds `scrub status` info in ceph status

MDS does not register with manager, therefore `task status`
(scrub status updates) were not getting displayed in ceph
status output.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit d6ad0a255a27c5e3da20c7ae03066a692a6d49bb)

 Conflicts:
src/mgr/DaemonServer.cc
src/mgr/DaemonServer.h

mgr/DaemonServer.{h,cc} deals with raw pointers while master uses ref_t<>
cast -- adjust to that. a minor conflict in the header and the metrics
templatization is not backported to nautilus. also, DaemonKey is a std::pair
in nautilus but a struct in master -- that requires a change in referencing
daemon type and name.

mon: filter out ceph normal ceph entity types when dumping service metadata

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit dfc056028e44cdbda84d7118e29120ae70edb35a)

mgr: filter out normal ceph services when processing service map

This is done is couple of places in ceph manager -- when culling
entries from service map and the other when dumping serice status.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 20139d12423fc6154b57499c576d1e4bb3f1eade)

 Conflicts:
src/mgr/DaemonServer.cc

Switch to non-initializer list in range-based for loop.

mgr: helper function to check if a service is a normal ceph service

This would be widely required since ceph metadata server entries are
maintained in service map (DaemonServer::pending_service_map). Such
normal ceph services would need to filtered when processing the service
map to avoid extraneous entries getting processed.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 79503fc16749ed0cfe8a89ea3b3c8c792d6b8809)

mds: establish session with mgr only after added to FSMap

Note that we now sub to the mgrmap after init because the MgrClient
connection to the mgr is driven by receipt of the MgrMap.

This is important so that the MDS does not have metadata with the mgr
when the mons are ignoring the MDS otherwise due to CompatSet
incompatibilities.

Fixes: https://tracker.ceph.com/issues/41538
Fixes: https://tracker.ceph.com/issues/42635
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit e765f2d533440cfc4189f36fcaba24617a302e84)

 Conflicts:
src/mds/MDSDaemon.cc

Nautilus uses Lock/Unlock rather than lock/unlock in master.

mds: do not register as a service daemon

This commit undoes the service daemon registration for the MDS. It doesn't look
absolutely necessary and it causes the MDS to be listed twice in the `ceph
versions` output:

    $ ceph versions
        ...
        "mds": {
            "ceph version v15.0.0-6915-g0143b904676 (0143b9046763ea1801efa8358a0c033ec862cea9) octopus (dev)": 3
        },
        "mds": {
            "unknown": 3
        },
        "overall": {
            "ceph version v15.0.0-6915-g0143b904676 (0143b9046763ea1801efa8358a0c033ec862cea9) octopus (dev)": 10,
            "unknown": 3
        }

Fixing that requires looking for duplicates or ignoring MDSs in the
service daemons when the mon processes `ceph versions`. I have a feeling
that it wasn't actually designed to be used by the MDS this way however.
Additionally, the reason for "unknown" version is because the metadata
sent to the mgr does not include "ceph_version".

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 046137f819aae72f1423e3feb213f0e46c97c9ce)

mds: do not try to diagnose cause of MDSMap removal

The Monitors send an empty MDSMap to an MDS it is removing. The MDS
can't diagnose the cause. Instead suggest looking at the cluster/monitor
logs.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c385178b0c04d09bc6774acf5bdb4537fa6803ad)

mds: fix handling of initial MDS states

Few things here:

- Make explicit the check for getting removed from the MDSMap. This was
  only done before by checking if MDS held a rank which does not check the
  case where a standby is removed from the FSMap.

- Use mds_info_t::dump to simplify various debug output.

- Add a few sanity asserts for invalid state transitions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 26a08df2adc7495fc660113cd26c33d5debd3ee6)

 Conflicts:
src/mds/MDSDaemon.h

Nautilus still uses "const_ref" rather than "cref_t<>" in master.

mds: remove unnecessary const qualifier

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 3355581a298963711ad058c6e47d21b7eab887a4)

mds: cleanup type decl and map iteration

This is a trivial refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit bafa3b731ff93a356fcf60ebbaf0e5f299182992)

mds: define stream operator for mds_info_t

This also cleans up the output to be more readable/useful in debug
output.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 1c56632e88a126d0eac75235d9a9716833cba6b7)

 Conflicts:
PendingReleaseNotes

Fix minor conflict in PendingReleaseNotes.

mds/MDSRank: report state to mgr as mds id, not rank

Introduced by 625dffe65c0f8001b3b6ca6d0b12732a1a103849, which added
periodic scrub stats reporting.

Fixes: https://tracker.ceph.com/issues/42494
Fixes: https://tracker.ceph.com/issues/41525
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit a1a220d137f7fa128d08f96d1eff83865388e5fc)

mgr: missing lock release in DaemonServer::handle_report()

Fixes: http://tracker.ceph.com/issues/42169
Introduced-by: 5c25a01864
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit cda3eadbfbd3e3a8a057cc7ed042ba0a6d7fef11)

mgr/ServiceMap: use plain gid

* partially revert 5c25a018. which is not backward compatible.
* change `ServiceMap::get_daemon()` so it returns a
  `pair<Daemon*,bool>`.

git is an `optional<uint64_t>`, so we cannot dump it without checking.

Fixes: http://tracker.ceph.com/issues/41424
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3e65551d0ab4acafaf4012601987b3391b2411ed)

doc: document scrub summary in ceph status output

Fixes: http://tracker.ceph.com/issues/36370
Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 05d17994a879995b56bda5d770a938d0aabaaed9)

test: extend scrub control test to validate mds task status

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 465a3adc6c31fd9b8359920ab47adf8e1f45d5f1)

mds: send scrub state changes to cluster log.

... also log new and completed scrubs.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 49520156414cb0667e8edd6ea4b31462e7cb7752)

mds: periodically sent mds scrub status to ceph manager

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 625dffe65c0f8001b3b6ca6d0b12732a1a103849)

 Conflicts:
src/mds/MDSRank.cc

mgr, mon: allow normal ceph services to register with manager

Additionally, introduce `task status` field in manager report
messages to forward status of executing tasks in daemons (e.g.,
status of executing scrubs in ceph metadata servers).

`task status` makes its way upto service map which is then used
to display the relevant information in ceph status.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 5c25a018643b10aa78db8270cae1476f71d8f4f4)

 Conflicts:
src/messages/MMgrReport.h
src/mgr/DaemonServer.cc
src/mgr/ServiceMap.h

pybind/mgr/*: fix config_notify handling of default values

After 5ace82e65c72847fb875fc01c419937a26a59d70 was merged, I found
three more instances of the code being patched.

The commit message of 5ace82e65c72847fb875fc01c419937a26a59d70 was/is:

"The default values are handled by mgr_module.py's _get_module_option();
the or here means that we break any non-true (0, false, none) value and
override it with the default."

Fixes: 5ace82e65c72847fb875fc01c419937a26a59d70
Fixes: https://tracker.ceph.com/issues/43746
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 30585f44de6147c824ed2df1a477ad17f7b51fff)

Conflicts:
src/pybind/mgr/osd_support/module.py
- file does not exist in nautilus

Merge pull request #33684 from k0ste/wip-44367-nautilus

nautilus: doc/mgr/telemetry: added device channel details

doc/mgr/telemetry: added device channel details

Plus details about license agreement.

Fixes: https://tracker.ceph.com/issues/43648
Signed-off-by: Yaarit Hatuka <yaarit@redhat.com>
(cherry picked from commit 66c41f3d0e516e217dc89ab20c53563ea10f97f7)

mgr: ceph_pg_* metrics contains last value instead of sum all of them

During evaluation of pool stats metrics contains last reported value instead of sum

Fixes: https://tracker.ceph.com/issues/44590
Signed-off-by: Jacek Suchenia <jacek.suchenia@gmail.com>
(cherry picked from commit ad4d790a35af483f3a240f247cda5754301a9199)

Merge pull request #34038 from rhcs-dashboard/wip-44334-nautilus

nautilus: mgr/dashboard: UI fixes

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33764 from shyukri/wip-44327-nautilus

nautilus: mgr/dashboard: 'destroyed' view in CRUSH map viewer

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33763 from shyukri/wip-43258-nautilus

nautilus: mgr/dashboard: Disable cache for static files

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by:  Tiago Melo <tmelo@suse.com>

Merge pull request #33739 from rhcs-dashboard/wip-44435-nautilus

nautilus: mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: fix error when enabling SSO with cert. file

Nautilus dedicated fix: added py2 compatibility code.
Also:
* Disabled security setting 'wantNameIdEncrypted': not all Identity Providers support this and we are already requiring encrypted assertions (which is the default).

Fixes: https://tracker.ceph.com/issues/44666
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

pybind/mgr/*: fix config_notify handling of default values

The default values are handled by mgr_module.py's _get_module_option();
the or here means that we break any non-true (0, false, none) value and
override it with the default.

https://tracker.ceph.com/issues/43746
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5ace82e65c72847fb875fc01c419937a26a59d70)

Conflicts:
src/pybind/mgr/cephadm/module.py
- nautilus has no "cephadm" module. It does have an "orchestrator_cli"
  module but it doesn't contain the code being patched
src/pybind/mgr/hello/module.py
- nautilus has a "hello" module, but it doesn't contain the code being
  patched

mgr/dashboard: Fix error when listing RBD while deleting or moving

This fix is based on a commit made to octopus, that fixed this problem in
octopus/master: 5627919c016785a27666b1992d336f8ff378a072.

Since the codebase is very different and a backport is not recommended or even
possible, I have created this commit with only the minimal code necessary.

Fixes: https://tracker.ceph.com/issues/42330
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #33738 from rhcs-dashboard/wip-44434-nautilus

nautilus: vstart.sh: set prometheus port for each mgr.

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #33530 from thmour/nautilus_test

nautilus: osd: fix racy accesses to OSD::osdmap.

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph_argparse: increment matchcnt on kwargs

Currently when you pass a param in on the ceph cli as a kwarg
(--<param_name>) the matchcnt isn't incremented in the validate method
which is used to choose the right command signature.

This means currently things fail like:

  ceph orchestrator rgw rm --realm_name=default --zone_name=default
  no valid command found; 1 closest matches:
  orchestrator rgw rm <realm_name> <zone_name>
  Error EINVAL: invalid command

The '--realm_name' and '--zone_name' isn't counted to the matchcnt, so
'orchestrator rgw rm' isn't picked as the valid command.

This patch simply corrects this by incrementing matchcnt on the kwarg
validate path before calling shortcircuiting the loop.

Fixes: https://tracker.ceph.com/issues/43803
Signed-off-by: Matthew Oliver <moliver@suse.com>
(cherry picked from commit cb37c9ee609864a078edf38d98608bd8cc18cbd7)

Conflicts:

  test: exclude helper method from nosetest discovery

  On nautilus the assertion helper was recognized by nosetest as a test
  even though it doens't start with test_ prefix. Explicitely decorate it
  with @nottest

Signed-off-by: Shyukri Shyukriev <shshyukriev@suse.com>

rgw: clear ent_list for each loop of bucket list

if ent_list is not cleared, the old element will be checked repeatedly
and will occupy more memory.
Fixes: http://tracker.ceph.com/issues/44394
Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>
(cherry picked from commit f63bf47aa464c345c907c748dfdbbc5a239d8488)

Merge pull request #34064 from smithfarm/wip-44145-nautilus

nautilus: rgw: set correct storage class for append

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33913 from shyukri/wip-42324-nautilus

nautilus: qa: get rid of iterkeys for py3 compatibility

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33649 from k0ste/wip-44364-nautilus

nautilus: mgr/telemetry: fix and document proxy usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33421 from k0ste/wip-44218-nautilus

nautilus: common/blkdev: fix some problems with smart scraping

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33420 from k0ste/wip-44219-nautilus

nautilus: mgr/pg_autoscaler: fix division by zero

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33409 from k0ste/wip-44199-nautilus

nautilus: mgr/telemetry: add 'last_upload' to status

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33355 from shyukri/wip-44141-nautilus

nautilus: rgw: sts: add all http args to req_info

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #33893 from rhcs-dashboard/wip-44200-nautilus

nautilus: mgr: Add get_rates_from_data to mgr_util.py

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33275 from smithfarm/wip-44081-nautilus

nautilus: mon/PGMap: fix summary display of >32bit pg states

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33261 from smithfarm/wip-43997-nautilus

nautilus: global: ensure CEPH_ARGS is decoded before early arg processing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33032 from shyukri/wip-42164-nautilus

nautilus: mgr/dashboard: REST API: OpenAPI docs require internet connection

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #32891 from shyukri/wip-42673-nautilus

nautilus: mgr/dashboard: Searchable objects for table

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34087 from jdurgin/wip-target-ratio-nautilus

nautilus: mgr/pg_autoscaler: treat target ratios as weights

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Pool read/write OPS shows too many decimal places
Fixes: https://tracker.ceph.com/issues/39714
Signed-off-by: anurag <anurag@localhost.localdomain>
(cherry picked from commit 27a2bbb12614b7aba0561c027346d9b5427f2405)
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
Conflicts:
        src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.spec.ts,
        src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table-key-value/table-key-value.component.spec.ts:
    added import of PipesModule to Angular unit tests

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: do not show API keys if user has only read-only privileges.

Fixes: https://tracker.ceph.com/issues/42475
Signed-off-by: Alfonso Martínez <almartin@redhat.com>
(cherry picked from commit 38d419620508af75f7c164b56d07d1ecde9173b4)

Conflicts:
- src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details/rgw-user-details.component.spec.ts
- src/pybind/mgr/dashboard/tests/test_rgw.py
Adapted (after cherry-pick) frontend & backend unit tests to nautilus code.

PendingReleaseNotes: mention target_size_ratio change

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 9118065c4c3803f8a2c26c28c69ffc9f14ff260a)

Merge pull request #34036 from ajarr/nautilus-volumes-cancel-clone

nautilus: fs subvolume clone cancel

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc: update autoscaler docs for target ratio

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 0a29cf7d6c9076c79416ef7868be3635921fd140)

mgr/pg_autoscaler: include effective target ratio in status

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 9758197afa856f3680d5b5d599c79353816e9927)

mgr/pg_autoscaler: add warning when target bytes and ratio are both set

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 772d7c1d3c1adf49821670da0f6cdb06e293dc0d)

mgr/pg_autoscaler: remove target ratio warning

Since the ratios are normalized, they cannot exceed 1.0 or overcommit
combined with target_bytes.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit d62c121ee3839a2fd4243a9b1c33aded1cc3db13)

qa: use ratios >1 in pg_autoscaler test

Also check for pg_num_target being set correctly, rather than pg_num,
so the test doesn't depend on merging/splitting speed.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
(cherry picked from commit 1ea8ce3480e96aa87b4d4f570022f9844bc54c17)