mgr/dashboard: Handle always-on Ceph Manager modules correctly

Currently always-on modules are not marked as enabled in the WebUI and can be disabled. This PR will fix that.
Note, this PR will NOT implement code that will prevent a developer from trying to disable an always-on module through the REST API. The Mgr Python extension will throw an adequate exception.

This PR will also do:
* Remove old code fragments from a previous Mgr Module management UI that is obsolete now.
* Cleanup code in BaseMgrModule code.

Fixes: https://tracker.ceph.com/issues/41648
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #30199 from IlsooByun/cmake_ptyhon_var

build/ops: fix build fail related to PYTHON_EXECUTABLE variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29986 into master

* refs/pull/29986/head:
mgr/pg_autoscaler: use 'stored' for pool_logical_used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30139 into master

* refs/pull/30139/head:
mgr/DaemonServer: skip adjusting pgp_num when merging is in-progress

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30061 from xiexingguo/wip-sparse-read

os, osd: readv

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30059 from xiexingguo/wip-clean-vs-repair

osd/PG: fix _finish_recovery vs repair race

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30190 from tchaikov/wip-python-rpm-macro

install-deps.sh: install `python*-devel` for python*rpm-macros

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

install-deps.sh: install `python*-devel` for python*rpm-macros

in 087ea813, we installed '*rpm-macros' for the macros, so we can have
access to the latest python packaging related macros for preparing the
build dependencies.

but we could run into https://bugs.centos.org/view.php?id=16379, if
we already have an old version of python-devel installed. as the newer
version of python-rpm-macros conflicts with it.

it was a chicken-and-egg problem, as we don't know the exact name of
*rpm-macros packages. that's why we chose to install all of them. but
we have to upgrade the existing python-devel package to resolve the
conflict. but the since there is no python3-devel in RHEL7/CentOS7,
what they have is python36-devel. so we have to hardwire the
`%{python3_pkgversion}` to "36" even before we have access to this
macro, and upgrade the python36-devel package beforehand. but this
renders installing the rpm-macro package less useful -- we intend to
use the macro offered by the package to figure out "36".

as a workaround, we pretend that we know the "main" version of python3
in current RHEL/CentOS. and always install python36-devel for
python-rpm-macros. as the former requires the latter.

once all python3*-devel on all builders are upgraded, we will be safe
to install '*rpm-macros' again without installing python36-devel first.
by then, we could revert this change, or continue installing
python36-devel until the distro bumps up the "main" python version to 3.7

Fixes: https://tracker.ceph.com/issues/41603
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22574 from aclamk/bs-allocator-ageing-test

bluestore/allocator: Ageing test for bluestore allocators.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30028 from tchaikov/wip-rpath

cmake: selectively rewrite install rpath

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #30015 from neha-ojha/wip-41429

mon/Monitor.cc: fix condition that checks for unrecognized auth mode

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30023 from tchaikov/wip-get-req

mon,osd: use get_req<> instead of static_cast<>(get_req())

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30210 from ctrlaltdel/doc-typo

doc: Fix a typo in balancer documentation

Reviewed-by: Lenz Grimmer <LGrimmer@suse.com>

Merge pull request #29520 from pcuzner/add-events-mgr-module

mgr/k8sevents: Add mgr module for kubernetes event integration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

doc: Fix a typo in balancer documentation

Signed-off-by: Francois Deppierraz <francois@ctrlaltdel.ch>

Merge PR #29786 into master

* refs/pull/29786/head:
allow for multiple "build" dirs to exist

Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: fix build fail related to PYTHON_EXECUTABLE variable

Fixes: https://tracker.ceph.com/issues/41676
Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>

allow for multiple "build" dirs to exist

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #29985 from xiexingguo/wip-41514-plus

osd/PrimaryLogPG: cancel in-flight manifest ops on interval changing; fix race with scrub

Reviewed-by: Myoungwon Oh <ommw@sk.com>

Merge pull request #30184 from tchaikov/wip-make-check

run-make-check.sh: extract run-make.sh

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30085 from xiexingguo/wip-41601

osd/PrimaryLogPG: update oi.size on write op implicitly truncating object up

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29842 into master

* refs/pull/29842/head:
kv/KeyValueDB: fix estimate_prefix_size()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29855 into master

* refs/pull/29855/head:
pybind/mgr: remove type conversions for ceph native options
mgr: return get_ceph_option result as typed Py object (not string)

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #30013 from ivancich/wip-beanjoy-update-attrib-tags

rgw: fix refcount tags to match and update object's idtag

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #30062 from david-z/wip-fix-test-assert

test: remove useless ASSERT_XXX macros for rgw test

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29673 from rafaelquint/imagetrash

mgr/dashboard: RBD Image Purge Trash, Move to Trash and Restore

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28755 from dzafman/wip-network

feature: Health warnings on long network ping times, add "dump_osd_network" to get a report

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

run-make-check.sh: extract run-make.sh

so we can reuse run-make.sh for building the artifact used by other
tests than "make check", for instance, dashboard's E2E test and
crimson's performance test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29402 from Devp00l/wip-40828

mgr/dashboard: Use onCancel on any modal event

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30064 from bk201/wip-41368

mgr/dashboard: fix cdEncode decorator is not working on class

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30145 from theanalyst/doc/releases/14.2.3

doc: releases: 14.2.3 dashboard note

https://github.com/ceph/ceph/pull/24179

mgr/dashboard: Empty datatable rendered before data has been f… (#30103)

mgr/dashboard: Empty datatable rendered before data has been fetched

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

test: remove useless ASSERT_XXX macros for rgw test

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #29981 from falcon78921/wip-ceph-conf-usage

ceph-conf: added --show-config-value to usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/k8sevents: Initial ceph -> k8s events integration

Adds a mgr module to provide integration between
Ceph and the kubernetes events API within the rook-ceph
namespace. It provides several commands to view event
status including;

ceph k8sevents list .... show all k8s related events
ceph k8sevents ceph .... show events generated from this
                         module
ceph k8sevents status .. show status of the tracker threads
                         and counts of tracked events

Events sent to kubernetes are sourced from the
a) clog: ceph healtchecks and admin commands
b) explicit checks: hosts, pools and OSD states are checked
   every 'n' seconds

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

ceph-conf: added --show-config-value to ceph-conf usage

--show-config-value is another way to lookup ceph.conf
values. Users may want to utilize --show-config-value instead of
--lookup because --show-config-value also shows global defaults.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: Document network performance monitoring

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #29847 from iotcg/mon_doc

doc: set ceph_perf_msgr_server arguments

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30136 from yuvalif/build_docs_on_fedora30

build-doc: allow building docs on fedora 30

Merge pull request #26755 from cbodley/wip-rgw-bilog-complete

rgw multisite: avoid writing bilog entries on PREPARE and CANCEL

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

osd doc mon mgr: To milliseconds for config value, user input and threshold out

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #29692 from mkogan1/wip-rgw-37615

rgw: fix cls_bucket_list_unordered() partial results

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/dashboard: Refactoring

I came across a yoda case and a test case that didn't use 'done' while
using an async call.

Fixes: https://tracker.ceph.com/issues/40828
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use onCancel on any modal event

The problem was, that if 'onCancel' was provided, it wasn't used in all
modal exit cases. Only for the close buttons (the X and the labeled
one), but not if you clicked outside the modal or hit escape.

'onCancel' is currently only used inside the user management form, where
the problem caused, that the submit button was not longer click able if
the modal was hidden through clicking outside of it or hitting escape.

Fixes: https://tracker.ceph.com/issues/40828
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add tests for confirmation modal

Fixes: https://tracker.ceph.com/issues/40828
Signed-off-by: Stephan Müller <smueller@suse.com>

doc: releases: 14.2.3 dashboard note

The comment unfortunately got buried in a resolved blog

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29973 from theanalyst/doc/releases/14.2.3

doc: release notes for v14.2.3 nautilus

Reviewed-By: Nathan Cutler <nculter@suse.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Lenz Grimmer <lgrimmer@suse.com>

doc: add a final set of notable changes to 14.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

build-doc: allow building docs on fedora 30

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

mgr/DaemonServer: skip adjusting pgp_num when merging is in-progress

otherwise there is a possibility we might change the up set of the
merge participants and hence have to back off the merge process.
Also this will cause some other (racy) corner cases that are hard
to take care of.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29882 from cyx1231st/wip-seastar-msgr-test-lossless-crimson

crimson/net: bug fixes from v2 failover tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30075 from rhcs-dashboard/fix-41600-master

mgr/dashboard: change warn_explicit to warn

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29929 from tspmelo/wip-ng-analytics

mgr/dashboard: Prevent angular of getting stuck during installation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29557 from Devp00l/wip-41166

mgr/dashboard: Fix CephFS chart

Reviewed-by: Ni-Feng Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

bluestore/allocator: Aging test for bluestore allocators.

Tests performance and fragmentation for data accumulation
and storage capacity extension scenario.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #29996 from iliul/export-standby-metadata

mgr/prometheus: export standby mds metadata

osd: add objects_readv_sync

However, this is only meaningful for ReplicatedBackend
since ECBackend is already reading everything in asynchronous way.
Also the osd_verify_sparse_read_holes option could be reliably
dropped as it seems never (ever?) actually get a chance to be
exercised ?

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/ReplicatedBackend: use readv to build push_op

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os: implement readv

Currently only works for bluestore.
The default version still reads each extent separately synchronously,
but I think that should not be a concern?

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30122 from tchaikov/wip-40646

install-deps: revert 47d4351d

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30119 from neha-ojha/wip-41385

osd/MissingLoc, PeeringState: remove osd from missing loc in purge_strays()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30072 from wjwithagen/wjw-wip_split_run-rbd-unit-tests

test/librbd: allow parallel runs of run-rbd-unit-tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps: revert 47d4351d

this change reverts 47d4351d3ad2787ae7781d868608eb0e3c7797c7

see also 33dfffc8

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30018 from xiexingguo/wip-38483

osd: prime splits/merges for any potential fabricated split/merge participant

Reviewed-by: Sage Weil <sage@redhat.com>

test/librbd: allow parallel runs of run-rbd-unit-tests

Running all tests sequential makes it the longest test of
`make check`, with each partial test taking around 500 sec.
Running 6 tests thus takes almost an hour.

Cut this down if ctest runs tests in parallel

Default behaviour of src/test/run-rbd-unit-tests.sh is kept:
  Without parameters the tests are run in sequence

To run unitttest_librbd with RBD_FEATURES, use `N` as parameter

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29990 from aclamk/fix-doc-ceph_kvstore_tool-stats

doc/ceph-kvstore-tool: add description for 'stats' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/man/ceph-kvstore-tool: fix typo

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd/MissingLoc, PeeringState: remove osd from missing loc in purge_strays()

We should always try to keep osds in missing_loc consistent with peer_missing
and peer_info. When we remove an osd from peer_missing and peer_info, we
should also remove it from missing_loc during purging strays.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29639 from baixueyu/wip-crypto-bug

rgw: get barbican secret key request maybe return error code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29776 from cbodley/wip-rgw-beast-static-flat-buffer

rgw: increase beast parse buffer size to 64k

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29852 from cbodley/wip-41395

rgw: move bucket reshard checks out of write path

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #29856 from cbodley/wip-41412

rgw: RGWCoroutine::call(nullptr) sets retcode=0

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #29894 from markhpc/wip-rgw-reshard-json

rgw/rgw_reshard: Don't dump RGWBucketReshard JSON in process_single_logshard

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30088 from tchaikov/wip-40646

ceph.spec.in: use g++ >= 8.3.1-3.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29742 from dang/push

rgw: make sure modelines are correct for all files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29199 from tianshan/fix_gc_tag_remove

rgw: gc remove tag after all sub io finish

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29628 from NancySu05/lihuanhuan

RGW: add bucket permission verify when copy obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29943 from markhpc/wip-rgw-optimize-get_val2

rgw/rgw_op: Remove get_val from hotpath via legacy options

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29654 from Songweibin/wip-rbd-list-descendants

 librbd: skip stale child with non-existent pool for list descendants

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30113 from tchaikov/wip-dmclock

dmclock:  pick up change to use specified C++ settings if any

Reviewed-by: Casey Bodley <cbodley@redhat.com>

dmclock:  pick up change to use specified C++ settings if any

see https://github.com/ceph/dmclock/pull/73

so dmclock won't add `-std=c++11` to CXX_FLAGS even if the parent project already
have `-std=c++17` in CXX_FLAGS.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: remove type conversions for ceph native options

We now return the correct Py types, just like we do for module options.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: return get_ceph_option result as typed Py object (not string)

We already return module options as the proper Py type; do the same for
native ceph options.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Prevent angular of getting stuck during installation

During npm packages installation Angular was getting stuck waiting for input.
Using "NG_CLI_ANALYTICS=false" will prevent that,

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29613 from trociny/wip-41206

librbd: behave more gracefully when data pool removed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #29835 into master

* refs/pull/29835/head:
osd/PeeringState: fix wrong history of merge target

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Empty datatable rendered before data has been fetched

Fixes: https://tracker.ceph.com/issues/41615
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #30097 from jan--f/ceph-backport-cp-indiviudal-commits

ceph-backport.sh: cherry-pick individual commits

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30063 from Rjerk/wip-silence-unused-warning

rgw/services: silence -Wunused-variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30065 from tchaikov/wip-rgw-test-wo-global

test: do not include unnecessary includes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-backport.sh: cherry-pick individual commits

In git 2.23.0 (possibly earlier too) cherry-pick seems to interact with
rev-list and submodules in unexpected ways, where the cherry-pick of a
commit range (<sha1>..<sha1>) fails with "changes would be overwritten"
in a submodule. CP'ing individual commits avoids this.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30060 from c744402859/master

rgw: fixed "unrecognized arg" error when using "radosgw-admin zone rm".

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30091 from joke-lee/doc-rgw-object-version

rgw: add --object-version in radosgw-admin help info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29957 from smithfarm/wip-ceph-backport-sh-improvements

script/ceph-backport.sh: wholesale refactor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/ceph-kvsore-tool: add description for 'stats' command

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #29665 from rishabh-d-dave/cv-remove-redundant-calls

ceph-volume: minor optimizations related to class Volumes's use

mgr/dashboard: fix cdEncode decorator is not working on class

Object.keys does not return all methods of a class.

Fixes: https://tracker.ceph.com/issues/41368
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

ceph.spec.in: use g++ >= 8.3.1-3.1

since https://bugzilla.redhat.com/show_bug.cgi?id=1726630 has been
fixed. and we have devtoolset-8-gcc-c++-8.3.1-3.1.el7.x86_64.rpm.

no reason to stick with devtoolset-8-gcc-c++-8.2.1-3 anymore.

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add --object-version in radosgw-admin help info

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #30086 from uniqueyehu/fix-complie-errors-dpdk

msg/async/dpdk: fix complie errors from fix FTBFS

Reviewed-by: Kefu Chai <kchai@redhat.com>