mgr/dashboard: Add info to Pools table

- Column 'Placement Groups' renamed to 'Pg Status':
It shows PG states.
- Created 'CephSharedModule' for shared services within ceph module.
- Created PgCategoryService & PgCategory model (logic encapsulation).
- Color consistency: PG chart (landing page) and cell text color are similar.

Fixes: https://tracker.ceph.com/issues/36740
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge PR #25573 into master

* refs/pull/25573/head:
osd: kill obsolete comments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23362 from luhp/master

common: fix a racing in PerfCounters::perf_counter_data_any_d::read_avg

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: kill obsolete comments

I added these comments a few years ago.
Since bluestore can read things that aren't committed and
https://github.com/ceph/ceph/pull/20177 should have made it work for
filestore too, this shouldn't matter any more.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25488 from trociny/wip-osd-perf-query-rbd-regex

mgr: improvements for dynamic osd perf counters

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25509 from trociny/wip-37596

librbd: disable image mirroring when moving to trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25462 from Songweibin/wip-rbd-invalid-pointer

 librbd: misc fix potential invalid pointer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25568 from cbodley/wip-rgw-pubsub-const-realm

rgw: fix compilation after pubsub conflict

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: pubsub uses const RGWRealm

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused get_tier_config()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23298 from yehudasa/wip-rgw-ps-3

rgw: pub-sub

qa/workunits/rbd: add trash move/restore mirror test

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: disable image mirroring when moving to trash

And enable when restoring if mirror pool mode set.

Fixes: https://tracker.ceph.com/issues/37596
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25357 from cbodley/wip-rgw-reshard-yield

rgw: add optional_yield to block_while_resharding()

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/osd_perf_query: generalize rbd regex for journal and data pool objects

And process collected stats accordingly.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: include only capturing groups into osd perf query subkeys

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/osd_perf_query: fix regression

(introduced when adding ALL_SUBKEYS_QUERY)

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/prometheus: fix get option methods names

(which has been renamed recently)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25412 from cbodley/wip-rgw-const-zone

rgw: zone service only provides const access to its data

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25549 from batrick/obsolete-comment

mds: remove obsolete comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #25550 from badone/wip-doc-remove-capability-reset-command

doc/user-management: Remove obsolete reset caps command

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/user-management: Remove obsolete reset caps command

This command has not worked since hammer and makes no sense in an
environment where we do not allow empty caps

Fixes: http://tracker.ceph.com/issues/37663
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #25454 from rzarzynski/wip-bl-drop-alloc-tracking

common: drop allocation tracking from bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: remove obsolete comments

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25333 from ivancich/wip-versioned-reshard-stats

rgw: fix stats for versioned buckets after reshard.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25356 from ivancich/wip-fix-rgw-options

rgw: move all reshard config options out of legacy_config_options

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25477 from alfredodeza/wip-rm37486

ceph-volume: set permissions right before prime-osd-dir

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional.lvm stop all osds before activating again to test permissions

Otherwise, ``ceph-volume lvm activate --all`` will skip OSDs that are
already running. So the test that verifies that permissions are
corrected will fail because ``activate`` will simply skip them

Signed-off-by: Alfredo Deza <adeza@redhat.com>

json formatter: reset pending string on dump_stream()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25504 from Devp00l/wip-summary-service-test-fix

mgr/dashboard: Cleanup of summary refresh test

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-volume lvm.activate ensure filestore osd path is chowned always

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #21305 into master

* refs/pull/21305/head:
ceph-create-keys: depreceate, print warning

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-create-keys: depreceate, print warning

We'll remove this post-nautilus or post-octopus, I guess?

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25411 from javacruft/py3-ceph-create-keys

ceph-create-keys: Misc Python 3 fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #25478 into master

* refs/pull/25478/head:
README.md: Add note about default do_cmake.sh debug builds.
doc/install: Add note about default do_cmake.sh debug builds.
do_cmake: Warn user about slow debug performance by default.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25334 from rhcs-dashboard/rgw-email-validation

mgr/dashboard: Validation for duplicate RGW user email

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #25502 from rhcs-dashboard/code-cleanup

mgr/dashboard: code cleanup

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25463 from tchaikov/wip-qa-thrash-old-clients

qa: downgrade librados2,librbd1 for thrash-old-clients tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: downgrade librados2,librbd1 for thrash-old-clients tests

librados2 and librbd1 are installed as a dependency of qemu-kvm.
qemu-kvm is installed by ceph-cm-ansible, see [1].

in thrash-old-clients, jewel packages are installed, but yum does
not allow downgrade unless it's required explicitly. in this change,
we downgrade librbd1 and librados2 to address this issue.

currently, the ceph packages shipped by CentOS/RHEL 7 are still an old
version of jewel. so this issue only kicks in when we try to install
hammer.

this change should address failures like

Command failed on smithi136 with status 1: '\n sudo yum -y install
rbd-fuse\n '

found in rados/thrash-old-clients tests.

---
[1]
https://github.com/ceph/ceph-cm-ansible/commit/3db1cbdc226ecdea532bcc7cd6e710b5ad2bb0d6#diff-f2b05d775fedff6c5c6689f564b32f1c

Fixes: http://tracker.ceph.com/issues/37618
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25433 from hsiang41/fix_37533

mgr/diskprediction_cloud: Remove needless library in the requirements file

Merge pull request #25510 from ivancich/wip-log-successful-resharding

rgw: feature -- log successful bucket resharding events

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #25222 into master

* refs/pull/25222/head:
mds: do not call Journaler::_trim twice

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25229 into master

* refs/pull/25229/head:
mds: fix incorrect l_pq_executing_ops statistics when meet an invalid item in purge queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25109 into master

* refs/pull/25109/head:
client: do not move f->pos untill success write

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25220 into master

* refs/pull/25220/head:
mds: queue inodes into MDCache::export_pin_queue after setattr is journaled

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25316 into master

* refs/pull/25316/head:
client: don't update importing caps' mseq when handing cap export
mds: don't exclude disallowed caps when recording wanted caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25389 into master

* refs/pull/25389/head:
mds: remove duplicated l_mdc_num_strays perfcounter set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25442 into master

* refs/pull/25442/head:
pybind/mgr/status: fix ceph fs status in py3 environments.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: include the ceph-facts role when needed during testing

The ceph-ansible stable-3.2 branch does not have the ceph-facts role
but master does. We need to make sure to import this role only when
it's supported.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #25402 into master

* refs/pull/25402/head:
pybind/mgr: rename get_option -> get_ceph_option
pybind/mgr: add get_ceph_option to standby module interface
ceph.spec.in: add xmlsec1 dependency for dashboard make check
mgr: rename internal ceph_set_config -> ceph_set_module_option
pybind/mgr: set_config -> set_module_option
pybind/mgr: get_config -> get_module_option
mgr: rename internal get_config -> get_module_option
pybind/mgr: OPTIONS -> MODULE_OPTIONS

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

rgw: feature -- log successful bucket resharding events

This change adds a single log entry at level 1 to indicate when a
bucket resharding successfully completes.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

README.md: Add note about default do_cmake.sh debug builds.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

doc/install: Add note about default do_cmake.sh debug builds.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

do_cmake: Warn user about slow debug performance by default.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #25325 from Devp00l/wip-issue-37469

 mgr/dashboard: Notification queue

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25494 from callithea/wip-e2e-script-rgw-conf

mgr/dashboard: Object Gateway user configuration

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25238 from jan--f/c-v-refactor-strategies-into-hierachy

ceph-volume: introduce class hierachy for strategies

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #25452 from jtlayton/wip-orchestrator

mgr/rook: fix API version and object types for recent rook changes

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: filestore strategy use strategy class hierarchy

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: bluestore strategy use strategy class hierarchy

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: add strategies.py to for shared code

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: Notification queue

It's now possible to send an array of notification configs to the notification
service. These array is cached for, by default 500ms, before this array
is unified and than converted to full notifications that are seen by the
user.

Useful if different service can come to the same conclusions regarding
the notification. This is the case for the prometheus notification service
and the prometheus alert service.

Fixes: https://tracker.ceph.com/issues/37469
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: CdNotificationConfig

This model will now be taken as single argument in the 'show' method
of the notification service. Through using this model it's easy to
define notifications before sending them out. Which can be helpful
if you have to process an array of possible notification and the
possibility that those notifications would look a like.

This scenario doesn't exist in the current code base, but it's the
case for new feature to show alerts from Prometheus.

Fixes: https://tracker.ceph.com/issues/37469
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #25390 from jan--f/c-v-revert-partition-as-disk

ceph-volume: revert partition as disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: code cleanup

Deleted duplicated ng-template.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #25276 from rhcs-dashboard/37401-chart-slice-hiding

mgr/dashboard: permanent pie chart slice hiding

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Cleanup of summary refresh test

Now the test is testing what it says.

Additionally resolved typescript compiler issues in the file and
removed not needed doc types in the corresponding service.

Fixes: https://tracker.ceph.com/issues/37619
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #25143 from nathan-weinberg/breadcrumb-block

mgr/dashboard: Added breadcrumb tests to Block menu items

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

librbd: misc fix potential invalid pointer

As described in commit: 5431fba5c5c9122418b6a03df4ed89f86fbb35e3, so do they.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

pybind/rbd: use the correct structure to calculate options size

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #25358 from trociny/wip-prometheus-rbd-stats

mgr/prometheus: provide RBD stats via osd dynamic perf counters

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume tests.functional set root:root in OSDs in filestore

To make sure that ceph-volume will be able to set the right permissions
when trying to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional set root:root in OSDs in bluestore

To make sure that ceph-volume will be able to set the right permissions
when trying to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25495 from ivancich/wip-rgw-add-missing-editor-directives

rgw: clean-up -- insure C++ source code files contain editor directives

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: clean-up -- insure C++ source code files contain editor directives

Most, although not all, source code files have a comment on the first
two lines describing the C++ formatting conventions used. This commit
adds that directive to the files lacking it. Also, vertical whitespace
was made more consistent. Note: this only touches files specifically
for RGW.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #25491 from smithfarm/wip-libradospp-devel

rpm: fix libradospp-devel runtime dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25436 from Songweibin/wip-rbd-silence-log

cls/rbd: silence the log of get metadata error

Merge pull request #25372 from iridescent-rsy/add-throttle-name

common: fix access and add name for the token bucket throttle

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #25445 from trociny/wip-python-rbd_list2

pybind/rbd: optimize rbd_list2

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Object Gateway user configuration

In order to be able to access and test the Object Gateway
within our end-to-end tests it needs to be configured properly.
This script will now take care of creating and configuring
a user and will set the corresponding credentials for the user.

Signed-off-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #25327 from Devp00l/wip-issue-37471

mgr/dashboard: Settings service

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25467 from rrmichel/patch-3

doc: Fixed syntax in iscsi initiator windows doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25487 from theanalyst/vstart-lc-level

vstart: rgw: disable the lc debug interval option

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: add inventory call to functional tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: add partition support to Device

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Revert "ceph-volume tests.util ensure partitions are listed as top-level keys"

This reverts commit 6dc01772245e557b73551111f7c48cc60aec0893.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Revert "ceph-volume util.disk include partitions as top level devices in get_devices"

This reverts commit 7f3c3598491e556ec1b93ea0a54f52144202fab6.
Fixes: http://tracker.ceph.com/issues/37506
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #25394 from LenzGr/master-documentation

doc: Added some hints on how to further accelerate builds with ccache

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Erwan Velu <erwan@redhat.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>

build/ops: rpm: fix libradospp-devel runtime dependency

The devel packages don't get the major version in their name.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-volume lvm.activate always chown osd path before osd-prime-dir in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25469 from sebastian-philipp/ceph-volume-fix-join

ceph-volume: Fix TypeError: join() takes exactly one argument (2 given)

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25401 from jmolmo/improve_wait_code

mgr/orchestrator: Simplify Orchestrator wait implementation

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

doc: Added some hints on how to further accelerate builds with ccache

Fixed incorrect note directives and added a section about how to speed
up development/scratch builds with ccache.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #25432 from rzarzynski/wip-bl-drop-pipesize

common: drop unused get_max_pipe_size() in buffer.cc.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25457 from trociny/wip-37581

pybind/mgr: drop unnecessary iterkeys usage to make py-3 compatible

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25471 from majianpeng/bluestore-open-blufs

os/bluestore: check return value of _open_bluefs.

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

ceph-volume: Fix TypeError: join() takes exactly one argument (2 given)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: rename get_option -> get_ceph_option

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr: add get_ceph_option to standby module interface

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add xmlsec1 dependency for dashboard make check

Signed-off-by: Sage Weil <sage@redhat.com>