mon/MgrMap: define ModuleOptions and include in map

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: expand type helpers and make them static

- make str_to_type and type_to_str symmetric
- keep the variant that provides C++ types, even though there are no users

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: pin enums to values

In case we encode these

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25190 into master

* refs/pull/25190/head:
mgr/prometheus: adjust to new 'df' fields
mon/Monitor: fix newline between df section
doc: update docs for new ceph df output
mon/PGMap: break down RAW usage by device class
mon/PGMap: tweak df headers
mon/PGMap: GLOBAL -> RAW STORAGE in 'df' output
mon/PGMap: dump_fs_stats -> dump_cluster_stats

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25615 into master

* refs/pull/25615/head:
common/options: fix bad default

Reviewed-by: Neha Ojha <nojha@redhat.com>

common/options: fix bad default

Got the fields wrong.

Fixes ff75a0b4f21eb14b11702ab2df82ed594f650fcd

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #25596 into master

* refs/pull/25596/head:
qa/suites/upgrade: fix wrt librados3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25486 into master

* refs/pull/25486/head:
common: enhance debug macro

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #25456 into master

* refs/pull/25456/head:
common/options: set mon or mgr service on these options
common/options: more mon options
common/options: kill old mon_max_pgmap_epochs option
common/options: mon options
mon: remove dead option mon_pg_min_inactive
mon: mon_osd_pool_ec_fast_read -> osd_pool_default_ec_fast_read
common/options: document mon options
common/options: messenger options
common/options: document misc
common/options: compression options
common/options: document monitor log options

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #25563 from rhcs-dashboard/bugfix-read-write-chart

mgr/dashboard: avoid blank content in Read/Write Card

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #25110 into master

* refs/pull/25110/head:
client: fix failure in quota size limitation when using samba

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25174 into master

* refs/pull/25174/head:
mds: fix bug filelock stuck at LOCK_XSYN leading client can't read data

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25481 into master

* refs/pull/25481/head:
mon: mark REMOVE_SNAPS messages as no_reply

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25548 from dzafman/wip-37653

osd: fix shard_info_wrapper encode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/upgrade: fix wrt librados3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25594 from alfredodeza/wip-rm37675

ceph-volume tests.functional fix typo when stopping osd.0 in filestore

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.strategies remove unused import

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25483 from dzafman/wip-hashinfo

ceph-objectstore-tool: Dump hashinfo

Reviewed-by: Neha Ojha <nojha@redhat.com>

ceph-volume tests.functional fix typo when stopping osd.0 in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25377 from aclamk/wip-kv-remove-validate

common/KeyValueDB: Get rid of validate parameter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: avoid blank content in Read/Write Card

The card has to show either a chart or 'N/A', but no blank content.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #25489 from rhcs-dashboard/36740-add-info-to-pools-table

mgr/dashboard: Add info to Pools table

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #25405 into master

* refs/pull/25405/head:
msg/async: truly use first addr if multiple incoming addrs provided

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25421 from xiexingguo/wip-37507

os/bluestore: add boundary check for cache-autotune related settings

Merge pull request #25422 from rzarzynski/wip-bl-drop-at_buffer_headtail

common: drop at_buffer_{head,tail} from buffer::ptr.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25474 from tchaikov/wip-update-seastar

cmake,seastar: pick up latest seastar

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #25564 into master

* refs/pull/25564/head:
qa/rados/upgrade: align thrashing with upgrade suite, don't import/export pgs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25468 from rjfd/wip-orch-service-descriptor

mgr/orchestrator: added useful attributes to ServiceDescription

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Add info to Pools table

- Column 'Placement Groups' renamed to 'Pg Status':
It shows PG states.
- Created 'CephSharedModule' for shared services within ceph module.
- Created PgCategoryService & PgCategory model (logic encapsulation).
- Color consistency: PG chart (landing page) and cell text color are similar.

Fixes: https://tracker.ceph.com/issues/36740
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

fmt: remove fmt submodule

since seastar contains fmt 5.2.1 as a submodule. there is no need
to have our own copy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update to accommodate seastar changes

- seastar now works with fmt 5.2.1
- seastar now contains fmt as a submodule, so use it instead of
  using our own submodule
- seastar now create include directory under gen/include
- we use std::option<> when using seastar, so enable
  Seastar_STD_OPTIONAL_VARIANT_STRINGVIEW accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup latest seastar

we have quite a few hacks to get seastar's cmake to play with the distro
we support, and since seastar's cmake now is completely rewritten. we
can now use the upstream's cmake without having to maintain our patches.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #25573 into master

* refs/pull/25573/head:
osd: kill obsolete comments

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #23362 from luhp/master

common: fix a racing in PerfCounters::perf_counter_data_any_d::read_avg

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: added service status attributes to service descriptor

Signed-off-by: Ricardo Dias <rdias@suse.com>

osd: kill obsolete comments

I added these comments a few years ago.
Since bluestore can read things that aren't committed and
https://github.com/ceph/ceph/pull/20177 should have made it work for
filestore too, this shouldn't matter any more.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25488 from trociny/wip-osd-perf-query-rbd-regex

mgr: improvements for dynamic osd perf counters

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25509 from trociny/wip-37596

librbd: disable image mirroring when moving to trash

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

include/rados: s/requires/req/

requires is a keyword in C++20. which is close =)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25462 from Songweibin/wip-rbd-invalid-pointer

 librbd: misc fix potential invalid pointer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25568 from cbodley/wip-rgw-pubsub-const-realm

rgw: fix compilation after pubsub conflict

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: pubsub uses const RGWRealm

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused get_tier_config()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23298 from yehudasa/wip-rgw-ps-3

rgw: pub-sub

qa/workunits/rbd: add trash move/restore mirror test

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: disable image mirroring when moving to trash

And enable when restoring if mirror pool mode set.

Fixes: https://tracker.ceph.com/issues/37596
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25357 from cbodley/wip-rgw-reshard-yield

rgw: add optional_yield to block_while_resharding()

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/osd_perf_query: generalize rbd regex for journal and data pool objects

And process collected stats accordingly.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr: include only capturing groups into osd perf query subkeys

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/osd_perf_query: fix regression

(introduced when adding ALL_SUBKEYS_QUERY)

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/prometheus: fix get option methods names

(which has been renamed recently)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25412 from cbodley/wip-rgw-const-zone

rgw: zone service only provides const access to its data

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/rados/upgrade: align thrashing with upgrade suite, don't import/export pgs

Don't import/export between versions

Fixes: http://tracker.ceph.com/issues/37665
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25549 from batrick/obsolete-comment

mds: remove obsolete comments

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #25550 from badone/wip-doc-remove-capability-reset-command

doc/user-management: Remove obsolete reset caps command

Reviewed-by: Jos Collin <jcollin@redhat.com>

doc/user-management: Remove obsolete reset caps command

This command has not worked since hammer and makes no sense in an
environment where we do not allow empty caps

Fixes: http://tracker.ceph.com/issues/37663
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #25454 from rzarzynski/wip-bl-drop-alloc-tracking

common: drop allocation tracking from bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: remove obsolete comments

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rados, librados: Get errors back to caller

Remove redundant error message

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: shard_info_wrapper encode/decode need to fall through to *_FINISH()

Fixes: http://tracker.ceph.com/issues/37653
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #25333 from ivancich/wip-versioned-reshard-stats

rgw: fix stats for versioned buckets after reshard.

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25356 from ivancich/wip-fix-rgw-options

rgw: move all reshard config options out of legacy_config_options

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

mgr/orchestrator: added useful attributes to ServiceDescription

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25477 from alfredodeza/wip-rm37486

ceph-volume: set permissions right before prime-osd-dir

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.functional.lvm stop all osds before activating again to test permissions

Otherwise, ``ceph-volume lvm activate --all`` will skip OSDs that are
already running. So the test that verifies that permissions are
corrected will fail because ``activate`` will simply skip them

Signed-off-by: Alfredo Deza <adeza@redhat.com>

json formatter: reset pending string on dump_stream()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/options: set mon or mgr service on these options

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: more mon options

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: kill old mon_max_pgmap_epochs option

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: mon options

Signed-off-by: Sage Weil <sage@redhat.com>

mon: remove dead option mon_pg_min_inactive

Signed-off-by: Sage Weil <sage@redhat.com>

mon: mon_osd_pool_ec_fast_read -> osd_pool_default_ec_fast_read

More consistent name!

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: document mon options

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: messenger options

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25504 from Devp00l/wip-summary-service-test-fix

mgr/dashboard: Cleanup of summary refresh test

Reviewed-by: Kanika Murarka <kmurarka@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-volume lvm.activate ensure filestore osd path is chowned always

Signed-off-by: Alfredo Deza <adeza@redhat.com>

common/options: document misc

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: compression options

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: document monitor log options

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #21305 into master

* refs/pull/21305/head:
ceph-create-keys: depreceate, print warning

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-create-keys: depreceate, print warning

We'll remove this post-nautilus or post-octopus, I guess?

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25411 from javacruft/py3-ceph-create-keys

ceph-create-keys: Misc Python 3 fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #25478 into master

* refs/pull/25478/head:
README.md: Add note about default do_cmake.sh debug builds.
doc/install: Add note about default do_cmake.sh debug builds.
do_cmake: Warn user about slow debug performance by default.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25334 from rhcs-dashboard/rgw-email-validation

mgr/dashboard: Validation for duplicate RGW user email

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #25502 from rhcs-dashboard/code-cleanup

mgr/dashboard: code cleanup

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #25463 from tchaikov/wip-qa-thrash-old-clients

qa: downgrade librados2,librbd1 for thrash-old-clients tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: downgrade librados2,librbd1 for thrash-old-clients tests

librados2 and librbd1 are installed as a dependency of qemu-kvm.
qemu-kvm is installed by ceph-cm-ansible, see [1].

in thrash-old-clients, jewel packages are installed, but yum does
not allow downgrade unless it's required explicitly. in this change,
we downgrade librbd1 and librados2 to address this issue.

currently, the ceph packages shipped by CentOS/RHEL 7 are still an old
version of jewel. so this issue only kicks in when we try to install
hammer.

this change should address failures like

Command failed on smithi136 with status 1: '\n sudo yum -y install
rbd-fuse\n '

found in rados/thrash-old-clients tests.

---
[1]
https://github.com/ceph/ceph-cm-ansible/commit/3db1cbdc226ecdea532bcc7cd6e710b5ad2bb0d6#diff-f2b05d775fedff6c5c6689f564b32f1c

Fixes: http://tracker.ceph.com/issues/37618
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25433 from hsiang41/fix_37533

mgr/diskprediction_cloud: Remove needless library in the requirements file

Merge pull request #25510 from ivancich/wip-log-successful-resharding

rgw: feature -- log successful bucket resharding events

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #25222 into master

* refs/pull/25222/head:
mds: do not call Journaler::_trim twice

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25229 into master

* refs/pull/25229/head:
mds: fix incorrect l_pq_executing_ops statistics when meet an invalid item in purge queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25109 into master

* refs/pull/25109/head:
client: do not move f->pos untill success write

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #25220 into master

* refs/pull/25220/head:
mds: queue inodes into MDCache::export_pin_queue after setattr is journaled

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25316 into master

* refs/pull/25316/head:
client: don't update importing caps' mseq when handing cap export
mds: don't exclude disallowed caps when recording wanted caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25389 into master

* refs/pull/25389/head:
mds: remove duplicated l_mdc_num_strays perfcounter set

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #25442 into master

* refs/pull/25442/head:
pybind/mgr/status: fix ceph fs status in py3 environments.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: include the ceph-facts role when needed during testing

The ceph-ansible stable-3.2 branch does not have the ceph-facts role
but master does. We need to make sure to import this role only when
it's supported.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #25402 into master

* refs/pull/25402/head:
pybind/mgr: rename get_option -> get_ceph_option
pybind/mgr: add get_ceph_option to standby module interface
ceph.spec.in: add xmlsec1 dependency for dashboard make check
mgr: rename internal ceph_set_config -> ceph_set_module_option
pybind/mgr: set_config -> set_module_option
pybind/mgr: get_config -> get_module_option
mgr: rename internal get_config -> get_module_option
pybind/mgr: OPTIONS -> MODULE_OPTIONS

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

rgw: feature -- log successful bucket resharding events

This change adds a single log entry at level 1 to indicate when a
bucket resharding successfully completes.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

README.md: Add note about default do_cmake.sh debug builds.

Signed-off-by: Mark Nelson <mnelson@redhat.com>