common: add ref header

This enables the use of these handy ref templates outside of Message.h.
This is necessary for a future rework of #26348.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29048 from rjfd/wip-dashboard-progress-tasks

mgr/dashboard: integrate progress mgr module events into dashboard tasks list

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #28006 from Devp00l/wip-39482

mgr/dashboard: Watch for pool pg's increase and decrease

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #29092 into master

* refs/pull/29092/head:
osd/ECTransaction,ReplicatedBackend: create op is new in octopus

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28935 from votdev/issue_40248

mgr/dashboard: Allow users to change their password on the UI

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #28928 from p-na/wip-pna-e2e-pools

mgr/dashboard: Write E2E tests for pool creation, deletion and verification

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Rafael Quintero <rquinter@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28882 from rjfd/wip-mgr-perf-counter-timestamp

mgr: return perf_counters data timestamps in nanosecs

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #29069 into master

* refs/pull/29069/head:
qa/tasks/ceph_manager: make upmap thrasher behave when no pools/pgs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29095 into master

* refs/pull/29095/head:
script/ptl-tool: update for python3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28900 into master

* refs/pull/28900/head:
doc: Update troubleshooting.rst
doc: drop kernel-client-troubleshooting.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

script/ptl-tool: update for python3

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29071 into master

* refs/pull/29071/head:
selinux: Allow ceph to read udev db

Reviewed-by: Sage Weil <sage@redhat.com>

osd/ECTransaction,ReplicatedBackend: create op is new in octopus

Not nautilus!

Fixes: https://tracker.ceph.com/issues/40755
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28987 into master

* refs/pull/28987/head:
Revert "mgr/dashboard: Add nolockdep option to e2e-script"
common: refactor handling of lockdep's group name in debug locks.
common: shared_mutex_debug doesn't use empty group name for lockdep.
common, crypto: give names to OpenSSL's mutexes.
common: tiny_vector – CPU-friendly container for mutexes & co.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29051 into master

* refs/pull/29051/head:
mon/OSDMonitor: tolerate duplicate MRemoveSnaps messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28468 from jmolmo/ansible_orchestrator

mgr/ansible: RGW service

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr: return perf_counters data timestamps in nanosecs

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend: move ellipsis to before progress in task execution description

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: test_pool: fix pool unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add rudimentary pool create/exist/delete test

Fixes: http://tracker.ceph.com/issues/38093
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge branch 'master' into ansible_orchestrator

mgr/dashboard: tests: fix tests by mocking progress service

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend/task.service: format task title for progress tasks

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: add progress module tasks to list of dashboard tasks

Fixes: https://tracker.ceph.com/issues/38202
Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: progress mgr module service

This service module implements helper methods to retrieve
tasks information from the progress mgr module.

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/progress: persist started_at and finished_at timestamps

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/progress: export more event properties to json

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add SSO guard service.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Allow users to change their password on the UI

Fixes: https://tracker.ceph.com/issues/40248
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28959 from tchaikov/wip-crimson-op-write

crimson/osd: differentiate write from writefull

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28861 from smanjara/wip-multi-meta

rgw/multisite: Don't allow certain radosgw-admin commands to run on non-master zone

Merge pull request #28809 from joscollin/wip-fix-rgw-warning

rgw: Silence warning: control reaches end of non-void function

Merge PR #28003 into master

* refs/pull/28003/head:
client: do not return EEXIST for mkdirs
pybind/mgr/volumes: print exceptions in purge thread
pybind/mgr/volumes: refactor trash readdir
pybind/mgr/volumes: use existing client provided recursive mkdir
pybind/mgr/volumes: cleanup fs removal
pybind/mgr/subvolumes: use bytes for paths
pybind/mgr/volumes: remove unused property
test: add basic purge queue validation test
mgr / volumes: schedule purge job for volumes on init
mgr / volumes: purge queue for async subvolume delete
mgr / volumes: maintain connection pool for fs volumes
mgr / volumes: wrap rmtree() call within try..except block
mgr / volumes: use negative error codes everywhere
test: cleanup removing all subvolumes before removing subvolume group

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

selinux: Allow ceph to read udev db

We are using libudev and reading the udev db files because of that. We
need to allow ceph to access these files in the SELinux policy.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge PR #29030 into master

* refs/pull/29030/head:
qa/tasks/ceph_deploy: assume systemd and simplify shutdown wonkiness

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #28994 from smithfarm/wip-40645

build/ops: run "npm ci" with a one-hour timeout

Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tasks/ceph_deploy: assume systemd and simplify shutdown wonkiness

The 'service' command was succeeding but not actually stopping ceph.  But
we can safely assume systemd now so get rid of the weirdness here.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28839 into master

* refs/pull/28839/head:
osd: support osd_repair_during_recovery

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #28907 into master

* refs/pull/28907/head:
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29027 into master

* refs/pull/29027/head:
common/options: Set concurrent bluestore rocksdb compactions to 2

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph_manager: make upmap thrasher behave when no pools/pgs

Fixes: https://tracker.ceph.com/issues/40635
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: pool: make _get and _wait_for_pgs class methods

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #28820 from trociny/wip-journal-fetch-corrupt-end

journal: return error after first corruption detected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

client: do not return EEXIST for mkdirs

Behavior should be similar to `mkdir -p`.

Introduced-by: 26905ca82776e8119a347a9a2e8d65c272c7ed62
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28633 from p-na/wip-pna-refactor-unit-tests

mgr/dashboard: Reduce code duplication through TableActionComponent UnitTests

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28999 from rafaelquint/bucketstest3

mgr/dashboard: RGW bucket E2E Tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Reduce code duplication through TableActionComponent Unit Tests

Fixes: http://tracker.ceph.com/issues/40399
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: E2E Tests for Create, Edit, Delete, and Invalid Input on RGW Buckets

Fixes: https://tracker.ceph.com/issues/40372
Fixes: https://tracker.ceph.com/issues/40484
Fixes: https://tracker.ceph.com/issues/40259
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #28879 from Devp00l/wip-40580

mgr/dashboard: Fix the table mouseenter event handling test

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28916 from p-na/wip-pna-fix-tag

mgr/dashboard: Add linter for unclosed HTML tags

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add missing closing tag for container

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #27463 from rjfd/wip-dashboard-ssl-cert

mgr/dashboard: commands to set SSL certificate and key

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28213 from sebastian-philipp/orchestrator_cache

mgr/orchestrator: Add cache for Inventory and Services

Reviewed-by: Tim Serong <tserong@suse.com>

pybind/mgr/volumes: print exceptions in purge thread

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: refactor trash readdir

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: use existing client provided recursive mkdir

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: cleanup fs removal

In Nautilus, the simplest sequence is:

    fs fail name
    fs rm name --yes-i-really-mean-it

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/subvolumes: use bytes for paths

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28700 from batrick/i39620

doc/cephfs: improve add/remove MDS section

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29033 from neha-ojha/wip-add-yt

doc/dev/developer_guide/index.rst: add youtube references

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #28649 from votdev/issue_40427

mgr/dashboard: Pool form uses different loading spinner

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mon/OSDMonitor: tolerate duplicate MRemoveSnaps messages

Fixes: http://tracker.ceph.com/issues/40774
Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/dashboard: update SSL configuration instructions

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: added CLI commands to set SSL certificate and key

Fixes: https://tracker.ceph.com/issues/39123
Signed-off-by: Ricardo Dias <rdias@suse.com>

Revert "mgr/dashboard: Add nolockdep option to e2e-script"

This reverts commit d92117d8ea7018c533cc422e31c09d7e45a0793a
at the rason for it has been fixed.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: refactor handling of lockdep's group name in debug locks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: shared_mutex_debug doesn't use empty group name for lockdep.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common, crypto: give names to OpenSSL's mutexes.

Fixes: http://tracker.ceph.com/issues/40698
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: tiny_vector – CPU-friendly container for mutexes & co.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Pool form uses different loading spinner

Use the 'cd-loading-panel' element to display the loading progress.

Fixes: https://tracker.ceph.com/issues/40427
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29043 from votdev/issue_40778

mgr/dashboard: Form fields do not show error messages/hints.

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #28951 into master

* refs/pull/28951/head:
mds: apply configuration changes through MDSRank
common/TrackedOp: make settings atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd/heartbeat: use v2 address also

as we support v2 protocol, so no need to pick v1 address.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDRepOpReply: include necessary header

for accessing the member variables of `MOSDRepOp`, we should include its
header file for its definition.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28964 from cbodley/wip-qa-workunit-s3tests

qa/rgw: update run-s3tests.sh

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Add linter for unclosed HTML tags

Fixes: http://tracker.ceph.com/issues/40686
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Form fields do not show error messages/hints.

Fixes: https://tracker.ceph.com/issues/40778
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28682 from cyx1231st/wip-seastar-msgr-socket2

crimson/net: implement preemptive shutdown/close

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: an MOSDOp should be waiting for min_epoch

for MOSDOp, get_min_epoch() is an alias of get_map_epoch(), but
get_min_epoch() is symantically more correct.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add a dedicate code path for WRITE

differentiate WRITE from WRITEALL.
and handles differnt cases of `os.oi.truncate_seq` caused by
`write_trunc` and/or `trunc` ops.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26104 into master

* refs/pull/26104/head:
client/Client: change dir's size delta to current write file size delta

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28683 into master

* refs/pull/28683/head:
mon/MDSMonitor: use stringstream instead of dout for mds repaired

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28710 into master

* refs/pull/28710/head:
cephfs-shell: Remove undefined variable files in do_rm()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28831 into master

* refs/pull/28831/head:
client: support the fallocate() when fuse version >= 2.9

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28906 into master

* refs/pull/28906/head:
cephfs-shell: Fix TypeError in poutput()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: apply configuration changes through MDSRank

This avoids the need to acquire the mds_lock prior to responding to config
changes. The MDSRank can create a finisher context that acquires the lock
later.

Fixes: https://tracker.ceph.com/issues/40694
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/TrackedOp: make settings atomic

To avoid locks on configuration changes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: improve add/remove MDS section

Include hardware details and update language for modern tools.

Fixes: http://tracker.ceph.com/issues/39620
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev/developer_guide/index.rst: add youtube references

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #28978 into master

* refs/pull/28978/head:
qa/tasks/mgr/dashboard/test_health: fix test_full_health test
mon: use per-pool stats only when all OSDs are reporting
osd: report whether we have per-pool stats
osd/osd_types: osd_stat_t: include num_per_pool_osds member

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29003 into master

* refs/pull/29003/head:
qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output
osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool
ceph-objectstore-tool: fix use of SnapSet::snaps
qa/standalone/scrub/osd-scrub-snaps: fix kv grep

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29013 into master

* refs/pull/29013/head:
qa/tasks/ceph_manager: 5s -> 15s for 'osd out' to be visible

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28998 from alfredodeza/wip-rm40664

ceph-volume  zap always skips block.db, leaves them around

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/options: Set concurrent bluestore rocksdb compactions to 2

Signed-off-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: controllers/grafana is not Python3 compatible (#28956)

mgr/dashboard: controllers/grafana is not Python3 compatible

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28979 from falcon78921/wip-doc-18197

doc: updated OpenStack rbd documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output

SnapSet now dumps just seq, not a (fake) SnapContext.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool

Some callers (ceph-objectstore-tool) instantiate SnapMapper with a bogus
pool id (0) and want to use it to adjust mappings.  We have the hobj
available, so use its pool member instead.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: fix use of SnapSet::snaps

Instead, use clone_snaps to identify clones.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Minor fixes (#29009)

mgr/dashboard: Minor fixes

- Fix radio input display
- Fix the close button on initiators card
- Fix missing 'fa-' prefix in spinner icon
- Fix label font-weight

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>