mgr/dashboard: Show iSCSI gateways status in the health page

Fixes: https://tracker.ceph.com/issues/39028
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #28882 from rjfd/wip-mgr-perf-counter-timestamp

mgr: return perf_counters data timestamps in nanosecs

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #29069 into master

* refs/pull/29069/head:
qa/tasks/ceph_manager: make upmap thrasher behave when no pools/pgs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29095 into master

* refs/pull/29095/head:
script/ptl-tool: update for python3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28900 into master

* refs/pull/28900/head:
doc: Update troubleshooting.rst
doc: drop kernel-client-troubleshooting.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

script/ptl-tool: update for python3

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29071 into master

* refs/pull/29071/head:
selinux: Allow ceph to read udev db

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28987 into master

* refs/pull/28987/head:
Revert "mgr/dashboard: Add nolockdep option to e2e-script"
common: refactor handling of lockdep's group name in debug locks.
common: shared_mutex_debug doesn't use empty group name for lockdep.
common, crypto: give names to OpenSSL's mutexes.
common: tiny_vector – CPU-friendly container for mutexes & co.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29051 into master

* refs/pull/29051/head:
mon/OSDMonitor: tolerate duplicate MRemoveSnaps messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28468 from jmolmo/ansible_orchestrator

mgr/ansible: RGW service

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr: return perf_counters data timestamps in nanosecs

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge branch 'master' into ansible_orchestrator

Merge pull request #28959 from tchaikov/wip-crimson-op-write

crimson/osd: differentiate write from writefull

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28861 from smanjara/wip-multi-meta

rgw/multisite: Don't allow certain radosgw-admin commands to run on non-master zone

Merge pull request #28809 from joscollin/wip-fix-rgw-warning

rgw: Silence warning: control reaches end of non-void function

Merge PR #28003 into master

* refs/pull/28003/head:
client: do not return EEXIST for mkdirs
pybind/mgr/volumes: print exceptions in purge thread
pybind/mgr/volumes: refactor trash readdir
pybind/mgr/volumes: use existing client provided recursive mkdir
pybind/mgr/volumes: cleanup fs removal
pybind/mgr/subvolumes: use bytes for paths
pybind/mgr/volumes: remove unused property
test: add basic purge queue validation test
mgr / volumes: schedule purge job for volumes on init
mgr / volumes: purge queue for async subvolume delete
mgr / volumes: maintain connection pool for fs volumes
mgr / volumes: wrap rmtree() call within try..except block
mgr / volumes: use negative error codes everywhere
test: cleanup removing all subvolumes before removing subvolume group

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

selinux: Allow ceph to read udev db

We are using libudev and reading the udev db files because of that. We
need to allow ceph to access these files in the SELinux policy.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge PR #29030 into master

* refs/pull/29030/head:
qa/tasks/ceph_deploy: assume systemd and simplify shutdown wonkiness

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #28994 from smithfarm/wip-40645

build/ops: run "npm ci" with a one-hour timeout

Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tasks/ceph_deploy: assume systemd and simplify shutdown wonkiness

The 'service' command was succeeding but not actually stopping ceph.  But
we can safely assume systemd now so get rid of the weirdness here.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28839 into master

* refs/pull/28839/head:
osd: support osd_repair_during_recovery

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #28907 into master

* refs/pull/28907/head:
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29027 into master

* refs/pull/29027/head:
common/options: Set concurrent bluestore rocksdb compactions to 2

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph_manager: make upmap thrasher behave when no pools/pgs

Fixes: https://tracker.ceph.com/issues/40635
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28820 from trociny/wip-journal-fetch-corrupt-end

journal: return error after first corruption detected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

client: do not return EEXIST for mkdirs

Behavior should be similar to `mkdir -p`.

Introduced-by: 26905ca82776e8119a347a9a2e8d65c272c7ed62
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28633 from p-na/wip-pna-refactor-unit-tests

mgr/dashboard: Reduce code duplication through TableActionComponent UnitTests

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28999 from rafaelquint/bucketstest3

mgr/dashboard: RGW bucket E2E Tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Reduce code duplication through TableActionComponent Unit Tests

Fixes: http://tracker.ceph.com/issues/40399
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: E2E Tests for Create, Edit, Delete, and Invalid Input on RGW Buckets

Fixes: https://tracker.ceph.com/issues/40372
Fixes: https://tracker.ceph.com/issues/40484
Fixes: https://tracker.ceph.com/issues/40259
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #28879 from Devp00l/wip-40580

mgr/dashboard: Fix the table mouseenter event handling test

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28916 from p-na/wip-pna-fix-tag

mgr/dashboard: Add linter for unclosed HTML tags

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add missing closing tag for container

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #27463 from rjfd/wip-dashboard-ssl-cert

mgr/dashboard: commands to set SSL certificate and key

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28213 from sebastian-philipp/orchestrator_cache

mgr/orchestrator: Add cache for Inventory and Services

Reviewed-by: Tim Serong <tserong@suse.com>

pybind/mgr/volumes: print exceptions in purge thread

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: refactor trash readdir

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: use existing client provided recursive mkdir

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: cleanup fs removal

In Nautilus, the simplest sequence is:

    fs fail name
    fs rm name --yes-i-really-mean-it

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/subvolumes: use bytes for paths

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28700 from batrick/i39620

doc/cephfs: improve add/remove MDS section

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29033 from neha-ojha/wip-add-yt

doc/dev/developer_guide/index.rst: add youtube references

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #28649 from votdev/issue_40427

mgr/dashboard: Pool form uses different loading spinner

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mon/OSDMonitor: tolerate duplicate MRemoveSnaps messages

Fixes: http://tracker.ceph.com/issues/40774
Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/dashboard: update SSL configuration instructions

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: added CLI commands to set SSL certificate and key

Fixes: https://tracker.ceph.com/issues/39123
Signed-off-by: Ricardo Dias <rdias@suse.com>

Revert "mgr/dashboard: Add nolockdep option to e2e-script"

This reverts commit d92117d8ea7018c533cc422e31c09d7e45a0793a
at the rason for it has been fixed.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: refactor handling of lockdep's group name in debug locks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: shared_mutex_debug doesn't use empty group name for lockdep.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common, crypto: give names to OpenSSL's mutexes.

Fixes: http://tracker.ceph.com/issues/40698
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: tiny_vector – CPU-friendly container for mutexes & co.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Pool form uses different loading spinner

Use the 'cd-loading-panel' element to display the loading progress.

Fixes: https://tracker.ceph.com/issues/40427
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29043 from votdev/issue_40778

mgr/dashboard: Form fields do not show error messages/hints.

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #28951 into master

* refs/pull/28951/head:
mds: apply configuration changes through MDSRank
common/TrackedOp: make settings atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd/heartbeat: use v2 address also

as we support v2 protocol, so no need to pick v1 address.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDRepOpReply: include necessary header

for accessing the member variables of `MOSDRepOp`, we should include its
header file for its definition.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28964 from cbodley/wip-qa-workunit-s3tests

qa/rgw: update run-s3tests.sh

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Add linter for unclosed HTML tags

Fixes: http://tracker.ceph.com/issues/40686
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Form fields do not show error messages/hints.

Fixes: https://tracker.ceph.com/issues/40778
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28682 from cyx1231st/wip-seastar-msgr-socket2

crimson/net: implement preemptive shutdown/close

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: an MOSDOp should be waiting for min_epoch

for MOSDOp, get_min_epoch() is an alias of get_map_epoch(), but
get_min_epoch() is symantically more correct.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add a dedicate code path for WRITE

differentiate WRITE from WRITEALL.
and handles differnt cases of `os.oi.truncate_seq` caused by
`write_trunc` and/or `trunc` ops.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #26104 into master

* refs/pull/26104/head:
client/Client: change dir's size delta to current write file size delta

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28683 into master

* refs/pull/28683/head:
mon/MDSMonitor: use stringstream instead of dout for mds repaired

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28710 into master

* refs/pull/28710/head:
cephfs-shell: Remove undefined variable files in do_rm()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28831 into master

* refs/pull/28831/head:
client: support the fallocate() when fuse version >= 2.9

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28906 into master

* refs/pull/28906/head:
cephfs-shell: Fix TypeError in poutput()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: apply configuration changes through MDSRank

This avoids the need to acquire the mds_lock prior to responding to config
changes. The MDSRank can create a finisher context that acquires the lock
later.

Fixes: https://tracker.ceph.com/issues/40694
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/TrackedOp: make settings atomic

To avoid locks on configuration changes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: improve add/remove MDS section

Include hardware details and update language for modern tools.

Fixes: http://tracker.ceph.com/issues/39620
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev/developer_guide/index.rst: add youtube references

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #28978 into master

* refs/pull/28978/head:
qa/tasks/mgr/dashboard/test_health: fix test_full_health test
mon: use per-pool stats only when all OSDs are reporting
osd: report whether we have per-pool stats
osd/osd_types: osd_stat_t: include num_per_pool_osds member

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29003 into master

* refs/pull/29003/head:
qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output
osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool
ceph-objectstore-tool: fix use of SnapSet::snaps
qa/standalone/scrub/osd-scrub-snaps: fix kv grep

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29013 into master

* refs/pull/29013/head:
qa/tasks/ceph_manager: 5s -> 15s for 'osd out' to be visible

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28998 from alfredodeza/wip-rm40664

ceph-volume  zap always skips block.db, leaves them around

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

common/options: Set concurrent bluestore rocksdb compactions to 2

Signed-off-by: Mark Nelson <mnelson@redhat.com>

mgr/dashboard: controllers/grafana is not Python3 compatible (#28956)

mgr/dashboard: controllers/grafana is not Python3 compatible

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28979 from falcon78921/wip-doc-18197

doc: updated OpenStack rbd documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/standalone/scrub/osd-scrub-snaps.sh: adjust expected output

SnapSet now dumps just seq, not a (fake) SnapContext.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/SnapMapper: to_raw_key: use hobject pool member, not SnapMapper:pool

Some callers (ceph-objectstore-tool) instantiate SnapMapper with a bogus
pool id (0) and want to use it to adjust mappings.  We have the hobj
available, so use its pool member instead.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-objectstore-tool: fix use of SnapSet::snaps

Instead, use clone_snaps to identify clones.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Minor fixes (#29009)

mgr/dashboard: Minor fixes

- Fix radio input display
- Fix the close button on initiators card
- Fix missing 'fa-' prefix in spinner icon
- Fix label font-weight

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #28945 into master

* refs/pull/28945/head:
os/bluestore: StupidAllocator stucks on 4GB allocations
os/bluestore: avoid length overflow in extents returned by Stupid

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28990 into master

* refs/pull/28990/head:
qa: vstart_runner fails because of string index out of range

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

qa/tasks/ceph_manager: 5s -> 15s for 'osd out' to be visible

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume tests: verify that wipefs tries several times

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap: retry wipefs several times to prevent race condition failures

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/tasks/mgr/dashboard/test_health: fix test_full_health test

New fields!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/scrub/osd-scrub-snaps: fix kv grep

SnapMapper keys are now SNA_, not MAP_.

Fixes: http://tracker.ceph.com/issues/40725
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Remove button and badge radius (#29001)

mgr/dashboard: Remove button and badge radius

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Fix form validation problems in RGW user form (#29000)

mgr/dashboard: Fix form validation problems in RGW user form

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/ansible: RGW service

RGW basic implementation

- Add one or more hosts running RGW daemon
- Allow to specify a multisite/multizone settings using a config file.
- Remove RGW daemon used to provide service for one zone.

Comments:

I see two use cases:

a. Install/Remove in new hosts the RGW daemons and configure them.
b. RGW management (CRUD zone groups/zones ... etc)

But the current orchestrators API seems to be oriented only to cover use case b.

What i have implemented in this moment covers use case a, despite the API is not friendly with that. I have added the possibility of getting a config file to specify hosts and settings.

rgw-admin commands covers all the functionality for managing RGW... So probably it does not have too much sense to implement anything around RGW management in orchestrators (use case b).

My doubts are:

1. The current API covers RGW management using a default configuration (user cannot configure multisite/multizone). Is this ok?

2. Shoud we add more API endpoints to allow add RGW nodes/hosts and manage the RGW config?

---------------------------------

Modifications suggested by @epuerta:
 Mainly:
 - New Inventory group class for manage hosts in Ansible Roles inside Ansible Inventory.
 - A redundant plus sign was found by @epuerta

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #28267 from flumm/ceph-volume-zap-fix

ceph-volume lvm.zap fix cleanup for db partitions

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Fix label font-weight

Some labels were not bold, because the css rule didn't apply to them.
I have improved the css rule and also add missing classes in some labels.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix missing 'fa-' prefix in spinner icon

This was causing the submit button to not show the spinner.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add nolockdep option to e2e-script (#28981)

mgr/dashboard: Add nolockdep option to e2e-script

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Fix the close button on initiators card

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix radio input display

We were using the wrong class for the labels.

Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: drop getting list-type when get_data is false

Signed-off-by: Jos Collin <jcollin@redhat.com>

rgw: Silence warning: control reaches end of non-void function

Build shows:
[ 53%] Building CXX object src/mds/CMakeFiles/mds.dir/JournalPointer.cc.o
ceph/src/rgw/rgw_rest_s3.cc: In member function ‘RGWOp* RGWHandler_REST_Bucket_S3::get_obj_op(bool)’:
ceph/src/rgw/rgw_rest_s3.cc:3588:5: warning: control reaches end of non-void function [-Wreturn-type]
   }   }
     ^
Fixes: 4ffc765c4c5debc665ade7769c4647c3a7278fd2
Fixes: http://tracker.ceph.com/issues/40747
Signed-off-by: Jos Collin <jcollin@redhat.com>