mgr/dashboard,grafana: remove shortcut menu

Remove shortcut menu (links) and add check in grafana CI script.

Fixes: https://tracker.ceph.com/issues/43091
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31883 from jan--f/ceph-volume-code-owner

github/codeowners: add ceph-volume

Merge PR #31892 into master

* refs/pull/31892/head:
ceph-daemon: seek relative to the start of file

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31899 into master

* refs/pull/31899/head:
mgr: drop session with Ceph daemon when not ready
mgr: improve debug message information

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30262 from sebastian-philipp/orchestrator-pomise-future

mgr/orchestrator: Make Completions composable

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/ssh: Disable host labels.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31896 from rsevero/missing_virtualenv

doc: add missing virtualenv for build-doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: use finalize to set TrivialReadCompletion result

This is needed to correctly set the state of the completion, so that
Completion.has_result returns true.  Without this, _orchestrator_wait()
will spin forever if given a TrivialReadCompletion.

Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add pretty_print()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Fix Promises

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

import typing optionally

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Fix testing error

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Fixed DeepSeaReadCompletion arguments

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: fix tests

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

make check fixes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchstrator: Removed all type anotations incompatible to Python 2

* Reduce mypy's type checking capability
* Remove useful code documentation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: Adapted to new Completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add cancel

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/orchestrator: Adapt teuthology test to new completions

Also: Adapt mgr/test_orchestrator to new completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ansible: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Adapt ssh orch to new Completions interface

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Adapt to new completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Introduce composable completions

Also:

* unify Read and Write completions
* Distinguish should_wait and is_finished
* Removed `all_hosts` parameter to `osd_create`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: rename wait() -> process()

Because wait() should actually not wait!

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31904 from tchaikov/wip-crimson-stop-msgr-after-stopping-osd

crimson: stop osd before stopping messengers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: stop osd before stopping messengers

so we can drain all pending tasks referencing messenger before
destroying it, otherwise they will be using a already-stopped
messenger when they are still alive.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31756 into master

* refs/pull/31756/head:
pybind / cephfs: remove static typing in LibCephFS.chown

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31828 into master

* refs/pull/31828/head:
mgr/orchestrator: include container_image in 'service ls' output
ceph-daemon: include container_image_name and container_image_id in 'ls' output
common/options: default to ceph/ceph:latest-master-devel (for now)

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31854 into master

* refs/pull/31854/head:
mgr/orchestrator_cli: print table for 'host ls'
mgr/ssh: populate labels
mgr/orchestrator: add 'labels' property to InventoryNode
mgr/ssh: add/remove labels
mgr/orchestrator: orchestrator host label {add,rm}
mgr/ssh: add explicit host list

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31864 into master

* refs/pull/31864/head:
osd/PeeringState: clear LAGGY and WAIT states on exiting Started

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #31865 into master

* refs/pull/31865/head:
cls/rbd: improve efficiency of mirror image status queries

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #31866 into master

* refs/pull/31866/head:
osd/SnapMirror: no need to record purged_snaps every epoch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: drop session with Ceph daemon when not ready

If the mgr is waiting on daemon metadata from the mons, it has no
DaemonState associated with the daemon yet. If we try to process this
MgrOpen, the metadata sent by the daemon (like its config) will not be
recorded.

Fixes: https://tracker.ceph.com/issues/43037
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: improve debug message information

Connection pointer is not helpful.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: missing virtualenv for build-doc

The install deps command recommended by admin/build-doc doesn't install virtualenv which is required by admin/build-doc.

After manually installing virtualenv, admin/build-doc runned without issues.

Signed-off-by: Rodrigo Severo <rodrigo@fabricadeideias.com>

Merge PR #27467 into master

* refs/pull/27467/head:
cephfs-shell: add snapshot management

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31871 from smithfarm/wip-43021

doc: mention --namespace option in rados manpage

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

ceph-daemon: seek relative to the start of file

python3 only allows seeks relative to the start of a text file

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc: mention --namespace option in rados manpage

Fixes: https://tracker.ceph.com/issues/43021
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/orchestrator_cli: print table for 'host ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: populate labels

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add 'labels' property to InventoryNode

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add/remove labels

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: orchestrator host label {add,rm}

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add explicit host list

Keep the host list in an explicit 'inventory' key, and provide a dict for
each host so that we can keep other metadata (like labels).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31837 into master

* refs/pull/31837/head:
ceph-daemon: add additional debug logging

Reviewed-by: Sage Weil <sage@redhat.com>

github/codeowners: add ceph-volume

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31874 from tchaikov/wip-crimson-cyanstore-restructure

crimson: move os/cyan_* down to os/cyanstore/*

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: move os/cyan_* down to os/cyanstore/*

prepare for bluestore and seastore

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31859 from rzarzynski/wip-crimson-drop_dup_failed

crimson/osd/osd_operation.h: clean up duplicative check.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31252 from wjwithagen/wjw-doc-ceph_volume_zfs_inventory

doc/ceph-volume: initial docs for zfs/inventory and zfs/api

Merge pull request #31821 from mgfritch/c-v-scan-help

ceph-volume: minor clean-up of `simple scan` subcommand help

ceph-daemon: add additional debug logging

Adds additional detail before invoking subprocess for the
`run`, `shell, `enter`, and `logs` commands

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31567 into master

* refs/pull/31567/head:
common: move gen_rand_alphanumeric() helpers into common
test: add unit tests for gen_rand_alphanumeric helpers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #31781 into master

* refs/pull/31781/head:
cmake/modules: look for GNU make first instead of BSD make
cmake: support parallel build for rocksdb

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31579 into master

* refs/pull/31579/head:
cmake: add cppcheck and iwyu static analysis targets

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #31775 into master

* refs/pull/31775/head:
mds: remove the incorrect comments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #31762 into master

* refs/pull/31762/head:
mgr/ssh: improve logging for mds removal
mgr/volumes: move up 'confirm' validation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

osd/SnapMirror: no need to record purged_snaps every epoch

These keys were a mirror of the mon's purged_epoch keys.  We were writing
them, but never looking at them.  We only need the per-pool keys that
are indexed by snapid for lookups, so we should stop storing the per-epoch
records.

Note that the OSDSuperblock has the purged_snaps_last that serves as our
position/cursor for ingesting the mon keys.

Fixes: https://tracker.ceph.com/issues/42012
Signed-off-by: Sage Weil <sage@redhat.com>

cls/rbd: improve efficiency of mirror image status queries

The removal of remote mirror image status records no longer
(accidentally) searches through nearly the full omap database and
getting the status of a single image no longer involves duplicate
lookups (to pull the fsid keys and then to read the data).

Fixes: https://tracker.ceph.com/issues/42576
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28625 from zhangsw/rgw-fix-get-object

rgw: ignore If-Unmodified-Since if If-Match exists, and ignore If-Modified-Since if If-None-Match exists.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/PeeringState: clear LAGGY and WAIT states on exiting Started

These flags were not getting cleared except in recheck_readable(), which
meant that a flag from a prior interval could bleed into a new interval.
More dangerously, in a mixed-version cluster, one interval might include
all octopus+ OSDs while the next might include a pre-octopus OSD, bypassing
most of the laggy recheck code.  This could lead to a stalled request
and/or requeue ordering bug when release_object_locks() looked at
is_laggy() and put a lock waiter on the waiting_for_readable list.

Fixes: https://tracker.ceph.com/issues/42978
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: minor clean-up of `simple scan` subcommand help

Fixes: https://tracker.ceph.com/issues/43017
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31780 into master

* refs/pull/31780/head:
mgr/volumes: remove unsed variable

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31824 into master

* refs/pull/31824/head:
cls/rbd: sanitize the mirror image status peer address after reading from disk

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Update translations (#31435)

mgr/dashboard: Update translations

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/orchestrator: include container_image in 'service ls' output

Rearrange the columns a bit too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: include container_image_name and container_image_id in 'ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: default to ceph/ceph:latest-master-devel (for now)

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/osd/osd_operation.h: clean up duplicative check.

`seastar::future<T>::available()` already verifies whether
a future instance stores exception inside. It's unnecessary
to ensure that with further call to `failed()`.

This tiny clean-up removes the duplicative check in
`with_blocking_future()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #31677 into master

* refs/pull/31677/head:
qa/standalone/ceph-helpers.sh: remove osd down check
qa/standalone/ceph-helpers.sh: destroy_osd: mark osd down
osd: add osd_fast_shutdown option (default true)

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31823 into master

* refs/pull/31823/head:
ceph-daemon: fix logrotate su line

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31835 into master

* refs/pull/31835/head:
mgr/ssh: add TemporaryDirectory impl for py2 compat

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31325 from cbodley/wip-rgw-sync-concurrent-versions

rgw multisite: fixes for concurrent version creation

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

cmake/modules: look for GNU make first instead of BSD make

  * BuildDPDK.cmake, BuildSPDK.cmake:
      * give priority to build with gmake
      * throw error if make not found

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: support parallel build for rocksdb

* cmake/modules/BuildRocksDB.cmake: if "make" detected
use concurrent processing, by passing the jobserver to
sub-make using `$(MAKE)`

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge PR #30972 into master

* refs/pull/30972/head:
mds: move 'traverse to auth' logic into MDCache::path_traverse
mds: cleanup Capability initialization
mds: cleanup code that kills session requests
mds: cleanup Server::try_open_auth_dirfrag()
mds: restrict path that walks into snapdir
mds: use single map to track pinned & auth_pinned objects
mds: define operator<(...) for MutationImpl::LockOp
mds: cleanup Locker::acquire_locks()
mds: introduce Locker::wrlock_try()
mds: explictly specify if MDCache::path_traverse() needs to check null dentry
mds: cleanup dentry non-readable check in MDCache::path_traverse()
mds: change MDCache::path_traverse()'s 'onfail' argument to flags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #31786 into master

* refs/pull/31786/head:
mgr: fix errors on using a reference in a Lambda function

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/standalone/ceph-helpers.sh: remove osd down check

A kill doesn't induce a mark-down of the OSD with osd_fast_shutdown=true.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add TemporaryDirectory impl for py2 compat

This didn't get copied over in 4bfcb5db6bdb833496aa840d6aac603c7c405277

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31761 from tchaikov/wip-cmake-include-libfmt

cmake: do not include ${CMAKE_SOURCE_DIR}/src/fmt/include

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #31711 from p-se/wip-pse-fix-osd-full-alert

monitoring: wait before firing osd full alert

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31692 from rzarzynski/wip-fips-zeroize-memset_bzero

FIPS: audit and switch some memset & bzero users

Reviewed-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #31705 from tchaikov/wip-mgr-orchestrator

mgr/orchestrator: do not try to iterate through None

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31691 from Anjingkegui/master.monv2change1

mon: remove the restriction of address type in init_with_hosts

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>

Merge pull request #31513 from ifed01/wip-ifed-check-bluefs-allocs

os/bluestore: check bluefs allocations on log replay

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #30204 from tchaikov/wip-test-aio-completion-release

test/librados: free AioCompletion using AioCompletion::release()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31703 from xiexingguo/wip-42577-plus

osd/PeeringState: do not exclude up from acting_recovery_backfill

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31693 from tchaikov/wip-42782

qa/workunits/rados/test_librados_build.sh: download from current branch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31243 from lxbsz/log

log: just return if t is empty

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31714 from lxbsz/preforker

preforker: remove useless code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31778 into master

* refs/pull/31778/head:
os/bluestore: pin onodes as they are added to the cache
Revert "Revert "Merge pull request #30964 from markhpc/wip-bs-cache-trim-pinned""

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31807 into master

* refs/pull/31807/head:
mgr/PyModule: correctly remove config options

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31806 into master

* refs/pull/31806/head:
spec,debian: ceph-mgr-ssh depends on openssh{-client{s}}

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge PR #31818 into master

* refs/pull/31818/head:
common/options: remove unused ms_msgr2_{sign,encrypt}_messages

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31825 into master

* refs/pull/31825/head:
msg/async: fix typo in Error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31831 into master

* refs/pull/31831/head:
ceph-daemon: make infer_fsid behave when /var/lib/ceph dne

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31832 into master

* refs/pull/31832/head:
ceph-daemon: fix extract_uid_gid

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: fix extract_uid_gid

This will get confused by the cephdaemon user!

Signed-off-by: Sage Weil <sage@redhat.com>