mgr/ssh: Disable host labels.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: use finalize to set TrivialReadCompletion result

This is needed to correctly set the state of the completion, so that
Completion.has_result returns true.  Without this, _orchestrator_wait()
will spin forever if given a TrivialReadCompletion.

Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add pretty_print()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Fix Promises

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

import typing optionally

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Fix testing error

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Fixed DeepSeaReadCompletion arguments

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: fix tests

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

make check fixes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchstrator: Removed all type anotations incompatible to Python 2

* Reduce mypy's type checking capability
* Remove useful code documentation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: Adapted to new Completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add cancel

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/orchestrator: Adapt teuthology test to new completions

Also: Adapt mgr/test_orchestrator to new completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ansible: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Adapt ssh orch to new Completions interface

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Adapt to new completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Introduce composable completions

Also:

* unify Read and Write completions
* Distinguish should_wait and is_finished
* Removed `all_hosts` parameter to `osd_create`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: rename wait() -> process()

Because wait() should actually not wait!

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31904 from tchaikov/wip-crimson-stop-msgr-after-stopping-osd

crimson: stop osd before stopping messengers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: stop osd before stopping messengers

so we can drain all pending tasks referencing messenger before
destroying it, otherwise they will be using a already-stopped
messenger when they are still alive.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31756 into master

* refs/pull/31756/head:
pybind / cephfs: remove static typing in LibCephFS.chown

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31828 into master

* refs/pull/31828/head:
mgr/orchestrator: include container_image in 'service ls' output
ceph-daemon: include container_image_name and container_image_id in 'ls' output
common/options: default to ceph/ceph:latest-master-devel (for now)

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31854 into master

* refs/pull/31854/head:
mgr/orchestrator_cli: print table for 'host ls'
mgr/ssh: populate labels
mgr/orchestrator: add 'labels' property to InventoryNode
mgr/ssh: add/remove labels
mgr/orchestrator: orchestrator host label {add,rm}
mgr/ssh: add explicit host list

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31864 into master

* refs/pull/31864/head:
osd/PeeringState: clear LAGGY and WAIT states on exiting Started

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #31865 into master

* refs/pull/31865/head:
cls/rbd: improve efficiency of mirror image status queries

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #31866 into master

* refs/pull/31866/head:
osd/SnapMirror: no need to record purged_snaps every epoch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #27467 into master

* refs/pull/27467/head:
cephfs-shell: add snapshot management

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31871 from smithfarm/wip-43021

doc: mention --namespace option in rados manpage

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

doc: mention --namespace option in rados manpage

Fixes: https://tracker.ceph.com/issues/43021
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/orchestrator_cli: print table for 'host ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: populate labels

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add 'labels' property to InventoryNode

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add/remove labels

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: orchestrator host label {add,rm}

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add explicit host list

Keep the host list in an explicit 'inventory' key, and provide a dict for
each host so that we can keep other metadata (like labels).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31837 into master

* refs/pull/31837/head:
ceph-daemon: add additional debug logging

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31874 from tchaikov/wip-crimson-cyanstore-restructure

crimson: move os/cyan_* down to os/cyanstore/*

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: move os/cyan_* down to os/cyanstore/*

prepare for bluestore and seastore

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31859 from rzarzynski/wip-crimson-drop_dup_failed

crimson/osd/osd_operation.h: clean up duplicative check.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31252 from wjwithagen/wjw-doc-ceph_volume_zfs_inventory

doc/ceph-volume: initial docs for zfs/inventory and zfs/api

Merge pull request #31821 from mgfritch/c-v-scan-help

ceph-volume: minor clean-up of `simple scan` subcommand help

ceph-daemon: add additional debug logging

Adds additional detail before invoking subprocess for the
`run`, `shell, `enter`, and `logs` commands

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31567 into master

* refs/pull/31567/head:
common: move gen_rand_alphanumeric() helpers into common
test: add unit tests for gen_rand_alphanumeric helpers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #31781 into master

* refs/pull/31781/head:
cmake/modules: look for GNU make first instead of BSD make
cmake: support parallel build for rocksdb

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31579 into master

* refs/pull/31579/head:
cmake: add cppcheck and iwyu static analysis targets

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #31775 into master

* refs/pull/31775/head:
mds: remove the incorrect comments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #31762 into master

* refs/pull/31762/head:
mgr/ssh: improve logging for mds removal
mgr/volumes: move up 'confirm' validation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

osd/SnapMirror: no need to record purged_snaps every epoch

These keys were a mirror of the mon's purged_epoch keys.  We were writing
them, but never looking at them.  We only need the per-pool keys that
are indexed by snapid for lookups, so we should stop storing the per-epoch
records.

Note that the OSDSuperblock has the purged_snaps_last that serves as our
position/cursor for ingesting the mon keys.

Fixes: https://tracker.ceph.com/issues/42012
Signed-off-by: Sage Weil <sage@redhat.com>

cls/rbd: improve efficiency of mirror image status queries

The removal of remote mirror image status records no longer
(accidentally) searches through nearly the full omap database and
getting the status of a single image no longer involves duplicate
lookups (to pull the fsid keys and then to read the data).

Fixes: https://tracker.ceph.com/issues/42576
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28625 from zhangsw/rgw-fix-get-object

rgw: ignore If-Unmodified-Since if If-Match exists, and ignore If-Modified-Since if If-None-Match exists.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/PeeringState: clear LAGGY and WAIT states on exiting Started

These flags were not getting cleared except in recheck_readable(), which
meant that a flag from a prior interval could bleed into a new interval.
More dangerously, in a mixed-version cluster, one interval might include
all octopus+ OSDs while the next might include a pre-octopus OSD, bypassing
most of the laggy recheck code.  This could lead to a stalled request
and/or requeue ordering bug when release_object_locks() looked at
is_laggy() and put a lock waiter on the waiting_for_readable list.

Fixes: https://tracker.ceph.com/issues/42978
Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: minor clean-up of `simple scan` subcommand help

Fixes: https://tracker.ceph.com/issues/43017
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31780 into master

* refs/pull/31780/head:
mgr/volumes: remove unsed variable

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31824 into master

* refs/pull/31824/head:
cls/rbd: sanitize the mirror image status peer address after reading from disk

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Update translations (#31435)

mgr/dashboard: Update translations

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/orchestrator: include container_image in 'service ls' output

Rearrange the columns a bit too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: include container_image_name and container_image_id in 'ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: default to ceph/ceph:latest-master-devel (for now)

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/osd/osd_operation.h: clean up duplicative check.

`seastar::future<T>::available()` already verifies whether
a future instance stores exception inside. It's unnecessary
to ensure that with further call to `failed()`.

This tiny clean-up removes the duplicative check in
`with_blocking_future()`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #31677 into master

* refs/pull/31677/head:
qa/standalone/ceph-helpers.sh: remove osd down check
qa/standalone/ceph-helpers.sh: destroy_osd: mark osd down
osd: add osd_fast_shutdown option (default true)

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31823 into master

* refs/pull/31823/head:
ceph-daemon: fix logrotate su line

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31835 into master

* refs/pull/31835/head:
mgr/ssh: add TemporaryDirectory impl for py2 compat

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31325 from cbodley/wip-rgw-sync-concurrent-versions

rgw multisite: fixes for concurrent version creation

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

cmake/modules: look for GNU make first instead of BSD make

  * BuildDPDK.cmake, BuildSPDK.cmake:
      * give priority to build with gmake
      * throw error if make not found

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: support parallel build for rocksdb

* cmake/modules/BuildRocksDB.cmake: if "make" detected
use concurrent processing, by passing the jobserver to
sub-make using `$(MAKE)`

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge PR #30972 into master

* refs/pull/30972/head:
mds: move 'traverse to auth' logic into MDCache::path_traverse
mds: cleanup Capability initialization
mds: cleanup code that kills session requests
mds: cleanup Server::try_open_auth_dirfrag()
mds: restrict path that walks into snapdir
mds: use single map to track pinned & auth_pinned objects
mds: define operator<(...) for MutationImpl::LockOp
mds: cleanup Locker::acquire_locks()
mds: introduce Locker::wrlock_try()
mds: explictly specify if MDCache::path_traverse() needs to check null dentry
mds: cleanup dentry non-readable check in MDCache::path_traverse()
mds: change MDCache::path_traverse()'s 'onfail' argument to flags

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #31786 into master

* refs/pull/31786/head:
mgr: fix errors on using a reference in a Lambda function

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/standalone/ceph-helpers.sh: remove osd down check

A kill doesn't induce a mark-down of the OSD with osd_fast_shutdown=true.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add TemporaryDirectory impl for py2 compat

This didn't get copied over in 4bfcb5db6bdb833496aa840d6aac603c7c405277

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31761 from tchaikov/wip-cmake-include-libfmt

cmake: do not include ${CMAKE_SOURCE_DIR}/src/fmt/include

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #31711 from p-se/wip-pse-fix-osd-full-alert

monitoring: wait before firing osd full alert

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31692 from rzarzynski/wip-fips-zeroize-memset_bzero

FIPS: audit and switch some memset & bzero users

Reviewed-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #31705 from tchaikov/wip-mgr-orchestrator

mgr/orchestrator: do not try to iterate through None

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31691 from Anjingkegui/master.monv2change1

mon: remove the restriction of address type in init_with_hosts

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>

Merge pull request #31513 from ifed01/wip-ifed-check-bluefs-allocs

os/bluestore: check bluefs allocations on log replay

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #30204 from tchaikov/wip-test-aio-completion-release

test/librados: free AioCompletion using AioCompletion::release()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31703 from xiexingguo/wip-42577-plus

osd/PeeringState: do not exclude up from acting_recovery_backfill

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31693 from tchaikov/wip-42782

qa/workunits/rados/test_librados_build.sh: download from current branch

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31243 from lxbsz/log

log: just return if t is empty

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31714 from lxbsz/preforker

preforker: remove useless code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31778 into master

* refs/pull/31778/head:
os/bluestore: pin onodes as they are added to the cache
Revert "Revert "Merge pull request #30964 from markhpc/wip-bs-cache-trim-pinned""

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31807 into master

* refs/pull/31807/head:
mgr/PyModule: correctly remove config options

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31806 into master

* refs/pull/31806/head:
spec,debian: ceph-mgr-ssh depends on openssh{-client{s}}

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge PR #31818 into master

* refs/pull/31818/head:
common/options: remove unused ms_msgr2_{sign,encrypt}_messages

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31825 into master

* refs/pull/31825/head:
msg/async: fix typo in Error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31831 into master

* refs/pull/31831/head:
ceph-daemon: make infer_fsid behave when /var/lib/ceph dne

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31832 into master

* refs/pull/31832/head:
ceph-daemon: fix extract_uid_gid

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: fix extract_uid_gid

This will get confused by the cephdaemon user!

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make infer_fsid behave when /var/lib/ceph dne

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31636 into master

* refs/pull/31636/head:
mgr/pg_autoscaler: default to pg_num[_min] = 16

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31796 into master

* refs/pull/31796/head:
PendingReleaseNotes: note about the removal of 'nvme' class
common/blkdev: drop is_nvme() method
os/bluestore/KernelDevice: get rid of 'nvme' type

Reviewed-by: Neha Ojha <nojha@redhat.com>

cls/rbd: sanitize the mirror image status peer address after reading from disk

RADOS upgrade tests were failing when OSDs were partially upgraded since the
entity_addr_t::type overload wasn't being recovered when re-read. Now we will
always sanitize the on-disk entity address after reading it to avoid such
issues of on-disk encoding/decoding.

Fixes: https://tracker.ceph.com/issues/42891
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31816 from jan--f/c-v-no-unittest-mock

ceph-volume: import mock.mock instead of unittest.mock (py2)

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30863 from matthewoliver/improve_do_cmake

do_cmake.sh: Add CEPH_GIT_DIR

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async: fix typo in Error message

And fix layout while there

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

PendingReleaseNotes: note about the removal of 'nvme' class

There isn't much to say here since this really shouldn't be happening in
the wild.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31502 into master

* refs/pull/31502/head:
qa/tasks/ceph2: get ceph-daemon from same place as ceph
qa/tasks/ceph2: use safe_while
qa/tasks/ceph2: pull image using sha1
qa/tasks/ceph2: docker needs quay.io/ prefix for image name
qa/workunits/rados/test_python: make sure rbd pool exists
qa/suites/rados/ssh: new tests!
qa/tasks/ceph2: pull ceph-ci/ceph:$branch
qa/tasks/ceph2: register_daemons after pods start
qa/tasks/ceph2: fix conf
qa/tasks/ceph2: add restart
qa/tasks/ceph2: pass ceph-daemon path to DaemonState
qa/tasks/ceph2: tolerate no mdss or 1 mgr
qa/tasks/ceph: replace wait_for_osds_up with manager.wait_for_all_osds_up
qa/tasks/ceph: wait-until-healthy
qa/tasks/ceph2: set up managers
qa/tasks/ceph2: use seed ceph.conf
qa/tasks/ceph: healthy: use manager helpers (instead of teuthology/misc ones)
qa/tasks/ceph2: name mds daemons
qa/tasks/ceph2: fix osd ordering
qa/tasks/ceph2: start up mdss
qa/tasks/ceph2: set up daemon handles and use them to stop
qa/tasks/ceph2: make it multicluster-aware
qa/tasks/ceph2: can bring up mon, mgr, osds!
qa/tasks/ceph2: basic task to bring up cluster with ceph-daemon and ssh

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: allow apply_olh_log to rewrite the same olh version

if two zones write an object version with the same olh epoch,
apply_olh_log needs to be able to write the more-current instance at the
same epoch

Fixes: https://tracker.ceph.com/issues/39142
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: apply_olh_log filters out older instances of same epoch

when apply_olh_log() detects multiple instances with the same epoch, we
only promote the one that the bucket index lists as more current

Fixes: https://tracker.ceph.com/issues/39142
Signed-off-by: Casey Bodley <cbodley@redhat.com>