mgr/test_orchestrator: add daemon_action

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasksmgr/test_orchestrator_cli: fix test_service_action

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/test_orchestrator_cli: disable test_osd_create exception check

I'm not sure what behavior we want here...

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: update_foo -> apply_foo

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: add force flag to remove_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: update

Most of these were broken due to CLI changes weeks ago.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix test_load_data

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: fix list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33456 from smithfarm/wip-44182

ceph-backport.sh: fix setup routine

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #33510 from krig/mypy-cache-gitignore

.gitignore: Add .mypy_cache to .gitignore

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33504 into master

* refs/pull/33504/head:
cephadm: allow skipping prepare_host in bootstrap step

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33495 into master

* refs/pull/33495/head:
mgr/cephadm: do not refresh device inventory on mgr restart
mgr/cephadm: make cache invalidate less racy
mgr/cephadm: fix last_device_update persistence

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>

Merge PR #33501 into master

* refs/pull/33501/head:
debian: modules-core replaces and breaks older ceph-mgr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33513 from liewegas/fix-44271

mgr/dashboard: behave when pwdUpdateRequired key is missing

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33440 from dillaman/wip-44103

rbd-mirror: provide initial snapshot replay status

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #33497 into master

* refs/pull/33497/head:
qa/suites/rados/singleton/all/thrash-eio: whitelist 'slow request'
qa: Whitelist 'slow request' within a bunch of tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33138 into master

* refs/pull/33138/head:
common/TextTable: only pad between columns
mgr/status: align with ceph table style
mgr/osd_perf_query: make table match ceph style
mgr: adjust tables to have 2 space column separation
common/TextTable: default to 2 spaces separating columns

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33413 into master

* refs/pull/33413/head:
test: verify purge queue w/ large number of subvolumes
test: pass timeout argument to mount::wait_for_dir_empty()
mgr/volumes: access volume in lockless mode when fetching async job

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/TextTable: only pad between columns

No need to pad on the right-hand side if there are no additional columns
that follow.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33113 into master

* refs/pull/33113/head:
doc/mgr/telemetry: added device channel details

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33439 into master

* refs/pull/33439/head:
common/options.cc: make rocksdb_delete_range_threshold very high

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd-mirror: implement basic status feedback for snapshot mirroring

The feedback includes the newest remote mirror snapshot timestamp,
the newest completely synced local mirror snapshot timestamp,
and optionally the in-progress sync snapshot timestamp and
percent complete.

Fixes: https://tracker.ceph.com/issues/44103
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved local to remote snapshot lookup to common function

This will be needed by the status formatter to lookup the remote
snapshot timestamp since the associated local snapshot timestamp
will be the time the snapshot was created on the local side.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: expand lock scope for snapshot replayer

The status callback will need access to the current state, so the
lock should be held while the snapshot scan is in-progress.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: periodically update sync position for snapshot-based images

This will allow the sync to restart at the last known successfully copied
object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33487 into master

* refs/pull/33487/head:
mgr/cephadm: upgrade: fix handling when target container id changes
mgr/cephadm: upgrade: handle stopped daemons

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33496 into master

* refs/pull/33496/head:
mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service
mgr/cephadm: remove apply_mon support
mgr/cephadm: use generics for add_mon
mgr/cephadm: use _apply_service for mgrs
mgr/cephadm: refactor most daemon add methods
mgr/cephadm: refactor _update_service and all apply methods
mgr/cephadm: fix get_unique_name when name in use

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33465 into master

* refs/pull/33465/head:
mgr/cephadm: implement placement by label
mgr/cephadm: fix self.log call in get_unique_name
mgr/cephadm: fix NodePlacement host names
mgr/tests/test_orchestrator: test PlacementSpec parsing
mgr/orch: remove label: prefix when parsing placementspec
mgr/orch: PlacementSpec: add all_hosts property

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33514 into master

* refs/pull/33514/head:
mgr/cephadm: drop mixin parent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33477 into master

* refs/pull/33477/head:
qa/tasks/mgr/test_orchestrator_cli: fix test_host_ls

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33474 into master

* refs/pull/33474/head:
mgr/orch: make arg hostname, not host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33418 into master

* refs/pull/33418/head:
mon: stash newer map on bootstrap when addr doesn't match

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33285 into master

* refs/pull/33285/head:
src/common: fix help text for echo option of cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33291 into master

* refs/pull/33291/head:
mds: fix use-after-free in Migrater

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service

These were the same.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove apply_mon support

Our apply method doesn't support removing mons at this point.  And using
it for adding mons is just an awkward version of 'daemon add'.

Update docs and cephadm.py task accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use generics for add_mon

Adjust apply_mon a bit so that the arg order for _create_mon is still
correct, and so that the name inference is done in the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use _apply_service for mgrs

Note that we are losing some of the special logic about removing standby
mgrs only.  This should be added back *after* we fix up the scheduler
to be more intelligent about choosing hosts that already host daemons,
and make removal pick hosts that aren't selected (by label, or by
scheduler, etc.).

A few bugs to track this:
 https://tracker.ceph.com/issues/44167
 https://tracker.ceph.com/issues/44252 (prefer standby mgrs *and* mdss)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor most daemon add methods

New _add_daemons() that shares what we can with _apply_service().

Unlike _apply_service(), the _add_daemon() path requires the user to
explicitly place daemons on hosts--no automatic placement is performed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor _update_service and all apply methods

- Use a common _apply_service() helper
- Consolidate _add_new_daemon logic into _apply_service
- Do the NodeAssignment all in one place

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix get_unique_name when name in use

If we are not adding a uniqueifying suffix, we need to error out instead
of looping.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33500 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/cephadm: do not refresh device inventory on mgr restart

The service inventory is more fluid and is faster to gather.  We also
make a time-saving assumption that we don't need to persist our cache
updates when making changes because we know a mgr restart will refresh.

Device inventory changes are much less frequent and slower.  Let's not
refresh them every restart.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make cache invalidate less racy

Consider a cache invalidation that races with an actual update:

- serve() refresh starts
- refresh runs cephadm ls
- add_daemon creates a new daemon
- add_daemon returns and invalidates the list (set last_udpate=None)
- serve() stores its ls result in the cache

In such a case the add result will get lost.

Fix this by taking a conservative strategy:

- invalidate adds host to a refresh list
- serve() removes an item from the refresh list and then does the ls,
then stores the result.

Any racing update will invalidate *after* it does it's work, which means
we will always do a final ls afterwards.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix last_device_update persistence

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop mixin parent

This was added a while back to let us block on completions, but that
code has since been removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: fix handling when target container id changes

If the same target_name pulls a new hash partway through the upgrade
(e.g., because the tag was updated in the registry), we restart the
upgrade.

Fix the code so that we update target_id instead of (unused) image_id.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: handle stopped daemons

A stopped daemon should have the correct target_name, and we should ensure
that the host has an up-to-date image, so that when it does start it
comes up with the new image.  If it has an old image name, we should
redeploy as per usual.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: behave when pwdUpdateRequired key is missing

Fixes: https://tracker.ceph.com/issues/44271
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/thrash-eio: whitelist 'slow request'

Signed-off-by: Sage Weil <sage@redhat.com>

qa: Whitelist 'slow request' within a bunch of tests

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge PR #33484 into master

* refs/pull/33484/head:
mgr/cephadm: catch exceptions when scraping ceph-volume inventory

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: implement placement by label

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix self.log call in get_unique_name

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix NodePlacement host names

This broke back in 82e4f1e0f1332f7d4f27dd0f83d394a6f14c6bd2, I think.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: test PlacementSpec parsing

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: remove label: prefix when parsing placementspec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec: add all_hosts property

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: Add .mypy-cache to .gitignore

Ignore the mypy cache directory.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge pull request #33481 from tchaikov/wip-crimson-cflags

cmake: move crimson-crush to crimson/

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test: verify purge queue w/ large number of subvolumes

Fixes: http://tracker.ceph.com/issues/44207
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: pass timeout argument to mount::wait_for_dir_empty()

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: access volume in lockless mode when fetching async job

Saw a deadlock when deleting lot of subvolumes -- purge threads were
stuck in accessing global lock for volume access. This can happen
when there is a concurrent remove (which renames and signals the
purge threads) and a purge thread is just about to scan the trash
directory for entries.

For the fix, purge threads fetches entries by accessing the volume
in lockless mode. This is safe from functionality point-of-view as
the rename and directory scan is correctly handled by the filesystem.
Worst case the purge thread would pick up the trash entry on next
scan, never leaving a stale trash entry.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #33429 from votdev/add_always_on_column

mgr/dashboard: Add 'Always-on' column to mgr module list

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

cephadm: allow skipping prepare_host in bootstrap step

There are situations that required packages are managed outside of
cephadm script, allowing skipping prepare_host in these situations.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

debian: modules-core replaces and breaks older ceph-mgr

in 589626464d475b4ad96905fbddf11a5782a040f0, the core modules are moved
into ceph-mgr-modules-core. but the upgrade path from older ceph-mgr was
broken by this change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: reduce the indent level

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use loop for applying new policies

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: make arg hostname, not host

For optional args, it can't be --host because this (currently) conflicts
with a core ceph option.  We change all of them for consistency.

This fixes 'ceph orch ps --host foo'.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/status: align with ceph table style

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/osd_perf_query: make table match ceph style

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: adjust tables to have 2 space column separation

Signed-off-by: Sage Weil <sage@redhat.com>

common/TextTable: default to 2 spaces separating columns

This is what other projects and libraries default to, and it is more
legible.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33485 into master

* refs/pull/33485/head:
qa/workunits/cephadm/test_adoption: run as root

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33467 into master

* refs/pull/33467/head:
doc/orchestrator: Substitute `host` for `node`
mgr/orchestrator: Substitute `node` with `host`, globally

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33328 into master

* refs/pull/33328/head:
osd/OSD: Log slow ops/types to cluster logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33380 into master

* refs/pull/33380/head:
common/admin_socket: Added printing of error message constucted in handlers.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/orchestrator: Substitute `host` for `node`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Substitute `node` with `host`, globally

Right now, there is a mix of `node` and `host`. Unify this to `host`

* mgr/rook is special, as Kubernetes nativaly uses "node"

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/workunits/cephadm/test_adoption: run as root

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33448 from dillaman/wip-librbd-group-snaps

librbd: fix broken group snapshot handling

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33335 from dillaman/wip-44043

rbd-mirror: apply image state during snapshot replay

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33451 from dillaman/wip-rbd-support-tasks

pybind/mgr/rbd_support: wait for latest OSD map prior to handling commands

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #31963 from yison/rwl-image-writeback-cache-seq2

rbd/cache: Replicated Write Log core codes part 2

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33461 into master

* refs/pull/33461/head:
cephadm: Simplify is_available
cephadm: Improve type signature of is_available
cephadm: Fix type errors found by mypy
cephadm: Enable and start chrony using any packager
cephadm: Add Packager for Zypper (openSUSE / SLES)
cephadm: Normalize mypy annotations

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33486 into master

* refs/pull/33486/head:
qa/suites/rados/cephadm/smoke: run a few orch commands
mgr/orch: fix 'host ls'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33475 into master

* refs/pull/33475/head:
qa/suites/rados/cephadm/upgrade: change start version

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33091 into master

* refs/pull/33091/head:
qa/suites/rados: disable device scraping
qa/standalone/ceph-helpers: disable device monitoring
qa/tasks/ceph.py: add pre-mgr-commands option for ceph task
mgr/devicehealth: set default monitoring to 'on'

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33235 into master

* refs/pull/33235/head:
os/bluestore/BlueFS: fixed printing stats

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33362 into master

* refs/pull/33362/head:
common/config :remove duplicaed code.
common/config: remove useless code.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33404 into master

* refs/pull/33404/head:
selinux: Allow ceph to setsched

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/cephadm/smoke: run a few orch commands

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix 'host ls'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33472 into master

* refs/pull/33472/head:
cephadm: allow users to provide their dashboard cert during bootstrap

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: change start version

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33469 into master

* refs/pull/33469/head:
mgr/cephadm: fix upgrade ok-to-stop condition check

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: catch exceptions when scraping ceph-volume inventory

This matches what we do with the background 'cephadm ls'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33473 from liewegas/remove-memstore-test

test/CMakeLists: disable memstore make check test

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: move crimson-crush to crimson/

so it's able to use crimson::cflags, this interface target could be
defined after crush/CMakeLists.txt is included by upper directory, so
just move it into crimson/CMakeLists.txt

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33106 from Yan-waller/wip-walle-fixbackfillundersized

osd/PG: restart peering for undersized PG on any down stray peer coming back

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>