doc: Add Dashboard Octopus release notes

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #33171 from rhcs-dashboard/wip-42914-fix-ui-issues

mgr/dashboard: UI fixes

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33405 from rhcs-dashboard/wip-44198-fix-style-issues

mgr/dashboard: 'destroyed' view in CRUSH map viewer

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #33459 from liewegas/doc-adopt

doc/mgr/cephadm: document adoption process

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #33454 from votdev/fix_mypy_issues

mgr/dashboard: Fix mypy issues and enable it by default

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33372 from pritha-srivastava/rgw-omap-offload-gc-fixes

Fixes related to omap offload and gc.

Merge pull request #33528 from cyx1231st/wip_crimson_osd_misc_fixes

crimson/osd: misc fixes and cleanup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33543 from badone/wip-doc-min_size-applicable-to-ec-pools

doc/rados/operations: min_size is applicable to EC

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados/operations: Detail for min_size on EC pools

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #32342 from dzafman/wip-43126

mon: Improvements to slow heartbeat health messages

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rados/operations: min_size is applicable to EC

min_size is *not* replicated pool specific.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #33379 from badone/wip-upgrade-ansible-version-for-ca

qa/ceph-ansible: Upgrade ansible version

Merge PR #33516 into master

* refs/pull/33516/head:
ceph.in: Fix name retval is not defined error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>

Merge PR #33525 into master

* refs/pull/33525/head:
osd/PeeringState.h: ignore RemoteBackfillReserved in WaitLocalBackfillReserved

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #33263 into master

* refs/pull/33263/head:
qa/vstart_runner.py: make run()'s interface same as teuthology's run

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33499 into master

* refs/pull/33499/head:
mount: fix the debug log when keyring getting secret failed

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #33515 into master

* refs/pull/33515/head:
mgr/cephadm: replace static SSL cert
mgr/cephadm: dynamically add Grafana data sources
mgr/cephadm: provide static default configs for grafana deployment
mgr/cephadm: provision grafana
mgr/orch: add grafana hooks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29816 from coder-hugo/feature/fix-ldap-auth-doc

rgw: Fix documentation for rgw_ldap_secret

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #33480 into master

* refs/pull/33480/head:
test: add new ceph_lseek test
client: fix error handling in lseek SEEK_HOLE/SEEK_DATA cases

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #33427 into master

* refs/pull/33427/head:
qa/cephfs: rewrite a bit of code xfstests_dev.py
qa/cephfs: update xfstests-dev deps for RHEL 8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32657 into master

* refs/pull/32657/head:
test: query using mds id, not rank
mgr: re-enable mds `scrub status` info in ceph status
mon: filter out ceph normal ceph entity types when dumping service metadata
mgr: filter out normal ceph services when processing service map
mgr: helper function to check if a service is a normal ceph service

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33449 into master

* refs/pull/33449/head:
mds: handle bad purge queue item encoding

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #33471 into master

* refs/pull/33471/head:
mds: relevel debug message levels for balancer/migrator
mds: inline mds_role_t stream operator

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #33535 from smithfarm/wip-rados-cephadm-smoke-152

qa: run cephadm/smoke on opensuse 15.2 instead of 15.1

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #33078 into master

* refs/pull/33078/head:
qa: avoid subtree rep in test_version_splitting

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #33456 from smithfarm/wip-44182

ceph-backport.sh: fix setup routine

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #33510 from krig/mypy-cache-gitignore

.gitignore: Add .mypy_cache to .gitignore

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33504 into master

* refs/pull/33504/head:
cephadm: allow skipping prepare_host in bootstrap step

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: replace static SSL cert

with dynamically generated one.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

qa: run cephadm/smoke on opensuse 15.2 instead of 15.1

Companion PR to https://github.com/ceph/ceph-build/pull/1509

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #33495 into master

* refs/pull/33495/head:
mgr/cephadm: do not refresh device inventory on mgr restart
mgr/cephadm: make cache invalidate less racy
mgr/cephadm: fix last_device_update persistence

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>

Merge PR #33501 into master

* refs/pull/33501/head:
debian: modules-core replaces and breaks older ceph-mgr

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: dynamically add Grafana data sources

depending on which Prometheus services have been deployed.

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: provide static default configs for grafana deployment

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/cephadm: provision grafana

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: Fix mypy issues and enable it by default

The decorator @no_type_check is used:
* To prevent a mypy error like 'error: INTERNAL ERROR -- Please try using mypy master on Github:'
* '#type: ignore' does not work, e.g. in broken lines

Fixes: https://tracker.ceph.com/issues/44269
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #33513 from liewegas/fix-44271

mgr/dashboard: behave when pwdUpdateRequired key is missing

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/orch: add grafana hooks

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

crimson/osd: hide add_blocker() and clear_blocker() in Operation

Cleanup. Hide add_blocker() and clear_blocker() as private members
because no one is using them anymore. They are not exception-safe. Also
take the chance to reorder Operation class members.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: use with_blocking_future in CompoundPeeringRequest

Cleanup. with_blocking_future() can cover all the use cases of
add_blocker() and clear_blocker(), and it is exception-safe and not
error-prone.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: fix osdmap update during osd start

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: fix min_peers related logic in heartbeat

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #33440 from dillaman/wip-44103

rbd-mirror: provide initial snapshot replay status

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #33497 into master

* refs/pull/33497/head:
qa/suites/rados/singleton/all/thrash-eio: whitelist 'slow request'
qa: Whitelist 'slow request' within a bunch of tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33138 into master

* refs/pull/33138/head:
common/TextTable: only pad between columns
mgr/status: align with ceph table style
mgr/osd_perf_query: make table match ceph style
mgr: adjust tables to have 2 space column separation
common/TextTable: default to 2 spaces separating columns

Reviewed-by: Sebastian Wagner <swagner@suse.com>

osd/PeeringState.h: ignore RemoteBackfillReserved in WaitLocalBackfillReserved

It is possible to dequeue an outstanding RemoteBackfillReserved, though we may have
already released reservations for that backfill target. Currently, if this happens
while we are in WaitLocalBackfillReserved, it can lead to a crash on the primary.
Prevent this by treating this condition as a no-op.

The longer term fix is to add a RELEASE_ACK mechanism, which prevents the primary
from scheduling a backfill retry until all the RELEASE_ACKs have been received.

Fixes: https://tracker.ceph.com/issues/44248
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #33413 into master

* refs/pull/33413/head:
test: verify purge queue w/ large number of subvolumes
test: pass timeout argument to mount::wait_for_dir_empty()
mgr/volumes: access volume in lockless mode when fetching async job

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/TextTable: only pad between columns

No need to pad on the right-hand side if there are no additional columns
that follow.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33113 into master

* refs/pull/33113/head:
doc/mgr/telemetry: added device channel details

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33439 into master

* refs/pull/33439/head:
common/options.cc: make rocksdb_delete_range_threshold very high

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd-mirror: implement basic status feedback for snapshot mirroring

The feedback includes the newest remote mirror snapshot timestamp,
the newest completely synced local mirror snapshot timestamp,
and optionally the in-progress sync snapshot timestamp and
percent complete.

Fixes: https://tracker.ceph.com/issues/44103
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved local to remote snapshot lookup to common function

This will be needed by the status formatter to lookup the remote
snapshot timestamp since the associated local snapshot timestamp
will be the time the snapshot was created on the local side.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: expand lock scope for snapshot replayer

The status callback will need access to the current state, so the
lock should be held while the snapshot scan is in-progress.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: periodically update sync position for snapshot-based images

This will allow the sync to restart at the last known successfully copied
object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33487 into master

* refs/pull/33487/head:
mgr/cephadm: upgrade: fix handling when target container id changes
mgr/cephadm: upgrade: handle stopped daemons

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33496 into master

* refs/pull/33496/head:
mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service
mgr/cephadm: remove apply_mon support
mgr/cephadm: use generics for add_mon
mgr/cephadm: use _apply_service for mgrs
mgr/cephadm: refactor most daemon add methods
mgr/cephadm: refactor _update_service and all apply methods
mgr/cephadm: fix get_unique_name when name in use

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33465 into master

* refs/pull/33465/head:
mgr/cephadm: implement placement by label
mgr/cephadm: fix self.log call in get_unique_name
mgr/cephadm: fix NodePlacement host names
mgr/tests/test_orchestrator: test PlacementSpec parsing
mgr/orch: remove label: prefix when parsing placementspec
mgr/orch: PlacementSpec: add all_hosts property

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33514 into master

* refs/pull/33514/head:
mgr/cephadm: drop mixin parent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: avoid subtree rep in test_version_splitting

Root subtree may be replicated which would open client sessions early.

Fixes: https://tracker.ceph.com/issues/43796
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33477 into master

* refs/pull/33477/head:
qa/tasks/mgr/test_orchestrator_cli: fix test_host_ls

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33474 into master

* refs/pull/33474/head:
mgr/orch: make arg hostname, not host

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/mgr/cephadm: document adoption process

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33418 into master

* refs/pull/33418/head:
mon: stash newer map on bootstrap when addr doesn't match

Reviewed-by: Neha Ojha <nojha@redhat.com>

test: add new ceph_lseek test

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: fix error handling in lseek SEEK_HOLE/SEEK_DATA cases

The error handling here looks quite wrong. If the offset represents
an area outside the file, then we want to just return -ENXIO
immediately.

Fixes: https://tracker.ceph.com/issues/44021
Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc/mgr/cephadm: fix headings, requirements

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33285 into master

* refs/pull/33285/head:
src/common: fix help text for echo option of cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: relevel debug message levels for balancer/migrator

Some of these messages are too verbose or inconsistent with other
messages. Also, clean up function name printing and be consistent
for all functions.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: inline mds_role_t stream operator

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33291 into master

* refs/pull/33291/head:
mds: fix use-after-free in Migrater

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph.in: Fix name retval is not defined error

In case of exceptions other than KeyboardInterrupt, retval is not defined.
Move exit call outside the finally block.

Fixes: https://tracker.ceph.com/issues/44275
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/cephadm: combine get_daemons_by_daemon -> get_daemons_by_service

These were the same.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove apply_mon support

Our apply method doesn't support removing mons at this point.  And using
it for adding mons is just an awkward version of 'daemon add'.

Update docs and cephadm.py task accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use generics for add_mon

Adjust apply_mon a bit so that the arg order for _create_mon is still
correct, and so that the name inference is done in the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use _apply_service for mgrs

Note that we are losing some of the special logic about removing standby
mgrs only.  This should be added back *after* we fix up the scheduler
to be more intelligent about choosing hosts that already host daemons,
and make removal pick hosts that aren't selected (by label, or by
scheduler, etc.).

A few bugs to track this:
 https://tracker.ceph.com/issues/44167
 https://tracker.ceph.com/issues/44252 (prefer standby mgrs *and* mdss)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor most daemon add methods

New _add_daemons() that shares what we can with _apply_service().

Unlike _apply_service(), the _add_daemon() path requires the user to
explicitly place daemons on hosts--no automatic placement is performed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refactor _update_service and all apply methods

- Use a common _apply_service() helper
- Consolidate _add_new_daemon logic into _apply_service
- Do the NodeAssignment all in one place

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix get_unique_name when name in use

If we are not adding a uniqueifying suffix, we need to error out instead
of looping.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33500 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/cephadm: do not refresh device inventory on mgr restart

The service inventory is more fluid and is faster to gather.  We also
make a time-saving assumption that we don't need to persist our cache
updates when making changes because we know a mgr restart will refresh.

Device inventory changes are much less frequent and slower.  Let's not
refresh them every restart.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make cache invalidate less racy

Consider a cache invalidation that races with an actual update:

- serve() refresh starts
- refresh runs cephadm ls
- add_daemon creates a new daemon
- add_daemon returns and invalidates the list (set last_udpate=None)
- serve() stores its ls result in the cache

In such a case the add result will get lost.

Fix this by taking a conservative strategy:

- invalidate adds host to a refresh list
- serve() removes an item from the refresh list and then does the ls,
then stores the result.

Any racing update will invalidate *after* it does it's work, which means
we will always do a final ls afterwards.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix last_device_update persistence

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop mixin parent

This was added a while back to let us block on completions, but that
code has since been removed.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: fix handling when target container id changes

If the same target_name pulls a new hash partway through the upgrade
(e.g., because the tag was updated in the registry), we restart the
upgrade.

Fix the code so that we update target_id instead of (unused) image_id.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade: handle stopped daemons

A stopped daemon should have the correct target_name, and we should ensure
that the host has an up-to-date image, so that when it does start it
comes up with the new image.  If it has an old image name, we should
redeploy as per usual.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: behave when pwdUpdateRequired key is missing

Fixes: https://tracker.ceph.com/issues/44271
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton/all/thrash-eio: whitelist 'slow request'

Signed-off-by: Sage Weil <sage@redhat.com>

qa: Whitelist 'slow request' within a bunch of tests

Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge PR #33484 into master

* refs/pull/33484/head:
mgr/cephadm: catch exceptions when scraping ceph-volume inventory

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: implement placement by label

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix self.log call in get_unique_name

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix NodePlacement host names

This broke back in 82e4f1e0f1332f7d4f27dd0f83d394a6f14c6bd2, I think.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/tests/test_orchestrator: test PlacementSpec parsing

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: remove label: prefix when parsing placementspec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec: add all_hosts property

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: Add .mypy-cache to .gitignore

Ignore the mypy cache directory.

Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>

Merge pull request #33481 from tchaikov/wip-crimson-cflags

cmake: move crimson-crush to crimson/

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test: verify purge queue w/ large number of subvolumes

Fixes: http://tracker.ceph.com/issues/44207
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: pass timeout argument to mount::wait_for_dir_empty()

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/volumes: access volume in lockless mode when fetching async job

Saw a deadlock when deleting lot of subvolumes -- purge threads were
stuck in accessing global lock for volume access. This can happen
when there is a concurrent remove (which renames and signals the
purge threads) and a purge thread is just about to scan the trash
directory for entries.

For the fix, purge threads fetches entries by accessing the volume
in lockless mode. This is safe from functionality point-of-view as
the rename and directory scan is correctly handled by the filesystem.
Worst case the purge thread would pick up the trash entry on next
scan, never leaving a stale trash entry.

Signed-off-by: Venky Shankar <vshankar@redhat.com>