mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Fixes: https://tracker.ceph.com/issues/44805
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34276 from badone/wip-fix-argument-ordering-docker-test

tests: Reinstate "opts" as final argument

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: Reinstate "opts" as final argument

a1e8f61cb72 broke the "opts" argument.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34252 from JackSlateur/doc-os-recommendations2

doc/start/os-recommendations.rst: recommend bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34267 from JackSlateur/doc-man-monmaptool

doc/man/8/monmaptool.rst: update synopsis etc with the latest options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man/8/monmaptool.rst: update the page with the latest options

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge pull request #34224 from markhpc/wip-bluefs-disable-buffered-io

common/options: Disable bluefs_buffered_io by default again.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33052 from ifed01/wip-ifed-fix-unused2

os/bluestore: fix 'unused' calculation

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34203 from majianpeng/bluefs-remove-unused-func

os/bluestore: remove unused func.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34061 into master

* refs/pull/34061/head:
mgr/orch: Add `ceph orch ls --export`
mgr/dashboard: adapt to new `ServiceDescription.to_json()`
python-common: add `service_name` to `ServiceSpec.to_json`
python-common: make ServiceSpec and ServiceDescription compatible
src/ceph.in: add yaml to known formats
mgr/orch: add yaml to `orch ls`
mgr/orch: remove `orch spec dump`
python-common: reorder RGWSpec arguments
python-common: prevent ServiceSpec of wrong type
pybind/mgr: tox.ini: omit cov report
mgr/cephadm: test describe_service
mgr/orch: ServiceDescription: change json representation
mgr/orch: ServiceDescription: Make spec a requirement

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34215 into master

* refs/pull/34215/head:
cephadm: Fix trucated output of `mgr dump`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34268 from poyea/docs-options-1

doc: Fix inconsistency in logging settings

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix inconsistency in logging settings

This patch fixes inconsistency in logging settings with options, namely `log_flush_on_exit` and `log_to_stderr`. This patch also adds `log_to_file` to the section.

Signed-off-by: John Law <johnlaw.po@gmail.com>

Merge PR #34062 into master

* refs/pull/34062/head:
cephadm: update check-host() to return all problems

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33943 from xxhdx1985126/wip-crimson-alienstore-bugs

crimson: fix bugs that come up when osds go through down/up

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: Fix trucated output of `mgr dump`

In case the process has terminated, we
have to make sure to read till EOF, otherwise
we might skip some bytes.

Fixes: https://tracker.ceph.com/issues/44642
Co-authored-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/start/os-recommendations.rst: recommend bluestore

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge PR #34247 into master

* refs/pull/34247/head:
mgr/dashboard: do not fail on user creation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33279 from rishabh-d-dave/fs-move-run_shell-and-related-methods

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Merge pull request #34021 from lxbsz/deleg_XXX

test/libcephfs: retry and wait for caps when setting the RD deleg

Merge pull request #33494 from lxbsz/mount

mount.ceph: stop printing warning message for unrecognized mount options

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #33740 from lxbsz/qa_test_timeout_failed

qa/test_config_session_timeout: fix options conflicting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34024 from vshankar/wip-ignore-mgrmap-when-laggy

mds: do not defer incoming mgrmap when mds is laggy

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

mgr/dashboard: do not fail on user creation

Accordingly to other Ceph commands do not fail on user
creation with a non-zero error code if the user already
exists.
Instead succeed and return the message 'User <username>
already exists'.

Fixes: https://tracker.ceph.com/issues/44502
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #33992 from rhcs-dashboard/44626-fix-notifications-e2e-tests

mgr/dashboard: fix notifications E2E tests

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34209 from smithfarm/wip-doc-multi-cluster-follow-on-fix

doc: fix cephadm link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34235 from toabctl/add-sle-15-sp2

qa/distros: add SLE-15-SP2

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32749 from james58899/fix-capacity

monitoring: Fix pool capacity incorrect

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #34233 from tchaikov/wip-qa-py3

qa/tasks: change to be py3 compatible and cleanups

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

qa/distros: add SLE-15-SP2

Ceph octopus is known to run on SLE-15-SP2 so add it to
qa/distros/all.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

cmake: add import-tasks to run-tox-qa test

to ensure that qa suites are python3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: use relative import

for better readability, and to ease the pain of developer to track back
to the top level python package for referencing a submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: fix print() to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33949 from toabctl/import-tasks-with-py3

qa: Add a tox env that can test importing files

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34141 from tchaikov/wip-test-threadsafe

test/objectstore: set "threadsafe" flag for ASSERT_DEATH tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson: fix pglog read_log_and_missing issue

Currently pglog's read_log_and_missing_crimson's implementation
would cause null pointer exception, since at the time, we don't the
key to start from

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add necessary FuturizedStore APIs for data recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: make sure pg meta is valid when loading pg

currently, PGMeta instances are all local variables in method,
which would lead to daggling reference/pointer problems when using
seastar::futures.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

common/options: Disable bluefs_buffered_io by default again.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #32093 into master

* refs/pull/32093/head:
doc/start: Update hardware recommendations.

Reviewed-by: Sage Weil <sage@redhat.com>

doc/start: Update hardware recommendations.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #34218 into master

* refs/pull/34218/head:
mgr/cephadm: Add missing decorator

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Add missing decorator

Caused by a conflict between

* #33581 (770816dc9f5b440fd1fc6cc507a06cd8573f6ea1) and
* #34091 (a78f42e225bbc041324dc3e565deb207feddad26)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33873 from lxbsz/qa_vs_failed

qa/tasks/vstart_runner: fix sh to make it compatible with teth

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33882 from rosinL/wip-fix-rdma-segfault

msg/async/rdma: add return value check after qp created

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34195 from tchaikov/wip-tasks-mgr-submodule

qa/tasks/mgr: partial revert of 'import with full path'

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34196 from majianpeng/bluestore-remove-incorrectly-comments

os/bluestore: remove incorrect comments.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34183 from neha-ojha/wip-doc-fix

doc/releases/octopus.rst: fix min_mon_release string

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: fix cephadm link

Fixes: 8fe460e7562b28e007179eeb533ef68a3e99b4c8
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #34091 into master

* refs/pull/34091/head:
mgr/cephadm: cleanup async_map_completion etc.
mgr/cephadm: replace async_completion with trivial_completion
mgr/cephadm: replace async_map_completion with a simple wrapper

Merge PR #34188 into master

* refs/pull/34188/head:
doc: do not encourage readers to run multiple clusters on same HW

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30839 from ifed01/wip-ifed-aloc_replay_improve

test/bmap_allocator_replay_test: improving allocator replay tool.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/orch: Add `ceph orch ls --export`

* defaults to `--format yaml`
* don't include `status`

```
$  ceph orch ls --export
placement:
  host_pattern: '*'
service_name: crash
service_type: crash
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: adapt to new `ServiceDescription.to_json()`

`ServiceDescription.to_json()` is now based on `ServiceSpec.to_json()`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: add `service_name` to `ServiceSpec.to_json`

To make it compatible to `Orchestrator.describe_service`.
Otherwise we have the awkward situation that users need to
pass `service_name` to `describe_service`, but `service_id` to apply

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: make ServiceSpec and ServiceDescription compatible

`ServiceSpec.from_json(ServiceDescription().to_json())` now works

as does

`ceph orch ls  --format yaml | ceph orch apply -i -`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

src/ceph.in: add yaml to known formats

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: add yaml to `orch ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: remove `orch spec dump`

as this is now a subset of `orch ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: reorder RGWSpec arguments

to match the argumens from `ServiceSpec`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: prevent ServiceSpec of wrong type

Some Python foo to make sure, we don't have an object
like `ServiceSpec('rgw')` of type `ServiceSpec`. Now we have:

>>> type(ServiceSpec('rgw')) == type(RGWSpec('rgw'))

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: tox.ini: omit cov report

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: test describe_service

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: ServiceDescription: change json representation

In order to match the new paradigm

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: ServiceDescription: Make spec a requirement

Because, a ServiceDescription is superset of a spec

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34187 from liewegas/fix-44759

osd/PeeringState: drop mimic assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: remove unused func.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #31453 from oneoneonepig/patch-2

doc: add sudo in ceph status command

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29443 from simon-rock/metatool_dev

tool: add cephfs-meta-injection to show and amend info of inode

Merge pull request #33198 from simon-rock/simon_work_get_dirfrag_bound

mds: optimize function, get_force_dirfrag_bound_set

Merge pull request #33643 from simon-rock/simon_destructor_of_session

mds : core dump if the resource is not released

Merge pull request #31668 from shenhang/deny_reconnect_on_reboot

mds: short reboot time

Merge pull request #33570 from shenhang/modifyactiverequest

mds: Using begin() and empty() to iterate the xlist

Merge pull request #33923 from systemcrash/patch-1

doc: fix spelling and grammar issues in ceph-deploy-osd.rst

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #31055 from smithfarm/wip-42357

tests: qa/tasks/ceph.py: fail if any osd role doesn't get a device

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33366 from jmolmo/rook_blinking_lights

mgr/rook: Blinking lights

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: Add a tox env that can test importing files

While switching to python3, we need to make sure that we can import
the qa/tasks (and others, but this starts with qa/tasks) on a python3
environment.
To test this, we need to install teuthology into the test
venv. Currently, teuthology is not py3 ready so this will fail.

To test the current state of the qa/tasks directory with the ongoing
work for python3 within teuthology, you can now do:

TEUTHOLOGY_GIT=git+https://github.com/kshtsk/teuthology.git@wip-py3-compat \
    tox -eimport-tasks

This is using the current branch from
https://github.com/ceph/teuthology/pull/1362 which does the work to
make teuthology python3 ready.

NOTE: This tox env is not activated by default. It's currently failing
but it provides a way to iterate over the failures and once we have
them fixed, we can activate the tox env during make-check.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

os/bluestore: remove incorrectly comments.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mgr/rook: Blinking lights

Blinking lights implementation

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

qa/tasks/mgr: partial revert of 'import with full path'

this change partially reverts #34139

using relative import helps with readability and ease the pain to write
down the full parent module name

in #34139, all relative imports were replaced with full path, because we
were using following code to verify if the code is python3 compatible:

```
  mod_spec = importlib.util.spec_from_file_location(mod_name, path)
  mod = importlib.util.module_from_spec(mod_spec)
  mod_spec.loader.exec_module(mod)
```

but this does not work with submodule which can import using relative
import without specifying the name of the package and its parent module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34184 from cbodley/wip-qa-rgw-test-multi-fix

qa/rgw: multisite test fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33581 into master

* refs/pull/33581/head:
mgr/cephadm: add watch_url to NFS bootstrap config
mgr/cephadm: check for an existing NFS config object
mgr/cephadm: combine `get_daemon_name` and `get_rados_user` funcs
mon: allow `auth caps` for `mgr` profile
mgr/cephadm: add deployment example to NFS docs
mgr/cephadm: add utils.py
mgr/cephadm: create/update keyring during nfs config
mgr/cephadm: pass NFSServiceSpec instead of pool/namespace args
mgr/cephadm: add nfs apply test
mgr/cephadm: fix nfs redeploy/reconfig
mgr/cephadm: ensure nfs uses the correct entity
mgr/cephadm: fix service name for nfs type
mgr/cephadm: apply nfs service spec
mgr/cephadm: don't create a unique nfs daemon name
mgr/cephadm: update docs
mgr/cephadm: add minimal caps for nfs client keyring
mgr/cephadm: add logging to NFSGanesha class
mgr/cephadm: create empty nfs rados config obj
mgr/cephadm: add nfs tests
mgr/cephadm: add a minimal ganesha.conf
mgr/cephadm: deploy nfs-ganesha service
mgr/cephadm: split out `config-and-keyrings` json
mgr/orchestrator: rearrange interface to match impl. order

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34181 into master

* refs/pull/34181/head:
mgr/cephadm: bind grafana to all interfaces

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #34182 into master

* refs/pull/34182/head:
mgr/cephadm: fix 'orch ps --refresh'

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33910 from xiexingguo/wip-44532-2

osd/PeeringState: do not trim pg log past last_update_ondisk

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>
Reviewed-by: Samuel Just <sjust@redhat.com>

doc: do not encourage readers to run multiple clusters on same HW

This material was first added on December 22, 2012 by
46d0334456c14da2bc8e012f121c545b1b33b95f

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/cephadm: add watch_url to NFS bootstrap config

watch for changes to the per-daemon RADOS conf file

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: check for an existing NFS config object

assume an existing NFS config if the named rados object already exists

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: combine `get_daemon_name` and `get_rados_user` funcs

Signed-off-by: Michael Fritch <mfritch@suse.com>

mon: allow `auth caps` for `mgr` profile

cephadm orchestrator needs to update the caps during reconfig/redeploy

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add deployment example to NFS docs

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add utils.py

move `name_to_config_section` and `assert_valid_host` into utils.py

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: create/update keyring during nfs config

the keyring might already exist from a prior config/reconfig/redeploy.

attempt to get_or_create the keyring first and than update the keyring
caps afterward

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: pass NFSServiceSpec instead of pool/namespace args

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: add nfs apply test

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix nfs redeploy/reconfig

search for the `NFSServiceSpec` instead of passing empty
`cephadm_args` during `_create_daemon`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: ensure nfs uses the correct entity

mon entity needs to start with `client.`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix service name for nfs type

the `nfs` service was incorrectly reported as `mds`

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: apply nfs service spec

orch apply nfs <svc_id> [<placement>]

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: don't create a unique nfs daemon name

only one nfs daemon will be deployed per host

Signed-off-by: Michael Fritch <mfritch@suse.com>