test/crimson: increase variance of stdev to 0.20

per Mark Nelson,

> yeah, 5% variation is way too low
> Sometimes we can stay within 5%, but especially if we are pushing the
> system hard we can see closer to 10-20% sometimes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34112 from trociny/wip-44610

pybind/rbd: ensure image is open before permitting operations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34019 from changchengx/blockguard

librbd: refine BlockGuard debug output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #34150 into master

* refs/pull/34150/head:
cephadm: rm-cluster clean up /etc/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34253 from cbodley/wip-qa-rgw-too-many-shards

qa/rgw: remove 1999-shard tests from rgw/verify

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #33557 from yuvalif/add_2pc_cls_queue

cls/queue: add 2-phase-commit queue implementation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33933 from cbodley/wip-rgw-doc-reshard-v2

doc/rgw: update multisite reshard design for cross-datalog-shard coordination

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #34260 into master

* refs/pull/34260/head:
Revert "Merge PR #34091 into master"

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34248 into master

* refs/pull/34248/head:
qa/tasks/cephadm: no need to explicitly reconfig
qa/tasks/cephadm: fetch final ceph.conf
qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34266 into master

* refs/pull/34266/head:
mgr/DaemonServer: fetch metadata for new daemons (e.g., mons)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34279 from votdev/issue_44805

mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Fixes: https://tracker.ceph.com/issues/44805
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34276 from badone/wip-fix-argument-ordering-docker-test

tests: Reinstate "opts" as final argument

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: Reinstate "opts" as final argument

a1e8f61cb72 broke the "opts" argument.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34252 from JackSlateur/doc-os-recommendations2

doc/start/os-recommendations.rst: recommend bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34267 from JackSlateur/doc-man-monmaptool

doc/man/8/monmaptool.rst: update synopsis etc with the latest options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man/8/monmaptool.rst: update the page with the latest options

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge pull request #34224 from markhpc/wip-bluefs-disable-buffered-io

common/options: Disable bluefs_buffered_io by default again.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33052 from ifed01/wip-ifed-fix-unused2

os/bluestore: fix 'unused' calculation

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34203 from majianpeng/bluefs-remove-unused-func

os/bluestore: remove unused func.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34061 into master

* refs/pull/34061/head:
mgr/orch: Add `ceph orch ls --export`
mgr/dashboard: adapt to new `ServiceDescription.to_json()`
python-common: add `service_name` to `ServiceSpec.to_json`
python-common: make ServiceSpec and ServiceDescription compatible
src/ceph.in: add yaml to known formats
mgr/orch: add yaml to `orch ls`
mgr/orch: remove `orch spec dump`
python-common: reorder RGWSpec arguments
python-common: prevent ServiceSpec of wrong type
pybind/mgr: tox.ini: omit cov report
mgr/cephadm: test describe_service
mgr/orch: ServiceDescription: change json representation
mgr/orch: ServiceDescription: Make spec a requirement

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34215 into master

* refs/pull/34215/head:
cephadm: Fix trucated output of `mgr dump`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34268 from poyea/docs-options-1

doc: Fix inconsistency in logging settings

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix inconsistency in logging settings

This patch fixes inconsistency in logging settings with options, namely `log_flush_on_exit` and `log_to_stderr`. This patch also adds `log_to_file` to the section.

Signed-off-by: John Law <johnlaw.po@gmail.com>

mgr/DaemonServer: fetch metadata for new daemons (e.g., mons)

We fetch metadata for mon/osd/mds daemons on startup.  If a new one comes
along *after* we start up, we need to fetch it on demand.  Otherwise,
notably, we will ignore any new mons added to the cluster:

- mon.a starts
- mgr starts, fetchs a's metadata
- mon.b added
- mon.b tries top open mgr connection, and is rejected each time

This can lead to follow-on badness when the mon tries to proxy mgr
commands or do other things.

Fix by fetching metadata on demand, like we already do in the report
path.

Fixes: https://tracker.ceph.com/issues/44798
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34062 into master

* refs/pull/34062/head:
cephadm: update check-host() to return all problems

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33943 from xxhdx1985126/wip-crimson-alienstore-bugs

crimson: fix bugs that come up when osds go through down/up

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: Fix trucated output of `mgr dump`

In case the process has terminated, we
have to make sure to read till EOF, otherwise
we might skip some bytes.

Fixes: https://tracker.ceph.com/issues/44642
Co-authored-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: remove 1999-shard tests from rgw/verify

causing test failures because the host runs out of disk space

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/start/os-recommendations.rst: recommend bluestore

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge PR #34247 into master

* refs/pull/34247/head:
mgr/dashboard: do not fail on user creation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33279 from rishabh-d-dave/fs-move-run_shell-and-related-methods

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Merge pull request #34021 from lxbsz/deleg_XXX

test/libcephfs: retry and wait for caps when setting the RD deleg

Merge pull request #33494 from lxbsz/mount

mount.ceph: stop printing warning message for unrecognized mount options

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #33740 from lxbsz/qa_test_timeout_failed

qa/test_config_session_timeout: fix options conflicting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34024 from vshankar/wip-ignore-mgrmap-when-laggy

mds: do not defer incoming mgrmap when mds is laggy

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Revert "Merge PR #34091 into master"

This reverts commit f865f3e0a0f3a646b093b3571ea76713eca1916c, reversing
changes made to 7ef5458e26ec7c0565509a7882fa31fa064eb49d.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: no need to explicitly reconfig

cephadm does this magically now.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: fetch final ceph.conf

Otherwise we'll distribute the initial ceph.conf, which only has the
bootstrap mon in it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes

Revert part of 96220c0c0574eb5b896023e1552f528bef9e1ca5 so that we still
distribute a *final* ceph.conf and admin keyring to all nodes, right after
all of the mons are up.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: do not fail on user creation

Accordingly to other Ceph commands do not fail on user
creation with a non-zero error code if the user already
exists.
Instead succeed and return the message 'User <username>
already exists'.

Fixes: https://tracker.ceph.com/issues/44502
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

cephadm: rm-cluster clean up /etc/ceph

rm-cluster command will delete config, keyring, and pub key files from /etc/ceph

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge pull request #33992 from rhcs-dashboard/44626-fix-notifications-e2e-tests

mgr/dashboard: fix notifications E2E tests

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34209 from smithfarm/wip-doc-multi-cluster-follow-on-fix

doc: fix cephadm link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34235 from toabctl/add-sle-15-sp2

qa/distros: add SLE-15-SP2

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32749 from james58899/fix-capacity

monitoring: Fix pool capacity incorrect

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #34233 from tchaikov/wip-qa-py3

qa/tasks: change to be py3 compatible and cleanups

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

qa/distros: add SLE-15-SP2

Ceph octopus is known to run on SLE-15-SP2 so add it to
qa/distros/all.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

cmake: add import-tasks to run-tox-qa test

to ensure that qa suites are python3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: use relative import

for better readability, and to ease the pain of developer to track back
to the top level python package for referencing a submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: fix print() to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33949 from toabctl/import-tasks-with-py3

qa: Add a tox env that can test importing files

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34141 from tchaikov/wip-test-threadsafe

test/objectstore: set "threadsafe" flag for ASSERT_DEATH tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson: fix pglog read_log_and_missing issue

Currently pglog's read_log_and_missing_crimson's implementation
would cause null pointer exception, since at the time, we don't the
key to start from

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add necessary FuturizedStore APIs for data recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: make sure pg meta is valid when loading pg

currently, PGMeta instances are all local variables in method,
which would lead to daggling reference/pointer problems when using
seastar::futures.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

common/options: Disable bluefs_buffered_io by default again.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #32093 into master

* refs/pull/32093/head:
doc/start: Update hardware recommendations.

Reviewed-by: Sage Weil <sage@redhat.com>

doc/start: Update hardware recommendations.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #34218 into master

* refs/pull/34218/head:
mgr/cephadm: Add missing decorator

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Add missing decorator

Caused by a conflict between

* #33581 (770816dc9f5b440fd1fc6cc507a06cd8573f6ea1) and
* #34091 (a78f42e225bbc041324dc3e565deb207feddad26)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33873 from lxbsz/qa_vs_failed

qa/tasks/vstart_runner: fix sh to make it compatible with teth

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33882 from rosinL/wip-fix-rdma-segfault

msg/async/rdma: add return value check after qp created

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34195 from tchaikov/wip-tasks-mgr-submodule

qa/tasks/mgr: partial revert of 'import with full path'

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34196 from majianpeng/bluestore-remove-incorrectly-comments

os/bluestore: remove incorrect comments.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34183 from neha-ojha/wip-doc-fix

doc/releases/octopus.rst: fix min_mon_release string

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: fix cephadm link

Fixes: 8fe460e7562b28e007179eeb533ef68a3e99b4c8
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #34091 into master

* refs/pull/34091/head:
mgr/cephadm: cleanup async_map_completion etc.
mgr/cephadm: replace async_completion with trivial_completion
mgr/cephadm: replace async_map_completion with a simple wrapper

Merge PR #34188 into master

* refs/pull/34188/head:
doc: do not encourage readers to run multiple clusters on same HW

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30839 from ifed01/wip-ifed-aloc_replay_improve

test/bmap_allocator_replay_test: improving allocator replay tool.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/orch: Add `ceph orch ls --export`

* defaults to `--format yaml`
* don't include `status`

```
$  ceph orch ls --export
placement:
  host_pattern: '*'
service_name: crash
service_type: crash
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: adapt to new `ServiceDescription.to_json()`

`ServiceDescription.to_json()` is now based on `ServiceSpec.to_json()`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: add `service_name` to `ServiceSpec.to_json`

To make it compatible to `Orchestrator.describe_service`.
Otherwise we have the awkward situation that users need to
pass `service_name` to `describe_service`, but `service_id` to apply

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: make ServiceSpec and ServiceDescription compatible

`ServiceSpec.from_json(ServiceDescription().to_json())` now works

as does

`ceph orch ls  --format yaml | ceph orch apply -i -`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

src/ceph.in: add yaml to known formats

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: add yaml to `orch ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: remove `orch spec dump`

as this is now a subset of `orch ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: reorder RGWSpec arguments

to match the argumens from `ServiceSpec`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: prevent ServiceSpec of wrong type

Some Python foo to make sure, we don't have an object
like `ServiceSpec('rgw')` of type `ServiceSpec`. Now we have:

>>> type(ServiceSpec('rgw')) == type(RGWSpec('rgw'))

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: tox.ini: omit cov report

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: test describe_service

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: ServiceDescription: change json representation

In order to match the new paradigm

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: ServiceDescription: Make spec a requirement

Because, a ServiceDescription is superset of a spec

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34187 from liewegas/fix-44759

osd/PeeringState: drop mimic assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: remove unused func.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #31453 from oneoneonepig/patch-2

doc: add sudo in ceph status command

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29443 from simon-rock/metatool_dev

tool: add cephfs-meta-injection to show and amend info of inode

Merge pull request #33198 from simon-rock/simon_work_get_dirfrag_bound

mds: optimize function, get_force_dirfrag_bound_set

Merge pull request #33643 from simon-rock/simon_destructor_of_session

mds : core dump if the resource is not released

Merge pull request #31668 from shenhang/deny_reconnect_on_reboot

mds: short reboot time

Merge pull request #33570 from shenhang/modifyactiverequest

mds: Using begin() and empty() to iterate the xlist

Merge pull request #33923 from systemcrash/patch-1

doc: fix spelling and grammar issues in ceph-deploy-osd.rst

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #31055 from smithfarm/wip-42357

tests: qa/tasks/ceph.py: fail if any osd role doesn't get a device

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33366 from jmolmo/rook_blinking_lights

mgr/rook: Blinking lights

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa: Add a tox env that can test importing files

While switching to python3, we need to make sure that we can import
the qa/tasks (and others, but this starts with qa/tasks) on a python3
environment.
To test this, we need to install teuthology into the test
venv. Currently, teuthology is not py3 ready so this will fail.

To test the current state of the qa/tasks directory with the ongoing
work for python3 within teuthology, you can now do:

TEUTHOLOGY_GIT=git+https://github.com/kshtsk/teuthology.git@wip-py3-compat \
    tox -eimport-tasks

This is using the current branch from
https://github.com/ceph/teuthology/pull/1362 which does the work to
make teuthology python3 ready.

NOTE: This tox env is not activated by default. It's currently failing
but it provides a way to iterate over the failures and once we have
them fixed, we can activate the tox env during make-check.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

os/bluestore: remove incorrectly comments.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mgr/rook: Blinking lights

Blinking lights implementation

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

qa/tasks/mgr: partial revert of 'import with full path'

this change partially reverts #34139

using relative import helps with readability and ease the pain to write
down the full parent module name

in #34139, all relative imports were replaced with full path, because we
were using following code to verify if the code is python3 compatible:

```
  mod_spec = importlib.util.spec_from_file_location(mod_name, path)
  mod = importlib.util.module_from_spec(mod_spec)
  mod_spec.loader.exec_module(mod)
```

but this does not work with submodule which can import using relative
import without specifying the name of the package and its parent module.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34184 from cbodley/wip-qa-rgw-test-multi-fix

qa/rgw: multisite test fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33581 into master

* refs/pull/33581/head:
mgr/cephadm: add watch_url to NFS bootstrap config
mgr/cephadm: check for an existing NFS config object
mgr/cephadm: combine `get_daemon_name` and `get_rados_user` funcs
mon: allow `auth caps` for `mgr` profile
mgr/cephadm: add deployment example to NFS docs
mgr/cephadm: add utils.py
mgr/cephadm: create/update keyring during nfs config
mgr/cephadm: pass NFSServiceSpec instead of pool/namespace args
mgr/cephadm: add nfs apply test
mgr/cephadm: fix nfs redeploy/reconfig
mgr/cephadm: ensure nfs uses the correct entity
mgr/cephadm: fix service name for nfs type
mgr/cephadm: apply nfs service spec
mgr/cephadm: don't create a unique nfs daemon name
mgr/cephadm: update docs
mgr/cephadm: add minimal caps for nfs client keyring
mgr/cephadm: add logging to NFSGanesha class
mgr/cephadm: create empty nfs rados config obj
mgr/cephadm: add nfs tests
mgr/cephadm: add a minimal ganesha.conf
mgr/cephadm: deploy nfs-ganesha service
mgr/cephadm: split out `config-and-keyrings` json
mgr/orchestrator: rearrange interface to match impl. order

Reviewed-by: Sebastian Wagner <swagner@suse.com>