doc/mgr/orchestrator: Update `Placement by pattern matching`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm/drivegroup: Don't introduce DriveGroups

We already have a name for it: a Service Specification of
type `osd`. We don't need to introduce a new name for it.

Well, they are "DriveGroups", but users don't need to know it.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm/drivegroups: Update to new yaml

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/mgr/orchestrator: use yaml syntax highlighting

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/mgr/orchestrator: Add "Service Specification"

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34259 from rzarzynski/wip-crimson-fix-bad-methods

crimson/osd: fix method's flag handling on CEPH_OSD_OP_CALL.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34305 from tchaikov/wip-crimson-stddev

test/crimson: increase variance of stdev to 0.20

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34167 from javacruft/32bit-fixes-march

mds: resolve type mismatches on 32 bit archs

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test/crimson: increase variance of stdev to 0.20

per Mark Nelson,

> yeah, 5% variation is way too low
> Sometimes we can stay within 5%, but especially if we are pushing the
> system hard we can see closer to 10-20% sometimes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34112 from trociny/wip-44610

pybind/rbd: ensure image is open before permitting operations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34019 from changchengx/blockguard

librbd: refine BlockGuard debug output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #34150 into master

* refs/pull/34150/head:
cephadm: rm-cluster clean up /etc/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34253 from cbodley/wip-qa-rgw-too-many-shards

qa/rgw: remove 1999-shard tests from rgw/verify

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #33557 from yuvalif/add_2pc_cls_queue

cls/queue: add 2-phase-commit queue implementation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33933 from cbodley/wip-rgw-doc-reshard-v2

doc/rgw: update multisite reshard design for cross-datalog-shard coordination

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #34260 into master

* refs/pull/34260/head:
Revert "Merge PR #34091 into master"

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34248 into master

* refs/pull/34248/head:
qa/tasks/cephadm: no need to explicitly reconfig
qa/tasks/cephadm: fetch final ceph.conf
qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34266 into master

* refs/pull/34266/head:
mgr/DaemonServer: fetch metadata for new daemons (e.g., mons)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34279 from votdev/issue_44805

mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Fixes: https://tracker.ceph.com/issues/44805
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34276 from badone/wip-fix-argument-ordering-docker-test

tests: Reinstate "opts" as final argument

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: Reinstate "opts" as final argument

a1e8f61cb72 broke the "opts" argument.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34252 from JackSlateur/doc-os-recommendations2

doc/start/os-recommendations.rst: recommend bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34267 from JackSlateur/doc-man-monmaptool

doc/man/8/monmaptool.rst: update synopsis etc with the latest options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man/8/monmaptool.rst: update the page with the latest options

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge pull request #34224 from markhpc/wip-bluefs-disable-buffered-io

common/options: Disable bluefs_buffered_io by default again.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33052 from ifed01/wip-ifed-fix-unused2

os/bluestore: fix 'unused' calculation

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34203 from majianpeng/bluefs-remove-unused-func

os/bluestore: remove unused func.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34061 into master

* refs/pull/34061/head:
mgr/orch: Add `ceph orch ls --export`
mgr/dashboard: adapt to new `ServiceDescription.to_json()`
python-common: add `service_name` to `ServiceSpec.to_json`
python-common: make ServiceSpec and ServiceDescription compatible
src/ceph.in: add yaml to known formats
mgr/orch: add yaml to `orch ls`
mgr/orch: remove `orch spec dump`
python-common: reorder RGWSpec arguments
python-common: prevent ServiceSpec of wrong type
pybind/mgr: tox.ini: omit cov report
mgr/cephadm: test describe_service
mgr/orch: ServiceDescription: change json representation
mgr/orch: ServiceDescription: Make spec a requirement

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34215 into master

* refs/pull/34215/head:
cephadm: Fix trucated output of `mgr dump`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34268 from poyea/docs-options-1

doc: Fix inconsistency in logging settings

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix inconsistency in logging settings

This patch fixes inconsistency in logging settings with options, namely `log_flush_on_exit` and `log_to_stderr`. This patch also adds `log_to_file` to the section.

Signed-off-by: John Law <johnlaw.po@gmail.com>

mgr/DaemonServer: fetch metadata for new daemons (e.g., mons)

We fetch metadata for mon/osd/mds daemons on startup.  If a new one comes
along *after* we start up, we need to fetch it on demand.  Otherwise,
notably, we will ignore any new mons added to the cluster:

- mon.a starts
- mgr starts, fetchs a's metadata
- mon.b added
- mon.b tries top open mgr connection, and is rejected each time

This can lead to follow-on badness when the mon tries to proxy mgr
commands or do other things.

Fix by fetching metadata on demand, like we already do in the report
path.

Fixes: https://tracker.ceph.com/issues/44798
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34062 into master

* refs/pull/34062/head:
cephadm: update check-host() to return all problems

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33943 from xxhdx1985126/wip-crimson-alienstore-bugs

crimson: fix bugs that come up when osds go through down/up

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix method's flag handling on CEPH_OSD_OP_CALL.

This patch fixes a regression introduced in d8d0ab1a and
makes `ClsHello.BadMethods` green again.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cephadm: Fix trucated output of `mgr dump`

In case the process has terminated, we
have to make sure to read till EOF, otherwise
we might skip some bytes.

Fixes: https://tracker.ceph.com/issues/44642
Co-authored-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: remove 1999-shard tests from rgw/verify

causing test failures because the host runs out of disk space

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/start/os-recommendations.rst: recommend bluestore

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge PR #34247 into master

* refs/pull/34247/head:
mgr/dashboard: do not fail on user creation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33279 from rishabh-d-dave/fs-move-run_shell-and-related-methods

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Merge pull request #34021 from lxbsz/deleg_XXX

test/libcephfs: retry and wait for caps when setting the RD deleg

Merge pull request #33494 from lxbsz/mount

mount.ceph: stop printing warning message for unrecognized mount options

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #33740 from lxbsz/qa_test_timeout_failed

qa/test_config_session_timeout: fix options conflicting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34024 from vshankar/wip-ignore-mgrmap-when-laggy

mds: do not defer incoming mgrmap when mds is laggy

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Revert "Merge PR #34091 into master"

This reverts commit f865f3e0a0f3a646b093b3571ea76713eca1916c, reversing
changes made to 7ef5458e26ec7c0565509a7882fa31fa064eb49d.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: no need to explicitly reconfig

cephadm does this magically now.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: fetch final ceph.conf

Otherwise we'll distribute the initial ceph.conf, which only has the
bootstrap mon in it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes

Revert part of 96220c0c0574eb5b896023e1552f528bef9e1ca5 so that we still
distribute a *final* ceph.conf and admin keyring to all nodes, right after
all of the mons are up.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: do not fail on user creation

Accordingly to other Ceph commands do not fail on user
creation with a non-zero error code if the user already
exists.
Instead succeed and return the message 'User <username>
already exists'.

Fixes: https://tracker.ceph.com/issues/44502
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

cephadm: rm-cluster clean up /etc/ceph

rm-cluster command will delete config, keyring, and pub key files from /etc/ceph

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge pull request #33992 from rhcs-dashboard/44626-fix-notifications-e2e-tests

mgr/dashboard: fix notifications E2E tests

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34209 from smithfarm/wip-doc-multi-cluster-follow-on-fix

doc: fix cephadm link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34235 from toabctl/add-sle-15-sp2

qa/distros: add SLE-15-SP2

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32749 from james58899/fix-capacity

monitoring: Fix pool capacity incorrect

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #34233 from tchaikov/wip-qa-py3

qa/tasks: change to be py3 compatible and cleanups

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

qa/distros: add SLE-15-SP2

Ceph octopus is known to run on SLE-15-SP2 so add it to
qa/distros/all.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

cmake: add import-tasks to run-tox-qa test

to ensure that qa suites are python3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: use relative import

for better readability, and to ease the pain of developer to track back
to the top level python package for referencing a submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: fix print() to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33949 from toabctl/import-tasks-with-py3

qa: Add a tox env that can test importing files

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34141 from tchaikov/wip-test-threadsafe

test/objectstore: set "threadsafe" flag for ASSERT_DEATH tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson: fix pglog read_log_and_missing issue

Currently pglog's read_log_and_missing_crimson's implementation
would cause null pointer exception, since at the time, we don't the
key to start from

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add necessary FuturizedStore APIs for data recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: make sure pg meta is valid when loading pg

currently, PGMeta instances are all local variables in method,
which would lead to daggling reference/pointer problems when using
seastar::futures.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

common/options: Disable bluefs_buffered_io by default again.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #32093 into master

* refs/pull/32093/head:
doc/start: Update hardware recommendations.

Reviewed-by: Sage Weil <sage@redhat.com>

doc/start: Update hardware recommendations.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #34218 into master

* refs/pull/34218/head:
mgr/cephadm: Add missing decorator

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Add missing decorator

Caused by a conflict between

* #33581 (770816dc9f5b440fd1fc6cc507a06cd8573f6ea1) and
* #34091 (a78f42e225bbc041324dc3e565deb207feddad26)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mds: resolve type mismatches on 32 bit archs

Ensure that size_t is cast to uint64_t on 32 bit architectures
to avoid compilation failures due mismatches between uint64_t
and size_t.

Signed-off-by: James Page <james.page@ubuntu.com>

Merge pull request #33873 from lxbsz/qa_vs_failed

qa/tasks/vstart_runner: fix sh to make it compatible with teth

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33882 from rosinL/wip-fix-rdma-segfault

msg/async/rdma: add return value check after qp created

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34195 from tchaikov/wip-tasks-mgr-submodule

qa/tasks/mgr: partial revert of 'import with full path'

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34196 from majianpeng/bluestore-remove-incorrectly-comments

os/bluestore: remove incorrect comments.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34183 from neha-ojha/wip-doc-fix

doc/releases/octopus.rst: fix min_mon_release string

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: fix cephadm link

Fixes: 8fe460e7562b28e007179eeb533ef68a3e99b4c8
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #34091 into master

* refs/pull/34091/head:
mgr/cephadm: cleanup async_map_completion etc.
mgr/cephadm: replace async_completion with trivial_completion
mgr/cephadm: replace async_map_completion with a simple wrapper

Merge PR #34188 into master

* refs/pull/34188/head:
doc: do not encourage readers to run multiple clusters on same HW

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30839 from ifed01/wip-ifed-aloc_replay_improve

test/bmap_allocator_replay_test: improving allocator replay tool.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/orch: Add `ceph orch ls --export`

* defaults to `--format yaml`
* don't include `status`

```
$  ceph orch ls --export
placement:
  host_pattern: '*'
service_name: crash
service_type: crash
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: adapt to new `ServiceDescription.to_json()`

`ServiceDescription.to_json()` is now based on `ServiceSpec.to_json()`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: add `service_name` to `ServiceSpec.to_json`

To make it compatible to `Orchestrator.describe_service`.
Otherwise we have the awkward situation that users need to
pass `service_name` to `describe_service`, but `service_id` to apply

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: make ServiceSpec and ServiceDescription compatible

`ServiceSpec.from_json(ServiceDescription().to_json())` now works

as does

`ceph orch ls  --format yaml | ceph orch apply -i -`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

src/ceph.in: add yaml to known formats

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: add yaml to `orch ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: remove `orch spec dump`

as this is now a subset of `orch ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: reorder RGWSpec arguments

to match the argumens from `ServiceSpec`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

python-common: prevent ServiceSpec of wrong type

Some Python foo to make sure, we don't have an object
like `ServiceSpec('rgw')` of type `ServiceSpec`. Now we have:

>>> type(ServiceSpec('rgw')) == type(RGWSpec('rgw'))

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: tox.ini: omit cov report

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: test describe_service

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: ServiceDescription: change json representation

In order to match the new paradigm

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orch: ServiceDescription: Make spec a requirement

Because, a ServiceDescription is superset of a spec

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34187 from liewegas/fix-44759

osd/PeeringState: drop mimic assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: remove unused func.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #31453 from oneoneonepig/patch-2

doc: add sudo in ceph status command

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29443 from simon-rock/metatool_dev

tool: add cephfs-meta-injection to show and amend info of inode

Merge pull request #33198 from simon-rock/simon_work_get_dirfrag_bound

mds: optimize function, get_force_dirfrag_bound_set

Merge pull request #33643 from simon-rock/simon_destructor_of_session

mds : core dump if the resource is not released

Merge pull request #31668 from shenhang/deny_reconnect_on_reboot

mds: short reboot time