Revert "qa/cephfs: move run_shell and related methods to mount.py"

This reverts commit 751d432a8696c7fdceb9d9707c2b3331878342ea.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Revert "vstart_runner.py: add methods for negative testing a cmd"

This reverts commit 48e0e1d7030bc4167bd3151696106a51f143bdba.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #34096 from smanjara/wip-dynamic-resharding

rgw: groundwork for supporting dynamic resharding in multisite environment

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34345 from tchaikov/wip-crimson-less-chatty

crimson: do not warn() under expected circumstances

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: do not use "warn()" when handling non-error

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: do not print error log at seeing unknown connection

it happens if a client or an peer osd drops the connection, so it's not
an error and hence we should not print this error message using
"error()".

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34341 from guits/update_cv_deploy

ceph-volume: update functional testing deploy.yml playbook

Merge pull request #34331 from zealoussnow/wip-fix-clone-error

test/common/unittest_blkdev: delete unused test file

Reviewed-by: Kefu Chai <kchai@redhat.com>

unittest_blkdev: delete unused test file

Signed-off-by: Leo Zhang <nguzcf@gmail.com>

ceph-volume: update functional testing deploy.yml playbook

This commit adds a call to `ceph-facts` role in the first play of this
playbook. This is needed so `ceph-validate` won't fail because of
following error:

```
fatal: [osd0]: FAILED! => {}

MSG:

'osd_pool_default_size' is undefined
```

`osd_pool_default_size` is set in ceph-facts.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #34232 from bk201/wip-44743

qa/tasks/mgr/dashboard/test_rbd: wait longer when purging

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

modified get_target_shard_id() to take bucket_index_normal_layout directly

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #34291 from dillaman/wip-44814

librbd: fix client backwards compatibility issues

Merge pull request #34295 from sebastian-philipp/raise-for-invalid-dg

python-common: raise on empty drive selections

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34026 from votdev/issue_44539_shorten_container_id

mgr/dashboard: shorten `Container ID` and `Container image ID` in Services page

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tasks/mgr/dashboard/test_rbd: wait longer when purging

Fixes: https://tracker.ceph.com/issues/44743
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
Co-authored-by: Kefu Chai <kefu@redhat.com>
Co-authored-by: Volker Theile <vtheile@suse.com>

Removed redundant encoding of layout.* and adjusted the corresponding decoding

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Moved ostream operator overloading to BucketIndexType() to rgw_bucket_layout header file

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

mgr/dashboard: shorten `Container ID` and `Container image ID` in Services page

Fixes: https://tracker.ceph.com/issues/44539
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34259 from rzarzynski/wip-crimson-fix-bad-methods

crimson/osd: fix method's flag handling on CEPH_OSD_OP_CALL.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34305 from tchaikov/wip-crimson-stddev

test/crimson: increase variance of stdev to 0.20

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #34167 from javacruft/32bit-fixes-march

mds: resolve type mismatches on 32 bit archs

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

test/crimson: increase variance of stdev to 0.20

per Mark Nelson,

> yeah, 5% variation is way too low
> Sometimes we can stay within 5%, but especially if we are pushing the
> system hard we can see closer to 10-20% sometimes.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34112 from trociny/wip-44610

pybind/rbd: ensure image is open before permitting operations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34019 from changchengx/blockguard

librbd: refine BlockGuard debug output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #34150 into master

* refs/pull/34150/head:
cephadm: rm-cluster clean up /etc/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

python-common: raise on emtpy drive groups

adds a test for https://tracker.ceph.com/issues/44758

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: re-add support for nautilus clients talking to jewel clusters

We want to support N - 3 client backward compatibility (special case
to support Jewel since it was a LTS release). The "get_snapshot_timestamp"
cls method does not exist in Jewel clusters so librbd should fallback
to excluding the op if it fails.

Note that this N - 3 also needs to apply for downstream releases as well,
which implies we still need Jewel for the time being.

Fixes: http://tracker.ceph.com/issues/39450
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c644121820b83c97e68f9896393a45cd34787672)

Conflicts:
src/test/librbd/image/test_mock_RefreshRequest.cc: tweaked to support pool configs

librbd: ignore -EOPNOTSUPP errors when retrieving image group membership

The Luminous release did not support adding images to a group (it only
included the bare-minimum support for creating groups). Commit f76df32666b
incorrectly dropped support for ignoring this possible failure. This
prevents Nautilus-release clients from opening images contained within
a Luminous-release cluster.

Fixes: http://tracker.ceph.com/issues/38834
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 6f29dc69a0db3b6d982c95ab9d3b9b00a7029e37)

Merge pull request #34253 from cbodley/wip-qa-rgw-too-many-shards

qa/rgw: remove 1999-shard tests from rgw/verify

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #33557 from yuvalif/add_2pc_cls_queue

cls/queue: add 2-phase-commit queue implementation

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33933 from cbodley/wip-rgw-doc-reshard-v2

doc/rgw: update multisite reshard design for cross-datalog-shard coordination

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #34260 into master

* refs/pull/34260/head:
Revert "Merge PR #34091 into master"

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34248 into master

* refs/pull/34248/head:
qa/tasks/cephadm: no need to explicitly reconfig
qa/tasks/cephadm: fetch final ceph.conf
qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #34266 into master

* refs/pull/34266/head:
mgr/DaemonServer: fetch metadata for new daemons (e.g., mons)

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: replace RGWBucketInfo's sharding-related fields with BucketLayout

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: add BucketLayout to RGWBucketInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use qualified encode/decode in putobj processor

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34279 from votdev/issue_44805

mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: 'Prometheus / All Alerts' page shows progress bar

Fixes: https://tracker.ceph.com/issues/44805
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #34276 from badone/wip-fix-argument-ordering-docker-test

tests: Reinstate "opts" as final argument

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: Reinstate "opts" as final argument

a1e8f61cb72 broke the "opts" argument.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34252 from JackSlateur/doc-os-recommendations2

doc/start/os-recommendations.rst: recommend bluestore

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34267 from JackSlateur/doc-man-monmaptool

doc/man/8/monmaptool.rst: update synopsis etc with the latest options

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man/8/monmaptool.rst: update the page with the latest options

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge pull request #34224 from markhpc/wip-bluefs-disable-buffered-io

common/options: Disable bluefs_buffered_io by default again.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33052 from ifed01/wip-ifed-fix-unused2

os/bluestore: fix 'unused' calculation

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34203 from majianpeng/bluefs-remove-unused-func

os/bluestore: remove unused func.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #34061 into master

* refs/pull/34061/head:
mgr/orch: Add `ceph orch ls --export`
mgr/dashboard: adapt to new `ServiceDescription.to_json()`
python-common: add `service_name` to `ServiceSpec.to_json`
python-common: make ServiceSpec and ServiceDescription compatible
src/ceph.in: add yaml to known formats
mgr/orch: add yaml to `orch ls`
mgr/orch: remove `orch spec dump`
python-common: reorder RGWSpec arguments
python-common: prevent ServiceSpec of wrong type
pybind/mgr: tox.ini: omit cov report
mgr/cephadm: test describe_service
mgr/orch: ServiceDescription: change json representation
mgr/orch: ServiceDescription: Make spec a requirement

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #34215 into master

* refs/pull/34215/head:
cephadm: Fix trucated output of `mgr dump`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34268 from poyea/docs-options-1

doc: Fix inconsistency in logging settings

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Fix inconsistency in logging settings

This patch fixes inconsistency in logging settings with options, namely `log_flush_on_exit` and `log_to_stderr`. This patch also adds `log_to_file` to the section.

Signed-off-by: John Law <johnlaw.po@gmail.com>

mgr/DaemonServer: fetch metadata for new daemons (e.g., mons)

We fetch metadata for mon/osd/mds daemons on startup.  If a new one comes
along *after* we start up, we need to fetch it on demand.  Otherwise,
notably, we will ignore any new mons added to the cluster:

- mon.a starts
- mgr starts, fetchs a's metadata
- mon.b added
- mon.b tries top open mgr connection, and is rejected each time

This can lead to follow-on badness when the mon tries to proxy mgr
commands or do other things.

Fix by fetching metadata on demand, like we already do in the report
path.

Fixes: https://tracker.ceph.com/issues/44798
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #34062 into master

* refs/pull/34062/head:
cephadm: update check-host() to return all problems

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33943 from xxhdx1985126/wip-crimson-alienstore-bugs

crimson: fix bugs that come up when osds go through down/up

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix method's flag handling on CEPH_OSD_OP_CALL.

This patch fixes a regression introduced in d8d0ab1a and
makes `ClsHello.BadMethods` green again.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cephadm: Fix trucated output of `mgr dump`

In case the process has terminated, we
have to make sure to read till EOF, otherwise
we might skip some bytes.

Fixes: https://tracker.ceph.com/issues/44642
Co-authored-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/rgw: remove 1999-shard tests from rgw/verify

causing test failures because the host runs out of disk space

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/start/os-recommendations.rst: recommend bluestore

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge PR #34247 into master

* refs/pull/34247/head:
mgr/dashboard: do not fail on user creation

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33279 from rishabh-d-dave/fs-move-run_shell-and-related-methods

qa/cephfs: move run_shell to mount.py and add methods for negative testing

Merge pull request #34021 from lxbsz/deleg_XXX

test/libcephfs: retry and wait for caps when setting the RD deleg

Merge pull request #33494 from lxbsz/mount

mount.ceph: stop printing warning message for unrecognized mount options

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #33740 from lxbsz/qa_test_timeout_failed

qa/test_config_session_timeout: fix options conflicting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34024 from vshankar/wip-ignore-mgrmap-when-laggy

mds: do not defer incoming mgrmap when mds is laggy

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Revert "Merge PR #34091 into master"

This reverts commit f865f3e0a0f3a646b093b3571ea76713eca1916c, reversing
changes made to 7ef5458e26ec7c0565509a7882fa31fa064eb49d.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: no need to explicitly reconfig

cephadm does this magically now.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: fetch final ceph.conf

Otherwise we'll distribute the initial ceph.conf, which only has the
bootstrap mon in it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: distribute ceph.conf and admin keyring to all nodes

Revert part of 96220c0c0574eb5b896023e1552f528bef9e1ca5 so that we still
distribute a *final* ceph.conf and admin keyring to all nodes, right after
all of the mons are up.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: do not fail on user creation

Accordingly to other Ceph commands do not fail on user
creation with a non-zero error code if the user already
exists.
Instead succeed and return the message 'User <username>
already exists'.

Fixes: https://tracker.ceph.com/issues/44502
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

cephadm: rm-cluster clean up /etc/ceph

rm-cluster command will delete config, keyring, and pub key files from /etc/ceph

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

Merge pull request #33992 from rhcs-dashboard/44626-fix-notifications-e2e-tests

mgr/dashboard: fix notifications E2E tests

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34209 from smithfarm/wip-doc-multi-cluster-follow-on-fix

doc: fix cephadm link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #34235 from toabctl/add-sle-15-sp2

qa/distros: add SLE-15-SP2

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #32749 from james58899/fix-capacity

monitoring: Fix pool capacity incorrect

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #34233 from tchaikov/wip-qa-py3

qa/tasks: change to be py3 compatible and cleanups

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

qa/distros: add SLE-15-SP2

Ceph octopus is known to run on SLE-15-SP2 so add it to
qa/distros/all.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

cmake: add import-tasks to run-tox-qa test

to ensure that qa suites are python3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr: use relative import

for better readability, and to ease the pain of developer to track back
to the top level python package for referencing a submodule

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephfs: fix print() to be py3 compatible

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33949 from toabctl/import-tasks-with-py3

qa: Add a tox env that can test importing files

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34141 from tchaikov/wip-test-threadsafe

test/objectstore: set "threadsafe" flag for ASSERT_DEATH tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson: fix pglog read_log_and_missing issue

Currently pglog's read_log_and_missing_crimson's implementation
would cause null pointer exception, since at the time, we don't the
key to start from

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: add necessary FuturizedStore APIs for data recovery

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: make sure pg meta is valid when loading pg

currently, PGMeta instances are all local variables in method,
which would lead to daggling reference/pointer problems when using
seastar::futures.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

common/options: Disable bluefs_buffered_io by default again.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #32093 into master

* refs/pull/32093/head:
doc/start: Update hardware recommendations.

Reviewed-by: Sage Weil <sage@redhat.com>

doc/start: Update hardware recommendations.

Signed-off-by: Mark Nelson <mnelson@redhat.com>

Merge PR #34218 into master

* refs/pull/34218/head:
mgr/cephadm: Add missing decorator

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: Add missing decorator

Caused by a conflict between

* #33581 (770816dc9f5b440fd1fc6cc507a06cd8573f6ea1) and
* #34091 (a78f42e225bbc041324dc3e565deb207feddad26)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mds: resolve type mismatches on 32 bit archs

Ensure that size_t is cast to uint64_t on 32 bit architectures
to avoid compilation failures due mismatches between uint64_t
and size_t.

Signed-off-by: James Page <james.page@ubuntu.com>

Merge pull request #33873 from lxbsz/qa_vs_failed

qa/tasks/vstart_runner: fix sh to make it compatible with teth

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33882 from rosinL/wip-fix-rdma-segfault

msg/async/rdma: add return value check after qp created

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34195 from tchaikov/wip-tasks-mgr-submodule

qa/tasks/mgr: partial revert of 'import with full path'

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34196 from majianpeng/bluestore-remove-incorrectly-comments

os/bluestore: remove incorrect comments.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34183 from neha-ojha/wip-doc-fix

doc/releases/octopus.rst: fix min_mon_release string

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: fix cephadm link

Fixes: 8fe460e7562b28e007179eeb533ef68a3e99b4c8
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #34091 into master

* refs/pull/34091/head:
mgr/cephadm: cleanup async_map_completion etc.
mgr/cephadm: replace async_completion with trivial_completion
mgr/cephadm: replace async_map_completion with a simple wrapper

Merge PR #34188 into master

* refs/pull/34188/head:
doc: do not encourage readers to run multiple clusters on same HW

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30839 from ifed01/wip-ifed-aloc_replay_improve

test/bmap_allocator_replay_test: improving allocator replay tool.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>