Merge pull request #37506 from rhcs-dashboard/fix-47478-master

mgr/dashboard: Improvements in cluster > OSDs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37649 from rhcs-dashboard/add-header-tooltip

mgr/dashboard: add description for CRUSH Ruleset

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37981 from tchaikov/wip-make-dist

make-dist: add liburing to dist tarball

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cmake: use src/liburing for building liburing if it exists

so we can build with liburing enabled if the dist tarball contains
liburing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: use a loop to collect the ingredients of the dist tarball

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: add liburing to dist tarball

since liburing is enabled by default, let's included it in the dist
tarball.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: extract download_from() out

so we can reuse it

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36998 into master

* refs/pull/36998/head:
doc/mgr/volumes: Document authorize/deauthorize cli commands
qa/tasks/cephfs: test `fs subvolume authorize/deauthorize`
mgr/volumes: Allow/deny auth IDs access to FS subvolumes
mon/MonCap: allow 'profile mgr' to create/update/del auth IDs

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37618 into master

* refs/pull/37618/head:
mds: throttle cap acquisition via readdir

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37662 into master

* refs/pull/37662/head:
mon/MDSMonitor: deal with subscription after being expired

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37600 from mgfritch/cephadm-nfs-rgw

mgr/cephadm: add RGW support for NFS ganesha

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37792 from p-se/monitoring-prom-many-to-many

mgr/cephadm: many-to-many Prometheus error

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37967 from linuxbox2/wip-rgwfile-zwrite

rgw_file: fix some zipper flow for RGWLibContinuedReq

Merge pull request #37968 from changchengx/paxos

mon/PaxosService: remove unnecessary code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37970 from toabctl/fix-orch-cli-doc-link

mgr/orchestratori: Fix doc link in README.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator_cli: Fix doc link in README.rst

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge pull request #37966 from tchaikov/wip-cmake-uring

cmake: set GIT_SHALLOW and UPDATE_DISCONNECTED

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

cmake: set GIT_SHALLOW and UPDATE_DISCONNECTED

* GIT_SHALLOW=TRUE, so we don't pull the full git history,
  as we don't care about it.
* UPDATE_DISCONNECTED=TRUE, to skip the UPDATE step, this change
  somehow works around
  https://gitlab.kitware.com/cmake/cmake/-/issues/19703. otherwise
  cmake keeps building liburing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

blk/kernel/io_uring: bump liburing to v0.7

* use functions exposed by liburing instead of using syscalls
* v0.7 is the latest release at the time of writing, as liburing is under
  active development. it'd be better to use a newer release.
* also use https://git.kernel.dk/liburing instead of
  http://git.kernel.dk/liburing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw_file: fix some zipper flow for RGWLibContinuedReq

Some bits of the standard Zipper conversions were missed for
the RGWLibContinuedReq case, where the setup is encapsulated in
the request, but execution is broken up in to steps.   This
currently affects only RGWWriteRequest.

Fixes: https://tracker.ceph.com/issues/48136
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37919 from tchaikov/wip-cmake-liburing

cmake: enable LIBURING bluestore backend by default

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #37929 from BryceCao/wip-modify-error-code-of-bucket-info-API

rgw : modify error message to NoSuchBucket when bucket doesn't exist in bucket info API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37913 from rhcs-dashboard/button-disable

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37832 from tchaikov/wip-doc-slow-ops

doc: reformat health-messages.rst and revise SLOW_OPS section

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37951 from agayev/misc-zbd-changes

doc, os/bluestore: Zoned storage related cleanup.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37637 from p-se/dashboard-smart-logging

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc/mgr/volumes: Document authorize/deauthorize cli commands

Fixes: https://tracker.ceph.com/issues/40401
Signed-off-by: Kotresh HR <khiremat@redhat.com>

qa/tasks/cephfs: test `fs subvolume authorize/deauthorize`

Fixes: https://tracker.ceph.com/issues/40401
Signed-off-by: Ramana Raja <rraja@redhat.com>
Signed-off-by: Kotresh HR <khiremat@redhat.com>

mgr/volumes: Allow/deny auth IDs access to FS subvolumes

... via the `ceph fs subvolume authorize/deauthorize` command.

Fixes: https://tracker.ceph.com/issues/40401
Signed-off-by: Ramana Raja <rraja@redhat.com>
Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #37454 from votdev/custom_container_docs

doc/mgr/orchestrator: Add hints related to custom containers to the docs

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37946 from bk201/wip-47373

mgr/dashboard: disable cluster selection in NFS export editing form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mon/MonCap: allow 'profile mgr' to create/update/del auth IDs

Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #37955 from tchaikov/wip-cmake-crimson-cleanup

cmake: rename crimson tests named like foo_bar to foo-bar

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37953 from tchaikov/wip-crimson-recovery-cleanup

test/crimson: do not support variadic future

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

cmake: do not name demo-like tests with unittest-*

these tests cannot be exercised without human intervention.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake/test: remove duplicated target_link_libraries()

it is a copy-n-paste error.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename crimson tests named like foo_bar to foo-bar

for two reasons:

* less typing: no need to press "shift" for inputting "_"
* more consistent with executable names like "ceph-conf"
* simpler to grep when compiling the tests. there is chance
  we need to kill the dead jobs on a jenkins worker node
  where it happens to be compiling the tests.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37940 from yuvalif/remove_lua_submodule

rpm,deb,cmake: remove the lua submodule

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #37869 into master

* refs/pull/37869/head:
qa: for the latest kclient it will also return EIO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37713 into master

* refs/pull/37713/head:
mds: fix wrong log text and allignment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37664 into master

* refs/pull/37664/head:
cephfs: release client dentry_lease before send caps release to mds

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #37654 into master

* refs/pull/37654/head:
mds: print the METRIC_TYPE for debug info
mds: add XXPayload:print support
mds: only update the requesting metrics

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #32479 into master

* refs/pull/32479/head:
mds: add dentry lease metric payload support
client: add _dentry_valid() helper support

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/common: refactor crimson::do_until()

* define `errorator_t` instead of `futurator` to simplify the code a
  little bit.
* use seastar helper function / types when appropriate

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: do not support variadic future

also add "unittest-seastar-errorator" to unit test.

this should address the build failure introduced by
688b95798bfd88b409abce9b70bb6d2933819bef

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37950 from tchaikov/wip-crimson-recovery-cleanup

crimson: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

mon/PaxosService: remove unnecessary code

The class declaration has been included in the header file.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

mon/MDSMonitor: deal with subscription after being expired

check expiration condition before taking further action

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge PR #37899 into master

* refs/pull/37899/head:
qa: avoid using ceph.dir.entries for empty check
qa: do not test dir size equals rbytes
qa: disable check for value kclient fails to get
qa: leave one standby available to avoid warning
qa: do not check access timestamps
qa: ignore skip errors for kclient
qa: add more clients for test_volume_client
qa: accept client returning pool id for layout
qa: ignore expected warning about client missing features
qa: do not fail on skip
qa: fix mount directory
qa: rename distro link to distros
qa: accept eio for mount failure
qa: unmount volumes before removal
qa: some sessionmap tests require fuse
qa: remove obsolete test
qa: do not fail on test skip
qa: use more OSDs for workload tests
qa: use 4 clients for volumes testing
qa: relocate tasks
qa: rename workload suite
qa: relocate new tasks from fs:basic_functional
qa: remove kcephfs suite
qa: move kcephfs:mixed-clients to fs
qa: remove dead kcephfs:cephfs suite
qa: merge snap tests for kcephfs/fs
qa: add new fs:shell sub-suite
qa: remove task-less kcephfs:recovery suite
qa: add new fs:volumes sub-suite
qa: merge fs:basic_functional and kcephfs:recovery
qa: move kcephfs:thrash to fs:thrash
qa: reorganize thrash tasks
qa: move multifs/failover tests
qa: run norstats workunit for klient
qa: merge workload tests from kcephfs with fs
qa: use consistent syntax
qa: remove inline testing
qa: move log-rotate config to common yaml
qa: remove fuse default permissions config
qa: fix whitespace

Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

os/bluestore: Cleanup.  Move ZBD related functions into one place and add zoned
prefix to all of them.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

doc: Update contact email address and documentation for zoned-storage.rst.

Signed-off-by: Abutalib Aghayev <agayev@psu.edu>

mgr/dashboard: add description for CRUSH Ruleset

Fixes: https://tracker.ceph.com/issues/48067
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Adds a description for CRUSH Ruleset column in pools table in the pool creation dialogue

Merge pull request #32394 from zhangsw/fix-appendobj-invalidposition

rgw: using strict_strtoll for append position. If not, any invalid value will be decoded as zero which is not correct.

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37703 from theanalyst/rgw/sal/init-quota-fixes

rgw: op: check for bucket existance as well in init_quota

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #37160 from linuxbox2/wip-rgwlc-tagfilter

rgwlc: fix clause counting in LCFilter_S3::decode_xml()

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

rpm/deb/cmake: remove the lua submodule

instead use the official lua-dev/el packages

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #37066 from smanjara/wip-metadata-reshard

rgw: don't allow overwrites to BucketInfo's layout field when sync fetches bucket metadata

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37892 from IlsooByun/fix_bucket_sync_err

rgw: keep syncstopped flag when copying bucket shard headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

osd/PGLog: do not use unique_ptr explicitly

* do not use unique_ptr<> explicitly, use `seastar::do_with()` for
  better readability
* use seastar::do_until() instead of seastar::repeat() for better
  readability. plain boolean is simpler than
  `seastar::stop_iteration::yes`
* do not capture variables using FuturizedStoreLogReader if we could
  pass them by instant parameters.
* rename "start()" to "read()". as "read" is more specific in this
  context.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PGLog: move PGLog::read_log_and_missing_crimson() into .cc

to speed up the compilation. and for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/PGLog: drop template parameter of FuturizedStoreLogReader

there is no need for the generic type at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use seastar:do_until() when appropriate

a plain boolean is easier to understand than
`seastar::stop_iteration::yes`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark CyanStore::CyanOmapIterator methods final

this class is not intended to be inherited by classes, and these methods
are "final", so mark them "final".

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark methods pure virtual

if the default implementation does not make sense in any case, just mark
it pure virtual.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: let OmapIterator mutators return future<>

instead of returning future<int> where the int represents an errno which
is not checked by any callers, we should just return a future<>, we
could erroratorize it later on, but returning an int is not the right
way to handle this, and it is not consistent with how we handle other
errors elsewhere in crimson.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mgr/orchestrator: Add hints related to custom containers to the docs

Fixes: https://tracker.ceph.com/issues/48113
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #37867 from zdover23/wip-doc-cephadm-install-oct-2020

doc/cephadm: improve install guide

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37944 from tchaikov/wip-crimson-recovery-cleanup

crimson/osd: extract read_omap_for_push_op()

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: disable cluster selection in NFS export editing form

We should not allow changing an export's cluster because an export ID
might live in one cluster but not in another one. Editing a non-existing
export in a cluster causes an error.

Fixes: https://tracker.ceph.com/issues/47373
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mds: add dentry lease metric payload support

Fixes: https://tracker.ceph.com/issues/43423
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: add _dentry_valid() helper support

Fixes: https://tracker.ceph.com/issues/43423
Signed-off-by: Xiubo Li <xiubli@redhat.com>

crimson/osd: extract read_omap_for_push_op()

extract read_omap_for_push_op() out of build_push_op() for better
readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: correct the function names in logging messages

it was a leftover of fbfc53c7081ea0b99fbc519d7fac9f5637cf5425

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #37881 into master

* refs/pull/37881/head:
doc/mgr/orchestrator: Update about nfs add in rook
mgr/rook: Refactor nfs add/apply methods
mgr/rook: Udpate to mypy py3 type style in _create(), _update() and apply()
mgr/rook: Remove unused 'current' argument from update_func method

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

qa: avoid using ceph.dir.entries for empty check

This avoids a bug [1] in the kernel client.

[1] https://tracker.ceph.com/issues/48104

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37825 into master

* refs/pull/37825/head:
doc: document MDS cache configuration
test: add DecayCounter test

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

qa: do not test dir size equals rbytes

Avoiding this failure:

2020-11-03T03:11:59.980 INFO:tasks.cephfs_test_runner:test_subvolume_resize_fail_quota_lt_used_size_no_shrink (tasks.cephfs.test_volumes.TestSubvolumes) ... FAIL
2020-11-03T03:11:59.981 INFO:tasks.cephfs_test_runner:
2020-11-03T03:11:59.981 INFO:tasks.cephfs_test_runner:======================================================================
2020-11-03T03:11:59.981 INFO:tasks.cephfs_test_runner:FAIL: test_subvolume_resize_fail_quota_lt_used_size_no_shrink (tasks.cephfs.test_volumes.TestSubvolumes)
2020-11-03T03:11:59.982 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-11-03T03:11:59.982 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-11-03T03:11:59.982 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201102.231317/qa/tasks/cephfs/test_volumes.py", line 1310, in test_subvolume_resize_fail_quota_lt_used_size_no_shrink
2020-11-03T03:11:59.982 INFO:tasks.cephfs_test_runner:    self.assertEqual(usedsize, susedsize)
2020-11-03T03:11:59.982 INFO:tasks.cephfs_test_runner:AssertionError: 8912896 != 1

From: /ceph/teuthology-archive/pdonnell-2020-11-03_02:22:55-fs:volumes-wip-pdonnell-testing-20201102.231317-distro-basic-smithi/5585383/teuthology.log

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: disable check for value kclient fails to get

See-also: https://tracker.ceph.com/issues/48075
Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: leave one standby available to avoid warning

Four file systems will use all MDS and generate this warning:

2020-11-02T03:48:33.407 INFO:teuthology.orchestra.run.smithi003.stdout:2020-11-02T03:24:21.817337+0000 mon.a (mon.0) 481 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not check access timestamps

kclient does not update these like ceph-fuse.

This avoids failures like:

2020-11-01T08:05:53.115 INFO:tasks.cephfs_test_runner:test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ... FAIL
2020-11-01T08:05:53.115 INFO:tasks.cephfs_test_runner:
2020-11-01T08:05:53.116 INFO:tasks.cephfs_test_runner:======================================================================
2020-11-01T08:05:53.116 INFO:tasks.cephfs_test_runner:FAIL: test_subvolume_clone_in_progress_getpath (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)
2020-11-01T08:05:53.116 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-11-01T08:05:53.116 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-11-01T08:05:53.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_volumes.py", line 2295, in test_subvolume_clone_in_progress_getpath
2020-11-01T08:05:53.117 INFO:tasks.cephfs_test_runner:    self._verify_clone(subvolume, snapshot, clone)
2020-11-01T08:05:53.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_volumes.py", line 154, in _verify_clone
2020-11-01T08:05:53.117 INFO:tasks.cephfs_test_runner:    self._verify_clone_attrs(path1, path2)
2020-11-01T08:05:53.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_volumes.py", line 107, in _verify_clone_attrs
2020-11-01T08:05:53.118 INFO:tasks.cephfs_test_runner:    self.assertEqual(sval, cval)
2020-11-01T08:05:53.118 INFO:tasks.cephfs_test_runner:AssertionError: 1604217951 != 1604217941

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore skip errors for kclient

To avoid this failure:

2020-11-01T07:21:35.117 INFO:tasks.cephfs_test_runner:test_volume_without_namespace_isolation (tasks.cephfs.test_volume_client.TestVolumeClient) ... ok
2020-11-01T07:21:35.118 INFO:tasks.cephfs_test_runner:
2020-11-01T07:21:35.118 INFO:tasks.cephfs_test_runner:======================================================================
2020-11-01T07:21:35.119 INFO:tasks.cephfs_test_runner:FAIL: test_evict_client (tasks.cephfs.test_volume_client.TestVolumeClient)
2020-11-01T07:21:35.119 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-11-01T07:21:35.119 INFO:tasks.cephfs_test_runner:Requires FUSE client to inject client metadata
2020-11-01T07:21:35.119 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-11-01T07:21:35.119 INFO:tasks.cephfs_test_runner:Ran 18 tests in 732.749s
2020-11-01T07:21:35.120 INFO:tasks.cephfs_test_runner:
2020-11-01T07:21:35.120 INFO:tasks.cephfs_test_runner:FAILED (failures=1)

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add more clients for test_volume_client

It requires 4.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: accept client returning pool id for layout

Resolving this failure:

2020-10-30T22:52:25.898 INFO:tasks.cephfs_test_runner:test_subvolume_group_create_with_desired_data_pool_layout (tasks.cephfs.test_volumes.TestSubvolumeGroups) ... FAIL
2020-10-30T22:52:25.898 INFO:tasks.cephfs_test_runner:
2020-10-30T22:52:25.899 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T22:52:25.899 INFO:tasks.cephfs_test_runner:FAIL: test_subvolume_group_create_with_desired_data_pool_layout (tasks.cephfs.test_volumes.TestSubvolumeGroups)
2020-10-30T22:52:25.899 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:52:25.899 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-10-30T22:52:25.900 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_volumes.py", line 568, in test_subvolume_group_create_with_desired_data_pool_layout
2020-10-30T22:52:25.900 INFO:tasks.cephfs_test_runner:    self.assertEqual(desired_pool, new_pool)
2020-10-30T22:52:25.900 INFO:tasks.cephfs_test_runner:AssertionError: '34' != 'new_pool'
2020-10-30T22:52:25.900 INFO:tasks.cephfs_test_runner:- 34
2020-10-30T22:52:25.900 INFO:tasks.cephfs_test_runner:+ new_pool

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: ignore expected warning about client missing features

2020-10-30T22:54:14.842 INFO:teuthology.orchestra.run.smithi168.stdout:2020-10-30T22:43:34.067194+0000 mds.a (mds.0) 1 : cluster [WRN] evicting session smithi168:0 (8757), missing required features '0x0000000000000400'

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not fail on skip

Avoiding these failures:

2020-10-30T22:49:00.248 INFO:tasks.cephfs_test_runner:test_client_release_bug (tasks.cephfs.test_client_limits.TestClientLimits) ...
2020-10-30T22:49:00.248 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T22:49:00.248 INFO:tasks.cephfs_test_runner:FAIL: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2020-10-30T22:49:00.249 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:49:00.249 INFO:tasks.cephfs_test_runner:Require FUSE client to inject client release failure
2020-10-30T22:49:00.249 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T22:49:00.250 INFO:tasks.cephfs_test_runner:FAIL: test_client_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
2020-10-30T22:49:00.250 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:49:00.251 INFO:tasks.cephfs_test_runner:Require FUSE client to inject client release failure
2020-10-30T22:49:00.251 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T22:49:00.252 INFO:tasks.cephfs_test_runner:FAIL: test_client_release_bug (tasks.cephfs.test_client_limits.TestClientLimits)
2020-10-30T22:49:00.252 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:49:00.252 INFO:tasks.cephfs_test_runner:Require FUSE client to inject client release failure
2020-10-30T22:49:00.252 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:49:00.252 INFO:tasks.cephfs_test_runner:Ran 8 tests in 368.825s
2020-10-30T22:49:00.253 INFO:tasks.cephfs_test_runner:
2020-10-30T22:49:00.253 INFO:tasks.cephfs_test_runner:FAILED (failures=3)
2020-10-30T22:49:00.253 INFO:tasks.cephfs_test_runner:
2020-10-30T22:49:00.253 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T22:49:00.253 INFO:tasks.cephfs_test_runner:FAIL: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2020-10-30T22:49:00.254 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:49:00.254 INFO:tasks.cephfs_test_runner:Require FUSE client to inject client release failure
2020-10-30T22:49:00.254 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T22:49:00.254 INFO:tasks.cephfs_test_runner:FAIL: test_client_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits)
2020-10-30T22:49:00.255 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:49:00.255 INFO:tasks.cephfs_test_runner:Require FUSE client to inject client release failure
2020-10-30T22:49:00.255 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T22:49:00.255 INFO:tasks.cephfs_test_runner:FAIL: test_client_release_bug (tasks.cephfs.test_client_limits.TestClientLimits)
2020-10-30T22:49:00.255 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T22:49:00.256 INFO:tasks.cephfs_test_runner:Require FUSE client to inject client release failure

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix mount directory

Otherwise we get this error:

2020-10-30T03:57:44.256 INFO:teuthology.run_tasks:Running task exec...
2020-10-30T03:57:44.265 INFO:teuthology.task.exec:Executing custom commands...
2020-10-30T03:57:44.266 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi095.front.sepia.ceph.com
2020-10-30T03:57:44.266 INFO:teuthology.orchestra.run.smithi095:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd $TESTDIR/mnt.* && dd if=/dev/zero of=./foo count=100'
2020-10-30T03:57:44.380 INFO:teuthology.orchestra.run.smithi095.stderr:bash: line 0: cd: too many arguments

From: /ceph/teuthology-archive/pdonnell-2020-10-30_02:26:51-fs-master-distro-basic-smithi/5573179/teuthology.log

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: rename distro link to distros

This needs to become any overrides present in "mount". Otherwise kclient
distro overrides have no effect.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: accept eio for mount failure

Example:

2020-10-30T03:37:33.752 INFO:tasks.cephfs_test_runner:======================================================================
2020-10-30T03:37:33.752 INFO:tasks.cephfs_test_runner:FAIL: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)
2020-10-30T03:37:33.752 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-10-30T03:37:33.753 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-10-30T03:37:33.753 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_multifs_auth.py", line 311, in test_mount_mon_and_osd_caps_present_mds_caps_absent
2020-10-30T03:37:33.753 INFO:tasks.cephfs_test_runner:    self.check_that_mount_failed_for_right_reason(retval[2])
2020-10-30T03:37:33.753 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_batrick_ceph_cephfs-qa-reorg/qa/tasks/cephfs/test_multifs_auth.py", line 269, in check_that_mount_failed_for_right_reason
2020-10-30T03:37:33.753 INFO:tasks.cephfs_test_runner:    raise AssertionError('can\'t find expected set of words in the '
2020-10-30T03:37:33.754 INFO:tasks.cephfs_test_runner:AssertionError: can't find expected set of words in the stderr
2020-10-30T03:37:33.754 INFO:tasks.cephfs_test_runner:self.errmsgs - ('permission denied', 'no mds server is up or the cluster is laggy', 'no such file or directory')
2020-10-30T03:37:33.754 INFO:tasks.cephfs_test_runner:stderr - mount error 5 = input/output error

From: /ceph/teuthology-archive/pdonnell-2020-10-30_02:26:51-fs-master-distro-basic-smithi/5573109/teuthology.log

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: unmount volumes before removal

To avoid potential failures/hangs in umount.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: some sessionmap tests require fuse

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove obsolete test

Since the work in [1], the kclient can be killed with network namespaces
without powercycling.

[1] https://tracker.ceph.com/issues/44044

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not fail on test skip

test_client_config only works with ceph-fuse.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use more OSDs for workload tests

Running out of space / slow requests are less likely this way.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use 4 clients for volumes testing

The test_volume_client.py tests require 4 clients.

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: relocate tasks

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: rename workload suite

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: relocate new tasks from fs:basic_functional

These were not tested with kcephfs before, let's see if there's any
bugs!

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove kcephfs suite

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: move kcephfs:mixed-clients to fs

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove dead kcephfs:cephfs suite

Fixes: https://tracker.ceph.com/issues/23718
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>