Eric Biggers [Sun, 13 Mar 2022 01:05:57 +0000 (17:05 -0800)]
fscrypt-crypt-util: add support for dumping key identifier
Add an option to fscrypt-crypt-util to make it compute the key
identifier for the given key. This will allow testing the correctness
of the filesystem's key identifier computation.
Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Eric Biggers [Sun, 13 Mar 2022 01:05:56 +0000 (17:05 -0800)]
fscrypt-crypt-util: refactor get_key_and_iv()
Split get_key_and_iv() into two distinct parts: (1) deriving the key and
(2) generating the IV. Also, check for the presence of needed options
just before they are used rather than doing it all up-front.
These changes should make this code much easier to understand.
Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Eric Biggers [Sun, 13 Mar 2022 01:05:55 +0000 (17:05 -0800)]
fscrypt-crypt-util: use an explicit --direct-key option
Make fscrypt-crypt-util use an option --direct-key to specify the use of
the DIRECT_KEY method for key derivation and IV generation. Previously,
this method was implicitly detected via --mode-num being given without
either --iv-ino-lblk-64 or --iv-ino-lblk-32, or --kdf=none being given
in combination with --file-nonce.
The benefit of this change is that it makes the various options to
fscrypt-crypt-util behave more consistently. --direct-key,
--iv-ino-lblk-64, and --iv-ino-lblk-32 now all work similarly (they
select a key derivation and IV generation method); likewise, --mode-num,
--file-nonce, --inode-number, and --fs-uuid now all work similarly (they
provide information that key derivation and IV generation may need).
Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Wed, 16 Mar 2022 22:13:26 +0000 (15:13 -0700)]
xfs/17[013]: fix intermittent failures when filesystem metadata gets large
These tests check that the filestreams allocator never shares an AG
across multiple streams when there's plenty of space in the filesystem.
Recent increases in metadata overhead for newer features (e.g. bigger
logs due to reflink) can throw this off, so add another AG to the
formatted filesystem to encourage it to avoid the AG with the log.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Wed, 16 Mar 2022 22:12:32 +0000 (15:12 -0700)]
xfs/076: only create files on the data device
This test checks that filesystems with sparse inode support can continue
to allocate inodes when free space gets fragmented. Inodes only exist
on the data device, so we need to ensure that realtime is not enabled on
the filesystem so that the rt metadata doesn't mess with the inode usage
percentage and cause a test failure.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Wed, 16 Mar 2022 03:30:25 +0000 (20:30 -0700)]
generic/673: fix golden output to reflect vfs setgid behavior
Filipe Manana pointed out[1] that the setgid dropping behavior encoded
in this generic test is based on some outdated XFS code, and not based
on what the VFS inode attribute change functions actually do. Now that
we're working on fixing that, we should update the golden output to
reflect what all filesystems are supposed to be doing.
Curiously, the file checksums always match, and it's not *forbidden* for
the page cache to have a page backing an unwritten extent that hasn't
been written.
The condition that this test cares about is that block 3 (192k-256k) are
reported by SEEK_DATA as data even if the data fork has a hole and the
COW fork has an unwritten extent. Matthew Wilcox thinks this is a side
effect of readahead.
To fix this occasional false failure, call SEEK_DATA and SEEK_HOLE only
on the offsets that we care about.
Suggested-by: Matthew Wilcox <willy@infradead.org> Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Wed, 16 Mar 2022 03:30:09 +0000 (20:30 -0700)]
generic/459: ensure that the lvm devices have been created
Once in a /very/ long while this test fails because _mkfs_dev can't find
the LVM thinp volume after it's been created. Since the "solution" du
jour seems to be to sprinkle udevadm settle calls everywhere, do that
here in the hopes that will fix it.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Filipe Manana [Thu, 3 Mar 2022 11:36:42 +0000 (11:36 +0000)]
generic: read multiple extents with io_uring
Test doing a read, with io_uring, over a file range that includes
multiple extents. The read operation triggers page faults when
accessing all pages of the read buffer except for the pages
corresponding to the first extent. We want to check that the
operation results in reading all the extents and that it returns the
correct data.
This is motivated by an issue found with MariaDB when using io_uring
and running on btrfs. There's a patch for btrfs to address the issue
ca93e44bfb5f ("btrfs: fallback to blocking mode when doing async dio
over multiple extents")
Masayoshi Mizuma [Wed, 16 Feb 2022 14:55:20 +0000 (09:55 -0500)]
xfs: test xfsdump with bind-mounted filesystem
commit 25195eb ("xfsdump: handle bind mount target") introduced
a bug of xfsdump which doesn't store the files to the dump file
correctly when the root inode number is changed.
The commit 25195eb is reverted, and commit 0717c1c ("xfsdump: intercept
bind mount targets") which is in xfsdump v3.1.10 fixes the bug to reject
the filesystem if it's bind-mounted.
Test that xfsdump can reject the bind-mounted filesystem.
Ojaswin Mujoo [Tue, 22 Feb 2022 17:50:53 +0000 (23:20 +0530)]
ext4: Test to ensure resize with sparse_super2 is handled correctly
Kernel currently doesn't support resize of EXT4 mounted with
sparse_super2 option enabled. Earlier, it used to leave the resize
incomplete and the fs would be left in an inconsistent state,
however commit b1489186cc83[1] fixed this to avoid the fs corruption
by clearly returning -EOPNOTSUPP.
Test to ensure that kernel handles resizing with sparse_super2
correctly. Run resize for multiple iterations because this sometimes
leads to kernel crash due to fs corruption, which we want to detect.
Ojaswin Mujoo [Tue, 22 Feb 2022 17:50:52 +0000 (23:20 +0530)]
src/ext4_resize.c: Refactor code and ensure accurate errno is returned
The current implementation of ext4_resize returned 1 whenever
there was an error. Modify this to return the correct error code.
This is important for tests that rely on correct error reporting, by
the kernel, for ext4 resize functionality.
The commands for package managers for both Ubuntu and RHEL weren't
up-to-date. A few packages doesn't exist anymore or missing from the
latest repositories (e.g. liburing-devel, btrfs-progs-devel).
Ubuntu's list missed a few packages listed in RHEL's list. There are
a few repeating steps. The indent of avaliable environment variables
is not clear.
This patch:
- Update package dependencies for Ubuntu/Debian/RHEL/CentOS/Fedora
- Unify list of packages between Ubuntu/Debian and RHEL/CentOS/Fedora
- Add list of tool packages for other FS
- Add links to tools not in the standard repository for RHEL/CentOS
- Drop xfsprogs-qa-devel reference for old systems
- Replace note about EPEL with installation step with link to Fedora
manual
- Add configuration examples in 'Setup Environment'
- Small clarification details, such as size of the partition and
KCONFIG_PATH description
- Removal of repeating steps (install administrative tools,
partitions content)
- Restructuring and formatting of "BUILDING THE FSQA SUITE" section
- Drop IRIXDEV references in variable set up
- Variables in 'Additional setup' divided into groups
- Move fsqa user/group creation and udf_test to 'Setup Environment'
section
When the helper function _scratch_mkfs_sized is called with small
filesystem size and FSTYP=btrfs, it calls mkfs.btrfs with --mixed option
to enable mixed mode. However, mkfs.btrfs with --mixed option fails for
zoned block devices since btrfs does not support mixed mode together
with zoned mode. To avoid this failure, do not set --mixed option when
the scratch device is a zoned block device.
Filipe Manana [Thu, 3 Mar 2022 11:40:21 +0000 (11:40 +0000)]
generic: move test case btrfs/261 into the generic group
The test case btrfs/261, part from its comments, doesn't really exercise
any behaviour that is btrfs specific, so, as Dave Chinner pointed out, it
can be moved into the generic group.
This change moves that test into the generic group and slightly adjust the
comments to make it clear which parts are btrfs specific.
Sidong Yang [Wed, 2 Mar 2022 14:05:48 +0000 (14:05 +0000)]
btrfs: add test for enable/disable quota and create/destroy qgroup repeatedly
Test enable/disable quota and create/destroy qgroup repeatedly in
parallel and confirm it does not cause kernel hang. It only happens
in kernel staring with kernel 5.17-rc3. This is a regression test
for the problem reported to linux-btrfs list [1].
The hang was recreated using the test case and fixed by kernel patch
titled
btrfs: qgroup: fix deadlock between rescan worker and remove qgroup
Nitesh Shetty [Tue, 1 Mar 2022 21:29:47 +0000 (02:59 +0530)]
generic/108: use sysfs values for logical,physical block size in scsi_debug
scsi_debug device used for test, is created with assumption of 512
bytes logical and physical block size.
This causes error in lvcreate step, when SCRATCH_DEV device lba is
not 512 bytes. This can be solved by reading block size from sysfs
of device. If sysfs is missing fallback to 512 bytes as default.
Jan Kara [Tue, 22 Feb 2022 10:36:58 +0000 (11:36 +0100)]
generic: Add test for seekdir
Add test checking functionality of seekdir. We check whether seekdir
gets us back to the directory entry it should and also whether
seeking to random positions does not crash the filesystem.
Unlike test generic/310 which also tests seeking, this test checks both
glibc readdir() function as well as getdents64() syscall directly. This
is because glibc readdir() implementation does a lot of caching and
processing internally thus hiding kernel from some possible problems.
Also test wider range of random offsets to have better chance of
hitting out of bound accesses or other bugs.
This is a regression test for a48fc69fe658 ("udf: Fix crash after
seekdir")
Signed-off-by: Jan Kara <jack@suse.cz> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
btrfs: add support for capturing metadumps of corrupted fses
Add the capability to capture btrfs metadumps when filesystem checks
fail, so that they can be used for further debugging. This is useful
for tests that _require_test and/or _require_scratch for which
filesystem checkers will run after a test completes and may
occasionally pick up inconsistencies.
Filipe Manana [Thu, 17 Feb 2022 12:14:21 +0000 (12:14 +0000)]
btrfs: log replay after fsync of file with prealloc extents beyond eof
Test that after a full fsync of a file with preallocated extents
beyond the file's size, if a power failure happens, the preallocated
extents still exist after we mount the filesystem.
This test currently fails and there is a patch for btrfs that fixes
this issue and has the following subject:
"btrfs: fix lost prealloc extents beyond eof after full fsync"
common: factor out xfs unique part from _filter_mkfs
Most of the code in the function _filter_mkfs is xfs unique. This is
misleading that the function would be dedicated for xfs. Clean up the
function by factoring out xfs unique part to _xfs_filter_mkfs in
common/xfs. While at the same time, fix indent in _xfs_filter_mkfs to be
consistent with other functions in common/xfs.
Suggested-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
The test cases xfs/015 calls _scratch_mkfs before _scratch_mkfs_sized,
and does not check return code of _scratch_mkfs_sized. Even if
_scratch_mkfs_sized failed, _scratch_mount after it cannot detect the
sized mkfs failure because _scratch_mkfs already created a file system
on the device. This results in unexpected test condition.
To avoid the unexpected test condition, check return code of
_scratch_mkfs_sized.
Suggested-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
The test cases ext4/021 calls _scratch_mkfs before _scratch_mkfs_sized,
and does not check return code of _scratch_mkfs_sized. Even if
_scratch_mkfs_sized failed, _scratch_mount after it cannot detect the
sized mkfs failure because _scratch_mkfs already created a file system
on the device. This results in unexpected test condition.
To avoid the unexpected test condition, check return code of
_scratch_mkfs_sized.
Suggested-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
The test cases generic/{171,172,173,174} call _scratch_mkfs before
_scratch_mkfs_sized, and they do not check return code of
_scratch_mkfs_sized. Even if _scratch_mkfs_sized failed, _scratch_mount
after it cannot detect the sized mkfs failure because _scratch_mkfs
already created a file system on the device. This results in unexpected
test condition of the test cases.
To avoid the unexpected test condition, check return code of
_scratch_mkfs_sized in the test cases.
Suggested-by: Naohiro Aota <naohiro.aota@wdc.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
The test case generic/204 calls _scratch_mkfs to get data block size and
i-node size of the filesystem and obtained data block size is passed to
the following _scratch_mfks_sized call as an option. However, the
_scratch_mkfs call is unnecessary since the sizes can be obtained by
_scratch_mkfs_sized call without the data block size option.
Also the _scratch_mkfs call is harmful when the _scratch_mkfs succeeds
and the _scratch_mkfs_sized fails. In this case, the _scratch_mkfs
leaves valid working filesystem on scratch device then following mount
and IO operations can not detect the failure of _scratch_mkfs_sized.
This results in the test case run with unexpected test condition.
Hence, remove the _scratch_mkfs call and the data block size option for
_scratch_mkfs_sized call.
Suggested-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Luis Chamberlain [Tue, 15 Feb 2022 01:48:43 +0000 (17:48 -0800)]
btrfs/237: adjust uuid heuristics
Using $(findmnt -n -o UUID mount-directory) only works on
util-linux v2.38, not on 2.37. Even debian-testing has util-linux
v2.37, so use a mechanism which has worked for longer to fetch the
UUID.
Without this the test fails on permission failure on accessing
the file /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold as $uuid is
empty. So while at it, add a check to ensure the UUID is never
empty.
Use $AWK_PROG in case the preferred awk program is not the
preferred awk program.
Cc: Karel Zak <kzak@redhat.com> Cc: Pankaj Raghav <p.raghav@samsung.com> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org> Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com> Tested-by: Adam Manzanares <a.manzanares@samsung.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
common/config: fix use of MODPROBE_RM_PATIENT_TIMEOUT_ARGS
When patient module removal support is detected on kmod
(note that this is not yet even merged onto kmod yet but
work in progress) the arguments required for it are not being
set yet. This was a typo, fix this.
This issue was spotted using shellcheck from blktests when adding
patient module removal support there. I'll post a patch next to
let folks evaluate if we should embrace shellcheck on fstests as
well.
Sun Ke [Tue, 8 Feb 2022 07:16:24 +0000 (15:16 +0800)]
common/attr: adbjust acl_max of f2fs
f2fs has set inline_xattr as a default option, and introduced a new
option named 'noinline_xattr' for disabling default inline_xattr
option. So in _acl_get_max we need to check 'noinline_xattr' string
in fs option, otherwise we may select the wrong max acl number since
we always found the string 'inline_xattr' in fs option.
Additionally, f2fs has changed disk layout of xattr block a bit, so
will contain one more entry in both inline and noinline xattr inode,
this patch will modify the max acl number to adjust it.
Suggested-by: Chao Yu <chao@kernel.org> Signed-off-by: Sun Ke <sunke32@huawei.com> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Qu Wenruo [Wed, 2 Feb 2022 04:29:29 +0000 (12:29 +0800)]
src/fssum: use newer EVP_* interface to replace deprecated MD5_* interace
In OpenSSL 3.0, MD_Init/Update/Final() interfaces are marked
deprecated, and we have to go EVP_DigestInit/Update/Final() instead.
Personally I'm not a fan of this, especially the new EVP_MD_CTX
structure can no longer be stack allocated, thus we have to
dynamically allocate and free EVP_MD_CTX in sum_init() and sum_free().
Hopes this is proper way to go and would solve the problem until
OpenSSL chooses to change their interface again.
Despite the regular file defragging, "btrfs filesystem defragment"
provides an option, -c, to convert all data extents (except holes
and preallocated ranges) to a new compression algorithm.
The special behavior here is, unlike regular defrag which is not
going to touch extents which are adjacent to preallocated/hole
ranges, with -c, all non-hole/non-preallocated extents should be
defragged and converted to the new compression algorithm.
This test case will ensure the old behavior is properly kept.
Currently both old kernels (v5.15 and older) and newer kernel with
refactored defrag (v5.16 and newer) can pass the tests.
Qu Wenruo [Fri, 28 Jan 2022 00:27:00 +0000 (08:27 +0800)]
btrfs: autodefrag with regular and hole extents
In v5.11~v5.15 kernels, there is a regression in autodefrag that if
a cluster (up to 256K in size) has even a single hole, the whole
cluster will be rejected.
This will greatly reduce the efficiency of autodefrag.
The behavior is fixed in v5.16 by a full rework, although the rework
itself has other problems, it at least solves the problem.
Here we add a test case to reproduce the case, where we have a 128K
cluster, the first half is fragmented extents which can be
defragged. The second half is hole.
Qu Wenruo [Fri, 28 Jan 2022 00:26:59 +0000 (08:26 +0800)]
btrfs: defrag with regular and preallocated extents
Recent v5.16 has some regression around btrfs autodefrag mount
option, and the extra scrutiny around defrag code exposes some
questionable behavior from the old code.
One behavior is to defrag extents along with the next preallocated
extent.
This behavior will cause extra IO and convert all the preallocated
extent to regular zero filled extents, rendering the preallocated
extent useless.
The kernel fix is titled:
btrfs: defrag: don't try to merge regular extents with preallocated extents
Zorro Lang [Sat, 29 Jan 2022 04:44:31 +0000 (12:44 +0800)]
fsx: disable allocsp_calls if -F is specified
As the fsx.c source code says:
int fallocate_calls = 1; /* -F flag disables */
int allocsp_calls = 1; /* -F flag disables */
The allocsp_calls and fallocate_calls should be disabled, if the -F
option is specified. But current fsx forgets to disable allocsp_calls
as is documented.
Signed-off-by: Zorro Lang <zlang@redhat.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Theodore Ts'o [Tue, 1 Feb 2022 23:19:36 +0000 (18:19 -0500)]
common: handle old versions of modprobe which don't support --help
There exists versions of modprobe out there which are so old that
modprobe --help isn't a thing. They're certainly not going to support
modprobe --remove-patiently, so test to make sure modprobe --help
works to avoid causing all tests to fail due to the error message
to stderr showing up in $seq.out.
common/dump: add explicit requirement for specific users and groups
Make the requirement for daemon:sys and bin:bin explicit, so that the
xfs tests that source common/dump and rely on _do_create_dumpdir_fill
and _create_dumpdir_symlinks do not fail on chown when those
users/groups are missing.
common/rc: fix group detection regex to strictly match required name
_require_group greps for the required group string in /etc/group but
this can partially match other groups or group member names and falsely
return success where it should fail.
Make the regex more specific so that it can unambigiously match only the
exact group name rather than any other group that happens to match as a
substring or any matching username from the group member list field.
Darrick J. Wong [Mon, 24 Jan 2022 18:37:35 +0000 (10:37 -0800)]
generic/273: use _get_file_block_size
This test calculates the amount of free space on a filesystem and uses
the block size to spread the work of filling the free space among a
bunch of threads. Unfortunately, the test /should/ be using the
allocation unit size, not the fs block size, which is why this test
fails on configurations such as XFS realtime with a 28k extent size.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Leah Rumancik <leah.rumancik@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Make it possible to always run all the tests in the testsuite that don't
require idmapped mounts. Now all filesystems can benefit from the
generic vfs tests that we currently implement. This means setgid
inheritance and other tests will be run for all filesystems not matter
if they support idmapped mounts or not.
To this end, the binary makes use of the fs_allow_idmap() helper we
introduced earlier to dynamically determine whether the underlying
filesystem supports idmapped mounts or not. It is therefore safe for
callers to call the binary even if the tested filesystem doesn't support
idmapped mounts.
Most of the tests that call the idmapped mount binary require idmapped
mount support and so they will continue to call
_require_idmapped_mounts. This will also ensure that we log a proper
message about skipping a whole test.
However, the generic/633 test includes idmapped mount specific and
generic tests. The generic tests can and should always be executed as
they test core vfs functionality that isn't tested anywhere else in
fstests. So here we can remove the _require_idmapped_mounts check from
the test and rely on the binary doing the right thing.
Move the check whether the underlying filesystem supports idmapped
mounts into a separate helper. We will use it in the following patch to
make it possible to always run all tests that don't require idmapped
mounts.
Lukas Czerner [Thu, 20 Jan 2022 10:20:25 +0000 (11:20 +0100)]
ext4/053: Test remount without changing mount options
With the recent ext4 mount api change we discovered a bugs that weren't
caught by this test. It was triggered by remounting the file system
either with the same mount options, or without specifying any mount
options at all. In this case we would expect the original mount options
to remain the same, however this was either not the case, or the remount
failed.
Add a remount test after a regular mount. Remount once with specifying
the original mount option and remount second time without specifying
anything. Test the active options after each test.
Additionally include all the combinations of data= options in the
remount test for the sake of completeness.
Filipe Manana [Tue, 18 Jan 2022 16:36:52 +0000 (16:36 +0000)]
btrfs: test that defrag on small files does not hang or crashes
Test that defragging files with very small sizes works and does not
result in any crash, hang or corruption.
This is motivated by a regression introduced in kernel 5.16 where
attempting to defrag a file with a size of 1 byte would result in
the kernel code hitting an "infinite" loop (iterating from 0 to
(u64)-1 in increments of 256K, which in practice is an eternity).
The regression is fixed by a patch with the following subject:
"btrfs: fix too long loop when defragging a 1 byte file"
btrfs: add test for quota disable in parallel with balance
Test quota disable during btrfs balance and confirm it does not
cause kernel hang. This is a regression test for the problem
reported to linux-btrfs list [1].
The hang was recreated using the test case and memory backed
null_blk device with 5GB size as the scratch device, and fixed by
kernel patch titled
btrfs: fix deadlock between quota disable and qgroup rescan worker
Darrick J. Wong [Thu, 20 Jan 2022 00:49:44 +0000 (16:49 -0800)]
xfs/107: fix formatting failures
Zorro Lang reported that the _scratch_mkfs_sized call in the new xfs/107
fstest sometimes fails on more exotic storage due to insufficient log
size on account of raid stripes, etc. These are side effects of the
filesystem being too small.
Change the filesystem size to 256M to avoid these problems, and change
the allocstale parameters to use the same file size (16M) as before.
Given that ALLOCSP produces stale disk contents pretty quickly this
shouldn't affect the test runtime too much.
Reported-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Tue, 18 Jan 2022 18:29:10 +0000 (10:29 -0800)]
iogen: upgrade to fallocate
Update this utility to use fallocate to preallocate/reserve space to a
file so that we're not so dependent on legacy XFS ioctls. Fix a minor
whitespace error while we're at it.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Yang Xu [Mon, 17 Jan 2022 07:36:54 +0000 (15:36 +0800)]
src/ext4_resize.c: set errno to 0 before the strtoull call
On my test machine, ext4/033 fails even use the non-overflow size.
It reports invalid new size when using strtoull because errno is 1.
As man-pages said "Since strtoul() can legitimately return 0 or ULONG_MAX
(ULLONG_MAX for strtoull()) on both success and failure, the calling program
should set errno to 0 before the call, and then determine if an error occurred
by checking whether errno has a nonzero value after the call".
So add a step to set errno to 0 before strtoull call.
Fixes: 92b9c0dedace ("ext4/033: test EXT4_IOC_RESIZE_FS by calling the ioctl directly") Signed-off-by: Yang Xu <xuyang2018.jy@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Theodore Ts'o <tytso@mit.edu> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Tue, 11 Jan 2022 21:50:30 +0000 (13:50 -0800)]
xfs: test fixes for new 5.17 behaviors
xfs/308 and xfs/130 are two tests that tried to mess with the refcount
btree to try to trip up the COW recovery code. Now that we've made COW
recovery only happen during log recovery, we must adjust these tests to
force a log recovery. Older kernels should be ok with this, since they
unconditionally try to recover COW on mount.
Add a helper function to unmount the filesystem with a dirty log and
convert the two tests to use it. While we're at it, remove an xfs_check
test because xfs_check refuses to run on a dirty fs, and nobody cares
about xfs_check anymore.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Tue, 11 Jan 2022 21:50:24 +0000 (13:50 -0800)]
xfs/220: fix quotarm syscall test
In commit 6ba125c9, we tried to adjust this fstest to deal with the
removal of the ability to turn off quota accounting via the Q_XQUOTAOFF
system call.
Unfortunately, the changes made to this test make it nonfunctional on
those newer kernels, since the Q_XQUOTARM command returns EINVAL if
quota accounting is turned on, and the changes filter out the EINVAL
error string.
Doing this wasn't /incorrect/, because, very narrowly speaking, the
intent of this test is to guard against Q_XQUOTARM returning ENOSYS when
quota has been enabled. However, this also means that we no longer test
Q_XQUOTARM's ability to truncate the quota files at all.
So, fix this test to deal with the loss of quotaoff in the same way that
the others do -- if accounting is still enabled after the 'off' command,
cycle the mount so that Q_XQUOTARM actually truncates the files.
While we're at it, enhance the test to check that XQUOTARM actually
truncated the quota files.
Fixes: 6ba125c9 ("xfs/220: avoid failure when disabling quota accounting is not supported") Cc: xuyang2018.jy@fujitsu.com Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by：Yang Xu <xuyang2018.jy@fujitsu.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Tue, 11 Jan 2022 21:50:19 +0000 (13:50 -0800)]
common/rc: fix unicode checker detection in xfs_scrub
_check_xfs_scrub_does_unicode is still less than adequate -- if running
ldd to report the xfs_scrub binary's dynamic library dependencies
doesn't work, we could still detect support by grepping for strings that
only appear when the unicode checker is built.
Note that this isn't the final word on all of this; I will make this
easier to discover in a future xfs_scrub release.
Cc: Theodore Ts'o <tytso@mit.edu> Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Qu Wenruo [Tue, 11 Jan 2022 00:36:46 +0000 (08:36 +0800)]
btrfs/011: continue the test if we failed to cancel the replace
Btrfs/011 test case has exposed quite some bugs in the past, but it also
has some tendency to cause false alert, as its way testing replace
cancel require the replace to be still running when we send the cancel
request.
But on a lot of cases, the replace can finish way faster than the wait
time, and cause false alert.
Commit fa85aa64 ("btrfs/011: Fill the fs to ensure we have enough data for
dev-replace") tries to address the problem by filling the fs, but there
is still no guarantee.
Although there is still some discussion on how to properly solve the
problem, there is one thing sure that we should continue the test
instead of abort, if the replace cancel failed.
A quick abort caused by finished replace will leave other profiles
untested, reducing the coverage.
This patch will still mark the test failed for a finished replace, but
at least ensure we have run the test for all the profiles.
Shiyang Ruan [Wed, 12 Jan 2022 09:28:46 +0000 (17:28 +0800)]
generic/670: allow _mread_range() races with reflink_range()
mread copies data one-byte-at-a-time, so it may races with
reflink_range() who invalidates page cache of the dest file.
Allow this race by adjusting the egrep regexp.
Reported-by: Darrick J. Wong <djwong@kernel.org> Suggested-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Theodore Ts'o [Sun, 9 Jan 2022 19:59:35 +0000 (14:59 -0500)]
common/rc: fix _check_xfs_scrub_does_unicode on newer versions of libc-bin
Debian 10 uses ldd from glibc 2.28, where as Debian 11 uses ldd from
glibc 2.31. Sometime between glibc 2.28 and 2.31, ldd has been
changed so that the message "not a dynamic executable" is sent stderr,
where before it was sent to stdout. As a result, it caused
regressions for tests such as generic/453 which uses
_check_xfs_scurb_does_unicode:
generic/453 5s ... [22:42:03] [22:42:08]- output mismatch (see /results/xfs/results-4k/generic/453.out.bad)
--- tests/generic/453.out 2022-01-08 15:15:15.000000000 -0500
+++ /results/xfs/results-4k/generic/453.out.bad 2022-01-08 22:42:08.596982251 -0500
@@ -4,3 +4,4 @@
Test files
Uniqueness of inodes?
Test XFS online scrub, if applicable
+ not a dynamic executable
...
Fix this by sending stderr from ldd to /dev/null. This is not a
perfect solution, since it means that even if xfs_scrub was compiled
with libicui18n, we will skip the online scrub portion of generic/453.
However, this fixes the regression when runtime OS is changed from
Debian Buster to Debian Bullseye when xfsprogs is built statically.
In the long run, it would be nice if we could determine whether
xfs_scrub has unicode support without using ldd --- perhaps by
signally this in the output of xfs_scrub -V --- but we'll need to
discuss this with the xfsprogs maintainers.
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
idmapped-mounts: add missing ownership comparisons to setgid tests
In some setgid tests we missed to check ownership right after file or
directory creation in order to verify whether gid ownership inheritance
from the parent directory to the newly created file or directory works
correctly. Add the missing ones.
Cc: Seth Forshee <sforshee@digitalocean.com> Cc: Eryu Guan <guaneryu@gmail.com> Cc: Christoph Hellwig <hch@lst.de> Cc: fstests@vger.kernel.org Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Jeff Layton [Thu, 6 Jan 2022 13:30:49 +0000 (08:30 -0500)]
ceph/001-003: skip if test_dummy_encryption is enabled
Some tests on ceph require changing the layout of new files, which
is forbidden when the files are encrypted. Skip these tests if the
test_dummy_encryption mount option is being used.
Generalize the _exclude_scratch_mount_option code and add a new
_exclude_test_mount_option call as well. Call the new function from the
ceph tests that should exclude test_dummy_encryption.
Cc: Luis Henriques <lhenriques@suse.de> Cc: Eric Biggers <ebiggers@kernel.org> Signed-off-by: Jeff Layton <jlayton@kernel.org> Reviewed-by: Eric Biggers <ebiggers@google.com> Reviewed-by: Luis Henriques <lhenriques@suse.de> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Darrick J. Wong [Tue, 4 Jan 2022 02:04:17 +0000 (18:04 -0800)]
xfs/014: try a few times to create speculative preallocations
This test checks that speculative file preallocations are transferred to
threads writing other files when space is low. Since we have background
threads to clear those preallocations, it's possible that the test
program might not get a speculative preallocation on the first try.
This problem has become more pronounced since the introduction of
background inode inactivation since userspace no longer has direct
control over the timing of file blocks being released from unlinked
files. As a result, the author has seen an increase in sporadic
warnings from this test about speculative preallocations not appearing.
Therefore, modify the function to try up to five times to create the
speculative preallocation before emitting warnings that then cause
golden output failures.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Anand Jain [Wed, 5 Jan 2022 06:03:20 +0000 (14:03 +0800)]
btrfs/254: fix non-unique fsid issue
This test case needs two scratch fs. For debugging purposes, a fsid is
kept known. However, mkfs-btrfs has a stringent and inconsistent
approach, to check if that fsid is already present on any of the other
disks in the system. (This does not apply to the virtual devices in the
system).
To avoid running into the non-unique fsid found error, remove the known
fsid.
Darrick J. Wong [Tue, 4 Jan 2022 02:07:46 +0000 (18:07 -0800)]
xfs/107: remove long-broken test
This poorly implemented test runs fsstress and embeds xfs_quota output
in the golden output. This means that it breaks as soon as anyone adds
a new operation to fsstress, since that perturbs the sequence of
operations that it runs, which will make the project quota report output
change.
Normally I'd just fix the test, but the golden output also encodes
output strings that xfs_quota hasn't printed since 2010. Clearly
nobody's running this test (including me, who has had it turned off for
five+ years) so just get rid of it.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Anand Jain [Fri, 10 Dec 2021 18:14:41 +0000 (02:14 +0800)]
btrfs: test cleaning up of the stale device
Recreating a new filesystem or adding a device to a mounted the
filesystem should remove the device entries under its previous fsid
even when confused with different device paths to the same device.
Shiyang Ruan [Tue, 14 Dec 2021 08:19:11 +0000 (16:19 +0800)]
generic: add mmap CoW test for mixed&source extents
Ensuring that copy on write in mmap mode works when the CoW range
originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks.
Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Shiyang Ruan [Tue, 14 Dec 2021 08:19:10 +0000 (16:19 +0800)]
generic: add direct-io CoW test for mixed&source extents
Ensuring that copy on write in direct-io mode works when the CoW
range originally covers multiple extents, mixed with reflinked,
unwritten, hole, regular and delalloc blocks.
Shiyang Ruan [Tue, 14 Dec 2021 08:19:09 +0000 (16:19 +0800)]
generic: add buffered-io CoW test for mixed&source extents
Ensuring that copy on write in buffered mode works when the CoW
range originally covers multiple extents, mixed with reflinked,
unwritten, hole, regular and delalloc blocks.
Shiyang Ruan [Tue, 14 Dec 2021 08:19:08 +0000 (16:19 +0800)]
generic: add mmap CoW test for delalloc&source extents
Ensuring that copy on write in mmap mode to the source file when the CoW range
covers delalloc blocks and regular shared blocks.
(MMAP version of generic/293,295)
Shiyang Ruan [Tue, 14 Dec 2021 08:19:07 +0000 (16:19 +0800)]
generic: add mmap CoW test for holes&source extents
Ensuring that copy on write in mmap mode to the source file when the CoW range
covers holes and regular shared blocks.
(MMAP version of generic/291,292)
Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Shiyang Ruan [Tue, 14 Dec 2021 08:19:06 +0000 (16:19 +0800)]
generic: add mmap CoW test for unwritten&source extents
Ensuring that copy on write in mmap mode to the source file when the CoW range
covers unwritten and regular shared blocks.
(MMAP version of generic/289,290)
Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Shiyang Ruan [Tue, 14 Dec 2021 08:19:05 +0000 (16:19 +0800)]
generic: add mmap CoW test for regular&source extents
Ensuring that copy on write in mmap mode to the source file when the
CoW range covers regular unshared and regular shared blocks. (MMAP
version of generic/284,287)
Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Shiyang Ruan [Tue, 14 Dec 2021 08:19:04 +0000 (16:19 +0800)]
generic: add mmap CoW test for mixed&destination extents
Ensuring that copy on write in mmap mode works when the CoW range
originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks. (MMAP version of generic/200,199)
Shiyang Ruan [Tue, 14 Dec 2021 08:19:03 +0000 (16:19 +0800)]
generic: add mmap CoW test for delalloc&destination extents
Ensuring that copy on write in mmap mode works when the CoW range
originally covers multiple extents, some delalloc, some not. (MMAP
version of generic/195,194)
Shiyang Ruan [Tue, 14 Dec 2021 08:19:01 +0000 (16:19 +0800)]
generic: add mmap CoW test for holes&destination extents
Ensuring that copy on write in mmap mode works when the CoW range
originally covers multiple extents, some holes, some not. (MMAP
version of generic/191,190)
Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Shiyang Ruan [Tue, 14 Dec 2021 08:19:00 +0000 (16:19 +0800)]
generic: add mmap CoW test for unwritten&destination extents
Ensuring that copy on write in mmap mode works when the CoW range
originally covers multiple extents, some unwritten, some not. (MMAP
version of generic/189,188)
Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Shiyang Ruan [Tue, 14 Dec 2021 08:18:59 +0000 (16:18 +0800)]
generic: add mmap CoW test for regular&destination extents
Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some regular, some not.
(MMAP version of generic/197,196)
Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Theodore Ts'o [Mon, 20 Dec 2021 20:40:59 +0000 (15:40 -0500)]
ext4/033: test EXT4_IOC_RESIZE_FS by calling the ioctl directly
E2fsprogs commits 4ea80d031c7e ("resize2fs: adjust new size of the
file system to allow a successful resize") and 50088b1996cc
("resize2fs: attempt to keep the # of inodes valid by removing the
last bg") will automatically reduce the requested new size of the file
system by up to a single block group to avoid overflowing the 32-bit
inode count. This interferes with ext4/033's test of kernel commit 4f2f76f75143 ("ext4: Forbid overflowing inode count when # resizing".)
Address this by creating a new test program, ext4_resize which calls
the EXT4_IOC_RESIZE_FS ioctl directly so we can correctly test the
kernel's online resize code.
Sun Ke [Thu, 16 Dec 2021 09:32:19 +0000 (17:32 +0800)]
ext4: illegal memory access caused by quota index information error
The quota index information in the image is tampered, causing
illegal memory access.
It is a regression test for kernel commits
- 9bf3d2033129 quota: check block number when reading the block in quota file
- d0e36a62bd4c quota: correct error number in free_dqentry()
Signed-off-by: Sun Ke <sunke32@huawei.com> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Brian Foster [Fri, 17 Dec 2021 17:18:26 +0000 (12:18 -0500)]
xfs: test COW writeback failure when overlapping non-shared blocks
Test that COW writeback that overlaps non-shared delalloc blocks
does not leave around stale delalloc blocks on I/O failure. This
triggers assert failures and free space accounting corruption on
XFS.
Fixed by upstream kernel commit 5ca5916b6bc9 ("xfs: punch out data
fork delalloc blocks on COW writeback failure").
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Theodore Ts'o [Sun, 19 Dec 2021 04:16:38 +0000 (23:16 -0500)]
common/filter: add _filter_bash()
This is needed to account for bash 5.1 adding line number annotation
when executing a command via the -c option and it fails. For example,
"bash -c 'echo foo > /'" will cause bash 5.1 to report:
Filipe Manana [Thu, 9 Dec 2021 14:44:06 +0000 (14:44 +0000)]
generic/335: explicitly fsync file foo when running on btrfs
The test is relying on the fact that an fsync on directory "a" will
result in persisting the changes of its subdirectory "b", namely the
rename of "/a/b/foo" to "/c/foo". For this particular filesystem layout,
that will happen on btrfs, because all the directory entries end up in
the same metadata leaf. However that is not a behaviour we can always
guarantee on btrfs. For example, if we add more files to directory
"a" before and after creating subdirectory "b", like this:
mkdir $SCRATCH_MNT/a
for ((i = 0; i < 1000; i++)); do
echo -n > $SCRATCH_MNT/a/file_$i
done
mkdir $SCRATCH_MNT/a/b
for ((i = 1000; i < 2000; i++)); do
echo -n > $SCRATCH_MNT/a/file_$i
done
mkdir $SCRATCH_MNT/c
touch $SCRATCH_MNT/a/b/foo
sync
# The rest of the test remains unchanged...
(...)
Then after fsyncing only directory "a", the rename of file "foo" from
"/a/b/foo" to "/c/foo" is not persisted.
Guaranteeing that on btrfs would be expensive on large directories, as
it would require scanning for every subdirectory. It's also not required
by posix for the fsync on a directory to persist changes inside its
subdirectories. So add an explicit fsync on file "foo" when the filesystem
being tested is btrfs.
Darrick J. Wong [Wed, 8 Dec 2021 16:45:18 +0000 (08:45 -0800)]
fsstress: consistently index the ops array by OP_ type
A mismerge during a git rebase some time ago broke fsstress in my
development tree, because it added OP_XCHGRANGE into the opt_y typedef
definition at a different offset than the actual entry in the ops array.
This broke the relationship ops[i].op == i.
Since most of fsstress.c blindly assumes that it's ok to index the ops
array by OP_ type, this off-by-one error meant that when I created an
fstest with "-f unlink=1", it actually set the frequency of the adjacent
operation (unresvsp) to 1. I didn't notice this until I started to
investigate how a filesystem created with "-z -f creat=4 -f unlink=4"
could end up with 1.8 million files after 30 seconds.
Eliminate the possibility for future screwups like this by using indexed
array initializers. This enables us to remove the separate op field in
struct opdesc, for a minor savings of memory footprint and reduction in
footgun opportunity.
While we're at it, reformat the ops table to be more pleasing to the
eye.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Reviewed-by: Eryu Guan <guaneryu@gmail.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
Luis Chamberlain [Mon, 15 Nov 2021 23:28:34 +0000 (15:28 -0800)]
common/module: fix patient module remover when module is not present
When module is not present and the open coded patient module
remover is called we'll end up in a loop which never ends.
Fix this.
I actually found this issue not in fstests, but when applying this
open coded solution to blktests. In fstest we tend to only call
module remove when we have a module loaded. blktests is different,
and so I immediately spotted the issue there.
Stefan Roesch [Fri, 3 Dec 2021 21:55:56 +0000 (13:55 -0800)]
btrfs: Add new test for setting the chunk size.
Add new testcase for testing the new btrfs sysfs knob to change the
chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block
type>/chunk_size.
The test case implements three different cases:
- Test allocation with the default chunk size
- Test allocation after increasing the chunk size
- Test allocation when the free space is smaller than the chunk size.
Note: this test needs to force the allocation of space. It uses the
/sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
Testing:
The test has been run with volumes of different sizes.
Signed-off-by: Stefan Roesch <shr@fb.com> Reviewed-by: Josef Bacik <josef@toxicpanda.com> Signed-off-by: Eryu Guan <guaneryu@gmail.com>
projects / xfstests-dev.git / log