fstests: update autoconf rules to define _GNU_SOURCE where needed

Starting from xfsprogs commit cb898f157f84 ("linux.h: Use off64_t
instead of loff_t") in v4.5-rc1, fstests failed to build because of
configure error:

    checking xfs/xfs.h usability... no
    checking xfs/xfs.h presence... yes
    configure: WARNING: xfs/xfs.h: present but cannot be compiled
    configure: WARNING: xfs/xfs.h:     check for missing prerequisite headers?
    configure: WARNING: xfs/xfs.h: see the Autoconf documentation
    configure: WARNING: xfs/xfs.h:     section "Present But Cannot Be Compiled"
    configure: WARNING: xfs/xfs.h: proceeding with the compiler's result
    checking for xfs/xfs.h... no

    FATAL ERROR: cannot find a valid <xfs/xfs.h> header file.
    Run "make install-qa" from the xfsprogs source.

and that's because off64_t is not defined without _GNU_SOURCE being set.

Fix it by updating the autoconf rules adding _GNU_SOURCE define for the
following checks:

xfs.h libxfs.h xfs_log_format.h xlog_assign_lsn xqm.h xfsctl

Suggested-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fstests: generic test for listing an inode's xattrs

Test that the filesystem's implementation of the listxattrs system call
lists all the xattrs an inode has.

This test is motivated by a bug found in btrfs, which is fixed by the
following patch for the linux kernel:

  "Btrfs: fix listxattrs not listing all xattrs packed in the same item"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common: filter warning messages from mkfs.ext4

Commit 4a5cda8102828 ("xfstests: do not unmount tmpfs during remount")
changed generic/135 to no longer redirect stderr to /dev/null when
running _scratch_mkfs.  This caused ext4 using a 1k block size to fail
when running this test because mkfs.ext4 would issue a warning message
about the use of a non-block size that wasn't expected by golden
output for the test.  Fix this by filtering out warning messages in
_scratch_mkfs_ext4.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

defrag: require extents support for ext4 defrag

The e4defrag program requires the use of fallocate, which in turn
means that the file system must have extents.  Enforce this
requirement so we don't get test failure noise when testing ext4's
ext3 compatibility.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/125: add _require_odirect

Also remove generic/125 from the auto group, and add it to the new
pnfs group.  This is to document where this test might be useful; it's
not really going to be useful for most normal on-disk file systems, so
remove it from the auto group.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/312: remove dependency on /proc/partitions

We don't need to do this test at all, since _scratch_mkfs_sized will
do this check for us.   This allows this test to work with tmpfs.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

gitignore: add src/test-nextquota

Commit 1dfb50585c09 (quota: test Q_GETNEXTQUOTA) added a new binary
without updating .gitignore.  Fix this.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fstests: btrfs/011 add debug logs to seqres.full

Signed-off-by: Anand Jain <anand.jain@oracle.com>

btrfs/011: use replace_options

This patch fixes test btrfs/011 which intended to use -r option
but was never used since its associated args 'replace_options'
didn't make it to the cli.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic: test file fsync after rename operation

Test that if we have a file F1 with two links, one in a directory A and
the other in directory B, if we remove the link in directory B, move some
other file F2 from directory B into directory C, fsync inode F1, power
fail and remount the filesystem, file F2 exists and is located only in
directory C.

This is motivated by a bug found in btrfs, which is fixed by the patch
(for the linux kernel) titled:

   "Btrfs: fix file loss on log replay after renaming a file and fsync"

Tested against ext3, ext4, xfs, f2fs and reiserfs.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic: test directory fsync after rename operation

Test that if we move one file between directories, fsync the parent
directory of the old directory, power fail and remount the filesystem,
the file is not lost and it's located at the destination directory.

This is motivated by a bug found in btrfs, which is fixed by the patch
(for the linux kernel) titled:

  "Btrfs: fix file loss on log replay after renaming a file and fsync"

Tested against ext3, ext4, xfs, f2fs and reiserfs.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common: fix _require_fs_space on remote file systems

Remove the -l flag to df so that it works properly on NFS and co.

This fixes various failures in new COW tests on NFS.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfs/24[356]: check for -c switch to xfs_io bmap command

Extend _require_xfs_io_command so that we can pass it a command line
argument to look for, and then use new capabililty in the relevant
tests to ensure that bmap knows how to dump CoW fork contents (bmap -c)

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

punch-alternating: use the block size reported by the fs for punching

When we're trying to punch alternating blocks out of a file, use the
bsize reported by fstatfs so that we can punch out single blocks.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/304: fix high offset

"Invalid argument" is a better response to an impossibly high offset
dedupe request than "extents don't match", so change the test.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

btrfs: test directory fsync after deleting snapshots

Test that if we fsync a directory that had a snapshot entry in it that
was deleted and crash, the next time we mount the filesystem, the log
replay procedure will not fail and the snapshot is not present anymore.

This issue is fixed by the following patch for the linux kernel:

  "Btrfs: fix unreplayable log after snapshot delete + parent dir fsync"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Tested-by: Liu Bo <bo.li.liu@oracle.com>
Reviewed-by: Liu Bo <bo.li.liu@oracle.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfs: new test default user/group quota

When default quota is set, all different quota types inherits the
same default value, include group quota. So if a user quota limit
larger than the default user quota value, it will still be limited
by the group default quota value.

An upstream patch for this bug:

  xfs: Split default quota limits by quota type

Signed-off-by: Zorro Lang <zlang@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfstests: remove _need_to_be_root

The check script requires that it be run as root, so adding
individualized checks for this in each teat is not needed.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfstests: increase tmpfs memory size

512M is not enough for generic/129.  Raise default tmpfs size to 1G.

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Junho Ryu <jayr@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfstests: add executable permission to tests

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/079,277: require chattr, not xattrs

Add a new helper, _require_chattr, which allows the test to explicitly
check to see if the file system supports a specific chattr flag, as
not all file systems support chattr +A or chattr +i, and the presence
of extended attribute support is has nothing to do with a specific
chattr flag being supported.

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/009: require fiemap

Require xfs_io commands fiemap and falloc as well as fzero: fzero
without falloc is unlikely, but tmpfs may later support fzero, though
probably never fiemap (and in v3.15 wrongly claimed to support fzero).

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/053: do not unmount before calling _check_scratch_fs()

Fix generic/053 so it works on tmpfs by relying on _check_scratch_fs
to unmount before checking the file system and remounting it
afterwards.  Many other tests rely on this, and since tmpfs does not
have a file system consistency checker, this allows the test to
succeed because the files don't disappear when the tmpfs file system
is unmounted.

Signed-off-by: Junho Ryu <jayr@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfstests: do not unmount tmpfs during remount

Several tests unmount then re-mount the scratch filesystem, to check
that the content is unchanged; but unmounting a tmpfs is designed to
lose its content, which causes such tests to fail unnecessarily.

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Junho Ryu <jayr@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

Rename _test_mount to _test_cycle_mount

This makes it clear when we are using "mount ; umount" versus "mount
-o remount" for most file systems.  The reason for this distinction is
(a) tests may want to test the difference between what happens on the
remount versus the munt paths, (b) with tmpfs, "mount ; umount" will
cause the contents of all of the files to disappear which makes many
tests sad, and (c) some mount options may not be changed using "mount
-o remount".

Currently _test_mount performs "_test_mount ; _test_umount"
so mechnically rename this function to _test_cycle_mount.  This was
done mechnically using the script fragment:

git grep -E "_test_remount" | \
awk -F: '{print $1}' | sort -u | grep -v tests/xfs/189 \
xargs sed -i 's/_test_remount/_test_cycle_mount/g'

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

Rename _scratch_mount to _scratch_cycle_mount

This makes it clear when we are using "mount ; umount" versus "mount
-o remount" for most file systems.  The reason for this distinction is
(a) tests may want to test the difference between what happens on the
remount versus the munt paths, (b) with tmpfs, "mount ; umount" will
cause the contents of all of the files to disappear which makes many
tests sad, and (c) some mount options may not be changed using "mount
-o remount".

Currently _scratch_mount performs "_scratch_mount ; _scratch_umount"
so mechnically rename this function to _scratch_cycle_mount.  This was
done mechnically using the script fragment:

git grep "_scratch_remount" | \
awk -F: '{print $1}' | sort -u | \
xargs sed -i 's/_scratch_remount/_scratch_cycle_mount/g'

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic: add _require_odirect to generic/113 and generic/214

generic/113 and generic/214 both use O_DIRECT at some stage in their
tests, so check O_DIRECT support before running them.

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic: use mount point instead of device name

A tmpfs mount does not involve any block device, its $SCRATCH_DEV is
nothing but a place-holder, so apply 'df' or 'stat' to its mount point
$SCRATCH_MNT instead of to $SCRATCH_DEV.

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Junho Ryu <jayr@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common: _scratch_mkfs_sized() for tmpfs

Enable _scratch_mkfs_sized() for use with tmpfs, so that tests which
use this helper can now run.

Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Junho Ryu <jayr@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

check: avoid spurious complaints that tests/$FSTYP/group does not exist

There are no tmpfs specific tests, so tests/tmpfs does not exist.
This commit avoids printing a spurious error message when running
specifying a group of tests (e.g., "check -g quick"):

  DEVICE: test:/tmp
  MK2FS OPTIONS:
  MOUNT OPTIONS: -o block_validity
  ./check: line 96: tests/tmpfs/group: No such file or directory
  FSTYP         -- tmpfs
  PLATFORM      -- Linux/i686 kvm-xfstests 4.5.0-rc2ext4-00002-g6df2762
  MKFS_OPTIONS  -- test:/scratch
  MOUNT_OPTIONS -- -o size=1G test:/scratch /test/scratch

  generic/001      [10:31:10][    5.811742] run fstests generic/001
      ...

Similar problems have been reported when testing nfs using xfstests.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

ext4/001: add output variant for nodelalloc mounts

If the delayed allocation is disabled, we need a slightly different
output for the delayed allocation portion of the tests.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Cc: Jan Kara <jack@suse.cz>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/158: test dedupe with destination offset past EOF

We were testing when the source file offset starts at EOF or beyond,
but not when the destination offset is beyond EOF or when the
destination offset is smaller than EOF but destination offset plus
dedup length is greater than EOF.

This is motivated by a bug in btrfs' extent_same (dedup) ioctl where
we allowed the destination offset to start at EOF and beyond (and
destination offset + length beyond EOF) for the case where the source
and destination files are the same (was not allowed for different
files used as source and destination). This also made the file's
metadata inconsistent when the dedup operation succeeded, which
happened when the source range corresponded to a file hole, prealloc
extent or a data extent filled with zeroes.

The btrfs issue is fixed by the following patch for the linux kernel:

  "Btrfs: fix extent_same allowing destination offset beyond i_size"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
[darrick.wong@oracle.com: fix merge conflicts with latest reflink patchbomb]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

ext4: test partial blocksize defrag integrity issue

Calling EXT4_IOC_MOVE_EXTENT on file not aligned with block size and
block size is smaller than page size would cause integrity issue on the
partial-blocksize part when copying data between orign file and donor
file.

This ext4 kernel patch would fix it, titled
"ext4: don't read blocks from disk after extents being swapped in
move_extent_per_page())"

Though this bug only happens in the blocksize smaller than pagesize
case, there's no harm to test on various block size fs, so no block size
is specified in the test, it depends on the test configurations.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfs: change return value check to golden image check

xfs/133 and xfs/138 use too much code to do "return value" check,
it's not necessary. For the code can be more readable and clear,
I change "return value" check to golden image check.

Signed-off-by: Zorro Lang <zlang@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/072: limit max cpu number to 8

Right now generic/072 scales the loop count based on the cpu count. But
on hosts with many cpus(100+), generic/072 runs for hours and generates
very high system load.

Given that the original bug can be reproduced easily on unpatched
kernel, the great number of loops and long run time are not needed. So
limiting the cpu number to 8 (which gives around 20 seconds run time on
my test vm with 8 vcpus) seems reasonable.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

aiocp: fix the gcc warnings

Eliminate a debug printf that was causing warnings and fix the other two
debug printfs to avoid tripping on return value warnings.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: kick the slow tests out of the 'quick' group

Since 'quick' tests are supposed to run in < 15s, kick out the ones
that can't finish that soon even on fast storage.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs: test source CoW across mixed block types with cowextsz set

Ensure that we can CoW the source file when the source file consists
of a range of mixed block types and there's a cowextsize hint set.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs: ensure that we don't crash when freeing the ag reservations on a ro mount

Christoph Hellwig discovered that the kernel crashed trying to free
the refcount btree per-ag reservation on a ro mount (because we don't
create the reservation except for rw mounts and ro->rw remounts).  So,
test this to make sure we never do that again. :)

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: test reflink+cow+enospc all at the same time

Set up an impossibly small filesystem and try to reflink and rewrite a
file on it to see what happens when we ENOSPC.  Basically
generic/16[67] but with a constrained fs size.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs: test xfs_getbmapx behavior with shared extents

Make sure that xfs_getbmapx behaves properly w.r.t. shared extents
and CoW fork reporting.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs: aio cow tests

Test what happens when AIO writes fail when we have a cowextsize hint
set on the files.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: test aio copy on write

Make sure that copy on write works with the AIO path.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs: test rmapbt functionality

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs: test the automatic cowextsize extent garbage collector

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: test CoW across a mixed range of block types with cowextsize set

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: test quota accounting

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: test xfs cow behavior when the filesystem crashes

Use the extent size hint to force leftover CoW reservations then
crash the filesystem to see how recovery works.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: high offset reflink and dedupe tests

Ensure that we can pass absurdly enormous offsets and lengths to
reflink/dedupe and it'll survive.

v2: Ask for dedupe in the dedupe test.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch@lst.de: call _require_test_dedupe]
Signed-off-by: Christoph Hellwig <hch@lst.de>

xfs: test fragmentation characteristics of copy-on-write

Perform copy-on-writes at random offsets to stress the CoW allocation
system.  Assess the effectiveness of the extent size hint at
combatting fragmentation via unshare, a rewrite, and no-op after the
random writes.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs/122: support refcount/rmap data structures

Include the refcount and rmap structures in the golden output.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfs: more reflink tests

Create a couple of XFS-specific tests -- one to check that growing
and shrinking the refcount btree works and a second one to check
what happens when we hit maximum refcount.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: ensure that we can handle reflinking a lot of extents

Update the existing stress tests to ensure that we can handle
reflinking the same block a million times, and that we can handle
reflinking million different extents.  Add a couple of tests to ensure
that we can ^C and SIGKILL our way out of long-running reflinks.

v2: Don't run the signal tests on NFS, as we cannot interrupt NFS
clone operations.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch@lst.de: don't run on NFS]
Signed-off-by: Christoph Hellwig <hch@lst.de>

reflink: test CoW operations against the source file

Ensure that CoW operations against shared blocks in the source file
work correctly.

v2: remove filefrag dependencies

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: test CoW behavior with IO errors

Test various scenarios (with dm-flakey) where we simulate write
failures during CoW, to see if the FS can get through it without
blowing up or corrupting data.  Plumb in a FS-generic method to
sort out repairing filesystems after they get hit by IO errors.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: test intersecting CoW and falloc/fpunch/fzero/fcollapse/finsert/ftrunc

Ensure that we correctly handle a CoW operation immediately followed
by a truncate, falloc, fpunch, fzero, fcollapse, and finsert operation
in the middle of the CoW'd region before any flush can occur.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

dio: unwritten conversion bug tests

Check that we don't expose old disk contents when a directio write to
an unwritten extent fails due to IO errors.  This primarily affects
XFS and ext4.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: add _require_odirect to the directio tests

For the tests that test O_DIRECT, we need to _require_odirect.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: fix style problems in existing tests

Fix style problems such as unnecessary use of quotes, add helper
variables to reduce visual clutter, and other minor fixes to make the
first batch of tests more closely resemble the second round tests.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: refactor mixed block creation code

Refactor the code that creates files with mixed block types that we feed
into CoW tests to make sure that we can tiptoe around that kind of stuff.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

common: provide a method to repair the scratch fs

Create a wrapper function that repairs any damage to the scratch
filesystem and returns a standard result.  We will use this to clean
up after IO error testing and other weird corruption tests.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

common: create _require_test_program to look for programs in src/

Create a helper that looks for a test program in src/ and fails the
test if it doesn't exist.  Refactor the existing testcases to use it.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

common/dmerror: add some more dmerror routines

Add functions to the dmerror routine so that we can load both the
error table and the linear table.  This will help us with EIO testing
of copy-on-write.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: remove unnecessary umounts from test code

The test harness already takes care of this, so get rid of it.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

reflink: remove redundant filesystem checks from the end of the tests

Turns out that check already runs _check_filesystems after each test,
so we don't need to do this at the end of each test.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfstests: make _scratch_mkfs_blocksized usable

The default mkfs.xfs options contain -b size=4096, so all tests
using _scratch_mkfs_blocksized won't actually run unless those
options are changed.  As we're trying to specificly test 1k
blocks we should always override the default option.

v2: Move the function to common/rc

Signed-off-by: Christoph Hellwig <hch@lst.de>
[darrick.wong@oracle.com: move function to common/rc]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

xfstests: filter whitespace in 128 and 132

Seems either I have a different lsattr version, or different mount points
cause differences in the golden output.  Send the lsattr output through
the whitespaces filter so that it works everywhere.

The lsattr output /does/ change depending on mountpoints.  Ick.  I'd
actually changed it to the long format output because line length in
the short format changes every time the flags change.

Signed-off-by: Christoph Hellwig <hch@lst.de>
[darrick.wong@oracle.com: update changelog]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>

generic/182: this is a dedupe test, check for dedupe

Since this test examines dedupe behavior, the documentation should
say 'dedupe', not 'reflink'.  Furthermore, the feature checks must
look for working dedupe functionality, not reflink functionality.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch@lst.de: add the test for dedupe support]
Signed-off-by: Christoph Hellwig <hch@lst.de>

Fix prerequisite packages to build fstests on Ubuntu

To build xfstests from minimal Ubuntu, it's necessary to install
some more packages in addition to ones mentioned in README.
Plus, libtool is not needed to specify here since libtool-bin
depends on it.

Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

aio-dio-eof-race should be ignored by git

src/aio-dio-regress/aio-dio-eof-race is a binary file built by make.
So it should not be tracked by git.

===============================================================================
$ make clean
...
$ git status
On branch sat-bugfixes
nothing to commit, working directory clean
$ make
...
$ git status
On branch sat-bugfixes
Untracked files:
  (use "git add <file>..." to include in what will be committed)

src/aio-dio-regress/aio-dio-eof-race
===============================================================================

Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fstests: btrfs, test for send with clone operations

Test that an incremental send operation which issues clone operations
works for files that have a full path containing more than one parent
directory component.

This used to fail before the following patch for the linux kernel:

  "[PATCH] Btrfs: send, fix extent buffer tree lock assertion failure"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

reflink: change to relative margins

NFS uses 32K blocks and thus the numbers used for sanity checking
various reflink operations are way off.  Change to relative margings
and increase them a bit to make NFS happy.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

reflink: also treat EINVAL as not supported

This can be returned by various implementations of the ioctl.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfs/139: fix scratch device filtering

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

tests: use lowercase variables for all the new reflink tests

Since $TESTDIR is a local variable, make it lowercase to avoid
confusion with $TEST_DIR.  While we're at it, make all the local
variables lowercase.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

dedupe: record dedupe errors in golden output

The xfs_io dedupe command originally didn't print dedupe errors
to stderr like you'd expect.  Since that was fixed, the golden
output should be changed to reflect that.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfs/122: update against xfsprogs 4.3

Make the xfsprogs sb layout and structure size checker work with
modern xfsprogs (i.e. 4.3+ and all the new v5 stuff).  The old test
wasn't picking up structures because they're no longer defined in the
"typedef struct xfs_foo { } xfs_foo_t;" style.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

reflink: fix off-by-one errors when iterating file blocks in a loop

When we're iterating file blocks in a loop (via seq), we have to
end at $nr-1, not $nr.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

tools/mvtest: call out to the correct group sort program

mvtest should call the actual name of the sort-group program.
Fix a cut-paste error in finding the destination group file.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

quota: test Q_GETNEXTQUOTA

The new Q_GETNEXTQUOTA quotactl (not yet merged) is designed
to take an ID as input ala Q_GETQUOTA, and return the quota
for the next active ID >= the input ID.  This lets us quickly
iterate over all existing quotas by leveraging the kernel's
knowledge of which quotas are allocated and active.

The test contains a new helper binary, test-nextquota, which
tests both the "vfs" and "xfs" versions of the quotactl.
It accepts an ID, and outputs the returned ID, ihard, and
isoft values for that quota.  It doesn't return block information
simply because that can vary depending on fs, block size, etc,
and we want something very consistent as output, for verifiation.

The test harness sets quotas for 100 random IDs, remounts,
and uses these quotactls to iterate over all the IDs we set,
using the test binary, making sure we get back what we expect.

Not the prettiest thing, but it works!

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fsx.c: Add header file stdbool.h

Compilation failed on RHEL6.7GA because stdbool.h is not included in
xfs/platform_defs-x86_64.h or xfs/linux.h on RHEL6.7GA, so fix it.

Signed-off-by: Xiao Yang <yangx.jy@cn.fujitsu.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

overlay: test copy up files from lower dir

Test copy up files from lower dir that are <, =, > 4G in size.

Kernel commit 0480334fa604 ("ovl: use O_LARGEFILE in ovl_copy_up()")
fixed bug when copying up large file(>4G) from lower dir.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common: export overlay lower/upper/work directories as variables

This turns hard coded lower/upper/work dir names to configurable
variables.

Suggested-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common: add overlayfs isupport in _scratch_cleanup_files()

All files in lower/upper dirs should be removed for overlayfs in
_scratch_cleanup_files(), not only files in merged dir, otherwise files
from lower dir won't be removed.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

btrfs: fix test failures after commit 27d077ec0bda

Commit 27d077ec0bda (common: use mount/umount helpers everywhere) made
a few btrfs tests fail (btrfs/003 and btrfs/011). These tests create
filesystems with multiple devices and test the device replace feature,
which need to unmount using the mount path ($SCRATCH_MNT) because
unmounting using one of the devices as an argument ($SCRATCH_DEV) does
not always work - after replace operations we get in /proc/mounts a
device other than $SCRATCH_DEV associated with the mount point
$SCRATCH_MNT (this is mentioned in a comment at btrfs/011 for example),
so we need to pass that other device to the umount program or pass it the
mount point.
Fix this by making _sctatch_unmount() pass $SCRATCH_MNT to umount instead
of $SCRATCH_DEV (when the filesystem being tested is btrfs).

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

new: create a default .out file

Creating a default .out file in "new" to ease the new test creation
process.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common: no need to parse config file in new

I saw "new" failed the TEST_DEV/SCRATCH_DEV validation when TEST_DEV or
SCRATCH_DEV points to directory, which is needed in overlayfs testing.

"new" only generates test template and doesn't need common/config to
parse and validate TEST_DEV/SCRATCH_DEV etc., so just returns after
setting all needed commands.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

generic/23[23]: some cleanups

1. remove $seqres.full before test
2. no need to check return values of _fsstress and _check_quota_usage
   calls, the golden image could catch the failures.
3. remove unused FSX related variables

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common: add names device mounted errors

This adds user friendly prompts to output the already mounted point
from _mount.  xfstests will do the cleanup (ie. umount) and user can
not get the mount name information when directory is already
mounted.

Signed-off-by: Jia He <hejianet@gmail.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fsync-tester: reopen files with correct flags

When "drop caches" tests are run, the test_fd is closed and
re-opened. It is reopened O_RDWR, so dropping O_DIRECT and any other
open flag that was set for test_fd. This fixes a random generic/311
failure due to file data corruption I've been seeing on subtests 11
and 13 when direct IO was specified.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

check: add exclude sections CLI parameter

When we have a config file with multiple sections, we might want to
exclude certain config sections from running. Rather than specifying
all the section we want to run, add a "-S <section>" option to build
up a list of sections to exclude.

This is useful if a given section config is known to cause a fatal
failure,but you still want to run all the other config sections.

Also add support to the setup program that emits the currently
configured setup for each section in the config file.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

setup: add section support

Make the setup command section aware so that it is easy to test
whether the section config code is generating the correct
configurations or not.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfs: support realtime/log device setup changes in config sections

Currently changing the devices used by "USE_EXTERNAL" environmental
variable is not supported by the config section parsing. Add the
functionality so that we can use config sections to test external
device configs successfully.

This required tracking down a bug in _check_xfs_filesystem() which
was causing a log device to be passed to a test device without an
external log device. This was caused by an uninitialised variable in
the function. I also added full output file removals to the first
couple of generic tests that were failing, because that's where the
check failure output ends up in this case.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fstests: canonicalize the mount points by removing the trailing "/"

removing all the trailing "/"s of mounting point name by readlink

Suggested-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Jia He <hejianet@gmail.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fstests: btrfs, verify fitrim does not discard reserved device area

Verify that when a fitrim operation is made against a btrfs filesystem,
the ranges [0, 64Kb[ and [68Kb, 1Mb[ of the device are not discarded,
they remain with the content they had before the fitrim operation. These
regions of the device are reserved for a boot loader to use at its will.

In the 4.3 linux kernel we got a regression that allowed a fitrim
operation to discard these reserved ranges of the device, resulting in
the filesystem becoming unbootable after a fitrim. The issue is fixed
by the following patch (targeted for 4.5 and 4.3/4.4 stable releases):

 "Btrfs: fix fitrim discarding device area reserved for boot loader's use"

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

xfs: test XFS torn log write detection

XFS torn log write detection includes a mechanism to inject CRC errors
into log records at runtime and shutdown the fs accordingly. This
ensures that the CRC verification pass on the subsequent mount discovers
an invalid record near the head of the log and considers it a torn
write.

This test runs a workload with error injection enabled and verifies that
the subsequent mount is successful. The test repeats for several
iterations using a random frequency factor for the error event each
time.

Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

f2fs: add Makefile for f2fs

Installation failed due to Makefile omission, so fix this.

Signed-off-by: Xiao Yang <yangx.jy@cn.fujitsu.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

shared: test mount needs_recovery extN partition with noload option

Once upon a time, ext4 encounter NULL pointer dereference under this
situaiton due to jump to a wrong label. Part of this commit fixed
this Oops:

744692d ext4: use ext4_get_block_write in buffer write

We can also run this test on ext2/3.

This case is based on a script from Monakhov Dmitriy @ openvz.

Cc: Monakhov Dmitriy <dmonakhov@openvz.org>
Signed-off-by: Xiong Zhou <xzhou@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

common/scsi_debug: fix removal of scsi_debug module

Occasionally scsi_debug cannot be removed because it's still in use and
causes xfs/279 to fail.

Now dryrun the removal by modprobe firstly then do the real rmmod.

Signed-off-by: Eryu Guan <eguan@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>

fstests: fix cleanup of test btrfs/003

If the test fails after removing a device and before adding it back, it
attempts to add back the device in its _cleanup() function. However this
is broken because the device identifier is stored in a variable local to
the function _test_replace() and not in a global variable. So make the
variable global instead of local.

Signed-off-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>