tests: reduce make check verbosity

Move check-local scripts

   src/test/run-cli-tests
   encode-decode-non-regression.sh
   test/encoding/readable.sh

to check_SCRIPTS. Their output is captured in .log file when running
with a recent automake. This reduces the output of make check by an
order of magnitude.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4711 from dachary/wip-ceph-detect-init

ceph-detect-init typo

Reviewed-by: Michal Jarzabek <stiopa@gmail.com>

ceph-detect-init: fix pep8 extra space

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-detect-init: run-tox.sh always succeeds

Because of the | grep, the status of tox is no longer the status of
run-tox.sh and errors are not reported as they should.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4704 from dachary/wip-ceph-helpers

{mon,osd}-test-helpers.sh are deprecated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4706 from stiopaa1/cephdetectinit

ceph_detect_init: added linux mint

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_detect_init: added linux mint

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

tests: {mon,osd}-test-helpers.sh are deprecated

Remove the helpers because they are not used any longer. They have been
deprecated by ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh remove redundant setup in main

And add setup/teardown in tests that do not have it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/mon-handle-forward.sh uses ceph-helpers.sh

Use ceph-helpers.sh instead of  mon/mon-test-helpers.sh.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * use explicit ports 7300 and 7301 instead of +1 so that grep
    will show that 7301 is used. This reduces the odds of a
    port collision when looking for a port that's not already
    used by an existing test.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-pool-create.sh uses ceph-helpers.sh

Use ceph-helpers.sh instead of  mon/mon-test-helpers.sh.

  * The expect_false() helper is replaced by a ! statement which is
    simpler and has the same effect.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * trim the --public-addr argument from the run_mon call

  * add || return 1 after run_mon where missing

  * add || return 1 to all commands in TEST_no_pool_delete

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-bench.sh uses ceph-helpers.sh

Remove useless inclusion of osd/osd-test-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-config.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * trim the --public-addr argument from the run_mon call

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-copy-from.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/vstart_wrapper.sh uses ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/misc.sh uses ceph-helpers.sh

Fix grep that assume the pool has id 0, because it may be different
when rbd is redefined to have less PG by the run_mon function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/erasure-code/test-erasure-code.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

  * replace check_osd_status with wait_for_osd

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-crush.sh uses ceph-helpers.sh

It is just a conversion and the behavior is not changed. It involves:

  * calling run_mon from inside each test instead of doing it in run

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-erasure-code-profile.sh uses ceph-helpers.sh

Instead of using mon-test-helpers.sh, primarily because the kill_daemon
function implemented in mon-test-helpers.sh is not as good as
ceph-helpers.sh.

Instead of having tests that share the same monitor, each test now runs
on a fresh monitor. The test writer no longer has to worry that it will
be re-using the pool or profile from a previous test. That causes
problems that are difficult to diagnose and the overhead of running a
new monitor is not so high.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: kill mon-test-helpers.sh call_TEST_functions

Remove call_TEST_functions because it is too complex and rarely used.
Replace the two instances where it was used by a few lines doing the
same thing.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh do not hardcode id a in run_mon

Fix hardcoding of id a in the run_mon function. The directory
in which the mon data is stored must be a sub-directory of the
directory given in argument.

If mon_initial_members is set, the rbd pool cannot be redefined, which
is ok because this is rare and it's only an optimization to reduce the
number of PG.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh shows ceph report if not clean

Instead of just pg dump and health detail, the wait_for_clean
function displays ceph report when it times out, to help figure out what
caused it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh implement wait_for_osd

The wait_for_osd to wait for an osd to go up and down is needed
internally, after running an osd. Move the inline snippet from run_osd
into a function so that it can be used by scripts as well.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4436 from BCLibCoop/bclibcoop/rgw-content_length

rgw: improve content-length env var handling

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4662 from dachary/wip-11359-tier-tests

tests: more robust tiering agent tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4684 from xinxinsh/wip-fiemap-fix

do not return non-exist extents when doing sparse read

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4698 from ceph/wip-wn2-comment

watch/notify: Clarify comment where notifies are completed

watch/notify: Clarify comment where notifies are completed

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #4653 from ceph/wip-client-misc

Wip client misc

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4651 from dachary/wip-erasure-code-benchmark

erasure-code: add --erased option to ceph_erasure_code_benchmark

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4654 from dachary/wip-erasure-code-bench

erasure-code: bench.sh compares isa & jerasure, vandermonde & cauchy

Reviewed-by: Andreas Peters <Andreas.Joachim.Peters@cern.ch>

Merge pull request #4694 from ceph/wip-workflow-doc

doc: s/relase/release/

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: s/relase/release/

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4226 from ceph/wip-11276

#11276: make ceph-fuse well behaved with pool quotas

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4622 from xiaoxichen/wip-rocksdb-conf

Wip rocksdb conf

Reviewed-by: Sage Weil <sage@redhat.com>

do not return non-exist extents when doing sparse read

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #4606 from ceph/wip-rgw-init

init-radosgw: check in /var/lib/ceph/radosgw ; unify w/ sysv version

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

init-radosgw.sysv: remove

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: unify init-radosgw[.sysv]

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: look in /var/lib/ceph/radosgw

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4680 from ceph/wip-fix-group-in-doxygen

doc: fix group in doxygen

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: customize css for the group title

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: use @name to define a group, not @group

we are able to output a specified group using the directive
of `doxygengroup` in breathe. this directive prints out the
description of the group. but it's not realistic to enumerate
all groups defined in source code in the rst files. but the
doxygen command @name also helps to group functions together.
the downside of this approach is that we can not add more
items to a group later on. but it should be fine with us,
since in our case, all the grouped items are living in a single
header file.

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4671 from dachary/wip-contributing

doc: recommendations related to performances / cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4674 from flaf/master

Doc: add a note about a CRUSH corner case with `ceph osd out`

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4673 from dachary/wip-bencher

tools: remove obsolete aio_bench argument placeholder

Reviewed-by: Sage Weil <sage@redhat.com>

Doc: add a note about a CRUSH corner case with `ceph osd out`

Signed-off-by: François Lafont <francois.lafont@ac-versailles.fr>

tools: remove obsolete aio_bench argument placeholder

It was a noop and has been removed by
1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4672 from dachary/wip-bencher

common: fix ObjBencher::aio_bench signature

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix ObjBencher::aio_bench signature

It was broken by 1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4592 from ceph/wip-7746

rbd: add disk usage tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4308 from dachary/wip-bencher

common: clarify some obj_bencher code

Reviewed-by: Dmitry Yatsushkevich <dmitry.yatsushkevich@gmail.com>

Merge pull request #4650 from stiopaa1/bufferIterator

buffer.h: Removed unnecessary copy constructor, assignment operator a…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: recommendations related to performances / cleanup

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4668 from ceph/wip-rados-binary-omapsetval

rados cli: setomapval can now read from stdin

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4473 from ceph/wip-10809-11135-10290

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test: Add config changes to all tests to avoid order dependency

ReplayCorrupt was crashing because journal_ignore_corruption wasn't set true
Improve ReplayCorrupt by checking setting of bool corrupt and use ASSERT_FALSE()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add admin socket feature set_recovery_delay

Set the value of osd_recovery_delay_start to specified seconds

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: For import/export --debug dump the log

Move log dumping code to common function dump_log()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use ceph-objectstore-tool --no-overwrite for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --dry-run in select operations

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test missing fd.close()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test error message

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: If object re-appears after removal, just skip it

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --no-overwrite flag for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Remove list-lost because now we have --dry-run flag

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --dry-run option

Fixes: #10290
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-info command to show object info

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Use empty string for <object> to specify pgmeta object

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add a couple of strategically placed prints

Signed-off-by: John Spray <john.spray@redhat.com>

ceph-objectstore-tool: Clean up error handling

Use negative errors throughout and positive error to specify exit status
cpp_strerror() handles negative errors
Clean up main() code to use local "ret"
Let end of main() decide how to treat the final value of "ret"

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Create section around log/missing/divergent_priors of --op log

Remove extra new lines in the output

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add divergent_priors handling

Add to --op log output
Add to metadata_section for export
Add filtering of divergent_priors in case of splits

Fixes: #11237
Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool: Remove duplicate debug messages, keep cmd/log/call together

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool import after split testing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use CEPH_DIR where appropriate

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Limit how long ceph-objectstore-tool test will wait for health

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add optional arg to vstart() to provide additional args to vstart

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --op dump-journal output

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Pep8 fixes for ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --force option which is used for import only

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Fix pgid scan to skip snapdirs

Long standing bug where it wasn't skipping snapdirs if !debug
For debug output use stderr like all other cases

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-journal op

Have dump-journal op directly dump a filestore journal without
mounting or try to dump after mounting.

Fixes: #11135
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal: Add _fdump() that takes Formatter instead of ostream

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test, overwrite OTHERFILE so second check is meaningful

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal: Conform to test infrastructure requirements

Only call RUN_ALL_TESTS() once

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal need to force aio because testing with a file

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal fix missing argument to FileJournal constructor

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal  Add check of journalq in WriteTrim test

Commit all journal entries and check for empty journalq

Signed-off-by: David Zafman <dzafman@redhat.com>

filestore: Change int open_journal() to void new_journal()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log output in FileJournal::read_entry() coming from do_read_entry()

Signed-off-by: David Zafman <dzafman@redhat.com>

os: Add some logging used only by test code

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add simple_dump() to FileJournal for unit testing

Allow a dump when entries aren't ObjectStore::Transaction's

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal clean-up

Move aio / directio adjustments to constructor
Indicate functions which only operate before journal is active
Make functions const when possible
Move push_back() of journalq to read_entry()
Change dump() to use do_read_entry() to minimize side effects

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Dump header in FileJournal::dump()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal::read_entry() can't use a zero seq to check for corruption

Seen when using dump() on a journal which has all entries committed

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix flushing in FileJournal::dump()

Use ostream arg since caller is specifying
Move flush after last section close

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: On any exit release CephContext so logging can flush

Add myexit() which will put() the g_ceph_context if present before exit
Must umount() once the objectstore has been mounted

Fixes: #10809
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Check for keyvaluestore experimental feature

Don't list keyvaluestore as a valid type if feature not enabled

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Eliminate obscure "Invalid params" error

Will fail later with "No such file or directory" for an empty data-path
because we've already checked for vm.count("data-path")

Signed-off-by: David Zafman <dzafman@redhat.com>