mds: use helper functions to access/modify SnapRealm::open

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove pruned parent snaprealm from snaprealm's open_past_parents

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: fix CInode::pop_projected_snaprealm()

CInode::pop_projected_snaprealm() was called after calling
CDentry::pop_projected_linkage(). So get_parent_inode()->find_snaprealm()
is not past parent snaprealm. The fix is set snaprealm->open to 'false'
and let SnapRealm::_open_parents() open the new past parent.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly open parents of snaprealm

make sure snaprealm's parents are opened before building
cached_snaps/cached_snap_context/cached_snap_trace

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: avoid calling StrayManager::eval_stray() recursively

StrayManager::eval_stray() may close dirfrags and unlink remote
dentries. These operations drop inode's reference, which may call
StrayManager::eval_stray() again.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly invalidate past child snaprealms

After updating an existing snapshot or deleting an old snapshot,
we should invalidate past child snaprealms' cached snap set and
snap context.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: unlink any stale remote snap dentry when evaluating stray inode

Remote snap dentry may hold reference to the stray directory inode,
the reference prevent the stray inode from being purged. After making
sure the stray directory inode is not referenced by any snapshot, we
know that all remote snap dentries should be stale, so we can unlink
them safely.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4703 from dachary/wip-make-check-verbose

tests: reduce make check verbosity

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: reduce make check verbosity

Move check-local scripts

   src/test/run-cli-tests
   encode-decode-non-regression.sh
   test/encoding/readable.sh

to check_SCRIPTS. Their output is captured in .log file when running
with a recent automake. This reduces the output of make check by an
order of magnitude.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4711 from dachary/wip-ceph-detect-init

ceph-detect-init typo

Reviewed-by: Michal Jarzabek <stiopa@gmail.com>

ceph-detect-init: fix pep8 extra space

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-detect-init: run-tox.sh always succeeds

Because of the | grep, the status of tox is no longer the status of
run-tox.sh and errors are not reported as they should.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4704 from dachary/wip-ceph-helpers

{mon,osd}-test-helpers.sh are deprecated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4706 from stiopaa1/cephdetectinit

ceph_detect_init: added linux mint

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_detect_init: added linux mint

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

tests: {mon,osd}-test-helpers.sh are deprecated

Remove the helpers because they are not used any longer. They have been
deprecated by ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh remove redundant setup in main

And add setup/teardown in tests that do not have it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/mon-handle-forward.sh uses ceph-helpers.sh

Use ceph-helpers.sh instead of  mon/mon-test-helpers.sh.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * use explicit ports 7300 and 7301 instead of +1 so that grep
    will show that 7301 is used. This reduces the odds of a
    port collision when looking for a port that's not already
    used by an existing test.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-pool-create.sh uses ceph-helpers.sh

Use ceph-helpers.sh instead of  mon/mon-test-helpers.sh.

  * The expect_false() helper is replaced by a ! statement which is
    simpler and has the same effect.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * trim the --public-addr argument from the run_mon call

  * add || return 1 after run_mon where missing

  * add || return 1 to all commands in TEST_no_pool_delete

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-bench.sh uses ceph-helpers.sh

Remove useless inclusion of osd/osd-test-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-config.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * trim the --public-addr argument from the run_mon call

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-copy-from.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/vstart_wrapper.sh uses ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/misc.sh uses ceph-helpers.sh

Fix grep that assume the pool has id 0, because it may be different
when rbd is redefined to have less PG by the run_mon function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/erasure-code/test-erasure-code.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

  * replace check_osd_status with wait_for_osd

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-crush.sh uses ceph-helpers.sh

It is just a conversion and the behavior is not changed. It involves:

  * calling run_mon from inside each test instead of doing it in run

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-erasure-code-profile.sh uses ceph-helpers.sh

Instead of using mon-test-helpers.sh, primarily because the kill_daemon
function implemented in mon-test-helpers.sh is not as good as
ceph-helpers.sh.

Instead of having tests that share the same monitor, each test now runs
on a fresh monitor. The test writer no longer has to worry that it will
be re-using the pool or profile from a previous test. That causes
problems that are difficult to diagnose and the overhead of running a
new monitor is not so high.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: kill mon-test-helpers.sh call_TEST_functions

Remove call_TEST_functions because it is too complex and rarely used.
Replace the two instances where it was used by a few lines doing the
same thing.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh do not hardcode id a in run_mon

Fix hardcoding of id a in the run_mon function. The directory
in which the mon data is stored must be a sub-directory of the
directory given in argument.

If mon_initial_members is set, the rbd pool cannot be redefined, which
is ok because this is rare and it's only an optimization to reduce the
number of PG.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh shows ceph report if not clean

Instead of just pg dump and health detail, the wait_for_clean
function displays ceph report when it times out, to help figure out what
caused it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh implement wait_for_osd

The wait_for_osd to wait for an osd to go up and down is needed
internally, after running an osd. Move the inline snippet from run_osd
into a function so that it can be used by scripts as well.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4436 from BCLibCoop/bclibcoop/rgw-content_length

rgw: improve content-length env var handling

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4662 from dachary/wip-11359-tier-tests

tests: more robust tiering agent tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4684 from xinxinsh/wip-fiemap-fix

do not return non-exist extents when doing sparse read

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4698 from ceph/wip-wn2-comment

watch/notify: Clarify comment where notifies are completed

watch/notify: Clarify comment where notifies are completed

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #4653 from ceph/wip-client-misc

Wip client misc

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4651 from dachary/wip-erasure-code-benchmark

erasure-code: add --erased option to ceph_erasure_code_benchmark

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4654 from dachary/wip-erasure-code-bench

erasure-code: bench.sh compares isa & jerasure, vandermonde & cauchy

Reviewed-by: Andreas Peters <Andreas.Joachim.Peters@cern.ch>

Merge pull request #4694 from ceph/wip-workflow-doc

doc: s/relase/release/

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: s/relase/release/

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4226 from ceph/wip-11276

#11276: make ceph-fuse well behaved with pool quotas

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4622 from xiaoxichen/wip-rocksdb-conf

Wip rocksdb conf

Reviewed-by: Sage Weil <sage@redhat.com>

do not return non-exist extents when doing sparse read

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #4606 from ceph/wip-rgw-init

init-radosgw: check in /var/lib/ceph/radosgw ; unify w/ sysv version

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

init-radosgw.sysv: remove

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: unify init-radosgw[.sysv]

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: look in /var/lib/ceph/radosgw

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4680 from ceph/wip-fix-group-in-doxygen

doc: fix group in doxygen

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: customize css for the group title

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: use @name to define a group, not @group

we are able to output a specified group using the directive
of `doxygengroup` in breathe. this directive prints out the
description of the group. but it's not realistic to enumerate
all groups defined in source code in the rst files. but the
doxygen command @name also helps to group functions together.
the downside of this approach is that we can not add more
items to a group later on. but it should be fine with us,
since in our case, all the grouped items are living in a single
header file.

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4671 from dachary/wip-contributing

doc: recommendations related to performances / cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4674 from flaf/master

Doc: add a note about a CRUSH corner case with `ceph osd out`

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4673 from dachary/wip-bencher

tools: remove obsolete aio_bench argument placeholder

Reviewed-by: Sage Weil <sage@redhat.com>

Doc: add a note about a CRUSH corner case with `ceph osd out`

Signed-off-by: François Lafont <francois.lafont@ac-versailles.fr>

tools: remove obsolete aio_bench argument placeholder

It was a noop and has been removed by
1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4672 from dachary/wip-bencher

common: fix ObjBencher::aio_bench signature

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix ObjBencher::aio_bench signature

It was broken by 1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4592 from ceph/wip-7746

rbd: add disk usage tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4308 from dachary/wip-bencher

common: clarify some obj_bencher code

Reviewed-by: Dmitry Yatsushkevich <dmitry.yatsushkevich@gmail.com>

Merge pull request #4650 from stiopaa1/bufferIterator

buffer.h: Removed unnecessary copy constructor, assignment operator a…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: recommendations related to performances / cleanup

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4668 from ceph/wip-rados-binary-omapsetval

rados cli: setomapval can now read from stdin

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4473 from ceph/wip-10809-11135-10290

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test: Add config changes to all tests to avoid order dependency

ReplayCorrupt was crashing because journal_ignore_corruption wasn't set true
Improve ReplayCorrupt by checking setting of bool corrupt and use ASSERT_FALSE()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add admin socket feature set_recovery_delay

Set the value of osd_recovery_delay_start to specified seconds

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: For import/export --debug dump the log

Move log dumping code to common function dump_log()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use ceph-objectstore-tool --no-overwrite for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --dry-run in select operations

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test missing fd.close()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test error message

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: If object re-appears after removal, just skip it

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --no-overwrite flag for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Remove list-lost because now we have --dry-run flag

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --dry-run option

Fixes: #10290
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-info command to show object info

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Use empty string for <object> to specify pgmeta object

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add a couple of strategically placed prints

Signed-off-by: John Spray <john.spray@redhat.com>

ceph-objectstore-tool: Clean up error handling

Use negative errors throughout and positive error to specify exit status
cpp_strerror() handles negative errors
Clean up main() code to use local "ret"
Let end of main() decide how to treat the final value of "ret"

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Create section around log/missing/divergent_priors of --op log

Remove extra new lines in the output

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add divergent_priors handling

Add to --op log output
Add to metadata_section for export
Add filtering of divergent_priors in case of splits

Fixes: #11237
Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool: Remove duplicate debug messages, keep cmd/log/call together

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool import after split testing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use CEPH_DIR where appropriate

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Limit how long ceph-objectstore-tool test will wait for health

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add optional arg to vstart() to provide additional args to vstart

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --op dump-journal output

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Pep8 fixes for ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --force option which is used for import only

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Fix pgid scan to skip snapdirs

Long standing bug where it wasn't skipping snapdirs if !debug
For debug output use stderr like all other cases

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-journal op

Have dump-journal op directly dump a filestore journal without
mounting or try to dump after mounting.

Fixes: #11135
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal: Add _fdump() that takes Formatter instead of ostream

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test, overwrite OTHERFILE so second check is meaningful

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal: Conform to test infrastructure requirements

Only call RUN_ALL_TESTS() once

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal need to force aio because testing with a file

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal fix missing argument to FileJournal constructor

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal  Add check of journalq in WriteTrim test

Commit all journal entries and check for empty journalq

Signed-off-by: David Zafman <dzafman@redhat.com>

filestore: Change int open_journal() to void new_journal()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log output in FileJournal::read_entry() coming from do_read_entry()

Signed-off-by: David Zafman <dzafman@redhat.com>

os: Add some logging used only by test code

Signed-off-by: David Zafman <dzafman@redhat.com>