Merge pull request #4517 from ceph/wip-11388-debian-argparse

#11388 debian: move ceph_argparse into ceph-common

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #4721 from ceph/wip-fix-concurrent.sh

Fix ceph.conf path in concurrent.sh - krbd qa suite

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

debian: move ceph_argparse into ceph-common

Prior to this commit, if a user installed the "ceph-common" Debian
package without installing "ceph", then /usr/bin/ceph would crash
because it was missing the ceph_argparse library.

Ship the ceph_argparse library in "ceph-common" instead of "ceph". (This
was the intention of the original commit that moved argparse to "ceph",
2a23eac54957e596d99985bb9e187a668251a9ec)

http://tracker.ceph.com/issues/11388 Refs: #11388

Reported-by: Jens Rosenboom <j.rosenboom@x-ion.de>
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4720 from athanatos/wip-clarify-DBObjectMap-sync

DBObjectMap::sync: add comment clarifying locking

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3946 from tchaikov/randomize-scrub-time

osd: Randomize scrub time

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd: use another name for randomize scrub option

s/osd_scrub_interval_limit/osd_scrub_interval_randomize_ratio/

Fixes: #10973
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: randomize scrub times to avoid scrub wave

- to avoid the scrub wave when the osd_scrub_max_interval reaches in a
  high-load OSD, the scrub time is randomized.
- extract scrub_load_below_threshold() out of scrub_should_schedule()
- schedule an automatic scrub job at a time which is uniformly distributed
  over [now+osd_scrub_min_interval,
        now+osd_scrub_min_interval*(1+osd_scrub_time_limit]. before
  this change this sort of scrubs will be performed once the hard interval
  is end or system load is below the threshold, but with this change, the
  jobs will be performed as long as the load is low or the interval of
  the scheduled scrubs is longer than conf.osd_scrub_max_interval. all
  automatic jobs should be performed in the configured time period, otherwise
  they are postponed.
- the requested scrub job will be scheduled right away, before this change
  it is queued with the timestamp of `now` and postponed after
  osd_scrub_min_interval.

Fixes: #10973
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: use __func__ in log messages

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: simplify OSD::scrub_load_below_threshold() a little bit

avoid unnecessary comparison

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4691 from varadakari/wip-kvs-objheader

KeyValueStore: optimize the object header writes

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

qa: unbreak concurrent.sh workunit

Signed-off-by: Vasu Kulkarni <vasu@redhat.com>

Merge pull request #4658 from ceph/wip-11481

#11481: MDS resilience to weird mdsmaps

Merge pull request #4722 from ceph/wip-rbd-xfstests-20150518

rbd: expunge xfstests generic/078

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: expunged xfstests generic/078

This tests RENAME_WHITEOUT, which was enabled for xfs in kernel commit
7dcf5c3e4527cfa2807567b00387cf2ed5e07f00. At first execution, it throws a BUG.
Subsequent executions appear to work correctly. This issue manifests for disks
and RBD instances.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #4705 from stiopaa1/exit

cryptic error message in ceph interactive mode

Reviewed-by: David Zafman <dzafman@redhat.com>

DBObjectMap::sync: add comment clarifying locking

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #4715 from ceph/wip-11641

mds: fix handling missing mydir dirfrag

mds: fix handling missing mydir dirfrag

This was broken by 96992466 aka "mds: handle missing mydir dirfrag"

The previous code was mistakenly treating a not-yet-loaded
dirfrag as a non-existent dirfrag, resulting in
inconsistent fragstats even when no objects had
actually been lost.

Fixes: #11641
Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4693 from varadakari/wip-kvdb-prefix

KeyValueStore: Fix the prefix comparion to avoid object leaks.

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4692 from varadakari/wip-kvs-iterator

wip-kvs-iterator

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4703 from dachary/wip-make-check-verbose

tests: reduce make check verbosity

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: reduce make check verbosity

Move check-local scripts

   src/test/run-cli-tests
   encode-decode-non-regression.sh
   test/encoding/readable.sh

to check_SCRIPTS. Their output is captured in .log file when running
with a recent automake. This reduces the output of make check by an
order of magnitude.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4711 from dachary/wip-ceph-detect-init

ceph-detect-init typo

Reviewed-by: Michal Jarzabek <stiopa@gmail.com>

ceph-detect-init: fix pep8 extra space

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph-detect-init: run-tox.sh always succeeds

Because of the | grep, the status of tox is no longer the status of
run-tox.sh and errors are not reported as they should.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4704 from dachary/wip-ceph-helpers

{mon,osd}-test-helpers.sh are deprecated

Reviewed-by: Kefu Chai <kchai@redhat.com>

cryptic error message in ceph interactive mode

Fixes: #11459
Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #4706 from stiopaa1/cephdetectinit

ceph_detect_init: added linux mint

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph_detect_init: added linux mint

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

tests: {mon,osd}-test-helpers.sh are deprecated

Remove the helpers because they are not used any longer. They have been
deprecated by ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh remove redundant setup in main

And add setup/teardown in tests that do not have it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/mon-handle-forward.sh uses ceph-helpers.sh

Use ceph-helpers.sh instead of  mon/mon-test-helpers.sh.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * use explicit ports 7300 and 7301 instead of +1 so that grep
    will show that 7301 is used. This reduces the odds of a
    port collision when looking for a port that's not already
    used by an existing test.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-pool-create.sh uses ceph-helpers.sh

Use ceph-helpers.sh instead of  mon/mon-test-helpers.sh.

  * The expect_false() helper is replaced by a ! statement which is
    simpler and has the same effect.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * trim the --public-addr argument from the run_mon call

  * add || return 1 after run_mon where missing

  * add || return 1 to all commands in TEST_no_pool_delete

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-bench.sh uses ceph-helpers.sh

Remove useless inclusion of osd/osd-test-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-config.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

  * trim the --public-addr argument from the run_mon call

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/osd/osd-copy-from.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/vstart_wrapper.sh uses ceph-helpers.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/misc.sh uses ceph-helpers.sh

Fix grep that assume the pool has id 0, because it may be different
when rbd is redefined to have less PG by the run_mon function.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/erasure-code/test-erasure-code.sh uses ceph-helpers.sh

The ceph-helpers.sh was already included, along with
mon/mon-test-helpers.sh and osd/osd-test-helpers.sh which creates
confusion.

  * replace check_osd_status with wait_for_osd

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-crush.sh uses ceph-helpers.sh

It is just a conversion and the behavior is not changed. It involves:

  * calling run_mon from inside each test instead of doing it in run

  * modifying the .asok and .log names to match the ceph-helpers.sh
    conventions

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: test/mon/osd-erasure-code-profile.sh uses ceph-helpers.sh

Instead of using mon-test-helpers.sh, primarily because the kill_daemon
function implemented in mon-test-helpers.sh is not as good as
ceph-helpers.sh.

Instead of having tests that share the same monitor, each test now runs
on a fresh monitor. The test writer no longer has to worry that it will
be re-using the pool or profile from a previous test. That causes
problems that are difficult to diagnose and the overhead of running a
new monitor is not so high.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: kill mon-test-helpers.sh call_TEST_functions

Remove call_TEST_functions because it is too complex and rarely used.
Replace the two instances where it was used by a few lines doing the
same thing.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh do not hardcode id a in run_mon

Fix hardcoding of id a in the run_mon function. The directory
in which the mon data is stored must be a sub-directory of the
directory given in argument.

If mon_initial_members is set, the rbd pool cannot be redefined, which
is ok because this is rare and it's only an optimization to reduce the
number of PG.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh shows ceph report if not clean

Instead of just pg dump and health detail, the wait_for_clean
function displays ceph report when it times out, to help figure out what
caused it.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-helpers.sh implement wait_for_osd

The wait_for_osd to wait for an osd to go up and down is needed
internally, after running an osd. Move the inline snippet from run_osd
into a function so that it can be used by scripts as well.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4436 from BCLibCoop/bclibcoop/rgw-content_length

rgw: improve content-length env var handling

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4662 from dachary/wip-11359-tier-tests

tests: more robust tiering agent tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4684 from xinxinsh/wip-fiemap-fix

do not return non-exist extents when doing sparse read

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4698 from ceph/wip-wn2-comment

watch/notify: Clarify comment where notifies are completed

watch/notify: Clarify comment where notifies are completed

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

Merge pull request #4653 from ceph/wip-client-misc

Wip client misc

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4651 from dachary/wip-erasure-code-benchmark

erasure-code: add --erased option to ceph_erasure_code_benchmark

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4654 from dachary/wip-erasure-code-bench

erasure-code: bench.sh compares isa & jerasure, vandermonde & cauchy

Reviewed-by: Andreas Peters <Andreas.Joachim.Peters@cern.ch>

Merge pull request #4694 from ceph/wip-workflow-doc

doc: s/relase/release/

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: fix handle_mds_map in standby_replay

Broken by "mds: validate the state+rank in MDS map"

Signed-off-by: John Spray <john.spray@redhat.com>

doc: s/relase/release/

Signed-off-by: Loic Dachary <ldachary@redhat.com>

KeyValueStore: Fix the prefix comparion to avoid object leaks.

Iterator becomes invalid due to a partial prefix comparision in
rmkeys_by_prefix, resulting in not deleting the objects from backend.
Modified the comparision to the given prefix.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

KeyValueStore: Initialize the iterator

Initialize the iterator in get_keys and get_keys_with_header.
This fixes the issues with ceph_test_rados_api_aio.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

KeyValueStore: optimize the object header writes

Don't write the object header when any strip is overwritten.
write header when new strip is added or any header fields are modified.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>

Merge pull request #4226 from ceph/wip-11276

#11276: make ceph-fuse well behaved with pool quotas

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4622 from xiaoxichen/wip-rocksdb-conf

Wip rocksdb conf

Reviewed-by: Sage Weil <sage@redhat.com>

do not return non-exist extents when doing sparse read

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #4606 from ceph/wip-rgw-init

init-radosgw: check in /var/lib/ceph/radosgw ; unify w/ sysv version

Reviewed-by: Travis Rhoden <trhoden@redhat.com>

init-radosgw.sysv: remove

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: unify init-radosgw[.sysv]

Signed-off-by: Sage Weil <sage@redhat.com>

init-radosgw: look in /var/lib/ceph/radosgw

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4680 from ceph/wip-fix-group-in-doxygen

doc: fix group in doxygen

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mds: validate the state+rank in MDS map

Especially:
 * once I have been assigned a rank, it
can't be taken away without restarting
the daemon.
 * once I have entered standby, I can
only go upwards through the states.

Fixes: #11481
Signed-off-by: John Spray <john.spray@redhat.com>

mds: respawn instead of suicide on blacklist

This was already the case in general, but the case
in RecoveryQueue slipped through.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: handle DNE beacon from MDS

...by calling fail_mds_gid for that MDS.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: on suicide(), send a DNE beacon to MDSMonitor

...using the same timeout beacon send routine that
was created for damaged().

Signed-off-by: John Spray <john.spray@redhat.com>

doc: customize css for the group title

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: use @name to define a group, not @group

we are able to output a specified group using the directive
of `doxygengroup` in breathe. this directive prints out the
description of the group. but it's not realistic to enumerate
all groups defined in source code in the rst files. but the
doxygen command @name also helps to group functions together.
the downside of this approach is that we can not add more
items to a group later on. but it should be fine with us,
since in our case, all the grouped items are living in a single
header file.

Fixes: #9442
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4671 from dachary/wip-contributing

doc: recommendations related to performances / cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4674 from flaf/master

Doc: add a note about a CRUSH corner case with `ceph osd out`

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4673 from dachary/wip-bencher

tools: remove obsolete aio_bench argument placeholder

Reviewed-by: Sage Weil <sage@redhat.com>

Doc: add a note about a CRUSH corner case with `ceph osd out`

Signed-off-by: François Lafont <francois.lafont@ac-versailles.fr>

tools: remove obsolete aio_bench argument placeholder

It was a noop and has been removed by
1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4672 from dachary/wip-bencher

common: fix ObjBencher::aio_bench signature

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix ObjBencher::aio_bench signature

It was broken by 1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4592 from ceph/wip-7746

rbd: add disk usage tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4308 from dachary/wip-bencher

common: clarify some obj_bencher code

Reviewed-by: Dmitry Yatsushkevich <dmitry.yatsushkevich@gmail.com>

Merge pull request #4650 from stiopaa1/bufferIterator

buffer.h: Removed unnecessary copy constructor, assignment operator a…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: recommendations related to performances / cleanup

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4668 from ceph/wip-rados-binary-omapsetval

rados cli: setomapval can now read from stdin

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4473 from ceph/wip-10809-11135-10290

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test: Add config changes to all tests to avoid order dependency

ReplayCorrupt was crashing because journal_ignore_corruption wasn't set true
Improve ReplayCorrupt by checking setting of bool corrupt and use ASSERT_FALSE()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add admin socket feature set_recovery_delay

Set the value of osd_recovery_delay_start to specified seconds

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: For import/export --debug dump the log

Move log dumping code to common function dump_log()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use ceph-objectstore-tool --no-overwrite for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --dry-run in select operations

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test missing fd.close()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test error message

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: If object re-appears after removal, just skip it

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --no-overwrite flag for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Remove list-lost because now we have --dry-run flag

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --dry-run option

Fixes: #10290
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-info command to show object info

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Use empty string for <object> to specify pgmeta object

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add a couple of strategically placed prints

Signed-off-by: John Spray <john.spray@redhat.com>