common: fix ObjBencher::aio_bench signature

It was broken by 1ac727982213bf676beefe9340d6822193dbb700

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4592 from ceph/wip-7746

rbd: add disk usage tool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4308 from dachary/wip-bencher

common: clarify some obj_bencher code

Reviewed-by: Dmitry Yatsushkevich <dmitry.yatsushkevich@gmail.com>

Merge pull request #4650 from stiopaa1/bufferIterator

buffer.h: Removed unnecessary copy constructor, assignment operator a…

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4668 from ceph/wip-rados-binary-omapsetval

rados cli: setomapval can now read from stdin

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4473 from ceph/wip-10809-11135-10290

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test: Add config changes to all tests to avoid order dependency

ReplayCorrupt was crashing because journal_ignore_corruption wasn't set true
Improve ReplayCorrupt by checking setting of bool corrupt and use ASSERT_FALSE()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add admin socket feature set_recovery_delay

Set the value of osd_recovery_delay_start to specified seconds

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: For import/export --debug dump the log

Move log dumping code to common function dump_log()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use ceph-objectstore-tool --no-overwrite for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --dry-run in select operations

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test missing fd.close()

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test error message

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: If object re-appears after removal, just skip it

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --no-overwrite flag for import-rados

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Remove list-lost because now we have --dry-run flag

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --dry-run option

Fixes: #10290
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-info command to show object info

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Use empty string for <object> to specify pgmeta object

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add a couple of strategically placed prints

Signed-off-by: John Spray <john.spray@redhat.com>

ceph-objectstore-tool: Clean up error handling

Use negative errors throughout and positive error to specify exit status
cpp_strerror() handles negative errors
Clean up main() code to use local "ret"
Let end of main() decide how to treat the final value of "ret"

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Create section around log/missing/divergent_priors of --op log

Remove extra new lines in the output

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add divergent_priors handling

Add to --op log output
Add to metadata_section for export
Add filtering of divergent_priors in case of splits

Fixes: #11237
Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool: Remove duplicate debug messages, keep cmd/log/call together

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph-objectstore-tool import after split testing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Use CEPH_DIR where appropriate

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Limit how long ceph-objectstore-tool test will wait for health

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add optional arg to vstart() to provide additional args to vstart

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test ceph-objectstore-tool --op dump-journal output

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Pep8 fixes for ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add --force option which is used for import only

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Fix pgid scan to skip snapdirs

Long standing bug where it wasn't skipping snapdirs if !debug
For debug output use stderr like all other cases

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add dump-journal op

Have dump-journal op directly dump a filestore journal without
mounting or try to dump after mounting.

Fixes: #11135
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal: Add _fdump() that takes Formatter instead of ostream

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix ceph-objectstore-tool test, overwrite OTHERFILE so second check is meaningful

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal: Conform to test infrastructure requirements

Only call RUN_ALL_TESTS() once

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal need to force aio because testing with a file

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal fix missing argument to FileJournal constructor

Signed-off-by: David Zafman <dzafman@redhat.com>

test: ceph_test_filejournal  Add check of journalq in WriteTrim test

Commit all journal entries and check for empty journalq

Signed-off-by: David Zafman <dzafman@redhat.com>

filestore: Change int open_journal() to void new_journal()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix log output in FileJournal::read_entry() coming from do_read_entry()

Signed-off-by: David Zafman <dzafman@redhat.com>

os: Add some logging used only by test code

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add simple_dump() to FileJournal for unit testing

Allow a dump when entries aren't ObjectStore::Transaction's

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal clean-up

Move aio / directio adjustments to constructor
Indicate functions which only operate before journal is active
Make functions const when possible
Move push_back() of journalq to read_entry()
Change dump() to use do_read_entry() to minimize side effects

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Dump header in FileJournal::dump()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: FileJournal::read_entry() can't use a zero seq to check for corruption

Seen when using dump() on a journal which has all entries committed

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix flushing in FileJournal::dump()

Use ostream arg since caller is specifying
Move flush after last section close

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: On any exit release CephContext so logging can flush

Add myexit() which will put() the g_ceph_context if present before exit
Must umount() once the objectstore has been mounted

Fixes: #10809
Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Check for keyvaluestore experimental feature

Don't list keyvaluestore as a valid type if feature not enabled

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Eliminate obscure "Invalid params" error

Will fail later with "No such file or directory" for an empty data-path
because we've already checked for vm.count("data-path")

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Check pgid validity earlier like we did before

This code was not doing the umount() even though it was moved after mount()

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-dencoder: Add ability to import from stdin by specifying "-"

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #4375 from tchaikov/wip-ceph-unrecognized-exception

ceph.in: do not throw on unknown errno

Reviewed-by: Loic Dachary <ldachary@redhat.com>

buffer.h: added some consts

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

buffer.h: Removed unnecessary copy constructor and assignment operator

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #4151 from dmitryya/issue_7401

obj_bencher: status_printer waits before the first calculation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4552 from stiopaa1/bufferLinkage

common/buffer.cc: change linkage of few variables to internal

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4663 from dachary/wip-11598-ceph-detect-init

ceph-detect-init: plain setup.py install on make install

Reviewed-by: Kernel Neophyte <neophyte.hacker001@gmail.com>

Merge pull request #4666 from SUSE/wip-11610-next

ceph.spec.in: include SUSE in _with_systemd

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

test/cli-integration/rbd: verify formatting for 'rbd disk-usage'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

ceph.in: do not throw on unknown errno

some of the errnos are not listed in errno.errorcode. if we happen
to run into them, print 'Unknown' instead.

Fixes: #11354
Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rados/test_rados_tool: added test for omapsetval via stdin

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: add disk usage tool

The new disk usage tool uses the new fast diff object map feature
(when enabled) to quickly calculate the provisioned vs actual usage
of images and associated snapshots within a pool.

Fixes: #7746
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: disable deep-flatten feature for the krbd fsx test

krbd doesn't support fast-diff or deep-flatten yet.  Had to do this
with two commits because fast-diff is in next and deep-flatten isn't.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rados cli: setomapval can now read from stdin

This will allow a user to store binary values within omap.  Since
RBD uses binary omap values, this will assist with certain recovery
operations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #4665 from ceph/wip-11551-next

tests: disable new RBD features for the krbd fsx test

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4644 from ceph/wip-11405

librbd: new QA client upgrade tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: disable fast-diff feature for the krbd fsx test

The kernel does not yet support fast-diff.

Fixes: #11551
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4627 from dachary/wip-install-deps-clean

install-deps.sh: only pip install after make clean

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: include SUSE in _with_systemd

The master specfile newly defines a _with_systemd variable that should be true
for the set of distros that are using systemd. Since this set of distros
includes SUSE/openSUSE (at least for the more recent versions where ceph is
supported), this commit sets _with_systemdto true on SUSE/openSUSE.

http://tracker.ceph.com/issues/11610 Fixes: #11610

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #4536 from ceph/wip-mds-snaps

Wip mds snaps

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-detect-init: plain setup.py install on make install

If no DESTDIR environment variable is set, just

  python setup.py install

instead of setting --root, --prefix or --install-layout. These options
are tested to do the right thing in the context of RPM and deb packaging
but they are presumably not what someone using make install manually
would want.

http://tracker.ceph.com/issues/11598 Fixes: #11598

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4646 from theanalyst/doc/release/v0.94.2

doc: more entries for release notes in v0.94.2

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: more entries for release notes in v0.94.2

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

install-deps.sh: only pip install after make clean

Do not re-install all python dependencies if there already exists a
wheelhouse directory. It makes it so running install-deps.sh twice will
only access the network the first time. The directories where the python
dependencies are installed are removed by make clean. Refreshing the
python dependencies cache can be done via make clean ; install-deps.sh

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4521 from tchaikov/wip-fix-malformed_input-macros

common: fix the macros for malformed_input::what()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

common: fix the macros for malformed_input::what()

the thrown exception of malformed_input should carry the function name in
which it was thrown. but what we have now is something like:
"buffer::malformed_input: __PRETTY_FUNCTION__ unknown encoding version >
100"
because __PRETTY_FUNCTION__ is not a macro any more. see
- https://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Function-Names.html
- https://gcc.gnu.org/onlinedocs/gcc/Function-Names.html
and it is not a string literal, so neither can we can concat it with
the literal err message.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4656 from ceph/wip-rbd-xfstests-201505

RBD: expunge test broken by xfs commit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4655 from ceph/wip-add-ceph-create-keys.rst

doc: add ceph-create-keys.rst to dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

RBD: expunge test broken by xfs commit

expunge generic/231, broken for disk and rbd by kernel commit 04162bb

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

doc: add ceph-create-keys.rst to dist tarball

Fixes: #10725
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4412 from tchaikov/wip-add-man-page

doc: add man page for ceph-create-keys

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4522 from ceph/wip-cephfs-reset-fixes

Fixes to resets of tables and journal with damaged metadata

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

tools: fix tabletool reset of nonexistent sessionmap

If the object didn't exist, the omap clear was failing
and preventing the subsueent omap set header from
executing.  Set the FAILOK flag on the omap clear
sub-operation.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: improve journaltool reset

Handle the case where no journal objects exist at all,
rather than just resetting a journal that already exists.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: expose static default file layout methods

So that tools can invoke these to get the defaults
without a fully baked MDCache.

Signed-off-by: John Spray <john.spray@redhat.com>

tools: fix tabletool reset snap

SnapServer has an encode method defined that
is different to encode_state, whereas in InoTable
the two were synonmous.  This code was working
previously for inotable but not for snapserver.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4652 from ceph/wip-11541

mds: clear CDir::STATE_REJOINUNDEF after fetching dirfrag

Reviewed-by: John Spray <john.spray@redhat.com>

mds: clear CDir::STATE_REJOINUNDEF after fetching dirfrag

Fixes: #11541
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #4545 from dachary/wip-mailmap

mailmap updates

mailmap: Jean-Charles Lopez affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Hervé Rousseau affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Rajesh Nambiar affiliation

Reviewed-by: Rajesh Nambiar <rajesh.n@msystechnologies.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Douglas Fuller affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Orit Wasserman affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Nathan Cutler affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Tim Serong affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Fabio Alessandro Locati affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mailmap: Henry Chang affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>