Fix ceph_scratchtoolpp to create and delete its own pool

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool: Bug fixes and test improvements

ceph_objectgstore_tool:
Fix bugs in the way collection_list_partial() was being called
which caused objects to be seen over and over again.

Unit test:
Fix get_objs() to walk pg tree for pg with sub-directories
Create more objects to test object listing code
Limit number of larger objects
Limit number of objects which get attributes and omaps

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph_objectstore_tool, test: Implement import-rados feature and unit test code

Fixes: #8276
Signed-off-by: David Zafman <david.zafman@inktank.com>

tools/rados: Improve trigger for op_size rounding info message

Signed-off-by: David Zafman <david.zafman@inktank.com>

test: ceph_objectstore_tool unit test improvements

Add namespaces to testing
Increase filesize so export will have multiple chunks
Put json for each object into the db dict

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_objectstore_tool: Add operation "rm-past-intervals" for testing purposes

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_objectstore_tool: Add past_intervals to export/import code

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_objectstore_tool: Minor improvements

Make all non-error non-debug output to stdout
Fix a message

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_objectstore_tool, test: Add list-pgs operations and unit test case

Signed-off-by: David Zafman <david.zafman@inktank.com>

Complete replacement of ceph_filestore_tool and ceph_filestore_dump
with unified ceph_objectstore_tool

Move list-lost-objects and fix-lost-objects features from
  ceph_filestore_tool to ceph_objectstore_tool as list-lost, fix-lost
Change --type to --op for info, log, export...operations
Add --type for the ObjectStore type (defaults to filestore)
Change --filestore-path to --data-path
Update installation, Makefile.am, and .gitignore
Fix and rename test case to match
  Add some additional invalid option checks

Signed-off-by: David Zafman <david.zafman@inktank.com>

Renames and removal towards a unified ceph_objectstore_tool

Rename ceph_filestore_dump.cc and ceph_filestore_dump.py
Remove ceph_filestore_tool.cc

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump.py: Replace ceph_filestore_dump.sh with better python version

Signed-off-by: David Zafman <david.zafman@inktank.com>

test: ceph_filestore_dump.sh test improvements

Add some usage error tests
Don't use the same var in second for loop
Add xattr/omap to rep pool and xattr to ec pool
Add list, get-bytes and set-bytes testing
Add list-attrs and get-attr

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add set-omaphdr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add get-omaphdr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add rm-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add set-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add get-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add rm-attr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add set-attr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add get-attr object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add set-bytes object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add get-bytes object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add list-omap object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add list-attrs object command

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add --type list to output objects in a pg in json

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add remove object command and usage for new commands

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add utility function get_fd_data()

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Fail import/export with a tty for stdin/stdout

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filstore_dump: Save if stdout is a tty and add routine to clean binary strings

Signed-off-by: David Zafman <david.zafman@inktank.com>

common: Add missing ghobject_t::decode() for json

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Add --skip-journal-replay and --skip-mount-omap

Signed-off-by: David Zafman <david.zafman@inktank.com>

os: Add optional flags to generic ObjectStore creation (SKIP_JOURNAL_REPLAY
and SKIP_MOUNT_OMAP)

Only FileStore cares about these flags, so passed on during create()

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Improve debug output by showing actual offset

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Use cerr now that we aren't closing stderr

Signed-off-by: David Zafman <david.zafman@inktank.com>

common,ceph_filestore_dump: Add ability for utilities to suppress library dout output

Suppress dout output with CODE_ENVIRONMENT_UTILITY_NODOUT
ceph_filestore_dump turns on dout output if --debug specified
When used it can still be enable with --log-to-stderr --err-to-stderr

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Export omap in batches for large omap case

New function get_omap_batch()
Create a TYPE_OMAP section for each batch

Signed-off-by: David Zafman <david.zafman@inktank.com>

ceph_filestore_dump: Remove unused bufferlist databl

Signed-off-by: David Zafman <david.zafman@inktank.com>

osd: make coll_t::META static to each file

Signed-off-by: Adam Crume <adamcrume@gmail.com>

Merge pull request #2273 from dachary/wip-mailmap

mailmap updates

Merge remote-tracking branch 'gh/wip-filejournal'

Reviewed-by: Sage Weil <sage@redhat.com>
Tested-by: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>

Merge remote-tracking branch 'gh/wip-lttng'

Revert "Merge pull request #2253 from adamcrume/wip-lttng"

This reverts commit 5b9ade5058f72116eb3dc920f442361c2a33b46b, reversing
changes made to b0aa846b3f81225a779de00100e15334fb8156b3.

I merged the wrong version of wip-lttng.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2274 from ceph/wip-9144

filestore: fix force_sync race with mount + umount + mount

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #2253 from adamcrume/wip-lttng

RBD-replay

Reviewed-by: Sage Weil <sage@redhat.com>

rbd_replay: Add unit test for batch_unreachable_from

This requires a fair amount of code reorganization, since the types in
the function signature were previously not in a header file.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add unit test for Ser

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Split --with-lttng and --with-babeltrace options

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove tracing-specfic local variables when lttng disabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add informational messages to configure.ac

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Only install rbd-replay-prep when enabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Check for babeltrace when autoconfiguring lttng

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Check that the main babeltrace header is present

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Test for sane lttng-gen-tp

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Don't build trace libraries when lttng is disabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix build error on 32-bit systems

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove -lboost_regex from unit test

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Don't build trace headers when lttng is disabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Update packaging

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Only require lttng for Fedora >= 20 and RHEL == 6 for RPM distros

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add distro packaging

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix "make distcheck"

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Support --with-lttng=check

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove tracing from libcommon

This is a short-term fix for issues caused by tracepoints in libcommon.
Code crashes at runtime if the same tracepoints are linked into the
program multiple times.  This happens with libcommon because it is
statically linked into dynamic libraries such as librados, then
statically linked into executables because symbols from libcommon are
not exposed in librados.  Therefore, any programs that use librados and
libcommon would crash because of duplicate tracepoints.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix bug in rbd-replay-prep, Thread::issued_io with wrong IO

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Remove extent tracepoints and inline extents

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Move tracepoints from internal.cc to librbd.cc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add man page and package binaries for rbd-replay-prep

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Convert prep-for-replay.py to rbd-replay-prep.cc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add man page, and add binaries to Debian packaging

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Remove outdated message that says regexes are used

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Replace Boost dependencies with custom string code

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Remove boost_regex from LIBADD and LDADD

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix compiler warning in unit tests

Was getting:
test/test_rbd_replay.cc:44:3: warning: converting ‘false’ to pointer type for argument 1 of ‘char testing::internal::IsNullLiteralHelper(testing::internal::Secret*)’ [-Wconversion-null]

Fixed by changing EXPECT_EQ(false, xxx) to EXPECT_FALSE(xxx).
For completeness, also changed EXPECT_EQ(true, xxx) to EXPECT_TRUE(xxx).

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Switch ImageNameMap from regex to plain string matching

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Switch ImageNameMap to using rbd_loc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add rbd_replay::rbd_loc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Use standard image@snap instead of image/snap

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix ceph_ctf_stringp

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Support replaying partial traces

Tracing may start after the application is started, and image open calls
may missed. To support replaying these traces, additional information is
traced, allowing missing open calls to be generated.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Use non-zero fake data

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add unit tests

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Preload liblttng-ust-fork.so in TESTS_ENVIRONMENT

This adds LD_PRELOAD=liblttng-ust-fork.so to TESTS_ENVIRONMENT.
This prevents lttng from complaining when processes are forked.
The complaints otherwise taint the output and cause tests to fail.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Disable LTTng by default, add --with-lttng configure option

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix 'make check' by setting up fake HOME

If the HOME variable is not set, LTTng complains, and the extra output
fails to match what the tests expect, so they fail.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --map-image option for image name mapping

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --pool option

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add assertions

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Default to not storing buffer content

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add command-line arguments to prep-for-replay.py

Includes:
--print-on-read
--print-on-write
--window

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix async writes in prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --read-only flag

Removes ignoreWrites flag in prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix examples/rbd-replay/replay

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Clean up prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Hash completions across multiple mutexes to reduce contention

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --latency-multiplier option to usage statement

Signed-off-by: Adam Crume <adamcrume@gmail.com>

doc: Add usage comment to librbd::RBD::AioCompletion

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix memory leak in PendingIO

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Check that return values are non-negative

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Support writing Actions to ostreams

Signed-off-by: Adam Crume <adamcrume@gmail.com>