Merge pull request #2288 from xiaoxichen/fix_add_simple_ruleset

CrushWrapper: pick a ruleset same as rule_id

Reviewed-by: Loic Dachary <loic-201408@dachary.org>

CrushWrapper: pick a ruleset same as rule_id

Originally in the add_simple_ruleset funtion, the ruleset_id
is not reused but rule_id is reused. So after some add/remove
against rules, the newly created rule likely to have
ruleset!=rule_id.

We dont want this happen because we are trying to hold the constraint
that ruleset == rule_id.

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #2273 from dachary/wip-mailmap

mailmap updates

Merge remote-tracking branch 'gh/wip-filejournal'

Reviewed-by: Sage Weil <sage@redhat.com>
Tested-by: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>

Merge remote-tracking branch 'gh/wip-lttng'

Revert "Merge pull request #2253 from adamcrume/wip-lttng"

This reverts commit 5b9ade5058f72116eb3dc920f442361c2a33b46b, reversing
changes made to b0aa846b3f81225a779de00100e15334fb8156b3.

I merged the wrong version of wip-lttng.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2274 from ceph/wip-9144

filestore: fix force_sync race with mount + umount + mount

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #2253 from adamcrume/wip-lttng

RBD-replay

Reviewed-by: Sage Weil <sage@redhat.com>

rbd_replay: Add unit test for batch_unreachable_from

This requires a fair amount of code reorganization, since the types in
the function signature were previously not in a header file.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add unit test for Ser

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Split --with-lttng and --with-babeltrace options

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove tracing-specfic local variables when lttng disabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add informational messages to configure.ac

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Only install rbd-replay-prep when enabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Check for babeltrace when autoconfiguring lttng

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Check that the main babeltrace header is present

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Test for sane lttng-gen-tp

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Don't build trace libraries when lttng is disabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix build error on 32-bit systems

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove -lboost_regex from unit test

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Don't build trace headers when lttng is disabled

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Update packaging

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Only require lttng for Fedora >= 20 and RHEL == 6 for RPM distros

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add distro packaging

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix "make distcheck"

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Support --with-lttng=check

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove tracing from libcommon

This is a short-term fix for issues caused by tracepoints in libcommon.
Code crashes at runtime if the same tracepoints are linked into the
program multiple times.  This happens with libcommon because it is
statically linked into dynamic libraries such as librados, then
statically linked into executables because symbols from libcommon are
not exposed in librados.  Therefore, any programs that use librados and
libcommon would crash because of duplicate tracepoints.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix bug in rbd-replay-prep, Thread::issued_io with wrong IO

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Remove extent tracepoints and inline extents

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Move tracepoints from internal.cc to librbd.cc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add man page and package binaries for rbd-replay-prep

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Convert prep-for-replay.py to rbd-replay-prep.cc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add man page, and add binaries to Debian packaging

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Remove outdated message that says regexes are used

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Replace Boost dependencies with custom string code

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Remove boost_regex from LIBADD and LDADD

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix compiler warning in unit tests

Was getting:
test/test_rbd_replay.cc:44:3: warning: converting ‘false’ to pointer type for argument 1 of ‘char testing::internal::IsNullLiteralHelper(testing::internal::Secret*)’ [-Wconversion-null]

Fixed by changing EXPECT_EQ(false, xxx) to EXPECT_FALSE(xxx).
For completeness, also changed EXPECT_EQ(true, xxx) to EXPECT_TRUE(xxx).

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Switch ImageNameMap from regex to plain string matching

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Switch ImageNameMap to using rbd_loc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add rbd_replay::rbd_loc

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Use standard image@snap instead of image/snap

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix ceph_ctf_stringp

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Support replaying partial traces

Tracing may start after the application is started, and image open calls
may missed. To support replaying these traces, additional information is
traced, allowing missing open calls to be generated.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Use non-zero fake data

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add unit tests

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Preload liblttng-ust-fork.so in TESTS_ENVIRONMENT

This adds LD_PRELOAD=liblttng-ust-fork.so to TESTS_ENVIRONMENT.
This prevents lttng from complaining when processes are forked.
The complaints otherwise taint the output and cause tests to fail.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Disable LTTng by default, add --with-lttng configure option

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Fix 'make check' by setting up fake HOME

If the HOME variable is not set, LTTng complains, and the extra output
fails to match what the tests expect, so they fail.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --map-image option for image name mapping

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --pool option

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add assertions

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Default to not storing buffer content

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add command-line arguments to prep-for-replay.py

Includes:
--print-on-read
--print-on-write
--window

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix async writes in prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --read-only flag

Removes ignoreWrites flag in prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix examples/rbd-replay/replay

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Clean up prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Hash completions across multiple mutexes to reduce contention

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --latency-multiplier option to usage statement

Signed-off-by: Adam Crume <adamcrume@gmail.com>

doc: Add usage comment to librbd::RBD::AioCompletion

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix memory leak in PendingIO

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Check that return values are non-negative

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Support writing Actions to ostreams

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Print "Still waiting for" more judiciously

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Change Worker::m_pending_ios from vector to map

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Printing pending IOs when worker threads waiting to die

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Bugfix: ensure PendingIOs are added before they're removed

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add rbd_replay subsystem

Signed-off-by: Adam Crume <adamcrume@gmail.com>

Add assertf macro

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Switch logging from cout to dout

To enable logs, we also have to use global_init to parse our
command-line args, so we now have other standard Ceph goodies
such as picking up config options from the environment.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add rbd-replay

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add more tracing to librbd

Includes:
- aio_complete
- aio_discard
- aio_flush
- aio_get_return_value
- aio_is_complete
- aio_read
- aio_wait_for_complete
- aio_write
- copy
- discard
- flush
- invalidate_cache
- rename
- resize
- Add ImageCtx pointer to trace points
- Add snap_name and readonly flag to trace point open_image_enter

Signed-off-by: Adam Crume <adamcrume@gmail.com>

tracing: instrument filestore

This adds objectstore tracepoints for the filestore. It'd be nice to add
these to the objectstore interface some how so we can get all
implementations for free, but that might just be a bit difficult
especially since each impl will apply transactions in a differnet way.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

lttng: Basic tracing of librbd

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove 'ver' from trace in code for CEPH_OSD_OP_NOTIFY

'ver' is obsolete and variable exists only for proper deserialization

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Split up libtracepoints

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Prevent deletion of tracing-common.h when cleaning

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Prevent creation of incorrect .o files when generating from .tp

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Trace librados C API

Signed-off-by: Adam Crume <adamcrume@gmail.com>

examples: Add example for the librados C API

Simply copied and modified the C++ API example.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add tracing-common.h

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Trace ReplicatedPG::do_osd_ops

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add rmw_flags to tracepoint in PG::queue_op

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Trace OpRequest

Signed-off-by: Adam Crume <adamcrume@gmail.com>

tracing: automake-ify tracepoint generation

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

lttng: Check for lttng/tracepoint.h when configuring

lttng: add pg and osd tracepoints

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

lttng: trace mutex::unlock

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

tracing: bootstrap lttng-ust with mutex events

See src/tracing/README.md

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

os/FileJournal: For journal-aio-mode, don't use aio when closing journal.

For jouranl-aio-mode when closing journal, the write_finish_thread_entry may exit before
write_thread_entry. This cause no one wait last aios to complete.
On some platform, after that the journal-header on journal corrupted.
To avoid this, when closing jouranl we don't use aio.

Fixes: 9073
Reported-by: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
Tested-by: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>

Merge pull request #2295 from dachary/wip-9153-jerasure-upgrade

erasure-code: do not preload the isa plugin

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: do not preload the isa plugin

Because it's not built for all architectures and distributions.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2219 from somnathr/wip-optracker-lock

TrackedOp: Removed redundant lock in OpTracker::_mark_event()

Reviewed-by: Sage Weil <sage@redhat.com>

TrackedOp:_dump_op_descriptor is renamed to _dump_op_descriptor_unlocked

Caller don't need to hold lock before calling _dump_op_descriptor(),so,
to reflect this it is renamed to _dump_op_descriptor_unlocked().

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

TrackedOp: Removed redundant lock in OpTracker::_mark_event()

ops_in_flight_lock seems redundant in OpTracker::_mark_event()
and this lock is highly contended for. Removing the same
is giving a significant performance boost.

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

Merge pull request #2282 from dachary/wip-9153-jerasure-upgrade

erasure-code: preload the jerasure plugin

Reviewed-by: Sage Weil <sage@redhat.com>

os/FileJournal: signal aio_cond even if seq is 0

This can happen if we write a journal but no events.

Reported-by: Somnath Roy <somnath.roy@sandisk.com>
Reported-by: Ma, Jianpeng <jianpeng.ma@intel.com>
Signed-off-by: Sage Weil <sage@redhat.com>

os/FileJournal: Update the journal header when closing journal

When closing journal, it should check must_write_header and update
journal header if must_write_header alreay set.
It can reduce the nosense journal-replay after restarting osd.

Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc/start/quick-ceph-deploy: missing {ceph-node} from mon create-initial

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #2283 from somnathr/wip-sd-9145

CollectionIndex: Collection name is added to the access_lock name

Reviewed-by: Samuel Just <sam.just@inktank.com>