rbd-replay: Add assertions

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Default to not storing buffer content

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add command-line arguments to prep-for-replay.py

Includes:
--print-on-read
--print-on-write
--window

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix async writes in prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --read-only flag

Removes ignoreWrites flag in prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix examples/rbd-replay/replay

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Clean up prep-for-replay.py

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Hash completions across multiple mutexes to reduce contention

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add --latency-multiplier option to usage statement

Signed-off-by: Adam Crume <adamcrume@gmail.com>

doc: Add usage comment to librbd::RBD::AioCompletion

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Fix memory leak in PendingIO

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Check that return values are non-negative

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Support writing Actions to ostreams

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Print "Still waiting for" more judiciously

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Change Worker::m_pending_ios from vector to map

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Printing pending IOs when worker threads waiting to die

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Bugfix: ensure PendingIOs are added before they're removed

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Add rbd_replay subsystem

Signed-off-by: Adam Crume <adamcrume@gmail.com>

Add assertf macro

Signed-off-by: Adam Crume <adamcrume@gmail.com>

rbd-replay: Switch logging from cout to dout

To enable logs, we also have to use global_init to parse our
command-line args, so we now have other standard Ceph goodies
such as picking up config options from the environment.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add rbd-replay

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add more tracing to librbd

Includes:
- aio_complete
- aio_discard
- aio_flush
- aio_get_return_value
- aio_is_complete
- aio_read
- aio_wait_for_complete
- aio_write
- copy
- discard
- flush
- invalidate_cache
- rename
- resize
- Add ImageCtx pointer to trace points
- Add snap_name and readonly flag to trace point open_image_enter

Signed-off-by: Adam Crume <adamcrume@gmail.com>

tracing: instrument filestore

This adds objectstore tracepoints for the filestore. It'd be nice to add
these to the objectstore interface some how so we can get all
implementations for free, but that might just be a bit difficult
especially since each impl will apply transactions in a differnet way.

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

lttng: Basic tracing of librbd

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Remove 'ver' from trace in code for CEPH_OSD_OP_NOTIFY

'ver' is obsolete and variable exists only for proper deserialization

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Split up libtracepoints

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Prevent deletion of tracing-common.h when cleaning

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Prevent creation of incorrect .o files when generating from .tp

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Trace librados C API

Signed-off-by: Adam Crume <adamcrume@gmail.com>

examples: Add example for the librados C API

Simply copied and modified the C++ API example.

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add tracing-common.h

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Trace ReplicatedPG::do_osd_ops

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Add rmw_flags to tracepoint in PG::queue_op

Signed-off-by: Adam Crume <adamcrume@gmail.com>

lttng: Trace OpRequest

Signed-off-by: Adam Crume <adamcrume@gmail.com>

tracing: automake-ify tracepoint generation

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

lttng: Check for lttng/tracepoint.h when configuring

lttng: add pg and osd tracepoints

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

lttng: trace mutex::unlock

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

tracing: bootstrap lttng-ust with mutex events

See src/tracing/README.md

Signed-off-by: Noah Watkins <noahwatkins@gmail.com>

Merge pull request #2295 from dachary/wip-9153-jerasure-upgrade

erasure-code: do not preload the isa plugin

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: do not preload the isa plugin

Because it's not built for all architectures and distributions.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2219 from somnathr/wip-optracker-lock

TrackedOp: Removed redundant lock in OpTracker::_mark_event()

Reviewed-by: Sage Weil <sage@redhat.com>

TrackedOp:_dump_op_descriptor is renamed to _dump_op_descriptor_unlocked

Caller don't need to hold lock before calling _dump_op_descriptor(),so,
to reflect this it is renamed to _dump_op_descriptor_unlocked().

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

TrackedOp: Removed redundant lock in OpTracker::_mark_event()

ops_in_flight_lock seems redundant in OpTracker::_mark_event()
and this lock is highly contended for. Removing the same
is giving a significant performance boost.

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>

Merge pull request #2282 from dachary/wip-9153-jerasure-upgrade

erasure-code: preload the jerasure plugin

Reviewed-by: Sage Weil <sage@redhat.com>

doc/start/quick-ceph-deploy: missing {ceph-node} from mon create-initial

Signed-off-by: Dan Mick <dan.mick@inktank.com>

Merge pull request #2283 from somnathr/wip-sd-9145

CollectionIndex: Collection name is added to the access_lock name

Reviewed-by: Samuel Just <sam.just@inktank.com>

CollectionIndex: Collection name is added to the access_lock name

The CollectionIndex constructor is changed to accept the coll_t
so that the collection name can be used to form access_lock(RWLock)
name.This is needed otherwise lockdep will report a recursive lock error
and assert. lockdep needs unique lock names for each Index object.

Fixes: #9145
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com>

erasure-code: preload the jerasure plugin

Load the jerasure plugin when ceph-osd starts to avoid the following
scenario:

* ceph-osd-v1 is running but did not load jerasure

* ceph-osd-v2 is installed being installed but takes time : the files
  are installed before ceph-osd is restarted

* ceph-osd-v1 is required to handle an erasure coded placement group and
  loads jerasure (the v2 version which is not API compatible)

* ceph-osd-v1 calls the v2 jerasure plugin and does not reference the
  expected part of the code and crashes

Although this problem shows in the context of teuthology, it is unlikely
to happen on a real cluster because it involves upgrading immediately
after installing and running an OSD. Once it is backported to firefly,
it will not even happen in teuthology tests because the upgrade from
firefly to master will use the firefly version including this fix.

While it would be possible to walk the plugin directory and preload
whatever it contains, that would not work for plugins such as jerasure
that load other plugins depending on the CPU features, or even plugins
such as isa which only work on specific CPU.

http://tracker.ceph.com/issues/9153 Fixes: #9153

Backport: firefly
Signed-off-by: Loic Dachary <loic-201408@dachary.org>

Merge pull request #2043 from guangyy/wip-pg-splitting

Support 'expected_num_objects' parameter when creating pool for pg folder splitting

Reviewed-by: Samuel Just <sam.just@inktank.com>

mon: fix signed/unsigned warnings

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2287 from ceph/wip-reweight-tunables

mon: make reweight-by-* sanity limits configurable

Reviewed-by: Greg Farnum <greg@inktank.com>

Merge pull request #2279 from ceph/wip-hadoop

fix and reorg hadoop workunits

Reviewed-by: Greg Farnum <greg@inktank.com>

mon: make reweight-by-* sanity limits configurable

Also drop the somewhat redundant osd_sum.kb check; the main thing we care
about here is

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2199 from ceph/wip-reweight

mon: allow reweighting of osds by pg (isntead of bytes used)

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>

mon/OSDMonitor: respect CRUSH weights for reweight-by-pg

Do not assume that all OSDs are weighted equally for reweight-by-pg.

Note that reweight-by-utilization already reweights based on the size of
the OSD volume; we presume that this is already reflected by the CRUSH
weights.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: reweight-by-pg for pool(s)

Allow the reweight-by-pg to look at a specific set of pools.  If the list
is ommitted, use PGs from all pools.  This allows you to focus on a
specific pool (the one that will dominate data usage).  Otherwise things
may not be quite right because other pools may have PGs that contain
much less data.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: adjust weights up, when possible

Note when OSDs are underloaded, as well.  If that is the case, adjust the
OSD reweight value if, if possible.  (It won't always be possible since
weights are capped at 1.)

Note that we set the underload threshold to the average, as we want to
aggressively adjust weights up (back to 1.0) whenever possible.  This gets
us a more efficient mapping calculation and reduces the amount of "noise"
in the weights.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: test reweight-by-pg

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: reweight-by-pg

This is just like reweight-by-utilization, but looks purely at the PG to
OSD mapping, not at the number of bytes used on the target disks.  This
allows the reweighting to be done before any data is written into the
cluster, when no data will need to migrate as a result of the reweight.

Signed-off-by: Sage Weil <sage@redhat.com>

Add tests for the collection hint OP: 1) Store Test 2) Idempotent Test.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Implement the collection hint transaction, add a new transation type as expected number of objects.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Add a new transaction OP (collection hint) to ObjectStore.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Add a new monitor command to let user specify the expected number of objects during pool creation.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Add a new field 'expected_num_objects' to pg_pool_t which denotes the expected number of objects on this pool.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Merge remote-tracking branch 'gh/next'

doc: Removed quick guide and wireshark from top-level IA.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc: Move wireshark documentation to dev.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>

doc/release-notes: v0.84

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2280 from ceph/wip-fs-docs

doc: add notes on using "ceph fs new"

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add notes on using "ceph fs new"

Signed-off-by: John Spray <john.spray@redhat.com>

0.84

qa/workunits/rbd/qemu-iotests: touch common.env

This seems to be necessary on trusty.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 055be68cf8e1b84287ab3631a02e89a9f3ae6cca)

qa/workunits/hadoop: move all hadoop tests into a hadoop/ dir

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/hadoop-wordcount: fix/use -rmr command

-rm -r -f ... doesn't seem to work; use -rmr instead.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/hadoop-wordcount: use -x

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd/qemu-iotests: touch common.env

This seems to be necessary on trusty.

Backport: firefly, dumpling
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2010 from ceph/wip-misplaced

osd: track misplaced objects separately from degraded objects

Reviewed-by: Samuel Just <sam.just@inktank.com>

qa/workunits/rest/test.py: use rbd instead of data pool for size tests

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rest/test.py: do snap test on our data2/3 pool

This way it works when a 'data' pool doesn't already exist.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rest/test.py: fix rd_kb -> rd_bytes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2272 from ceph/wip-8621

Wip 8621

Reviewed-by: Sage Weil <sage@redhat.com>

osd: fix theoretical use-after-free of OSDMap

In practice, the map will remain pinned for a while, but this
will make coverity happy.

*** CID 1231685:  Use after free  (USE_AFTER_FREE)
/osd/OSD.cc: 6223 in OSD::handle_osd_map(MOSDMap *)()
6217
6218           if (o->test_flag(CEPH_OSDMAP_FULL))
6219            last_marked_full = e;
6220           pinned_maps.push_back(add_map(o));
6221
6222           bufferlist fbl;
>>>     CID 1231685:  Use after free  (USE_AFTER_FREE)
>>>     Calling "encode" dereferences freed pointer "o".
6223           o->encode(fbl);
6224
6225           hobject_t fulloid = get_osdmap_pobject_name(e);
6226           t.write(coll_t::META_COLL, fulloid, 0, fbl.length(), fbl);
6227           pin_map_bl(e, fbl);
6228           continue;

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2259 from ceph/wip-9039

Wip 9039

Reviewed-by: Sage Weil <sage@redhat.com>

vstart.sh: make filestore fd cache size smaller

I hit the fd limit on a vstart cluster with the default 128; reduce this
to 16.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: track stuck undersized

Signed-off-by: Sage Weil <sage@redhat.com>

mon: track pgs that get stuck degraded

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track last_fullsized in pg_stat_t

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track last_undegraded pg stat

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: add last_undegraded, last_undersized to pg_stat_t

Keep track of the last time the PG was known to not be degraded or
undersized.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: track PG_STATE_UNDERSIZED separately from DEGRADED

DEGRADED means there are objects without complete reduncancy; also check
for needs_recovery().

UNDERSIZED means acting set is too small.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: add PG_STATE_UNDERSIZED

This is a distinct concept from degraded.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: account for misplaces separately than degraded

A degraded object does not have enough replicas or shards, while a
misplaced object is not stored in the correct place.  Account for them
separately.

Signed-off-by: Sage Weil <sage@redhat.com>

librados: approximate legacy 'degraded' value

The librados API returns a degraded count and no misplaced count.  Sum them
to approximate the old behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

mon: warn about misplaced objects, just like degraded

Signed-off-by: Sage Weil <sage@redhat.com>

osd: num_objects_misplaced

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #2217 from ceph/wip-problem-osds

mon: 'ceph osd blocked-by' for histogram of peers OSDs are waiting for

Reviewed-by: Samuel Just <sam.just@inktank.com>

qa/workunits/rest/test.py: fix 'df' test to use total_used_bytes

This changed back in ee2dbdb0f5e54fe6f9c5999c032063b084424c4c

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "os/FileJournal: Update the journal header when closing journal"

This reverts commit 4eb18dd487da4cb621dcbecfc475fc0871b356ac.

This may be responsible for #9073.  Until that is resolved, revert.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #2271 from ceph/wip-9053

paxos: fix problem with disjoint quorum members

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>