Owen Synge [Tue, 5 Aug 2014 15:28:16 +0000 (17:28 +0200)]
Do not make directories by mistake.
Rational: I found I had created a series of OSD directories under "/dev/" when disks I thought existed did not exist.
Warning: This change will be noticed by end users and may effect deployment infrastructures.
rgw: call processor->handle_data() again if needed
Fixes: #8937
Following the fix to #8928 we end up accumulating pending data that
needs to be written. Beforehand it was working fine because we were
feeding it with the exact amount of bytes we were writing.
Loic Dachary [Fri, 30 May 2014 13:24:25 +0000 (15:24 +0200)]
erasure-code: HTML display of benchmark results
The ceph_erasure_code_benchmark output is converted into a JSON series
suitable to display in HTML with the http://www.flotcharts.org/
library. A self contained copy of the HTML,JS,CSS files is included for
durability and can be used from the source tree with:
CEPH_ERASURE_CODE_BENCHMARK=src/ceph_erasure_code_benchmark \
PLUGIN_DIRECTORY=src/.libs \
qa/workunits/erasure-code/bench.sh fplot jerasure |
tee qa/workunits/erasure-code/bench.js
Loic Dachary [Tue, 27 May 2014 19:45:19 +0000 (21:45 +0200)]
COPYING: Cloudwatt copyright is inline
Remove partial list of contributions since Cloudwatt copyright has been
placed in the copyright notices of the files where works covered by
copyright have been included.
Loic Dachary [Tue, 27 May 2014 17:25:22 +0000 (19:25 +0200)]
erasure-code: rework benchmark suite
Expand the default suite to enumerate all cases that are relevant to the
current code base so that it is easier to consume. Namely it means
* iterating over object sizes of 4KB (what is used by default) and
1MB (what was previous benchmarked)
* grouping results in series that would make sense to plot to get the
behavior of a given technique for a series of K/M values and all
possible erasures.
Instead of specifying the iterations to run, set the size of the total
data set to be exercised and compute the iterations by dividing it by
the object size. Since the object size varies, it is impractical to
preset the number of iterations and get meaningful results.
The PARAMETERS environment variable is added to enable the caller to
inject --parameter jerasure-variant=generic, for instance.
The packets size is calculated based on the other parameters. The
options are limited when packets are small (4KB) and it would not make a
real difference to give control over it. The packet size is capped to
a maximum of 3100 bytes which is roughly what has been found to be an
optimal value for large packets (1MB).
Loic Dachary [Fri, 30 May 2014 12:33:15 +0000 (14:33 +0200)]
erasure-code: control jerasure plugin variant selection
The jerasure-variant parameter is interpreted as the name of the plugin
variant to be loaded regardless of the available CPU features. The
values can be sse3, sse4, generic. It is undocumented and meant for
benchmarking purposes, primarily to force the generic plugin to be
loaded when the sse4 would be chosen.
Loic Dachary [Tue, 27 May 2014 16:40:45 +0000 (18:40 +0200)]
erasure-code: implement alignment on chunk sizes
jerasure expects chunk sizes that are aligned on the largest possible
vector size that could be used by SSE instructions, when available (
LARGEST_VECTOR_WORDSIZE == 16 bytes ).
For techniques derived from Cauchy, encoding and decoding is done by
subdividing the chunk into packets of packetsize bytes. The operations
are done w * packetsize bytes at a time. It follows that each chunk must
have a size that is a multiple of w * packetsize bytes.
For techniques derived from Vandermonde, it is enough for a chunk to be
a multiple of w * LARGEST_VECTOR_WORDSIZE.
ErasureCodeJerasure::get_alignment returns a size alignment constraint
that has to be enforced as a multiple of the object size. The resulting
object size then has to match the chunk constraints described above
although they have no relationship with K. For Cauchy, it leads to
excessive padding, making it impossible to set sensible parameters for
when the object size is small.
When the per_chunk_alignement data member is true, the semantic of
ErasureCodeJerasure::get_alignment is changed to return a size alignment
constraint to be enforced as a multiple of the chunk size. The
ErasureCodeJerasure::get_chunk_size method is modified to use the new
semantic when appropriate.
The jerasure-per-chunk-alignement parameter is parsed to set
per_chunk_alignement for the Vandermonde and Cauchy techniques.
The memory address of a chunk is implicitly aligned to a page boundary
because it is allocated with buffer::create_page_aligned.
Loic Dachary [Tue, 27 May 2014 16:36:09 +0000 (18:36 +0200)]
erasure-code: cauchy techniques allow w 8,16,32
Enforce the restriction at initialization time, the same way it is done
for Reed Solomon. Choosing a w value different from 8,16,32 will lead to
memory corruption that cannot easily be traced to the cause.
Ma Jianpeng [Wed, 30 Jul 2014 03:03:17 +0000 (11:03 +0800)]
ECBackend: Don't directly use get_recovery_chunk_size() in RecoveryOp::WRITING state.
We cannot guarantee that conf->osd_recovery_max_chunk don't change when
recoverying a erasure object.
If change between RecoveryOp::READING and RecoveryOp::WRITING, it can cause this bug:
2014-07-30 10:12:09.599220 7f7ff26c0700 -1 osd/ECBackend.cc: In function
'void ECBackend::continue_recovery_op(ECBackend::RecoveryOp&,
RecoveryMessages*)' thread 7f7ff26c0700 time 2014-07-30 10:12:09.596837
osd/ECBackend.cc: 529: FAILED assert(pop.data.length() ==
sinfo.aligned_logical_offset_to_chunk_offset(
after_progress.data_recovered_to -
op.recovery_progress.data_recovered_to))
Sage Weil [Mon, 4 Aug 2014 01:26:34 +0000 (18:26 -0700)]
msg/SimpleMessenger: drop msgr lock when joining a Pipe
Avoid this deadlock:
- a fault
- delay thread entry gets a fast dispatch message
- drops delay_lock
- calls into fast_dispatch
- reaper tries to reap the pipe
- pipe->join()
- delay_thread->join()
- blocks waiting for delay_thread to exit
- delay thread / fast dispatch blocks on msgr->lock trying to mark_down
The solution is to drop the msgr lock while joining the thread. This will
allow the join() to complete. Adjust the reaper thread to recheck the
exit condition since the lock may have been dropped. The other two callers
do not care.
Fixes: #8891 Signed-off-by: Sage Weil <sage@redhat.com>
Zhiqiang Wang [Fri, 1 Aug 2014 08:09:50 +0000 (16:09 +0800)]
osd: add local_mtime to struct object_info_t
This fixes a bug when the time of the OSDs and clients are not
synchronized (especially when client is ahead of OSD), and the cache
tier dirty ratio reaches the threshold, the agent skips the flush work
because it thinks the object is too young.
Sage Weil [Fri, 1 Aug 2014 03:59:49 +0000 (20:59 -0700)]
osd: do not leak Session* ref in _send_boot()
The get_priv() call returns a ref; make sure we drop it if it exists.
This doesn't happen on every run because usually it is NULL and we take
the other path; it's only after the OSD has been marked down that we reach
the second path.
Ma Jianpeng [Thu, 31 Jul 2014 02:19:32 +0000 (10:19 +0800)]
ReplicatedPG: For async-read, set the real result after completing read.
When reading an object from replicated pool, ceph uses sync mode,
so it can set the results in execute_ctx correctly.
However, For the async-read in EC Pool, current code didn't set the
real results after read in complete_read_ctx.
Signed-off-by: Ma Jianpeng <jianpeng.ma@intel.com>
Sage Weil [Thu, 31 Jul 2014 18:02:55 +0000 (11:02 -0700)]
mon/OSDMonitor: warn when cache pools do not have hit_sets configured
Give users a clue when cache pools are enabled but the hit_set is not
configured. Note that technically this will work, but not well, so for
now let's just steer them away.
Sage Weil [Thu, 31 Jul 2014 16:13:11 +0000 (09:13 -0700)]
osd/ReplicatedPG: check agent_mode if agent is enabled but hit_sets aren't
It is probably not a good idea to try to run the tiering agent without a
hit_set to inform its actions, but it is technically possible. For
example, one could simply blindly evict when we reach the full point.
However, this doesn't work because the agent mode is guarded by a hit_set
check, even though agent_setup() is not. Fix that.
projects / ceph.git / log