Ilya Dryomov [Mon, 23 Dec 2013 16:12:56 +0000 (18:12 +0200)]
crush: use kernel-doc consistently
kernel-doc syntax is "@arg: desc", not "@param arg desc". In addition,
these comments are usually placed around function definitions instead
of function declarations. Follow these guidelines to shrink the diff.
Ilya Dryomov [Mon, 23 Dec 2013 16:12:56 +0000 (18:12 +0200)]
crush/mapper: unsigned -> unsigned int
Kernel implementation is located in net/, and use of "unsigned int" is
preferred to bare "unsigned" in net tree (as proven by several net/
cleanups). Follow this guideline to shrink the diff.
Loic Dachary [Thu, 12 Dec 2013 22:14:02 +0000 (23:14 +0100)]
osd: erasure code benchmark workunit
Display benchmark results for the default erasure code plugins, in a tab
separated CSV file. The first two column contain the amount of KB
that were coded or decoded, for a given combination of parameters
displayed in the following fields.
seconds KB plugin k m work. iter. size eras.
1.2 10 example 2 1 encode 10 1024 0
0.5 10 example 2 1 decode 10 1024 1
It can be used as input for a human readable report. It is also intented
to be used to show if a given version of an erasure code plugin performs
better than another.
The last column ( not shown above for brievety ) is the exact command
that was run to produce the result so it can be copy / pasted to
reproduce them or to profile.
Only the jerasure techniques mentionned in
https://www.usenix.org/legacy/events/fast09/tech/full_papers/plank/plank_html/
are benchmarked, the others are assumed to be less interesting.
Loic Dachary [Fri, 13 Dec 2013 23:41:03 +0000 (00:41 +0100)]
osd: set erasure code packet size default to 2048
As shown in
https://www.usenix.org/legacy/events/fast09/tech/full_papers/plank/plank_html/
under "Impact of the Packet Size", the optimal for is in the order of 1k
rather than the current default of 8. Benchmarks are required to find
the actual optimum.
Loic Dachary [Fri, 13 Dec 2013 13:07:37 +0000 (14:07 +0100)]
osd: better performances for the erasure code example
The XOR based example is ten times slower than it could because it uses
the buffer::ptr[] operator. Use a temporary char * instead. It performs
as well as jerasure Reed Solomon when decoding with a single erasure:
Loic Dachary [Thu, 12 Dec 2013 13:03:26 +0000 (14:03 +0100)]
osd: conditionally disable dlclose of erasure code plugins
When profiling, tools such as valgrind --tool=callgrind require that the
dynamically loaded libraries are not dlclosed so they can collect usage
information.
The public ErasureCodePluginRegistry::disable_dlclose boolean is introduced
for this purpose.
Alexandre Oliva [Thu, 19 Dec 2013 16:09:46 +0000 (08:09 -0800)]
mds: fix Resetter locking
ceph-mds --reset-journal didn't work; it would deadlock waiting for
the osdmap. Comparing the init code in the Dumper (that worked) with
that in the Resetter (that didn't), I noticed the lock had to be
released before waiting for the osdmap.
Now the resetter works. However, both the resetter and the dumper
fail an assertion after they've performed their task; I didn't look
into it:
../../src/msg/SimpleMessenger.cc: In function 'void SimpleMessenger::reaper()' t
hread 7fdc188d27c0 time 2013-12-19 04:48:16.930895
../../src/msg/SimpleMessenger.cc: 230: FAILED assert(!cleared)
ceph version 0.72.1-6-g6bca44e (6bca44ec129d11f1c4f38357db8ae435616f2c7c)
1: (SimpleMessenger::reaper()+0x706) [0x880da6]
2: (SimpleMessenger::wait()+0x36f) [0x88180f]
3: (Resetter::reset()+0x714) [0x56e664]
4: (main()+0x1359) [0x562769]
5: (__libc_start_main()+0xf5) [0x3632e21b45]
6: /l/tmp/build/ceph/build/src/ceph-mds() [0x564e49]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to int
erpret this.
2013-12-19 04:48:16.934093 7fdc188d27c0 -1 ../../src/msg/SimpleMessenger.cc: In
function 'void SimpleMessenger::reaper()' thread 7fdc188d27c0 time 2013-12-19 04
:48:16.930895
../../src/msg/SimpleMessenger.cc: 230: FAILED assert(!cleared)
osd: OSD: reflect OSDMap EC flag being set by setting on-disk feature
If OSDMap has the EC feature set, then update our superblock to
reflect as such, making our on-disk format incompatible with previous
OSDs without EC support.
Fixes: 6028 Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Add the osd's features to the osd's extra info field in the OSDMap
so we can track which OSDs are able to deal with Erasure Codes.
This will allow us to decide whether or not we are ready to set EC
whenever the user asks us to set EC on a pool -- which shall be
handled by a subsequent commit.
Fixes: 6028 Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
Loic Dachary [Wed, 18 Dec 2013 16:16:08 +0000 (17:16 +0100)]
crush: silence error messages in unit tests
The error messages are intentional when error conditions are
created. They will create false positive in the gitbuilder parser when
the string error is found.
The --debug-crush flag is detected to allow the caller to reset the
verbosity level.
Sage Weil [Tue, 17 Dec 2013 17:28:43 +0000 (09:28 -0800)]
mon: warn if crush has non-optimal tunables
Allow warning to be disabled via ceph.conf. Link to the docs from the
warning detail. Add a section to the docs specifically about what to do
about the warning.
Laurent Barbe [Wed, 18 Dec 2013 13:20:24 +0000 (14:20 +0100)]
upstart: add rbdmap script
Upstart script for mapping / unmapping rbd device based on /etc/ceph/rbdmap file.
It does not mount or unmount filesystem, this part should be performed by _netdev option in fstab.
Loic Dachary [Tue, 17 Dec 2013 19:26:01 +0000 (20:26 +0100)]
erasure-code: tests must use aligned buffers
The underlying code assumes the memory buffer is aligned on a long
boundary which is not always the case. Using buffer::create_page_aligned
which calls posix_memalign ensure the allocated buffer starts at an
address that is properly aligned.
Loic Dachary [Mon, 16 Dec 2013 16:13:27 +0000 (17:13 +0100)]
qa: vstart wrapper helper for unittests
Primarily useful to run scripts from qa/workunits as part of make check.
vstart_wrapper.sh starts a vstart.sh cluster, runs the command given in
argument and tearsdown cluster when it completes.
The vstart_wrapped_tests.sh script contains the list of scripts that
need the vstart_wrapper.sh to run. It would not be necessary if automake
allowed passing argument to tests scripts. It also adds markers to the
output to facilitate searching the output because it can be very verbose.
This wrapper is kept simple and will probably evolve into something more
sophisticated depending on the scripts being added to
vstart_wrapper_tests.sh. There are numerous options, ranging from
parsing the yaml from ceph-qa-suite to figure out the configuration
cluster to converting the same yaml into a puppet manifest that is
applied locally or even driving OpenStack instances to avoid messing
with the local machine. But this would probably be overkill at this
point.
Ilya Dryomov [Tue, 17 Dec 2013 15:42:30 +0000 (17:42 +0200)]
rbd: make coverity happy
A recent coverity run found two "defects" in rbd.cc:
** CID 1138367: Time of check time of use (TOCTOU)
/rbd.cc: 2024 in do_kernel_rm(const char *)()
2019 const char *fname = "/sys/bus/rbd/remove_single_major";
2020 if (stat(fname, &sbuf)) {
2021 fname = "/sys/bus/rbd/remove";
2022 }
2023
2024 int fd = open(fname, O_WRONLY);
2025 if (fd < 0) {
** CID 1138368: Time of check time of use (TOCTOU)
/rbd.cc: 1735 in do_kernel_add(const char *, const char *, const char *)()
same as above, s/remove/add
There is nothing racey going on here, and this is not an instance of
TOCTOU, but, instead of silencing coverity with annotatations, redo
this with two open() calls.
Loic Dachary [Mon, 16 Dec 2013 15:27:34 +0000 (16:27 +0100)]
vstart/stop: use pkill instead of killall
killall fails to kill all OSDs when called as a oneliner. Replace with a
loop using pkill that retries until there are no more process to kill by
the required name.
Loic Dachary [Mon, 16 Dec 2013 13:36:26 +0000 (14:36 +0100)]
qa: recursively remove .gcno and .gcda
Instead of removing them only in the current directory. Leftovers
prevent running make check-coverage properly because lcov fails
when stumbling on old .gcno files with
lcov -d . -c -i -o check-coverage_base_full.lcov
Processing os/BtrfsFileStoreBackend.gcno
geninfo: ERROR: ceph/src/os/BtrfsFileStoreBackend.gcno: reached
unexpected end of file
Sage Weil [Tue, 17 Dec 2013 05:33:07 +0000 (21:33 -0800)]
crush/mapper: generalize descend_once
The legacy behavior is to make the normal number of tries for the
recursive chooseleaf call. The descend_once tunable changed this to
making a single try and bail if we get a reject (note that it is
impossible to collide in the recursive case).
The new set_chooseleaf_tries lets you select the number of recursive
chooseleaf attempts for indep mode, or default to 1. Use the same
behavior for firstn, except default to total_tries when the legacy
tunables are set (for compatibility). This makes the rule step
override the (new) default of 1 recursive attempt, keeping behavior
consistent with indep mode.
projects / ceph.git / log