Sage Weil [Fri, 20 Sep 2013 00:57:14 +0000 (17:57 -0700)]
common/bloom_filter: unit tests
Fun facts:
- fpp = false positive probability
- fpp is a function of insert count only
- at .1% fpp, we pay about 2 bytes per insert
- at 1-2% fpp, we pay about 1 byte per insert
- at 15% fpp, we pay about .5 bytes per insert
Sage Weil [Wed, 18 Sep 2013 21:50:05 +0000 (14:50 -0700)]
common/bloom_filter: make optimal parameter calculation static
We pass the ctor our target behavior and calculate parameters based on
that. Avoid storing the target behavior, and make that calc a static
method. And add a new ctor that takes the parameters explicitly.
Sage Weil [Wed, 25 Sep 2013 22:47:28 +0000 (15:47 -0700)]
mon/PGMonitor: fix segfault when osdmap and pgmap stats are out of sync
The leader mon is responsible for propagating changes in the osdmap
into the pgmap (e.g., when a pool or osd goes away). If that hasn't
happened yet, the pg_map map will be out of sync with the current set
of pools. We shouldn't crash.
Sage Weil [Wed, 25 Sep 2013 20:37:13 +0000 (13:37 -0700)]
Makefile: don't use srcdir in TESTS
src/Makefile-env.am:31: error: using '$(srcdir)' in TESTS is currently broken: '$(srcdir)/unittest_bufferlist.sh'
src/Makefile.am:1: 'src/Makefile-env.am' included from here
src/Makefile-env.am:31: error: using '$(srcdir)' in TESTS is currently broken: '$(srcdir)/test/encoding/check-generated.sh'
src/Makefile.am:1: 'src/Makefile-env.am' included from here
The substr_of bufferlist method is used to reduce the copies to the
strict minimum. It is both able to split bufferptr when they are larger
than a chunk and to concatenate them when they are smaller.
autoconf: remove get_command_descriptions dependency to gtest
It is not required and a leftover of the time when it was a unit test
which was a mistake to begin with. It has been reported to create
dependencies problems in some cases, when gest is not available at the
time get_command_descriptions is compiled.
The TEST_F(ErasureCodePluginRegistryTest, factory_mutex) was bugous and
only succeeded by chance. The sleep was on the factory constructor which
was never called. An erasure code plugin that hangs forever on load is
created instead.
The sleep_forever.detach is replaced by pthread_cancel to interrupt the
thread that hangs forever. If not, gtest will try to join the thread and
never exit.
Andreas-Joachim Peters pointed out that the ErasureCodeJerasure method
pad_in_length is very inefficient for no reason. It is replaced by the
alignment method and the calculation is inlined in the encode method.
The general idea is to have a series of commands, in the same order as
they show in mon/MonCommands.h, as if they were input to the ceph
client. For each command a valid combination is verified. And at least
one validation error is checked to produce a validation error. For
instance:
['pg', 'stat']
is a valid command and the validate_command function is expected to
return a value that is not None or {}. The command
['pg', 'stat', 'toomany' ]
is also given to validate_command to check that an error occurs when
an extra argument is given.
The TestArparse class implements a few methods to reduce the verbosity
of the tests. It does not provide many methods : only those that
significantly reduce the verbosity have been implemented. The drawback
of writing too many convenience methods is that they are more difficult
to read and maintain.
The signature dictionary is made a global variable so that
it is only extracted once for all classes. It is immutable.
http://tracker.ceph.com/issues/6274 refs #6274
Reviewed-by: Dan Mick <dan.mick@inktank.com> Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com> Signed-off-by: Loic Dachary <loic@dachary.org>
If the / is missing in an EntityAddress, an ArgumentValid exception must
be raised so that it can be caught in the same way other argument
validation exceptions are.
http://tracker.ceph.com/issues/6274 refs #6274
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com> Signed-off-by: Loic Dachary <loic@dachary.org>
pybind: unit tests for ceph_argparse::parse_json_funcsigs
Run parse_json_funcsigs against the builtin commands found
in mon/MonCommands.h. Although it does not provide for a full
validation, it will detect some degenerate cases.
It is expected to raise if a space is missing at the end of an option
specification ( see https://github.com/ceph/ceph/pull/585 ) and this
case is tested. New tests of the same kind can be added whenever an
undetected error is found. Ideally a validation function is implemented
and it would be updated ( with an associated test ) when a new
pathological case is found.
The json string given to parse_json_funcsigs is obtained from
the support program get_command_descriptions.
The python-nose dependencies are added to the build requirements in
debian/control and ceph.spec.in because make check should always be run
at built time.
http://tracker.ceph.com/issues/6274 refs #6274
Reviewed-by: Dan Mick <dan.mick@inktank.com> Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com> Signed-off-by: Loic Dachary <loic@dachary.org>
The get_command_descriptions function is not designed to be tested in
C++ because all the validation happens in pybind/ceph_argparse.py. The
get_command_descriptions program is designed to be used by python unit
tests as a mean to get a JSON dump of the content of mon/MonCommands.h
mon: convenience function to convert commands to json
The get_command_descriptions is added to Monitor.h and contains
the code previously inlined in Monitor::handle_command to implement
the get_command_descriptions command. It is intended for tests to
convert command descriptions into json, including error cases.
http://tracker.ceph.com/issues/6274 refs #6274
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com> Signed-off-by: Loic Dachary <loic@dachary.org>
The check_SCRIPTS is added in Makefile-env.am to list the tests that do
not require compilation. The scripts listed in check-local and in the
TESTS variable use check_SCRIPTS instead.
The PYTHONPATH environment variable is added to Makefile-env.am and
includes the pybind directory so that python unit tests can load the
libraries from sources.
http://tracker.ceph.com/issues/6274 refs #6274
Reviewed-by: Roald J. van Loon <roaldvanloon@gmail.com> Signed-off-by: Loic Dachary <loic@dachary.org>
libkeyutils is only available on linux. before the automake refactoring
secret.c was compiled into rbd and ceph_mount targets which are linux
only targets. secret.c was moved to libcommon during the refactoring,
but the conditional compilation was lost.
ErasureCode: minimum_to_decode unit tests and optimization
The minimum_to_decode function simply returns the available chunks if no
recovery is necessary.
Add unit tests covering all minimum_to_decode situations:
* trying to read nothing
* read a chunk if none are available
* reading a subset of the available chunks
* read a missing chunk if there is less than k chunks available.
* reading when a chunk must be recovered
we save old data pools in both inode_backtrace_t::old_pools and
inode_t::old_pools. We have the inode in the cache when purging
stray, so no need to fetch backtrace to find old data pools.
Now we call MDCache::maybe_eval_stray() in MDSCacheObject::put().
So there is no need to call MDCache::maybe_eval_stray() after
releasing inode/dentry's refernece.
mds: evaluate stray when releasing inode/dentry's reference
Current method to purge stray inode is call MDCache::maybe_eval_stray()
after releasing a reference to the stray inode/dentry. It's difficult
to make this method work correct, because there are so many places that
can release reference.
This patch solves the issue by calling MDCache::maybe_eval_stray()
in MDSCacheObject::put(). This avoids adding code that calls
MDCache::maybe_eval_stray() to each place that releases reference.
Sage Weil [Sat, 21 Sep 2013 04:06:09 +0000 (21:06 -0700)]
ceph_test_rados: fix COPY_FROM completion
Fix the copy_from operation to not remove the objects from the in_use list
until after the entire operation is complete. In particular, the racing
read was completing and removing the dest oid from the in-use list before
the copy-from completed. This keeps the model in sync with what the OSD
is actually doing.
If another new read started up, it would grab the previous value from the
model and expect to see that, but would instead see the updated value.
Fixes: #6176
Backport: dumpling
We take different code paths in copy_obj, make sure we close the handle
when we exit the function. Move the call to finish_get_obj() out of
copy_obj_data() as we don't create the handle there, so that should
makes code less confusing and less prone to errors.
Also, note that RGWRados::get_obj() also calls finish_get_obj(). For
everything to work in concert we need to pass a pointer to the handle
and not the handle itself. Therefore we needed to also change the call
to copy_obj_data().
projects / ceph.git / log