Loic Dachary [Mon, 10 Feb 2014 22:42:38 +0000 (23:42 +0100)]
common: admin socket fallback to json-pretty format
If the format argument to a command sent to the admin socket is not
among the supported formats ( json, json-pretty, xml, xml-pretty ) the
new_formatter function will return null and the AdminSocketHook::call
function must fall back to a sensible default.
The CephContextHook::call and HelpHook::call failed to do that and a
malformed format argument would cause the mon to crash. A check is added
to each of them and fallback to json-pretty if the format is not
recognized.
To further protect AdminSocketHook::call implementations from similar
problems the format argument is checked immediately after accepting the
command in AdminSocket::do_accept and replaced with json-pretty if it is
not known.
A test case is added for both CephContextHook::call and HelpHook::call
to demonstrate the problem exists and is fixed by the patch.
Three other instances of unsafe calls to new_formatter were found and
a fallback to json-pretty was added. All other calls have been audited
and appear to be safe.
Florian Haas [Mon, 10 Feb 2014 23:04:06 +0000 (00:04 +0100)]
doc: highlight that "raw" is the only useful RBD format for QEMU
Explain why people should be using the "raw" image format for RBD
volumes created for use by QEMU: using any other format adds only
overhead, but no extra value (since RBDs are also CoW and
thin-provisioned), plus the Qcow2 storage driver is not migration safe
when caching is enabled, whereas the RBD driver is.
Also, fix a minor glitch in the example qemu-img commands ("-f rbd"
and "-O rbd" should really be "-f raw" and "-O raw").
Finally, drop the "-f" option altogether on qemu-img commands where it
makes no sense (info and resize).
Josh Durgin [Thu, 6 Feb 2014 01:22:14 +0000 (17:22 -0800)]
msg/Pipe: add option to restrict delay injection to specific msg type
This makes it possible to test timeouts reliably by delaying certain
messages effectively forever, but still being able to e.g. connect and
authenticate to the monitors.
Josh Durgin [Tue, 4 Feb 2014 01:59:21 +0000 (17:59 -0800)]
Objecter: implement mon and osd operation timeouts
This captures almost all operations from librados other than mon_commands().
Get the values for the timeouts from the Objecter constructor, so only
librados uses them.
Add C_Cancel_*_Op, finish_*_op(), and *_op_cancel() for each type of
operation, to mirror those for Op. Create a callback and schedule it
in the existing timer thread if the timeouts are specified.
Ilya Dryomov [Mon, 10 Feb 2014 17:34:44 +0000 (19:34 +0200)]
configure: fix F_SETPIPE_SZ detection
Currently CEPH_HAVE_SETPIPE_SZ is not set even if F_SETPIPE_SZ is
available, because AC_COMPILE_IFELSE test program as written always
fails to compile. F_SETPIPE_SZ is a macro, so use AC_EGREP_CPP which
works on the preprocessor output instead of trying to compile.
Sage Weil [Tue, 4 Feb 2014 20:14:14 +0000 (12:14 -0800)]
crush: fix off-by-one errors in total_tries refactor
Back in 27f4d1f6bc32c2ed7b2c5080cbd58b14df622607 we refactored the CRUSH
code to allow adjustment of the retry counts on a per-pool basis. That
commit had an off-by-one bug: the previous "tries" counter was a *retry*
count, not a *try* count, but the new code was passing in 1 meaning
there should be no retries.
Fix the ftotal vs tries comparison to use < instead of <= to fix the
problem. Note that the original code used <= here, which means the
global "choose_total_tries" tunable is actually counting retries.
Compensate for that by adding 1 in crush_do_rule when we pull the tunable
into the local variable.
This was noticed looking at output from a user provided osdmap.
Unfortunately the map doesn't illustrate the change in mapping behavior
and I haven't managed to construct one yet that does. Inspection of the
crush debug output now aligns with prior versions, though.
Sage Weil [Tue, 4 Feb 2014 00:19:07 +0000 (16:19 -0800)]
osdmaptool: add --test-map-pgs mode
This command will map all pgs from all pools (or just one pool) to osds
and summarize the placement and calculate the actual standard deviation and
the expected value.
Loic Dachary [Thu, 6 Feb 2014 10:00:33 +0000 (11:00 +0100)]
erasure-code: move source files to a dedicated directory
The src/erasure-code directory contains the erasure-code plugin system
and the jerasure plugin. It is moved out of OSD because it now belongs
to a convenience library ( LIBERASURE_CODE ) which is used both by OSDs
and MONs.
Loic Dachary [Wed, 5 Feb 2014 19:44:32 +0000 (20:44 +0100)]
mon: test osd pool create pg_pool_t::stripe_width behavior
* Check that the default from the configuration options is found in the
output of osd dump
* Check that specifying an undersized osd_pool_erasure_code_stripe_width
value is taken into account and padded.
Loic Dachary [Wed, 5 Feb 2014 19:40:46 +0000 (20:40 +0100)]
mon: osd pool create sets pg_pool_t::stripe_width
It does nothing if the pool is replicated. Otherwise it uses
osd_pool_erasure_code_stripe_width as the desired stripe width and run
it by get_chunk_size() on the erasure code plugin to get the actual
stripe_width. It will always be >= 0 to the desired stripe_width, padded
to match the alignment constraints imposed by the erasure code plugin.
Loic Dachary [Wed, 5 Feb 2014 19:31:18 +0000 (20:31 +0100)]
mon: add erasure-code pg_pool_t::stripe_width
Contains the actual stripe size used by erasure coded pools. It is
initialized to zero by default and has to be explicitly set by erasure
coded pools. get/set methods are added inline.
Yehuda Sadeh [Tue, 4 Feb 2014 18:34:02 +0000 (10:34 -0800)]
rgw: fix rgw_read_user_buckets() use of max param
Fixes: #7336
The rgw_read_user_buckets() treated the max param as the max number of
entries to request in a single op, but always fetched the entire list
of buckets. This is wrong, as it should have treated it as the total
number of entries requested. All the callers assume the latter.
Loic Dachary [Mon, 3 Feb 2014 13:00:41 +0000 (14:00 +0100)]
erasure-code: add ceph_erasure_code debug command
It loads a designated erasure-code plugin and calls its
methods. It is convenient to figure out and tune the number of data
chunks, the size of an aligned chunk etc. For instance:
Yan, Zheng [Fri, 10 Jan 2014 01:55:50 +0000 (09:55 +0800)]
mds: avoid sending duplicated discovers during recovery
If MDS just entered the rejoin state, it should not kick discovers
because the discovers were just sent. Similarly, if MDS just entered
the clientreplay state, it should not call MDS::handle_mds_recovery()
because MDS::recovery_done() has already recovered the table server.
Also make MDCache::handle_mds_recovery() not wake the discover waiters
up. Because the MDCache::kick_discovers re-sends the discovers, their
replies will wake the discover waiter up.
Loic Dachary [Tue, 4 Feb 2014 00:31:52 +0000 (01:31 +0100)]
mon: check cluster features before rule create-erasure
Encapsulate the logic used when creating an erasure coded pool into the
check_cluster_features helper.
check_cluster_features(CEPH_FEATURE_CRUSH_V2) is required for crush rule
create-erasure because it is expected that the erasure code plugin will
use indep instead of firstn and expect the V2 behavior and not the
legacy behavior.
The CEPH_FEATURE_CRUSH_V2 is added to CEPH_FEATURE_OSD_ERASURE_CODES
when an erasure coded pool is created. It is necessary because pool
won't function properly if given an indep ruleset that does not
implement the V2 behavior.
Loic Dachary [Sun, 2 Feb 2014 10:35:13 +0000 (11:35 +0100)]
mon: osd-pool-create test no longer use hardcoded ruleset
For erasure-code the ruleset must be specified instead of relying on an
hardcoded value. Adapt the test to this for tests that do not otherwise
change behavior.
Loic Dachary [Sun, 2 Feb 2014 09:40:37 +0000 (10:40 +0100)]
mon: test erasure code pool creation
* The sequence now is a) create ruleset, b) create pool.
* Check that not specifying the ruleset when an erasure coded pool is
created fails
* Check that specifying a non existent ruleset when a pool is created
fails
* Check that osd dump shows the expected ruleset. It assumes ruleset
numbers are allocated in sequence, i.e. the ruleset will be 1.
Loic Dachary [Sun, 2 Feb 2014 08:38:23 +0000 (09:38 +0100)]
unittests: reduce paxos propose interval to increase speed
The MONs are stressed more often and there is less aggregation of the
pending requests. But the unit tests are only meant to verify that a
known code path exists and performs as expected, therefore it will not
make a difference. And if it does, it is a bug that needs fixing.
Loic Dachary [Wed, 29 Jan 2014 13:52:22 +0000 (14:52 +0100)]
pybind: fix tests that do not fail as expected
A missing argument make the test fail indeed, but the intended test is
to demonstrate something else ( either character validation or excess of
arguments etc. ). The result is {} instead of None which is what should
have been expected in the first place.
Ideally there would be a more verbose way to check for syntactic errors
to make such mistakes less probable.
Loic Dachary [Sun, 2 Feb 2014 09:05:59 +0000 (10:05 +0100)]
mon: compute the ruleset of erasure-coded pools
The default ruleset of an erasure coded pool may depend on the
parameters used to configure it. In the case of a pyramidal /
hierarchical plugin, the desired ruleset will, for instance, chose from
datacenters and then from racks and disperse local coding chunks among
them.
For this reason the default ruleset cannot be hardcoded in config_opts
as it is for replicated pools. Instead, the "crush_ruleset" property is
interpreted to be the name of an existing crush ruleset to be used.
If the corresponding ruleset is found in a pending crushmap, the
prepare_pool_crush_ruleset will return EAGAIN. The "osd pool create"
caller is modified to handle the EAGAIN error and reschedules the message.
projects / ceph.git / log