Loic Dachary [Thu, 6 Feb 2014 10:00:33 +0000 (11:00 +0100)]
erasure-code: move source files to a dedicated directory
The src/erasure-code directory contains the erasure-code plugin system
and the jerasure plugin. It is moved out of OSD because it now belongs
to a convenience library ( LIBERASURE_CODE ) which is used both by OSDs
and MONs.
Loic Dachary [Wed, 5 Feb 2014 19:44:32 +0000 (20:44 +0100)]
mon: test osd pool create pg_pool_t::stripe_width behavior
* Check that the default from the configuration options is found in the
output of osd dump
* Check that specifying an undersized osd_pool_erasure_code_stripe_width
value is taken into account and padded.
Loic Dachary [Wed, 5 Feb 2014 19:40:46 +0000 (20:40 +0100)]
mon: osd pool create sets pg_pool_t::stripe_width
It does nothing if the pool is replicated. Otherwise it uses
osd_pool_erasure_code_stripe_width as the desired stripe width and run
it by get_chunk_size() on the erasure code plugin to get the actual
stripe_width. It will always be >= 0 to the desired stripe_width, padded
to match the alignment constraints imposed by the erasure code plugin.
Loic Dachary [Wed, 5 Feb 2014 19:31:18 +0000 (20:31 +0100)]
mon: add erasure-code pg_pool_t::stripe_width
Contains the actual stripe size used by erasure coded pools. It is
initialized to zero by default and has to be explicitly set by erasure
coded pools. get/set methods are added inline.
Loic Dachary [Mon, 3 Feb 2014 13:00:41 +0000 (14:00 +0100)]
erasure-code: add ceph_erasure_code debug command
It loads a designated erasure-code plugin and calls its
methods. It is convenient to figure out and tune the number of data
chunks, the size of an aligned chunk etc. For instance:
Yan, Zheng [Fri, 10 Jan 2014 01:55:50 +0000 (09:55 +0800)]
mds: avoid sending duplicated discovers during recovery
If MDS just entered the rejoin state, it should not kick discovers
because the discovers were just sent. Similarly, if MDS just entered
the clientreplay state, it should not call MDS::handle_mds_recovery()
because MDS::recovery_done() has already recovered the table server.
Also make MDCache::handle_mds_recovery() not wake the discover waiters
up. Because the MDCache::kick_discovers re-sends the discovers, their
replies will wake the discover waiter up.
Loic Dachary [Tue, 4 Feb 2014 00:31:52 +0000 (01:31 +0100)]
mon: check cluster features before rule create-erasure
Encapsulate the logic used when creating an erasure coded pool into the
check_cluster_features helper.
check_cluster_features(CEPH_FEATURE_CRUSH_V2) is required for crush rule
create-erasure because it is expected that the erasure code plugin will
use indep instead of firstn and expect the V2 behavior and not the
legacy behavior.
The CEPH_FEATURE_CRUSH_V2 is added to CEPH_FEATURE_OSD_ERASURE_CODES
when an erasure coded pool is created. It is necessary because pool
won't function properly if given an indep ruleset that does not
implement the V2 behavior.
Loic Dachary [Sun, 2 Feb 2014 10:35:13 +0000 (11:35 +0100)]
mon: osd-pool-create test no longer use hardcoded ruleset
For erasure-code the ruleset must be specified instead of relying on an
hardcoded value. Adapt the test to this for tests that do not otherwise
change behavior.
Loic Dachary [Sun, 2 Feb 2014 09:40:37 +0000 (10:40 +0100)]
mon: test erasure code pool creation
* The sequence now is a) create ruleset, b) create pool.
* Check that not specifying the ruleset when an erasure coded pool is
created fails
* Check that specifying a non existent ruleset when a pool is created
fails
* Check that osd dump shows the expected ruleset. It assumes ruleset
numbers are allocated in sequence, i.e. the ruleset will be 1.
Loic Dachary [Sun, 2 Feb 2014 08:38:23 +0000 (09:38 +0100)]
unittests: reduce paxos propose interval to increase speed
The MONs are stressed more often and there is less aggregation of the
pending requests. But the unit tests are only meant to verify that a
known code path exists and performs as expected, therefore it will not
make a difference. And if it does, it is a bug that needs fixing.
Loic Dachary [Wed, 29 Jan 2014 13:52:22 +0000 (14:52 +0100)]
pybind: fix tests that do not fail as expected
A missing argument make the test fail indeed, but the intended test is
to demonstrate something else ( either character validation or excess of
arguments etc. ). The result is {} instead of None which is what should
have been expected in the first place.
Ideally there would be a more verbose way to check for syntactic errors
to make such mistakes less probable.
Loic Dachary [Sun, 2 Feb 2014 09:05:59 +0000 (10:05 +0100)]
mon: compute the ruleset of erasure-coded pools
The default ruleset of an erasure coded pool may depend on the
parameters used to configure it. In the case of a pyramidal /
hierarchical plugin, the desired ruleset will, for instance, chose from
datacenters and then from racks and disperse local coding chunks among
them.
For this reason the default ruleset cannot be hardcoded in config_opts
as it is for replicated pools. Instead, the "crush_ruleset" property is
interpreted to be the name of an existing crush ruleset to be used.
If the corresponding ruleset is found in a pending crushmap, the
prepare_pool_crush_ruleset will return EAGAIN. The "osd pool create"
caller is modified to handle the EAGAIN error and reschedules the message.
Loic Dachary [Sun, 2 Feb 2014 08:59:52 +0000 (09:59 +0100)]
mon: erasure code plugin loader helper
The get_erasure_code helper loads the erasure code plugin found in the
erasure-code-plugin string of the properties argument. It is meant to be
used to query the plugin to determine the desired size of a pool, the
more suitable ruleset to use etc.
Loic Dachary [Sun, 2 Feb 2014 08:56:13 +0000 (09:56 +0100)]
mon: pool create helper for crush ruleset
The crush ruleset of the replicated pools are by default set to
osd_pool_default_crush_replicated_ruleset but it may vary depending on
the pool type. Create a helper to compute the crush ruleset.
Loic Dachary [Sun, 2 Feb 2014 08:51:50 +0000 (09:51 +0100)]
mon: pool creation helper for size
The size of the replicated pools are by default set to
osd_pool_default_size but it may vary depending on the pool type. Create
a helper to compute the pool size.
Loic Dachary [Sat, 1 Feb 2014 09:21:00 +0000 (10:21 +0100)]
mon: no default ruleset except for replicated pools
Remove the hardcoded default ruleset for erasure coded pools and only
keep it for replicated pools. Move the logic up in the prepare_new_pool
method so that an error code can be returned before allocating the new
pending pool in case the ruleset is not initialized.
Loic Dachary [Sat, 1 Feb 2014 09:09:12 +0000 (10:09 +0100)]
mon: helper for pool properties parsing
Add the prepare_pool_properties to convert the properties vector into a
properties map suitable for either initializing the pg_pool_t member or
an erasure code plugin.
It is based on CrushWrapper::add_simple_ruleset, using a "default" root
and "host" failure domain by default. They can be overridden with
erasure-code parameters ( erasure-code-ruleset-root and
erasure-code-ruleset-failure-domain respectively ).
Sage Weil [Tue, 4 Feb 2014 05:12:41 +0000 (21:12 -0800)]
client: fix warnings
client/Client.cc: In member function 'int Client::_read(Fh*, int64_t, uint64_t, ceph::bufferlist*)':
warning: client/Client.cc:5893:27: comparison between signed and unsigned integer expressions [-Wsign-compare]
client/Client.cc: In member function 'int Client::_write(Fh*, int64_t, uint64_t, const char*)':
warning: client/Client.cc:6235:30: comparison between signed and unsigned integer expressions [-Wsign-compare]
Sage Weil [Mon, 3 Feb 2014 21:19:14 +0000 (13:19 -0800)]
mon: fix 'mds set allow_new_snaps'
We had already added this as a flag (set/unset) when I generalized the
'mds set_max_mds' to be 'ceph mds set <var> <val>'. Switch the snaps
flag to be a key/value to with true/false (similar to the hashpspool
pool flag) since there are fewer users and the var/val approach is more
general.
Signed-off-by: Sage Weil <sage@inktank.com> Reviewed-by: Greg Farnum <greg@inktank.com>
Loic Dachary [Sun, 2 Feb 2014 00:09:52 +0000 (01:09 +0100)]
mon: do not force proposal when no osds
If there are no OSDs, there is no need to propose to paxos. It does not
hurt on a production cluster but it matters when running tests because
it effectively alway ignores --paxos-propose-interval.
Sage Weil [Mon, 3 Feb 2014 16:54:14 +0000 (08:54 -0800)]
client: use 64-bit value in sync read eof logic
The file size can jump to a value that is very much larger than our current
position (for example, it could be a disk image file that gets a sparse
write at a large offset). Use a 64-bit value so that 'some' doesn't
overflow.
Signed-off-by: Sage Weil <sage@inktank.com> Reviewed-by: John Spray <john.spray@inktank.com>
Somnath Roy [Wed, 22 Jan 2014 18:30:32 +0000 (10:30 -0800)]
Pipe, cephx: Message signing under config option
The config option was present earlier but the option is checked
not in the beginning, so, it was doing some stuff. Now the config
option guard is at the very beginning of the function.
Signed-off-by: Somnath Roy <somnath.roy@sandisk.com> Signed-off-by: Greg Farnum <greg@inktank.com>
Greg Farnum [Tue, 7 Jan 2014 18:05:57 +0000 (10:05 -0800)]
TrackedOp: optionally disable the actual tracking operations
To avoid op contention on global locks, optionally disable the
op tracking. Create an "osd_op_tracker" config to control it, then
in the OpTracker constructor set a bool. If it's set, the OpTracker
doesn't actually maintain its lists and avoids taking any locks. We
maintain the within-op tracking for now since it shouldn't contend,
but we can turn that off later on if we like.
Greg Farnum [Tue, 7 Jan 2014 17:43:28 +0000 (09:43 -0800)]
Throttler: optionally disable use of perfcounters
These can be expensive enough that we want to disable them. We're already
mostly set up for it to be optional, so just plug in a config option and
move a timestamp read under the "if (logger)" guard to set it up!
- fix a couple of typo for repo configuration and service restart
- the ceph package must be installed on RPM distro since the init
script relies on ceph-conf
- Note on radosgw service name for RPM distro
David Zafman [Wed, 29 Jan 2014 03:18:32 +0000 (19:18 -0800)]
osd: Move the rest of scrubbing routines to the backend
Move enum scrub_error_type to osd_types.h
Move PG::_compare_scrub_objects to ReplicatedBackend::be_compare_scrub_objects
Move PG::_select_auth_object to ReplicatedBackend::be_select_auth_object
Move PG::_compare_scrubmaps to ReplicatedBackend::be_compare_scrubmaps
Signed-off-by: David Zafman <david.zafman@inktank.com>
Sage Weil [Fri, 31 Jan 2014 15:19:10 +0000 (07:19 -0800)]
os/KeyValueStore: fix warning
./os/KeyValueStore.h: In member function ‘std::string KeyValueStore::strip_object_key(uint64_t)’:
warning: ./os/KeyValueStore.h:173:31: format ‘%ld’ expects argument of type ‘long int’, but argument 4 has type ‘uint64_t {aka long long unsigned int}’ [-Wformat=]
Sage Weil [Thu, 30 Jan 2014 23:13:05 +0000 (15:13 -0800)]
mon/OSDMonitor: encode full OSDMap with same feature bits as the Incremental
Each monitor is independently encoding the full OSDMap and storing it in
its local store. Sometime this happens when we do not have a valid value
for quorum_features (for example, it can happen during sync).
Instead, use the feature bits the Incremental was encoded with for the full
OSDMap so that they always match.
Note that this conveniently the *only* place in the mon where we encode
the full OSDMap, so we're capturing all paths. Yay!
Sage Weil [Thu, 30 Jan 2014 23:09:58 +0000 (15:09 -0800)]
OSDMap: note encoding features in Incremental encoding
The monitor will need to know what features the incremental was encoded
with so that it can encode the OSDMap using the same bits. Introduce a
member that is set during decode. During encode, encoding the value passed
in by the caller.
projects / ceph.git / log