Josh Durgin [Tue, 18 Feb 2014 06:42:13 +0000 (22:42 -0800)]
ReplicatedPG: fix successful write condition
ctx->modify no longer implies that the operation is a write,
if it ever did. These days op->may_write() is checked reliably
on the OSD, so just use that instead.
Josh Durgin [Tue, 18 Feb 2014 06:35:13 +0000 (22:35 -0800)]
ReplicatedPG: clear osd op reply output for writes
Since the vector of OSDOps used by the reply is the same
as the one processed by do_osd_ops() now, any output data
needs to be cleared for writes. To be compatible with current behavior,
allow writes that aren't applying anything or have failed to return data
still.
Add a new parameter to the MOSDOpReply constructor to determine
whether the output data should be cleared. Clear it for successful
writes, and remove a redundant result < 0 -> result > 0 check in the
process. This was caught by ceph_test_cls_hello and its
writes_dont_return_data method.
Josh Durgin [Fri, 7 Feb 2014 02:20:06 +0000 (18:20 -0800)]
ReplicatedPG: set individual osd op rvals
This takes care of things that can fail before transaction is
executed, like omap comparison. Getting individual rvals from a
transaction requires more refactoring, so I'm leaving it for a later
cleanup.
Josh Durgin [Fri, 7 Feb 2014 02:17:06 +0000 (18:17 -0800)]
ReplicatedPG: don't copy the ops vector into OpContext
There's no need to copy it, and if we refer to the same vector from
the original MOSDOp, we can modify rvals for individual ops, since the
MOSDOp ops vector is copied to the MOSDOpReply when it is constructed.
Josh Durgin [Fri, 7 Feb 2014 02:50:50 +0000 (18:50 -0800)]
ReplicatedPG: avoid undefined behavior in xattr comparison
Reading past the end of a pointer returned by string.data() in c++98
is undefined. While we're fixing this, also allow comparison of xattrs
containing null bytes.
Josh Durgin [Fri, 7 Feb 2014 04:10:15 +0000 (20:10 -0800)]
librados: add omap object operations to the c api
Create an iterator type similar to XattrIter for returning results.
It just wraps the std::map to preserve sorting.
When getting omap keys or values, use an extra callback to set up the
iterator, and default optional parameters to the empty string.
Since it's hard to test read or write omap ops in isolation, combine
them all in the c read ops test case.
One difference between this and the c++ api is that omap_cmp only
allows one assertion per sub-op, for simplicity of the
interface. Multiple omap_cmp operations can still be added to the same
op to get the same effect.
Josh Durgin [Fri, 7 Feb 2014 04:03:20 +0000 (20:03 -0800)]
librados: add read xattr functions for c object operations
Only implement string xattr comparison by since integer xattr
comparison assumes ceph-encoded integers in little-endian 64 bit
format, which should not be exposed to librados users.
Add an extra callback for getxattr so that we can initialize the
iterator.
Josh Durgin [Fri, 7 Feb 2014 03:55:30 +0000 (19:55 -0800)]
Objecter: add a way to chain callbacks
The librados C api needs to do extra things like converting c++ data
structures or setting lengths, but some objecter operations already
have out_handlers that librados shouldn't override.
Add a method that chains contexts together by using a new context that
calls both the original and the newly added one.
Josh Durgin [Fri, 7 Feb 2014 03:51:59 +0000 (19:51 -0800)]
librados: add exec to the c object operations api
Add two versions: one that allocates a buffer of the appropriate
length for the user, but relies on the user to free it, and one that
uses a user-supplied buffer but may fail if it is too short.
Reuse the bufferlist -> buffer conversion context added for reads into
the user supplied buffer.
The librados-allocated buffer can be handled just like librados
allocated buffers used by the various command functions, so just reuse
do_out_buffer() for them.
Josh Durgin [Fri, 7 Feb 2014 03:25:31 +0000 (19:25 -0800)]
librados: add exec to the c write operations api
Nothing special needed here, just copying the input buffer and passing
things through. Don't allow output data since it's not usually
available for writes and unreliable when it is.
Josh Durgin [Fri, 7 Feb 2014 03:19:51 +0000 (19:19 -0800)]
librados: add individual op flags for c write operations
Move flag validation to a static function so it can be shared with the
c++ api. Refer to the new C constants from the c++ api so that it's
easy to keep them in sync.
Josh Durgin [Fri, 7 Feb 2014 03:12:58 +0000 (19:12 -0800)]
librados: add global op flags to the c api
The c++ api did not include these in the sync versions of the calls,
so add a flags argument to the IoCtxImpl functions for those, with
default arguments to avoid changing the many many callers.
Josh Durgin [Fri, 7 Feb 2014 03:03:56 +0000 (19:03 -0800)]
librados: remove constant for CMPXATTR_NOP
This constant was just added in the public header to match the
operations in rados.h, and was not included in a stable release yet.
NOP is misleading, since it's actually uninterpreted anywhere, and
using it would always return EIO from the OSD. It's not particularly
useful, so just remove it. Adding a cmpxattr with FAIL_OK is
equivalent.
Josh Durgin [Mon, 3 Feb 2014 09:00:49 +0000 (01:00 -0800)]
test/librados: refactor setup/teardown in tiering tests
These tests can either use the standard RadosTest, with one pool, or a
new test case with two pools.
The new test case has to copy a couple lines in static methods
(working around lack of virtual static methods is more complex than
it's worth here), but the rest is straight forward. Rename
base_pool_name and base_ioctx to pool_name and ioctx to match the
member variables of the parent class RadosTest.
Josh Durgin [Mon, 3 Feb 2014 08:56:27 +0000 (00:56 -0800)]
test/librados: use standard rados test case for snapshot tests
Split these into two categories, self-managed and pool snapshots,
since they are mutually exclusive for a single pool. Remove snapshots
and objects at the end of tests, regardless of pass/fail, so the same
pool can be reused.
Josh Durgin [Mon, 3 Feb 2014 08:54:21 +0000 (00:54 -0800)]
test/librados: use a test case instead of duplicating code everywhere
These tests don't have any special requirements on their ioctx and
pools, so they can also run much faster by using different namespaces
per test instead of new pools. For the few tests that do depend on
namespace, reset the ioctx to the default namespace.
Josh Durgin [Mon, 3 Feb 2014 17:01:20 +0000 (09:01 -0800)]
test/librados: create general test case classes
Using a test case allows us to remove a boatload of boilerplate code
in all the tests, and focus them more on what they're actually
testing.
Create one for C tests, and one for C++ tests, with the same
functionality:
- create a pool when the test case starts
- between individual tests, create an ioctx and set its namespace uniquely
- delete objects from the default namespace during individual test teardown
- delete the pool only when the whole test case is finished
In gtest, a test case is the whole set of tests declared as part of
the same class using TEST_F(). Many tests create and delete individual
pools, but this is unnecessary for independent operation in most
cases, since we can use namespaces for that now. Since pool creation
and deletion dominates test run time, using these test cases makes
running many of the tests much faster.
Josh Durgin [Thu, 6 Feb 2014 23:09:57 +0000 (15:09 -0800)]
test/librados: move test.cc into its own library
This way it doesn't have to be compiled many times, and it's easier
to add new general functionality to new files without adding those
.cc files to every test program.
Josh Durgin [Thu, 30 Jan 2014 14:55:29 +0000 (06:55 -0800)]
Objecter: keep ObjectOperation rval pointers for writes
Just before sending an op, prepare_mutate_op() is called, creating a
new Op. prepare_read_op() already copied over all the out-params
correctly, but for write operations the individual op return value
pointers were not copied, so they would not be filled in. With this
fixed, librados users can get the per-op return codes again.
Yehuda Sadeh [Fri, 31 Jan 2014 00:05:42 +0000 (16:05 -0800)]
rgw: scalable manifest object
Define a new manifest structure. The idea is that the manifest defines a
set of rules for structuring the object parts. There are a few terms to
note:
- head: the head part of the object, which is the part that contains
the first chunk of data. An object might not have a head (as in the
case of multipart-part objects.
- stripe: data portion of a single rgw object that resides on a single
rados object.
- part: a collection of stripes that make a contiguous part of an
object. A regular object will only have one part (although might have
many stripes), a multipart object might have many parts. Each part
has a fixed stripe size, although the last stripe of a part might
be smaller than that. Consecutive parts may be merged if their stripe
value is the same.
Samuel Just [Wed, 29 Jan 2014 21:38:04 +0000 (13:38 -0800)]
PG,ReplicatedPG: Generalize missing_loc for ECBackend
Prior to EC pools, unfound => missing. Now, unfound (unreadable,
really) is dependent on the PGBackend requirements for reconstituting
an object. This also means recovering an object missing on a replica
but not the primary requires tracking the missing_loc set.
Thus, rather than maintaining missing_loc only for objects missing
on the primary, the MissingLoc structure will track all missing
objects actingbackfill-wide until each object is recovered.
For simplicity, since we don't really know what objects need recovery
until activation (and since we can't do anything with that information
prior to activation anyway), we defer populating the missing_loc
information until activation.
We need peers to rollback divergent log entries before we attempt to
read the relevant objects. The simplest way to accomplish this seems to
be the simply choose to always activate peers if search_for_missing
turns up missing objects.
Due to EC pools, missing is necessary, but not sufficient for readability.
Thus, we instead check is_unreadable for cases where we need to read the object
and reserve is_missing for cases where we need the object context.
wait_for_missing_object becomes waiting_for_unreadable_object in order to avoid
having another layer of waiting_for_* maps. These ops may be requeued
either when the primary is recovered or when the object is no longer degraded,
depending on when the object becomes readable.
Samuel Just [Wed, 12 Feb 2014 18:53:13 +0000 (10:53 -0800)]
PG: allow PGBackend to set criteria for PG up-ness
ECBackend needs to be able to require that a readable
set of the most recent interval to write be available
in order to ensure that it rolls back the log far
enough.
Samuel Just [Tue, 28 Jan 2014 00:52:05 +0000 (16:52 -0800)]
PGBackend: add some additional helpers.
ECBackend's primary specific logic mostly won't treat the
primary shard specially, so it'll be handy to have primary
agnostic helpers for get_shard_info and get_shard_missing.
Samuel Just [Wed, 12 Feb 2014 18:44:45 +0000 (10:44 -0800)]
osd/: extend pg_interval_t to include primary
Otherwise, we cannot correctly determine up_from/up_thru for
old intervals. Also, we need this information to determine
when a new interval starts due to a new primary without a
change in the acting set.
Samuel Just [Thu, 16 Jan 2014 23:27:36 +0000 (15:27 -0800)]
messages/: include shard information in various pg messages
We can no longer use the messenger source information to determine
the origin of the message since an osd might have more than one
shard of a particular pg. Thus, we need to include a pg_shard_t
from field to indicate origin. Similarly, pg_t is no longer
sufficient to specify the destination pg, we instead use spg_t.
In the event that we get a message from an old peer, we default
from to pg_shard_t(get_source().num(), ghobject_t::no_shard())
and spg_t to spg_t(pgid, ghobject_t::no_shard()). This suffices
because non-NO_SHARD shards can only appear once ec pools have
been enabled -- and doing that bans unenlightened osds.
Samuel Just [Thu, 23 Jan 2014 21:32:21 +0000 (13:32 -0800)]
ReplicatedPG,osd_types: seperate require_rollback from ec_pool
It's handy to allow a pool to answer false to ec_pool() and
true to require_rollback() in order to allow a replicated
pool to test the rollback mechanisms without allowing
non-NO_SHARD shards.
projects / ceph.git / log