]> git.apps.os.sepia.ceph.com Git - ceph.git/log
ceph.git
11 years agotest/librados: remove unused lines and fix return code for cmp guards 1256/head
Josh Durgin [Tue, 18 Feb 2014 01:09:26 +0000 (17:09 -0800)]
test/librados: remove unused lines and fix return code for cmp guards

Now that return values are actually set by the osd and client, fix up
the tests that were checking the old behavior.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoReplicatedPG: fix successful write condition
Josh Durgin [Tue, 18 Feb 2014 06:42:13 +0000 (22:42 -0800)]
ReplicatedPG: fix successful write condition

ctx->modify no longer implies that the operation is a write,
if it ever did. These days op->may_write() is checked reliably
on the OSD, so just use that instead.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoReplicatedPG: clear osd op reply output for writes
Josh Durgin [Tue, 18 Feb 2014 06:35:13 +0000 (22:35 -0800)]
ReplicatedPG: clear osd op reply output for writes

Since the vector of OSDOps used by the reply is the same
as the one processed by do_osd_ops() now, any output data
needs to be cleared for writes. To be compatible with current behavior,
allow writes that aren't applying anything or have failed to return data
still.

Add a new parameter to the MOSDOpReply constructor to determine
whether the output data should be cleared. Clear it for successful
writes, and remove a redundant result < 0 -> result > 0 check in the
process.  This was caught by ceph_test_cls_hello and its
writes_dont_return_data method.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoReplicatedPG: set individual osd op rvals
Josh Durgin [Fri, 7 Feb 2014 02:20:06 +0000 (18:20 -0800)]
ReplicatedPG: set individual osd op rvals

This takes care of things that can fail before transaction is
executed, like omap comparison. Getting individual rvals from a
transaction requires more refactoring, so I'm leaving it for a later
cleanup.

Fixes: #6483
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoReplicatedPG: don't copy the ops vector into OpContext
Josh Durgin [Fri, 7 Feb 2014 02:17:06 +0000 (18:17 -0800)]
ReplicatedPG: don't copy the ops vector into OpContext

There's no need to copy it, and if we refer to the same vector from
the original MOSDOp, we can modify rvals for individual ops, since the
MOSDOp ops vector is copied to the MOSDOpReply when it is constructed.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoReplicatedPG: avoid undefined behavior in xattr comparison
Josh Durgin [Fri, 7 Feb 2014 02:50:50 +0000 (18:50 -0800)]
ReplicatedPG: avoid undefined behavior in xattr comparison

Reading past the end of a pointer returned by string.data() in c++98
is undefined. While we're fixing this, also allow comparison of xattrs
containing null bytes.

Fixes: #7250
Backport: dumpling
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoqa: add librados c object operations tests to librados test script
Josh Durgin [Fri, 7 Feb 2014 02:12:54 +0000 (18:12 -0800)]
qa: add librados c object operations tests to librados test script

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agopackaging: add test programs for librados c object operations
Josh Durgin [Fri, 7 Feb 2014 02:12:33 +0000 (18:12 -0800)]
packaging: add test programs for librados c object operations

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: bump version number
Josh Durgin [Fri, 7 Feb 2014 04:11:10 +0000 (20:11 -0800)]
librados: bump version number

This covers the many recent changes leading up to the firefly release,
including the c object operations api.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add omap object operations to the c api
Josh Durgin [Fri, 7 Feb 2014 04:10:15 +0000 (20:10 -0800)]
librados: add omap object operations to the c api

Create an iterator type similar to XattrIter for returning results.
It just wraps the std::map to preserve sorting.

When getting omap keys or values, use an extra callback to set up the
iterator, and default optional parameters to the empty string.

Since it's hard to test read or write omap ops in isolation, combine
them all in the c read ops test case.

One difference between this and the c++ api is that omap_cmp only
allows one assertion per sub-op, for simplicity of the
interface. Multiple omap_cmp operations can still be added to the same
op to get the same effect.

Fixes: #7194
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add read xattr functions for c object operations
Josh Durgin [Fri, 7 Feb 2014 04:03:20 +0000 (20:03 -0800)]
librados: add read xattr functions for c object operations

Only implement string xattr comparison by since integer xattr
comparison assumes ceph-encoded integers in little-endian 64 bit
format, which should not be exposed to librados users.

Add an extra callback for getxattr so that we can initialize the
iterator.

Fixes: #7193
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoObjecter: add a way to chain callbacks
Josh Durgin [Fri, 7 Feb 2014 03:55:30 +0000 (19:55 -0800)]
Objecter: add a way to chain callbacks

The librados C api needs to do extra things like converting c++ data
structures or setting lengths, but some objecter operations already
have out_handlers that librados shouldn't override.

Add a method that chains contexts together by using a new context that
calls both the original and the newly added one.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add exec to the c object operations api
Josh Durgin [Fri, 7 Feb 2014 03:51:59 +0000 (19:51 -0800)]
librados: add exec to the c object operations api

Add two versions: one that allocates a buffer of the appropriate
length for the user, but relies on the user to free it, and one that
uses a user-supplied buffer but may fail if it is too short.

Reuse the bufferlist -> buffer conversion context added for reads into
the user supplied buffer.

The librados-allocated buffer can be handled just like librados
allocated buffers used by the various command functions, so just reuse
do_out_buffer() for them.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add read to c object operation api
Josh Durgin [Fri, 7 Feb 2014 03:45:53 +0000 (19:45 -0800)]
librados: add read to c object operation api

Do the usual bufferlist to buffer conversion in a callback from the
objecter before the librados user gets called.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoObjecter: add a context argument to ObjectOperation::read()
Josh Durgin [Fri, 7 Feb 2014 03:40:11 +0000 (19:40 -0800)]
Objecter: add a context argument to ObjectOperation::read()

The c api will need this to be able to set the output buffer length
and potentially copy from bufferlist to buffer.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add simple atomic read operations to c api
Josh Durgin [Fri, 7 Feb 2014 03:37:41 +0000 (19:37 -0800)]
librados: add simple atomic read operations to c api

stat, assert_exists, and set_flags are simple and need no extra
infrastrucutre.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add exec to the c write operations api
Josh Durgin [Fri, 7 Feb 2014 03:25:31 +0000 (19:25 -0800)]
librados: add exec to the c write operations api

Nothing special needed here, just copying the input buffer and passing
things through. Don't allow output data since it's not usually
available for writes and unreliable when it is.

Fixes: #7195
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add individual op flags for c write operations
Josh Durgin [Fri, 7 Feb 2014 03:19:51 +0000 (19:19 -0800)]
librados: add individual op flags for c write operations

Move flag validation to a static function so it can be shared with the
c++ api. Refer to the new C constants from the c++ api so that it's
easy to keep them in sync.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add global op flags to the c api
Josh Durgin [Fri, 7 Feb 2014 03:12:58 +0000 (19:12 -0800)]
librados: add global op flags to the c api

The c++ api did not include these in the sync versions of the calls,
so add a flags argument to the IoCtxImpl functions for those, with
default arguments to avoid changing the many many callers.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: remove constant for CMPXATTR_NOP
Josh Durgin [Fri, 7 Feb 2014 03:03:56 +0000 (19:03 -0800)]
librados: remove constant for CMPXATTR_NOP

This constant was just added in the public header to match the
operations in rados.h, and was not included in a stable release yet.
NOP is misleading, since it's actually uninterpreted anywhere, and
using it would always return EIO from the OSD. It's not particularly
useful, so just remove it. Adding a cmpxattr with FAIL_OK is
equivalent.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add some comments about constants in librados.h
Josh Durgin [Fri, 7 Feb 2014 02:58:26 +0000 (18:58 -0800)]
librados: add some comments about constants in librados.h

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agolibrados: add section docs for object operations
Josh Durgin [Mon, 3 Feb 2014 19:46:59 +0000 (11:46 -0800)]
librados: add section docs for object operations

The doxygen end section of watch notify was accidentally moved when
these were added, so fix that at the same time.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: add ObjectReadOperation tests
Josh Durgin [Fri, 7 Feb 2014 00:32:21 +0000 (16:32 -0800)]
test/librados: add ObjectReadOperation tests

These will help prevent regressions and demonstrate what should be
expected by the c api.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: refactor setup/teardown in tiering tests
Josh Durgin [Mon, 3 Feb 2014 09:00:49 +0000 (01:00 -0800)]
test/librados: refactor setup/teardown in tiering tests

These tests can either use the standard RadosTest, with one pool, or a
new test case with two pools.

The new test case has to copy a couple lines in static methods
(working around lack of virtual static methods is more complex than
it's worth here), but the rest is straight forward. Rename
base_pool_name and base_ioctx to pool_name and ioctx to match the
member variables of the parent class RadosTest.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: use standard rados test case for snapshot tests
Josh Durgin [Mon, 3 Feb 2014 08:56:27 +0000 (00:56 -0800)]
test/librados: use standard rados test case for snapshot tests

Split these into two categories, self-managed and pool snapshots,
since they are mutually exclusive for a single pool. Remove snapshots
and objects at the end of tests, regardless of pass/fail, so the same
pool can be reused.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: use a test case instead of duplicating code everywhere
Josh Durgin [Mon, 3 Feb 2014 08:54:21 +0000 (00:54 -0800)]
test/librados: use a test case instead of duplicating code everywhere

These tests don't have any special requirements on their ioctx and
pools, so they can also run much faster by using different namespaces
per test instead of new pools. For the few tests that do depend on
namespace, reset the ioctx to the default namespace.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados/cmd: remove unnecessary pool creation
Josh Durgin [Sun, 2 Feb 2014 17:45:31 +0000 (09:45 -0800)]
test/librados/cmd: remove unnecessary pool creation

Several tests here don't use the pool at all.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: use connect_cluster_pp() instead of duplicating it
Josh Durgin [Sun, 2 Feb 2014 22:33:31 +0000 (14:33 -0800)]
test/librados: use connect_cluster_pp() instead of duplicating it

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: add a connect_cluster() helper
Josh Durgin [Sun, 2 Feb 2014 22:32:13 +0000 (14:32 -0800)]
test/librados: add a connect_cluster() helper

This mirrors the c++ version, connect_cluster_pp(), and removes the
same code from create_one_pool().

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: create general test case classes
Josh Durgin [Mon, 3 Feb 2014 17:01:20 +0000 (09:01 -0800)]
test/librados: create general test case classes

Using a test case allows us to remove a boatload of boilerplate code
in all the tests, and focus them more on what they're actually
testing.

Create one for C tests, and one for C++ tests, with the same
functionality:

- create a pool when the test case starts
- between individual tests, create an ioctx and set its namespace uniquely
- delete objects from the default namespace during individual test teardown
- delete the pool only when the whole test case is finished

In gtest, a test case is the whole set of tests declared as part of
the same class using TEST_F(). Many tests create and delete individual
pools, but this is unnecessary for independent operation in most
cases, since we can use namespaces for that now. Since pool creation
and deletion dominates test run time, using these test cases makes
running many of the tests much faster.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agotest/librados: move test.cc into its own library
Josh Durgin [Thu, 6 Feb 2014 23:09:57 +0000 (15:09 -0800)]
test/librados: move test.cc into its own library

This way it doesn't have to be compiled many times, and it's easier
to add new general functionality to new files without adding those
.cc files to every test program.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoObjecter: keep ObjectOperation rval pointers for writes
Josh Durgin [Thu, 30 Jan 2014 14:55:29 +0000 (06:55 -0800)]
Objecter: keep ObjectOperation rval pointers for writes

Just before sending an op, prepare_mutate_op() is called, creating a
new Op. prepare_read_op() already copied over all the out-params
correctly, but for write operations the individual op return value
pointers were not copied, so they would not be filled in. With this
fixed, librados users can get the per-op return codes again.

Partially fixes: #6483
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
11 years agoMerge pull request #1214 from ceph/wip-rgw-manifest-2
Josh Durgin [Tue, 18 Feb 2014 21:07:53 +0000 (13:07 -0800)]
Merge pull request #1214 from ceph/wip-rgw-manifest-2

Wip rgw manifest 2
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
11 years agodencoder: fix for new rgw manifest code 1214/head
Yehuda Sadeh [Sat, 8 Feb 2014 01:00:07 +0000 (17:00 -0800)]
dencoder: fix for new rgw manifest code

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
11 years agocls/rgw: fix debug output
Yehuda Sadeh [Sat, 8 Feb 2014 00:34:52 +0000 (16:34 -0800)]
cls/rgw: fix debug output

print the correct error value

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
11 years agotest/rgw: manifest unitest
Yehuda Sadeh [Mon, 3 Feb 2014 22:58:00 +0000 (14:58 -0800)]
test/rgw: manifest unitest

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
11 years agorgw: scalable manifest object
Yehuda Sadeh [Fri, 31 Jan 2014 00:05:42 +0000 (16:05 -0800)]
rgw: scalable manifest object

Define a new manifest structure. The idea is that the manifest defines a
set of rules for structuring the object parts. There are a few terms to
note:
 - head: the head part of the object, which is the part that contains
   the first chunk of data. An object might not have a head (as in the
   case of multipart-part objects.
 - stripe: data portion of a single rgw object that resides on a single
   rados object.
 - part: a collection of stripes that make a contiguous part of an
   object. A regular object will only have one part (although might have
   many stripes), a multipart object might have many parts. Each part
   has a fixed stripe size, although the last stripe of a part might
   be smaller than that. Consecutive parts may be merged if their stripe
   value is the same.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
11 years agoMerge pull request #1227 from ceph/wip-7394
Gregory Farnum [Tue, 18 Feb 2014 20:34:29 +0000 (12:34 -0800)]
Merge pull request #1227 from ceph/wip-7394

Improve the OSD and generic monitor timeout mechanisms

Reviewed-by: Sage Weil <sage@inktank.com>
11 years agoMerge remote-tracking branch 'upstream/wip-ecbackend-for-review'
Samuel Just [Tue, 18 Feb 2014 18:56:02 +0000 (10:56 -0800)]
Merge remote-tracking branch 'upstream/wip-ecbackend-for-review'

Reviewed-by: Sage Weil <sage@inktank.com>
11 years agoMerge pull request #1255 from ceph/wip-cache-perf
Sage Weil [Tue, 18 Feb 2014 16:22:17 +0000 (08:22 -0800)]
Merge pull request #1255 from ceph/wip-cache-perf

osd: add perfcounters for new cache and agent events

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
11 years agoosd/ReplicatedPG: clean up agent skip debug output 1255/head
Sage Weil [Mon, 17 Feb 2014 21:05:58 +0000 (13:05 -0800)]
osd/ReplicatedPG: clean up agent skip debug output

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoosd: l_osd_agent_{wake,skip,flush,evict}
Sage Weil [Mon, 17 Feb 2014 20:23:44 +0000 (12:23 -0800)]
osd: l_osd_agent_{wake,skip,flush,evict}

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoosd: l_osd_tier_[dirty|clean]
Sage Weil [Mon, 17 Feb 2014 20:20:28 +0000 (12:20 -0800)]
osd: l_osd_tier_[dirty|clean]

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoosd: l_osd_tier_whiteout
Sage Weil [Mon, 17 Feb 2014 20:19:36 +0000 (12:19 -0800)]
osd: l_osd_tier_whiteout

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoosd: l_osd_tier_evict
Sage Weil [Mon, 17 Feb 2014 20:18:49 +0000 (12:18 -0800)]
osd: l_osd_tier_evict

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoosd: l_osd_tier_[try_]flush[_fail]
Sage Weil [Mon, 17 Feb 2014 18:55:54 +0000 (10:55 -0800)]
osd: l_osd_tier_[try_]flush[_fail]

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoosd: l_osd_copyfrom
Sage Weil [Mon, 17 Feb 2014 18:52:36 +0000 (10:52 -0800)]
osd: l_osd_copyfrom

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoRadosModel: only output if missing header is actually a problem 1228/head
Samuel Just [Sun, 16 Feb 2014 01:33:39 +0000 (17:33 -0800)]
RadosModel: only output if missing header is actually a problem

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoObjecter: track primary explicitly to detect changing primaries
Samuel Just [Sun, 16 Feb 2014 01:33:07 +0000 (17:33 -0800)]
Objecter: track primary explicitly to detect changing primaries

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedPG: add some debugging if waiting_for_commit is non-empty
Samuel Just [Sun, 16 Feb 2014 01:32:34 +0000 (17:32 -0800)]
ReplicatedPG: add some debugging if waiting_for_commit is non-empty

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd/: instantiate the right backend based on pool
Samuel Just [Mon, 27 Jan 2014 21:37:08 +0000 (13:37 -0800)]
osd/: instantiate the right backend based on pool

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoECBackend: flesh out ECBackend implementation
Samuel Just [Thu, 6 Feb 2014 02:49:41 +0000 (18:49 -0800)]
ECBackend: flesh out ECBackend implementation

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd/ECTransaction: add type encapsulating EC transaction
Samuel Just [Sat, 1 Feb 2014 04:20:55 +0000 (20:20 -0800)]
osd/ECTransaction: add type encapsulating EC transaction

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd/ECUtil: add ec encode/decode util helpers
Samuel Just [Fri, 20 Dec 2013 00:15:27 +0000 (16:15 -0800)]
osd/ECUtil: add ec encode/decode util helpers

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoObjectStore: add bufferlist/string getattr
Samuel Just [Sat, 15 Feb 2014 00:13:01 +0000 (16:13 -0800)]
ObjectStore: add bufferlist/string getattr

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agobuffer: allow caller to initialize bufferhash state
Samuel Just [Mon, 17 Feb 2014 01:42:30 +0000 (17:42 -0800)]
buffer: allow caller to initialize bufferhash state

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd_types: add swap, claim_append to ObjectModDesc
Samuel Just [Mon, 17 Feb 2014 05:26:41 +0000 (21:26 -0800)]
osd_types: add swap, claim_append to ObjectModDesc

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoOSD,PG: pass through EC messages
Samuel Just [Sat, 1 Feb 2014 04:06:58 +0000 (20:06 -0800)]
OSD,PG: pass through EC messages

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agomessages,osd: add EC messages and associated types
Samuel Just [Fri, 20 Dec 2013 00:15:03 +0000 (16:15 -0800)]
messages,osd: add EC messages and associated types

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agosrc/osd/: add ECBackend stubs
Samuel Just [Thu, 12 Dec 2013 19:17:04 +0000 (11:17 -0800)]
src/osd/: add ECBackend stubs

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGBackend: make getattrs virtual
Samuel Just [Sun, 16 Feb 2014 03:08:42 +0000 (19:08 -0800)]
PGBackend: make getattrs virtual

This way, the backend can choose to filter private attrs.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedPG: fix size checking for ECBackend pools
Samuel Just [Sun, 16 Feb 2014 03:08:13 +0000 (19:08 -0800)]
ReplicatedPG: fix size checking for ECBackend pools

Allow the backend to adjust the logical object size before
checking against the ondisk size.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGBackend/ReplicatedBackend: factor out logical to ondisk size mapping
Samuel Just [Sat, 8 Feb 2014 22:04:14 +0000 (14:04 -0800)]
PGBackend/ReplicatedBackend: factor out logical to ondisk size mapping

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGBackend/ReplicatedBackend: move the backend agnostic code into PGBackend for scrub
Samuel Just [Sat, 8 Feb 2014 21:52:28 +0000 (13:52 -0800)]
PGBackend/ReplicatedBackend: move the backend agnostic code into PGBackend for scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedBackend: factor out be_deep-scrub
Samuel Just [Sat, 8 Feb 2014 21:40:50 +0000 (13:40 -0800)]
ReplicatedBackend: factor out be_deep-scrub

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd/: replace bufferptr with bufferlist for attrs in recovery
Samuel Just [Thu, 30 Jan 2014 21:06:50 +0000 (13:06 -0800)]
osd/: replace bufferptr with bufferlist for attrs in recovery

Now we won't have to keep converting back and forth!

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG::discover_all_missing: request a FULLLOG
Samuel Just [Mon, 10 Feb 2014 22:05:09 +0000 (14:05 -0800)]
PG::discover_all_missing: request a FULLLOG

Otherwise, we cannot correct for divergent objects in the missing
set.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG,ReplicatedPG: Generalize missing_loc for ECBackend
Samuel Just [Wed, 29 Jan 2014 21:38:04 +0000 (13:38 -0800)]
PG,ReplicatedPG: Generalize missing_loc for ECBackend

Prior to EC pools, unfound => missing.  Now, unfound (unreadable,
really) is dependent on the PGBackend requirements for reconstituting
an object.  This also means recovering an object missing on a replica
but not the primary requires tracking the missing_loc set.

Thus, rather than maintaining missing_loc only for objects missing
on the primary, the MissingLoc structure will track all missing
objects actingbackfill-wide until each object is recovered.

For simplicity, since we don't really know what objects need recovery
until activation (and since we can't do anything with that information
prior to activation anyway), we defer populating the missing_loc
information until activation.

We need peers to rollback divergent log entries before we attempt to
read the relevant objects.  The simplest way to accomplish this seems to
be the simply choose to always activate peers if search_for_missing
turns up missing objects.

Due to EC pools, missing is necessary, but not sufficient for readability.
Thus, we instead check is_unreadable for cases where we need to read the object
and reserve is_missing for cases where we need the object context.

wait_for_missing_object becomes waiting_for_unreadable_object in order to avoid
having another layer of waiting_for_* maps.  These ops may be requeued
either when the primary is recovered or when the object is no longer degraded,
depending on when the object becomes readable.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG: allow PGBackend to set criteria for PG up-ness
Samuel Just [Wed, 12 Feb 2014 18:53:13 +0000 (10:53 -0800)]
PG: allow PGBackend to set criteria for PG up-ness

ECBackend needs to be able to require that a readable
set of the most recent interval to write be available
in order to ensure that it rolls back the log far
enough.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGBackend: add some additional helpers.
Samuel Just [Tue, 28 Jan 2014 00:52:05 +0000 (16:52 -0800)]
PGBackend: add some additional helpers.

ECBackend's primary specific logic mostly won't treat the
primary shard specially, so it'll be handy to have primary
agnostic helpers for get_shard_info and get_shard_missing.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG,PGBackend: expose acting and backfill_targets
Samuel Just [Tue, 28 Jan 2014 00:52:20 +0000 (16:52 -0800)]
PG,PGBackend: expose acting and backfill_targets

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoos/: allow objects with NO_GEN but non-NO_SHARD shard
Samuel Just [Sat, 1 Feb 2014 05:42:45 +0000 (21:42 -0800)]
os/: allow objects with NO_GEN but non-NO_SHARD shard

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG: skip CRUSH_ITEM_NONE in acting set in various places
Samuel Just [Wed, 12 Feb 2014 18:46:09 +0000 (10:46 -0800)]
PG: skip CRUSH_ITEM_NONE in acting set in various places

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd/: extend pg_interval_t to include primary
Samuel Just [Wed, 12 Feb 2014 18:44:45 +0000 (10:44 -0800)]
osd/: extend pg_interval_t to include primary

Otherwise, we cannot correctly determine up_from/up_thru for
old intervals.  Also, we need this information to determine
when a new interval starts due to a new primary without a
change in the acting set.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoObjecter: handle primary other than acting[0]
Samuel Just [Wed, 12 Feb 2014 18:48:56 +0000 (10:48 -0800)]
Objecter: handle primary other than acting[0]

EC pool particularly might have CRUSH_ITEM_NONE as acting[0] which
is not a valid osd.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoOSDMap: handle CRUSH_ITEM_NONE in acting sets
Samuel Just [Wed, 12 Feb 2014 18:45:07 +0000 (10:45 -0800)]
OSDMap: handle CRUSH_ITEM_NONE in acting sets

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoOSDMap: fill in get_primary_shard
Samuel Just [Thu, 23 Jan 2014 22:52:01 +0000 (14:52 -0800)]
OSDMap: fill in get_primary_shard

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG: introduce calc_ec_acting
Samuel Just [Thu, 23 Jan 2014 21:09:54 +0000 (13:09 -0800)]
PG: introduce calc_ec_acting

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG: refactor calc_acting to handle pg_shard_t changes
Samuel Just [Thu, 23 Jan 2014 18:12:45 +0000 (10:12 -0800)]
PG: refactor calc_acting to handle pg_shard_t changes

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd/: refer to pg by spg_t and replica by pg_shard_t
Samuel Just [Tue, 28 Jan 2014 20:50:05 +0000 (12:50 -0800)]
osd/: refer to pg by spg_t and replica by pg_shard_t

We may have multiple pg shards on the same osd for an
ec pool.  Thus, replicas must be referred to by <osd, shard>
and pgs by <pgid, shard>.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agomessages/: include shard information in various pg messages
Samuel Just [Thu, 16 Jan 2014 23:27:36 +0000 (15:27 -0800)]
messages/: include shard information in various pg messages

We can no longer use the messenger source information to determine
the origin of the message since an osd might have more than one
shard of a particular pg.  Thus, we need to include a pg_shard_t
from field to indicate origin.  Similarly, pg_t is no longer
sufficient to specify the destination pg, we instead use spg_t.
In the event that we get a message from an old peer, we default
from to pg_shard_t(get_source().num(), ghobject_t::no_shard())
and spg_t to spg_t(pgid, ghobject_t::no_shard()).  This suffices
because non-NO_SHARD shards can only appear once ec pools have
been enabled -- and doing that bans unenlightened osds.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedBackend: excise OSDService*
Samuel Just [Sun, 16 Feb 2014 01:43:19 +0000 (17:43 -0800)]
ReplicatedBackend: excise OSDService*

This should eventually make it easier to mock out a PGBackend::Listener.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGBackend/ReplicatedBackend: move temp handling into PGBackend
Samuel Just [Thu, 23 Jan 2014 01:56:30 +0000 (17:56 -0800)]
PGBackend/ReplicatedBackend: move temp handling into PGBackend

Temp handling is also the same in ReplicatedBackend as in ECBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGBackend: use whoami_shard().shard in rollback*, objects*
Samuel Just [Sat, 1 Feb 2014 06:52:37 +0000 (22:52 -0800)]
PGBackend: use whoami_shard().shard in rollback*, objects*

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedBackend/PGBackend: move objects_* methods into PGBackend
Samuel Just [Thu, 23 Jan 2014 01:50:04 +0000 (17:50 -0800)]
ReplicatedBackend/PGBackend: move objects_* methods into PGBackend

These also are unchanged in ECBackend.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGBackend/ReplicatedBackend: move rollback methods into PGBackend
Samuel Just [Thu, 23 Jan 2014 01:46:22 +0000 (17:46 -0800)]
PGBackend/ReplicatedBackend: move rollback methods into PGBackend

These will end up essentially unchanged in ECBackend as it turns out.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd_types: introduce spg_t, pg_shard_t
Samuel Just [Wed, 11 Dec 2013 23:03:44 +0000 (15:03 -0800)]
osd_types: introduce spg_t, pg_shard_t

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoObjectStore: use ghobject_t for setattrs
Samuel Just [Mon, 16 Dec 2013 23:40:04 +0000 (15:40 -0800)]
ObjectStore: use ghobject_t for setattrs

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agosrc/osd: pass in version to recover to recover_object
Samuel Just [Fri, 27 Dec 2013 19:26:11 +0000 (11:26 -0800)]
src/osd: pass in version to recover to recover_object

Otherwise we need to do some annoying contortions to
get it out of the missing sets.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd/: Add a shard parameter to SnapMapper to handle multiple pg shards
Samuel Just [Thu, 6 Feb 2014 02:42:34 +0000 (18:42 -0800)]
osd/: Add a shard parameter to SnapMapper to handle multiple pg shards

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoTestPGLog: remove test with DELETE op with prior_version = eversion_t()
Samuel Just [Mon, 17 Feb 2014 22:20:44 +0000 (14:20 -0800)]
TestPGLog: remove test with DELETE op with prior_version = eversion_t()

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPGLog::merge_old_entry: simplify the oe.version > ne.version case
Samuel Just [Mon, 10 Feb 2014 22:07:07 +0000 (14:07 -0800)]
PGLog::merge_old_entry: simplify the oe.version > ne.version case

If ne.version < oe.version, the correct answer is to rollback oe.version
if possible regardless of what the entries are.

Also, rearrange to deal with the fact that we cannot roll back a missing
object.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoosd: l_osd_tier_promote
Sage Weil [Mon, 17 Feb 2014 18:50:39 +0000 (10:50 -0800)]
osd: l_osd_tier_promote

Signed-off-by: Sage Weil <sage@inktank.com>
11 years agoMerge pull request #1128 from ceph/wip-dirfrags
Yan, Zheng [Mon, 17 Feb 2014 23:15:58 +0000 (07:15 +0800)]
Merge pull request #1128 from ceph/wip-dirfrags

Fixes for mds cluster + directory fragments

11 years agoPG: just ignore rollbacks on objects we have already failed to rollback
Samuel Just [Fri, 7 Feb 2014 01:26:24 +0000 (17:26 -0800)]
PG: just ignore rollbacks on objects we have already failed to rollback

The relevant changes to missing and/or the filestore will have
already been made.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoPG::remove_snap_mapped_object: use pg_whoami.shard
Samuel Just [Mon, 10 Feb 2014 22:04:30 +0000 (14:04 -0800)]
PG::remove_snap_mapped_object: use pg_whoami.shard

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedPG: take snapset_obc write lock where appropriate
Samuel Just [Sun, 16 Feb 2014 01:44:57 +0000 (17:44 -0800)]
ReplicatedPG: take snapset_obc write lock where appropriate

Otherwise, we might read it for backfill before it's fully created
on a peer.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedPG: release backfill state on_global_recover not peer recover
Samuel Just [Tue, 11 Feb 2014 02:35:47 +0000 (18:35 -0800)]
ReplicatedPG: release backfill state on_global_recover not peer recover

There may now be multiple backfill peers.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedPG: take clone write lock as well in make_writeable
Samuel Just [Tue, 11 Feb 2014 02:34:52 +0000 (18:34 -0800)]
ReplicatedPG: take clone write lock as well in make_writeable

Otherwise, we might start backfilling the clone before the op is
complete.

Signed-off-by: Samuel Just <sam.just@inktank.com>
11 years agoReplicatedPG,osd_types: seperate require_rollback from ec_pool
Samuel Just [Thu, 23 Jan 2014 21:32:21 +0000 (13:32 -0800)]
ReplicatedPG,osd_types: seperate require_rollback from ec_pool

It's handy to allow a pool to answer false to ec_pool() and
true to require_rollback() in order to allow a replicated
pool to test the rollback mechanisms without allowing
non-NO_SHARD shards.

Signed-off-by: Samuel Just <sam.just@inktank.com>