DeterministicOpSequence: writing to object being cloned in the same tx.
We write to the object being cloned prior to the clone to ensure we are
cloning a valid range of bytes.
The write and the clone were being done in two distinct transactions,
which would trigger a diff mismatch if a failure happened to occur within
the write tx. By not differentiating the transactions when building a
pristine copy, we were executing one more transaction (the clone_range one)
than the transactions that were executed in the failed filestore, thus
triggering the mismatch (one more object in the pristine filestore than on
the failed filestore).
Now we issue a single transaction, containing both the clone_range()
preceded by a write().
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
Sage Weil [Thu, 12 Apr 2012 17:40:08 +0000 (10:40 -0700)]
filestore: name internally
We need to allow the perfcounter name to be controlled so that we can have
two instances of FileStore in the same process that don't step on each
other. Default to 'filestore'.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
test_idempotent_sequence: Generate a reproducible sequence of txs.
With this test we aim at reproducing the same sequence of transactions
as long as we are provided with the same seed between runs.
We also allow failures to be injected onto the filestore if the
--filestore-kill-at <VAL> argument is passed, and we provide verification
when --test-verify-at <VAL> is provided.
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
VerifyFileStore: Check if two FileStore's match after applying a set of operations.
With DeterministicOpSequence we are able to reproduce exactly the same
sequence of operations, over and over. However, if the filestore fails
(e.g., because we injected a failure), we want to check if it is kept
consistent after replaying its journal.
With VerifyFileStore, which extends DeterministicOpSequence, we are able
to bring a brand new filestore to the state the failed filestore would
reach had it not failed. We can then compare to check if the failure
introduced inconsistencies after replaying the journal.
(This is still work in progress and not fully functional)
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
TestFileStoreState: Represent a FileStore's state to be used by tests.
Instead of having each test creating the same representation of a
FileStore's state, with a map/set of collections and objects, as well as
multiple init() functions for each test that are in all similar in
nature, provide this in a single class that can be inheritted by test
classes.
Sage Weil [Sat, 14 Apr 2012 00:11:54 +0000 (17:11 -0700)]
filestore: two-phase guard
For certain operations (collection_add) we need a two-phase guard, and an
"in-progress" state.
- before exposing an object in a new location, we need to mark it so that
old operations affecting the target name don't touch the new object.
- can't just set the guard before starting or else we can't distinguish
between a collection_add that was in-progress and one that happend a
long time ago.
We may need the same for collection_rename().
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Sat, 14 Apr 2012 00:17:36 +0000 (17:17 -0700)]
filestore: simple failure injections via --filestore-kill-at <n>
This will make filestore suicide (_exit(1)) on the n'th potential failure
call site. We can potentially fail:
- before a transaction
- between each op
- at the end
Additionally, we instrument the guards:
- before/after/inside _set_replay_guard
- between significant steps of callers of _set_replay_guard
All instrumentation points are inside _do_transactions(), so if everything
is done in a single sequencer (or from a single thread) the failure
point is deterministic.
That said, use an atomic so we will still reliably fail (at some point)
when there are multiple filestore threads in action.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Fri, 13 Apr 2012 18:30:49 +0000 (11:30 -0700)]
filestore: replay collection_move using add+remove
This approximates the buggy collection_move. It is still buggy. It is
only there to replay old journals.
Rip out buggy (and now unused) collection_move code.
For the record, the problem there is that a crash between setting the guard
and unlinking the old name will not remove the old name on replay because
the guard for the link stage is indistinguishable from that for the unlink
stage.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Fri, 13 Apr 2012 16:56:04 +0000 (09:56 -0700)]
filestore: implement collection_move() as add + remove
This ensures we get add and remove steps with different spos values, which
makes the guard work. The collection_move implementation breaks on replay
because those values match, so the just-set guard prevents unlink from
happening.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Tue, 10 Apr 2012 22:24:49 +0000 (15:24 -0700)]
filestore: fix collection_move guard
We had a sequence like:
1- write A block 1
2- write A block 2
3- write A block 3
4- write A block 4
5- move A -> B
- link B
- unlink A
- set guard on B <crash>
- replay 3, 4, 5
with the result being B with only half of its content. The problem is that
we destroyed the old link _and_ didn't guard the new content. Instead,
set the guard before the link, and replay the unlink step here
unconditionally.
Fixes: #2178 Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Fri, 6 Apr 2012 16:33:43 +0000 (09:33 -0700)]
encoding: use iterator to copy_in encoded length
This gives us a pointer to the position into the list where the final
length value will be copied. Previously we used bl.copy_in(), which takes
a byte offset and needs iterator over the bufferlist to seek to the
correct position, resulting in O(n^2) encoding time for large structures.
Fixes: #2161 Reported-by: Jim Schutt <jaschut@sandia.gov> Diagnosed-by: Ake van der Meer <petrabbit@xs4all.nl> Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Tue, 3 Apr 2012 21:21:53 +0000 (14:21 -0700)]
rgw: throttle at num_threads * 2
If we throttle at num_threads, then nothing gets into the workqueue until
a worker thread is idle, which means you pay the latency of setting it up
and queueing it. This way we keep some requests ready to go.
Greg Farnum [Wed, 28 Mar 2012 22:06:32 +0000 (15:06 -0700)]
msgr: clean up Pipe::do_sendmsg.
Document it as with the tcp stuff, remove an if(0)'d debugging block,
and remove the useless "sd" parameter since it's always the same as
the Pipe's sd member.
Greg Farnum [Tue, 27 Mar 2012 19:57:14 +0000 (12:57 -0700)]
msgr: make a bunch of stuff private.
Why were all these data members public? They're accessed by Pipes
and the Accepter and stuff, so maybe that's why...but that's all
internal interface stuff.
Convert ms_addr and _my_name to be references to their fields in
the entity_inst_t my_inst.
This way we can use const references for accessing all of them,
instead of the bizarre distinction we had before for get_myinst().
Greg Farnum [Mon, 19 Mar 2012 20:12:14 +0000 (13:12 -0700)]
msgr: change the signature of get_myaddr()
Return a const reference to the actual address, instead of copying it.
All current users are happy with this, and I can't see a good reason
to copy it instead.
Greg Farnum [Thu, 8 Mar 2012 00:43:04 +0000 (16:43 -0800)]
msgr: get_connection() is required to establish a connection if none exists.
Making an allowance for lossy server connections is silly. Just don't
ask for the Connection in that case. (There aren't any users who
rely on the previous behavior.)
Document that requirement in Messenger.h!
Greg Farnum [Sat, 31 Mar 2012 00:07:19 +0000 (17:07 -0700)]
ceph_mon: fix fsid parsing.
fsid is a field in the CephContext _conf structure and is parsed by
the standard options parsing library before it gets to the ceph_mon
custom parsing.
Instead do the standard parsing, and check that member directly
to decide if we want to (over)write the monmap's fsid.
Sage Weil [Fri, 30 Mar 2012 23:14:05 +0000 (16:14 -0700)]
osd: update_stats() on reads too
Update pg stats on any op completion (read or write), not just writes. Do
the calls with log_op_stats() for consistency's sake. Skip if the request
was an error.
Fixes: #2209 Signed-off-by: Sage Weil <sage.weil@dreamhost.com> Reviewed-by: Samuel Just <samuel.just@dreamhost.com>
Tommi Virtanen [Wed, 28 Mar 2012 20:55:01 +0000 (13:55 -0700)]
doc: Convert the mailing list mention to not be a section heading.
If toctree is inside a section, the subtree is inside the section too.
We don't want all of dev/* to be under "Mailing list".
I have not found a decent workaround for this. The toplevel toctree
avoids this purely by the fact that it is the topmost toctree. Right
now that means you should 1) avoid having more than a few paragraphs of
text before the toctree for that subtree (put most of the content after
the toctree; clumsy if the toctree is long), or 2) put the toptree
immediately after the document title, make it :hidden:, and let the
reader use links in the text or the ToC in the sidebar to navigate.
See start/index for an example of this.
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>
projects / ceph.git / log