Sage Weil [Thu, 12 Apr 2012 17:40:08 +0000 (10:40 -0700)]
filestore: name internally
We need to allow the perfcounter name to be controlled so that we can have
two instances of FileStore in the same process that don't step on each
other. Default to 'filestore'.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
test_idempotent_sequence: Generate a reproducible sequence of txs.
With this test we aim at reproducing the same sequence of transactions
as long as we are provided with the same seed between runs.
We also allow failures to be injected onto the filestore if the
--filestore-kill-at <VAL> argument is passed, and we provide verification
when --test-verify-at <VAL> is provided.
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
VerifyFileStore: Check if two FileStore's match after applying a set of operations.
With DeterministicOpSequence we are able to reproduce exactly the same
sequence of operations, over and over. However, if the filestore fails
(e.g., because we injected a failure), we want to check if it is kept
consistent after replaying its journal.
With VerifyFileStore, which extends DeterministicOpSequence, we are able
to bring a brand new filestore to the state the failed filestore would
reach had it not failed. We can then compare to check if the failure
introduced inconsistencies after replaying the journal.
(This is still work in progress and not fully functional)
Signed-off-by: Joao Eduardo Luis <jecluis@gmail.com>
TestFileStoreState: Represent a FileStore's state to be used by tests.
Instead of having each test creating the same representation of a
FileStore's state, with a map/set of collections and objects, as well as
multiple init() functions for each test that are in all similar in
nature, provide this in a single class that can be inheritted by test
classes.
Sage Weil [Sat, 14 Apr 2012 00:11:54 +0000 (17:11 -0700)]
filestore: two-phase guard
For certain operations (collection_add) we need a two-phase guard, and an
"in-progress" state.
- before exposing an object in a new location, we need to mark it so that
old operations affecting the target name don't touch the new object.
- can't just set the guard before starting or else we can't distinguish
between a collection_add that was in-progress and one that happend a
long time ago.
We may need the same for collection_rename().
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Sat, 14 Apr 2012 00:17:36 +0000 (17:17 -0700)]
filestore: simple failure injections via --filestore-kill-at <n>
This will make filestore suicide (_exit(1)) on the n'th potential failure
call site. We can potentially fail:
- before a transaction
- between each op
- at the end
Additionally, we instrument the guards:
- before/after/inside _set_replay_guard
- between significant steps of callers of _set_replay_guard
All instrumentation points are inside _do_transactions(), so if everything
is done in a single sequencer (or from a single thread) the failure
point is deterministic.
That said, use an atomic so we will still reliably fail (at some point)
when there are multiple filestore threads in action.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Fri, 13 Apr 2012 18:30:49 +0000 (11:30 -0700)]
filestore: replay collection_move using add+remove
This approximates the buggy collection_move. It is still buggy. It is
only there to replay old journals.
Rip out buggy (and now unused) collection_move code.
For the record, the problem there is that a crash between setting the guard
and unlinking the old name will not remove the old name on replay because
the guard for the link stage is indistinguishable from that for the unlink
stage.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Fri, 13 Apr 2012 16:56:04 +0000 (09:56 -0700)]
filestore: implement collection_move() as add + remove
This ensures we get add and remove steps with different spos values, which
makes the guard work. The collection_move implementation breaks on replay
because those values match, so the just-set guard prevents unlink from
happening.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Tue, 10 Apr 2012 22:24:49 +0000 (15:24 -0700)]
filestore: fix collection_move guard
We had a sequence like:
1- write A block 1
2- write A block 2
3- write A block 3
4- write A block 4
5- move A -> B
- link B
- unlink A
- set guard on B <crash>
- replay 3, 4, 5
with the result being B with only half of its content. The problem is that
we destroyed the old link _and_ didn't guard the new content. Instead,
set the guard before the link, and replay the unlink step here
unconditionally.
Fixes: #2178 Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Fri, 6 Apr 2012 16:33:43 +0000 (09:33 -0700)]
encoding: use iterator to copy_in encoded length
This gives us a pointer to the position into the list where the final
length value will be copied. Previously we used bl.copy_in(), which takes
a byte offset and needs iterator over the bufferlist to seek to the
correct position, resulting in O(n^2) encoding time for large structures.
Fixes: #2161 Reported-by: Jim Schutt <jaschut@sandia.gov> Diagnosed-by: Ake van der Meer <petrabbit@xs4all.nl> Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Tue, 3 Apr 2012 21:21:53 +0000 (14:21 -0700)]
rgw: throttle at num_threads * 2
If we throttle at num_threads, then nothing gets into the workqueue until
a worker thread is idle, which means you pay the latency of setting it up
and queueing it. This way we keep some requests ready to go.
Greg Farnum [Wed, 28 Mar 2012 22:06:32 +0000 (15:06 -0700)]
msgr: clean up Pipe::do_sendmsg.
Document it as with the tcp stuff, remove an if(0)'d debugging block,
and remove the useless "sd" parameter since it's always the same as
the Pipe's sd member.
Greg Farnum [Tue, 27 Mar 2012 19:57:14 +0000 (12:57 -0700)]
msgr: make a bunch of stuff private.
Why were all these data members public? They're accessed by Pipes
and the Accepter and stuff, so maybe that's why...but that's all
internal interface stuff.
Convert ms_addr and _my_name to be references to their fields in
the entity_inst_t my_inst.
This way we can use const references for accessing all of them,
instead of the bizarre distinction we had before for get_myinst().
Greg Farnum [Mon, 19 Mar 2012 20:12:14 +0000 (13:12 -0700)]
msgr: change the signature of get_myaddr()
Return a const reference to the actual address, instead of copying it.
All current users are happy with this, and I can't see a good reason
to copy it instead.
Greg Farnum [Thu, 8 Mar 2012 00:43:04 +0000 (16:43 -0800)]
msgr: get_connection() is required to establish a connection if none exists.
Making an allowance for lossy server connections is silly. Just don't
ask for the Connection in that case. (There aren't any users who
rely on the previous behavior.)
Document that requirement in Messenger.h!
Greg Farnum [Sat, 31 Mar 2012 00:07:19 +0000 (17:07 -0700)]
ceph_mon: fix fsid parsing.
fsid is a field in the CephContext _conf structure and is parsed by
the standard options parsing library before it gets to the ceph_mon
custom parsing.
Instead do the standard parsing, and check that member directly
to decide if we want to (over)write the monmap's fsid.
Sage Weil [Fri, 30 Mar 2012 23:14:05 +0000 (16:14 -0700)]
osd: update_stats() on reads too
Update pg stats on any op completion (read or write), not just writes. Do
the calls with log_op_stats() for consistency's sake. Skip if the request
was an error.
Fixes: #2209 Signed-off-by: Sage Weil <sage.weil@dreamhost.com> Reviewed-by: Samuel Just <samuel.just@dreamhost.com>
Tommi Virtanen [Wed, 28 Mar 2012 20:55:01 +0000 (13:55 -0700)]
doc: Convert the mailing list mention to not be a section heading.
If toctree is inside a section, the subtree is inside the section too.
We don't want all of dev/* to be under "Mailing list".
I have not found a decent workaround for this. The toplevel toctree
avoids this purely by the fact that it is the topmost toctree. Right
now that means you should 1) avoid having more than a few paragraphs of
text before the toctree for that subtree (put most of the content after
the toctree; clumsy if the toctree is long), or 2) put the toptree
immediately after the document title, make it :hidden:, and let the
reader use links in the text or the ToC in the sidebar to navigate.
See start/index for an example of this.
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>
Sage Weil [Fri, 30 Mar 2012 16:51:45 +0000 (09:51 -0700)]
filestore: set guard on collection_move
During recovery we submit transactions like:
- delete a/foo
- move tmp/foo to a/foo
This prevents the EEXIST check in collection_move from doing any good,
since the destination never exists. We need to do that remove at least
sometimes, because we may be overwriting an existing/older version of the
object.
So,
- set the guard after we do the move, so that
- the delete won't be repated, and
- the EEXIST check will work
Also check the guard for good measure (although that doesn't do anything
specifically useful in this scenario).
Fixes: #2164 Signed-off-by: Sage Weil <sage@newdream.net> Reviewed-by: Josh Durgin <josh.durgin@dreamhost.com> Reviewed-by: Samuel Just <samuel.just@dreamhost.com>
Sage Weil [Thu, 29 Mar 2012 05:32:30 +0000 (22:32 -0700)]
osd: discard heartbeat_peer in note_down_osd
Discard the heartbeat_peer as soon as we find out, along with queued
failures, or else the heartbeat_check may come along (without map_lock)
and requeue a failure. And then later, when we try to report it, we'll
osdmap->get_inst() on a now-down OSD and fail miserably.
Reported-by: Wido den Hollander <wido@widodh.nl> Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
test: test_workload_gen: Add callback for collection destruction.
When we remove a collection, we must cleanup after the coll_entry_t we
once had on the available collections set. For some reason, we weren't
doing this.
This commit adds a new callback, which inherits from the 'OnReadable'
callback on the WorkloadGenerator class, that will be responsible for
deleting the coll_entry_t once we know the collection transaction
destroying the collection has finished.
test: test_workload_gen: Change CLI option and add '--help' usage.
With this commit, we support the following options (and old ones are no
longer available):
--test-num-colls VAL Set the number of collections
--test-num-objs-per-coll VAL Set the number of objects per
collection
--test-destroy-coll-per-N-trans VAL Set how many transactions to run
before destroying a collection.
And --help will show the program's usage description.
test: test_workload_gen: Default arguments, and minor changes.
Besides adding support for default arguments, passed onto global_init(),
this commit fixes a conflict in Makefile.am, and a missing lib
dependency. Also, we didn't used to pay attention to the return values
from store->mkfs() and store->mount(), and now do.
test: test_workload_gen: CodeStyle compliance and cleanup.
This commit aims at the compliance with Ceph's CodeStyle, as well
as cleaning up some lingering unused code.
Also, now we allow changing the default OSD data and journal
locations, as well as the OSD journal size, by providing the
options '--osd-data <PATH>', '--osd-journal <PATH>' and
'--osd-journal-size <VAL>' on the CLI arguments. If not provided,
these will default to 'workload_gen_dir', 'workload_gen_journal'
and '400', respectively.
projects / ceph.git / log