Samuel Just [Wed, 8 May 2013 23:19:34 +0000 (16:19 -0700)]
PG,OSD: delay ops for map prior to queueing in the OpWQ
Previously, we simply queued ops in the OpWQ without checking. The PG
would then check in do_request whether the message should wait for a new
map. Unfortunately, this has the side effect that any op requeued for
any reason must also requeue the waiting_for_map queue.
Now, we will check before queueing the op whether it must wait on a map.
To avoid contention, there is now a map_lock which must be held along
with the PG lock in order to update the osdmap_ref. The map_lock also
protects the waiting_for_map list and queueing PG ops at the back of
the OpWQ. A few details:
1) It is no longer necessary to requeue waiting_for_map in on_change()
since the other ops are queued at the front.
2) Once waiting_for_map is non-empty, all ops are delayed to simplify
ordering.
3) waiting_for_map may now be non-empty during split, so we must split
waiting_for_map along with waiting_for_active. This must be done
under the map_lock.
The bug which uncovered this involved an out of order op as follows:
client.4208.0:2378 (e252) arrives, object is degraded
client.4208.0:2379 (e253) arrives, waits for map
client.4208.0:2378 (e252) is requeued after recovery
client.4208.0:2379 (e253) is requeued on map arrival
client.4208.0:2379 is processed
client.4208.0:2378 is processed
Fixes: #4955 Signed-off-by: Samuel Just <sam.just@inktank.com>
Sage Weil [Wed, 8 May 2013 21:54:33 +0000 (14:54 -0700)]
ceph-create-keys: gracefully handle no data from admin socket
Old ceph-mon (prior to 393c9372f82ef37fc6497dd46fc453507a463d42) would
return an empty string and success if the command was not registered yet.
Gracefully handle that case by retrying.
If we still fail to parse, exit entirely with EINVAL.
Fixes: #4952 Signed-off-by: Sage Weil <sage@inktank.com> Reviewed-by: Dan Mick <dan.mick@intank.com>
Samuel Just [Tue, 7 May 2013 23:41:22 +0000 (16:41 -0700)]
OSD: handle stray snap collections from upgrade bug
Previously, we failed to clear snap_collections, which causes split to
spawn a bunch of snap collections. In load_pgs, we now clear any such
snap collections and then snap_collections field on the PG itself.
Related: #4927 Signed-off-by: Samuel Just <sam.just@inktank.com> Reviewed-by: Sage Weil <sage@inktank.com>
Sage Weil [Mon, 6 May 2013 18:40:52 +0000 (11:40 -0700)]
ceph-disk: use separate lock files for prepare, activate
Use a separate lock file for prepare and activate to avoid deadlock. This
didn't seem to trigger on all machines, but in many cases, the prepare
process would take the file lock and later trigger a udev event and the
activate would then block on the same lock, either when we explicitly call
'udevadm settle --timeout=10' or when partprobe does it on our behalf
(without a timeout!). Avoid this by using separate locks for prepare
and activate. We only care if multiple activates race; it is
okay for a prepare to be in progress and for an activate to be kicked
off.
Sage Weil [Fri, 3 May 2013 23:20:26 +0000 (16:20 -0700)]
mon: fix init sequence when not daemonizing
We made the common_init_finish and chdir conditional on daemonize in commit 2e0dd5ae6c8751e33d456b2b06c1204b63db959a, breaking init (asok at least)
when -f is specified (as with upstart).
Signed-off-by: Sage Weil <sage@inktank.com> Reviewed-by: Greg Farnum <greg@inktank.com>
Sage Weil [Fri, 3 May 2013 18:29:24 +0000 (11:29 -0700)]
mon: fork early to avoid leveldb static env state
leveldb has static state that prevents it from recreating its worker thread
after our fork(), even when we close and reopen the database (tsk tsk!).
Avoid this by forking early, before we touch leveldb.
Hide the details in a Preforker class. This is modeled after what
ceph-fuse already does; we should convert it later.
Signed-off-by: Sage Weil <sage@inktank.com> Reviewed-by: Samuel Just <sam.just@inktank.com> Reviewed-by: Greg Farnum <greg@inktank.com>
"complete list of distributions" should be complete list of releases since we already know what distributions are supported and the list specifies releases. (Wheezy, Precise etc.)
Samuel Just [Wed, 1 May 2013 21:59:08 +0000 (14:59 -0700)]
OSD: load_pgs() should fill in start_split honestly
In load_pgs(), we previously called assigned children starting
at the loaded pg created between its stored epoch and the current
osdmap to have that pg as their parent. This is not correct, some
of the children may have been split in subsequent epochs from children
split in earlier epochs. Instead, do each map individually.
Greg Farnum [Wed, 1 May 2013 21:10:31 +0000 (14:10 -0700)]
dumper: fix Objecter locking
Locking expectations changed at some point, and the Dumper wasn't
updated to comply:
1) We need to take the lock for Objecter, as it
doesn't do so on its own any more.
2) We need to drop the lock in several places so that Objecter
can take delivery of messages
Signed-off-by: Greg Farnum <greg@inktank.com> Reviewed-by: Sage Weil <sage@inktank.com>
Sage Weil [Wed, 1 May 2013 17:57:35 +0000 (10:57 -0700)]
mon/Paxos: update first_committed when we trim
The Paxos::trim() -> ::trim_to() path trims old states but does not
update first_committed. This misinforms later paxos rounds such that
peers think they can participate and end up with COMMIT messages
following the COLLECT/LAST exchange that are for future commits they
can't do anything with and then crash out when they get the BEGIN:
Sage Weil [Wed, 1 May 2013 04:16:16 +0000 (21:16 -0700)]
mon/Paxos: don't ignore peer first_committed
We go to the effort of keeping a map of the peer's first/last committed
so that we can send the right commits during the first phase of paxos,
but we forgot to record the first value. This appears to simply be an
oversight. It is mostly harmless; it just means we send extra states
that the peer already has.
Signed-off-by: Sage Weil <sage@inktank.com> Reviewed-by: Greg Farnum <greg@inktank.com>
Samuel Just [Tue, 30 Apr 2013 22:48:10 +0000 (15:48 -0700)]
OSD: clean up in progress split state on pg removal
There are two cases: 1) The parent pg has not yet initiated the split 2) The
parent pg has initiated the split.
Previously in case 1), _remove_pg left the entry for its children in the
in_progress_splits map blocking subsequent peering attempts.
In case 1), we need to unblock requests on the child pgs for the parent on
parent removal. We don't need to bother waking requests since any requests
received prior to the remove_pg request are necessarily obsolete.
In case 2), we don't need to do anything: the child will complete the split on
its own anyway.
Thus, we now track pending_splits vs in_progress_splits. Children in
pending_splits are in state 1), in_progress_splits in state 2). split_pgs
bumps pgs from pending_splits to in_progress_splits atomically with respect to
_remove_pg since the parent pg lock is held in both places.
Fixes: #4813 Signed-off-by: Samuel Just <sam.just@inktank.com> Reviewed-by: Greg Farnum <greg@inktank.com>