Sage Weil [Tue, 10 Jan 2012 21:23:00 +0000 (13:23 -0800)]
osd: fail to peer if interval lacks any !incomplete replicas
We need at least one non-incomplete replica during a rw interval in order
to peer. The backfilling/incomplete replicas get log entries, but not
all object writes, so they are (mostly) excluded from the peering process
(find_best_info(), in particular).
We can't do this during the PriorSet calculation because we don't have
their PG::Info yet. But, once we get it, we need to make sure at least one
of the replicas during the last rw interval is not incomplete, or else we
should mark the pg DOWN (just like the PriorSet calculation does).
This logic mostly mirrors that of PriorSet, but additionally requires
the replicas be !incomplete.
Sage Weil [Mon, 9 Jan 2012 00:23:55 +0000 (16:23 -0800)]
osd: populate_obc_watchers when object pulled to primary
We don't care about degraded state, only whether the object is on the
primary so that we can load the object_info_t.
In particular, this avoids problems with backfill, where an object is
not degraded and populated, is then degraded while we backfill to the
target, and then not degraded again, and populate_obc_watchers() is called
a second time.
Fixes: #1903 Signed-off-by: Sage Weil <sage@newdream.net>
Sage Weil [Sat, 7 Jan 2012 01:18:01 +0000 (17:18 -0800)]
osd: clean up src_oid, src_obc map key calculation
Be consistent about how we generate the src_oid and src_oloc, so that we
feed good value into find_object_context and use a consistent key for
the src_obc map<>. This fixes a crash in do_osd_ops() due to a missing
src_obc key when the get_src_oloc() normalizes the key in do_op() but not
in do_osd_ops().
Also use a nicer name.
Fixes: #1897 Signed-off-by: Sage Weil <sage@newdream.net>
Sage Weil [Fri, 6 Jan 2012 19:38:15 +0000 (11:38 -0800)]
objecter: ignore replies from old request attempts
If we know the request attempt, ignore old attempts.
If we do not know the attempt (because the server is old), accept the
reply. This could lead to doing some ACK callbacks we shouldn't in
extreme failure/recovery scenarios, but that is better than doing
the callbacks out of order.
Partially fixes: #1490 Signed-off-by: Sage Weil <sage@newdream.net>
Greg Farnum [Thu, 5 Jan 2012 23:29:32 +0000 (15:29 -0800)]
mon: elector needs to reset leader_acked on every election start
Otherwise you never reset the leader_acked after a failed
election attempt, so if mon 0 is available on the first round
but then fails, you never make progress!
Greg Farnum [Thu, 5 Jan 2012 22:03:43 +0000 (14:03 -0800)]
mon: kill client sessions when we're not in quorum
After a timeout of 2*mon_lease length (ie, two election rounds),
kill existing client sessions so they can reconnect to a
monitor that's (hopefully) remained in the quorum. Let any
new client sessions stick around for a mon_lease interval, then
do the same to them.
Sage Weil [Wed, 4 Jan 2012 21:21:36 +0000 (13:21 -0800)]
mon: rev cluster protocol
The OSDMap NEW and AUTOOUT bit additions subtely change the decoding of
the incremental maps in a reasonably harmless way in that the bits get
implicitly cleared whenever the OSD weight changes from non-zero. The
monitors need to agree on this behavior to avoid odd behavior. We don't
care what clients see, since those bits are informational only.
Sage Weil [Wed, 4 Jan 2012 20:56:15 +0000 (12:56 -0800)]
mon: track auto-marked-out osds
Mark OSDs that were automatically marked OUT by the monitor because they
were down for too long. Clear the bit as soon as they are no longer out,
as soon as the weight is changed from 0.
Sage Weil [Wed, 4 Jan 2012 17:42:02 +0000 (09:42 -0800)]
osd: initialize backfill_pos on activate
Handling of writes depends on backfill_pos being initialized (to know what
is between the leading and trailing edge of the backfill), so it needs to
be initialized at activate time to avoid badness on writes prior to
recovery starting.
- initialize during activate to last_backfill
- update on receiving the digest to maintain the invariant that
backfill_pos = min(peer_backfill_info.start, backfill_info.start)
in recover_backfill().
Sage Weil [Sun, 1 Jan 2012 04:44:05 +0000 (20:44 -0800)]
osd: do not use incomplete peer for best info/log
For one, their stats are incomplete; if we use them we'll screw up everyone
else. For another, it doesn't do us any good if they are a bit ahead of
the peers: we/they may not even have the objects their newer log says were
updated. The only real use is if their log extends farther back in time,
but that is a problem in general that we'll eventually solve in other ways.
On the other hand, having the pg_stats sum only through last_backfill may
not have been the best choice; we could avoid that part of things by adding
a objects_backfilled field. But this is probably a good idea anyway.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Josh Durgin [Fri, 30 Dec 2011 02:36:54 +0000 (18:36 -0800)]
RadosModel: check for out of order replies within WriteOps
A single WriteOp already does multiple aio_writes. Each aio_write
gets a unique tid that is checked upon completion. There's no reason
to loop over the ranges twice since we can use the done flag instead
of the set of completions in WriteOp::finished().
Josh Durgin [Fri, 30 Dec 2011 02:30:57 +0000 (18:30 -0800)]
RadosModel: allow TestOps to pass data to their finish methods
This will allow nested writes to keep track of which write actually
completed. Also remove finish() and _finish() from TestOp subclasses
that had the same implementation as the superclass.
Greg Farnum [Sat, 24 Dec 2011 00:41:38 +0000 (16:41 -0800)]
osd: add a monitor timeout via MPGStatsAck messages
Keep track of when we have outstanding updates, and while we do, make
sure the monitor responds within a timeout (default 30 seconds). If
it doesn't, reconnect!
Sage Weil [Sat, 31 Dec 2011 23:09:58 +0000 (15:09 -0800)]
osd: trigger RecoveryFinished event on recovery completion
Unconditionally trigger the RecoveryFinished event when start_recvoery_ops
thinks it may be done. This lets us trigger the acting change (if needed),
or call finish_recovery() if needed.
This fixes the case where we are backfilling with up == acting, complete,
but don't call finish_recovery() or clear the backfill|degraded bits.
At some point we may want to move the is_all_uptodate() checks to the
caller.
Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
Sage Weil [Fri, 30 Dec 2011 23:51:45 +0000 (15:51 -0800)]
osd: do not backfill if any objects are missing on the primary
Someday we need to do something smarter so that a single unfound object
doesn't hold up replication of other objects. For now, this is the
simplest thing to do.
Florian Haas [Thu, 29 Dec 2011 19:58:01 +0000 (20:58 +0100)]
Add OCF-compliant resource agent for Ceph daemons
Add a wrapper around the ceph init script that makes
MDS, OSD and MON configurable as Open Cluster Framework
(OCF) compliant cluster resources. Allows Ceph
daemons to tie in with cluster resource managers that
support OCF, such as Pacemaker (http://www.clusterlabs.org).
Disabled by default, configure --with-ocf to enable.
Sage Weil [Fri, 30 Dec 2011 01:15:07 +0000 (17:15 -0800)]
mon: make full ratio config change callback safe
We can't propose_pending() from any context; do this in the tick() thread,
with the proper locking. Among other things, this fixes the crash on
startup that is now triggered due to eba235f2.
projects / ceph.git / log