Dan Mick [Sat, 1 Dec 2012 02:11:09 +0000 (18:11 -0800)]
librbd: handle parent change while async I/Os are in flight
During a test_librbd_fsx run including flatten, ImageCtx->parent
was being dereferenced while null. Between the time the parent
overlap is calculated and the time the guard+write completes
with ENOENT and submits the copyup+write, the parent image
could have changed (by resize) or been made irrelevant (by
child flatten) such that the parent overlap is now incorrect.
Handle "no parent" by just sending the copyup+write; the copyup
part will be a no-op. Move to WRITE_FLAT state in this case
because there's no more child to deal with.
Handle "overlap changed" by recalculating overlap before
reading parent data; if none is left, don't read, but rather
just clear m_object_image_extents, in which case the copyup
will again be a no-op because it will be of zero length.
However we still have a parent, so stay in WRITE_COPYUP state
and come back through as usual.
Signed-off-by: Dan Mick <dan.mick@inktank.com> Fixes: #3524
Yan, Zheng [Tue, 4 Dec 2012 08:09:48 +0000 (16:09 +0800)]
mds: journal remote inode's projected parent
Server::_rename_prepare() adds remote inode's parent instead of
projected parent to the journal. So during journal replay, the
journal entry for the rename operation will wrongly revert the
remote inode's projected rename. This issue can be reproduced by:
Yan, Zheng [Tue, 4 Dec 2012 08:09:47 +0000 (16:09 +0800)]
mds: don't create bloom filter for incomplete dir
Creating bloom filter for incomplete dir that was added by log
replay will confuse subsequent dir lookup and can create null
dentry for existing file. The erroneous null dentry confuses the
fragstat accounting and causes undeletable empty directory.
The fix is check if the dir is complete before creating the bloom
filter. For the MDCache::trim_non_auth{,_subtree} cases, just do
not call CDir::add_to_bloom because bloom filter is useless for
replica.
Yan, Zheng [Tue, 4 Dec 2012 08:09:48 +0000 (16:09 +0800)]
mds: journal remote inode's projected parent
Server::_rename_prepare() adds remote inode's parent instead of
projected parent to the journal. So during journal replay, the
journal entry for the rename operation will wrongly revert the
remote inode's projected rename. This issue can be reproduced by:
Yan, Zheng [Tue, 4 Dec 2012 08:09:47 +0000 (16:09 +0800)]
mds: don't create bloom filter for incomplete dir
Creating bloom filter for incomplete dir that was added by log
replay will confuse subsequent dir lookup and can create null
dentry for existing file. The erroneous null dentry confuses the
fragstat accounting and causes undeletable empty directory.
The fix is check if the dir is complete before creating the bloom
filter. For the MDCache::trim_non_auth{,_subtree} cases, just do
not call CDir::add_to_bloom because bloom filter is useless for
replica.
Sam Lang [Thu, 29 Nov 2012 20:32:32 +0000 (14:32 -0600)]
client: Fix ceph_mount() when subdir is specified
If a subdirectory is specified to ceph_mount, the
root inode does not have an ino of CEPH_INO_ROOT, so
cwd will fail to ever find root and eventially hits
an assertion in in->get_first_parent(). This fix uses
the inode stored in the root member instead, ensuring
that we stop wherever the mount is rooted.
Sage Weil [Wed, 28 Nov 2012 04:43:38 +0000 (20:43 -0800)]
mds: use TMAP_RMSLOPPY op when removing dentries
After replay, we don't know if the dentry removal has already been
committed. Use a sloppy removal so that we succeed even if we are
repeating the operation.
Conveniently, the previous implementation (pre v0.55) silently ignored
tmap op codes it did not understand, which means this new RMSLOPPY will
be interpreted the same as an actual RMSLOPPY. That means an v0.55
mds can run against an older osd (say, argonaut) without problems.
Rename applied_seq to max_applied_seq, since it is a bound; there may be
seq's < max_applied_seq that are not applied. This aligns the naming with
max_applying_seq.
Sage Weil [Sun, 2 Dec 2012 15:29:46 +0000 (07:29 -0800)]
os/FileStore: only wait for applying ops to complete before commit
We can have a large number of operations in the op_wq waiting to be applied
to the fs. Currently, when we want to commit, we want for them *all* to
apply. This can take a very long time (the default queue length is 500
operations!).
Instead, mark an Op as started ("applying") when the thread pool actually
starts to apply it. At that point, only wait for applying ops to complete.
We let any threads with an op seq < max_applying_seq begin as well so that
we have a proper ordering/barrier. When those flush, applied_seq will ==
max_applying_seq, and that becomes the committing_seq value.
Note that 'applied_seq' is still maintain, but serves no real purpose
except to populate our asserts with sanity checks. max_applying_seq serves
the purpose applied_seq used to.
This removes once unnecessary source of latency associated with fs
commits.
Sage Weil [Sat, 1 Dec 2012 05:43:06 +0000 (21:43 -0800)]
OutputDataSocket: fix uninit var
CID 745933 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
At (2): Non-static class member "data_size" is not initialized in this constructor nor in any functions that it calls.
Sage Weil [Sat, 1 Dec 2012 05:41:54 +0000 (21:41 -0800)]
rgw: fix uninit var
CID 745935 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
At (2): Non-static class member "expiration" is not initialized in this constructor nor in any functions that it calls.
At (2): Non-static class member "min_len" is not initialized in this constructor nor in any functions that it calls.
At (4): Non-static class member "max_len" is not initialized in this constructor nor in any functions that it calls.
At (6): Non-static class member "ret" is not initialized in this constructor nor in any functions that it calls.
At (8): Non-static class member "len" is not initialized in this constructor nor in any functions that it calls.
At (10): Non-static class member "ofs" is not initialized in this constructor nor in any functions that it calls.
At (12): Non-static class member "supplied_md5_b64" is not initialized in this constructor nor in any functions that it calls.
At (14): Non-static class member "supplied_etag" is not initialized in this constructor nor in any functions that it calls.
CID 745934 (#1 of 1): Uninitialized pointer field (UNINIT_CTOR)
At (16): Non-static class member "data_pending" is not initialized in this constructor nor in any functions that it calls.
Sage Weil [Sat, 1 Dec 2012 05:39:05 +0000 (21:39 -0800)]
test/osdc/FakeWriteback: fix uninit var
CID 745936 (#1 of 1): Uninitialized scalar field (UNINIT_CTOR)
At (2): Non-static class member "m_off" is not initialized in this constructor nor in any functions that it calls.
Yan, Zheng [Mon, 19 Nov 2012 02:43:46 +0000 (10:43 +0800)]
mds: fix freeze inode deadlock
CInode::freeze_inode() is used in the case of cross authority rename.
Server::handle_slave_rename_prep() calls it to wait for all other
operations on source inode to complete. This happens after all locks
for the rename operation are acquired. But to acquire locks, we need
auth pin locks' parent objects first. So there is an ABBA deadlock
if someone auth pins the source inode after locks for rename are
acquired and before Server::handle_slave_rename_prep() is called.
The fix is freeze and auth pin the source inode at the same time.
This patch introduces CInode::freeze_auth_pin(), it waits for all
other MDRequests to release auth pins, then change the inode to
FROZENAUTHPIN state, this state prevents other MDRequests from
getting new auth pins.
Yan, Zheng [Mon, 19 Nov 2012 02:43:48 +0000 (10:43 +0800)]
mds: use rdlock_try() when checking NULL dentry
Use rdlock_try() instead can_read() when path_traverse encounters
a NULL dentry. This can partly avoid infinitely waiting for the
dentry to become readable when the dentry is replica.
Strictly speaking, use rdlock_try() is still enough because auth
MDS may drop the REQRDLOCK message in some cases.
Yan, Zheng [Mon, 19 Nov 2012 02:43:47 +0000 (10:43 +0800)]
mds: allow open_remote_ino() to open xlocked dentry
discover_ino() has a parameter want_xlocked. The parameter indicates
if remote discover handler can proceed when xlocked dentry is
encountered. open_remote_ino() uses discover_ino() to find non-auth
inode, but always set 'want_xlocked' to false. This may cause
dead lock in some corner cases. For example:
we rename a inode's primary dentry to one of its remote dentry and
send slave request to one witness MDS. but before the slave request
reaches the witness MDS, the inode is trimmed from the witness MDS'
cache. Then when the slave request arrives, open_remote_ino() will
be called during traversing the destpath. open_remote_ino() calls
discover_ino() with 'want_xlocled=false' to find the inode.
discover_ino() sends MDiscover message to the inode's authority MDS.
The handler of MDiscover message finds the inode's primary dentry
is xlocked and it sleeps.
The fix is add a parameter 'want_xlocked' to open_remote_ino() and
make open_remote_ino() pass the parameter to discover_ino().
Yan, Zheng [Mon, 19 Nov 2012 02:43:44 +0000 (10:43 +0800)]
mds: fix open_remote_inode race
discover_ino() may return -ENOENT if it races with other FS activities.
so use C_MDC_RetryOpenRemoteIno instead of C_MDC_OpenRemoteIno as
onfinish callback.
Yan, Zheng [Mon, 19 Nov 2012 02:43:43 +0000 (10:43 +0800)]
mds: consider revoking caps in imported caps as issued
The clients may already send caps release message to the exporting
MDS, so the importing MDS waits for the release message forever.
consider revoking caps as issued can avoid this issue.
Yan, Zheng [Mon, 19 Nov 2012 02:43:42 +0000 (10:43 +0800)]
mds: drop locks if requiring auth pinning new objects.
Locker::acquire_locks() skip auth pinning replica object if we only
request a rdlock and the lock is read-lockable. To get all locks,
we may call Locker::acquire_locks() several times, locks in replca
objects may become not read-lockable between calls. So it is
possible we need auth pin new objects after already take some locks.
Yan, Zheng [Mon, 19 Nov 2012 02:43:40 +0000 (10:43 +0800)]
mds: don't forward client request from MDS
Forwarding client request that was from MDS will trigger assertion
in MDS::forward_message_mds(). MDS only send client requests for
stray migration/reintegration, so it's safe to drop them.
Yan, Zheng [Mon, 19 Nov 2012 02:43:39 +0000 (10:43 +0800)]
mds: call eval() after caps are exported
For an inode just changed authority, if the new auth MDS want to
change a lock in the inode from 'sync' to 'lock' state before caps
are exported. The lock in replica can be in 'sync->lock' state
because client caps prevent it from transitting to 'lock' state.
So we should call eval() after clearing client caps.
Yan, Zheng [Mon, 19 Nov 2012 02:43:37 +0000 (10:43 +0800)]
mds: Don't acquire replica object's versionlock
Both CInode and CDentry's versionlocks are of type LocalLock.
Acquiring LocalLock in replica object is useless and problematic.
For example, if two requests try acquiring a replica object's
versionlock, the first request succeeds, the second request
is added to wait queue. Later when the first request finishes,
MDCache::request_drop_foreign_locks() finds the lock's parent is
non-auth, it skips waking requests in the wait queue. So the
second request hangs.
Yan, Zheng [Mon, 19 Nov 2012 02:43:36 +0000 (10:43 +0800)]
mds: allow try_eval to eval unstable locks in freezing object
Unstable locks hold auth_pins on the object, it prevents the freezing
object become frozen and then unfreeze. So try_eval() should not wait
for freezing object
Sage Weil [Sat, 1 Dec 2012 04:23:52 +0000 (20:23 -0800)]
msg/Pipe: flush delayed messages when stealing/failing pipes
If we are failing a pipe, flush the incoming messages before we try to
reconnect. Similarly, flush queued messages on an existing pipe beore we
replace it. This ensures that when we get a socket failure and reconnect
the delayed messages are handled in the normal fashion.
Specifically, it fixes a situation like:
- read msg, update in_seq etc.
- delay msg
- pipe faults
- peer reconnects, we replace existing pipe, discard delayed msgs
- peer resends msgs
- we discard, because they are < in_seq
Samuel Just [Fri, 30 Nov 2012 22:04:53 +0000 (14:04 -0800)]
ReplicatedPG: only increment active_scrub on primary for final push
We only queue the _applied_recovered_object callback on the primary for the
final push. It is this callback which decrements active_pushes. It's ok to
not increment active_pushes for the intermediate pushes since these only affact
a temp file.
Sage Weil [Fri, 30 Nov 2012 00:45:52 +0000 (16:45 -0800)]
client: only dump cache on umount if we time out
We don't want to dump the cache every time an item is trimmed and the
mount_cond gets signaled; this can make umount crazy-slow when logging is
turned up.
Instead, only dump if we wait 5 seconds without making any progress on
shrinking the cache.
Sage Weil [Tue, 27 Nov 2012 23:36:11 +0000 (15:36 -0800)]
msg/Pipe: refactor msgr delays
- move all delay state into a single class
- create thread once and only once per Pipe
- adjust debug levels
- discard messages at the appropriate times
Greg Farnum [Tue, 27 Nov 2012 18:05:47 +0000 (10:05 -0800)]
msgr: add a delay_until queue that is used to delay deliveries.
Its life-cycle matches that of delay_queue, and the delayed_delivery
function respects it. For now queue_received is just setting it to
delay everything by 1 second.
Greg Farnum [Tue, 27 Nov 2012 17:44:19 +0000 (09:44 -0800)]
msgr: clear out the delay queue when stop()ing
After some brief thought, I believe deleting any messages in the
delay queue is correct -- we are trying to simulate line delays
in delivery and so anything still in the queue has supposedly
not arrived yet. So delete them when we stop the Pipe for
any reason.
Greg Farnum [Tue, 27 Nov 2012 19:02:07 +0000 (11:02 -0800)]
msgr: move the delay queue initialization into start_reader
The Pipe doesn't know the peer type in the constructor. It
doesn't always know in start_reader either, so this needs more work,
but at least it knows more frequently than it did.
Greg Farnum [Wed, 21 Nov 2012 18:54:06 +0000 (10:54 -0800)]
messenger: add the shell of a system to delay incoming Message delivery
When ms_inject_delay_type matches that of the incoming Connection,
the Pipe sets up a delay queue that it shuttles all Messages through.
This lets us check cleanup and some notification code but doesn't
actually generate any delays.
Sam Lang [Thu, 29 Nov 2012 18:19:51 +0000 (12:19 -0600)]
client: Fix for #3490 and config option to test
If the mds revokes our cache cap, and we follow
the _read_sync() path, on a zero-byte file the
osd returns ENOENT. We need to replace ENOENT
with a return of 0 in this case.
Samuel Just [Wed, 28 Nov 2012 23:10:43 +0000 (15:10 -0800)]
PG: scrubber.end should be exactly a boundary
Let scrubber.end be (foo, HEAD, 10) where the oid is foo , HEAD is the
snap, and 10 is the hash and scrubber.begin similarly be (bar, 5, 1).
After choosing to scan [(bar, 5, 1), (foo, HEAD, 10)), we block writes
on that interval.
1) A write might then come in for foo (which isn't blocked) which
creates a new snap (foo, 400, 10) which happens to fall in the interval.
This will result in a crash in _scrub() when it attempts to compare
clones since it will get (foo, 400, 10) but not the head object
(foo, HEAD, 10).
2) Alternately, the write from 1) has already happened. When we scan
the log, we find 34'10 and 34'11 are the clone operation creating
(foo, 400, 10) and the modify on (foo, HEAD, 10) respectively. Both
primary and replica will wait for last_update_applied to be 34'10
before scanning, but last_update_applied will in fact skip to 34'11
since 34'10 and 34'11 happened in the same transaction. This can
result in IO hanging on the scrubber interval.
Instead, we ensure that scrubber.end is exactly a hash boundary
(min hobject_t a with the specified hash). No such object can
exist since we don't create objects with empty oids, so no writes
can occur on that object.
Samuel Just [Wed, 28 Nov 2012 00:00:03 +0000 (16:00 -0800)]
OSD: history.last_epoch_started should start at 0
history.last_epoch_started marks a lower bound on the last epoch at
which the pg went active. As with info.last_epoch_started, it should be
0 prior to the first activation.
Samuel Just [Wed, 21 Nov 2012 21:59:22 +0000 (13:59 -0800)]
PG: maintain osd local last_epoch_started for find_best_info
In order to proceed with peering, we need an osd with a log including
the last commit sent to a client. This translates to the oldest
last_update from the infos of the most recent acting set to go active.
history.last_epoch_started gives us a lower bound on the last time the
entire acting set persisted authoratative logs/infos. However, it
doesn't indicate anything about the info/log on the osd which sent it.
Thus, we will maintain an osd local info.last_epoch_started to determine
which osds were actually active (and thus have the required log
entries). The max info.last_epoch_started in the prior set gives us an
upper bound on the last interval during which writes occurred. The min
last_update among the infos with that last_epoch_started must therefore
be an upper bound on the oldest operation which clients consider
committed. Any osd with an info.last_updated past that version must be
sufficient.
The observed bug was there was an empty pg info with a
last_epoch_started at the most recent interval which pushed
min_last_update_acceptable to eversion_t(). There were two down osds,
but peering proceeded since the backfill peer did survive. However,
its info was later disregarded due to incomplete. An empty osd was
then chosen as the best_info since it's last_update was equal to
min_last_update_acceptable. This caused the contents of the pg to be
lost.
projects / ceph.git / log