From: Sage Weil <sage@redhat.com>
Date: Mon, 28 Nov 2016 20:56:29 +0000 (-0500)
Subject: os/bluestore/KernelDevice: fix race in aio_thread vs aio_wait
X-Git-Tag: v11.1.0~125^2
X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=refs%2Fpull%2F12204%2Fhead;p=ceph.git

os/bluestore/KernelDevice: fix race in aio_thread vs aio_wait

The caller is free to destroy the aio vector contents
once aio_wait completes.  This is exactly what
BlueFS::_fsync() does.  Delay the num_running dec
(which is what aio_wait waits for) until after we've
examined the aios.

Fixes: http://tracker.ceph.com/issues/17824
Signed-off-by: Sage Weil <sage@redhat.com>
---

diff --git a/src/os/bluestore/KernelDevice.cc b/src/os/bluestore/KernelDevice.cc
index b49032b2405..56173bb8e4a 100644
--- a/src/os/bluestore/KernelDevice.cc
+++ b/src/os/bluestore/KernelDevice.cc
@@ -262,12 +262,14 @@ void KernelDevice::_aio_thread()
 	  std::lock_guard<std::mutex> l(debug_queue_lock);
 	  debug_aio_unlink(*aio[i]);
 	}
-	int left = --ioc->num_running;
 	int r = aio[i]->get_return_value();
 	dout(10) << __func__ << " finished aio " << aio[i] << " r " << r
 		 << " ioc " << ioc
 		 << " with " << left << " aios left" << dendl;
 	assert(r >= 0);
+	int left = --ioc->num_running;
+	// NOTE: once num_running is decremented we can no longer
+	// trust aio[] values; they my be freed (e.g., by BlueFS::_fsync)
 	if (left == 0) {
 	  // check waiting count before doing callback (which may
 	  // destroy this ioc).