osd: log inconsistent shard sizes

When an OSD asserts because it cannot recover from an unexpected erasure
coded shard size, the object needs to be manually fixed (i.e. the shard
must be removed so it can be reconstructed). Unfortunately the assert
does not display the name of the object and it is not displayed unless
the OSD logs are >= 10, which is uncommon on an actual cluster because
it creates log files that are too big.

The problem has been fixed in infernalis as part of
https://github.com/ceph/ceph/pull/5173 and backporting it is non
trivial.

The error condition is modified to make it easy for the system
administrator to identify which object is causing the crash and what
steps must be taken to fix the issue.

http://tracker.ceph.com/issues/14009 Refs: #14009

Signed-off-by: Loic Dachary <loic@dachary.org>

diff --git a/src/osd/ECBackend.cc b/src/osd/ECBackend.cc
index 3b517402bcef79943953ceb1208e6a02e8766110..00cbd54c6d7411833d2dd567ac2b1428727cdc95 100644 (file)
--- a/src/osd/ECBackend.cc
+++ b/src/osd/ECBackend.cc
@@ -361,7 +361,17 @@ void ECBackend::handle_recovery_read_complete(
     from[i->first.shard].claim(i->second);
   }
   dout(10) << __func__ << ": " << from << dendl;
-  ECUtil::decode(sinfo, ec_impl, from, target);
+  if (ECUtil::decode(sinfo, ec_impl, from, target) != 0) {
+    derr << __func__ << ": inconsistent shard sizes " << hoid << " "
+        << " the offending shard must be manually removed "
+        << " after verifying there are enough shards to recover "
+        << "(" << to_read.get<0>()
+        << ", " << to_read.get<1>()
+        << ", " << to_read.get<2>()
+        << ")"
+        << dendl;
+    assert(0);
+  }
   if (attrs) {
     op.xattrs.swap(*attrs);
 

diff --git a/src/osd/ECUtil.cc b/src/osd/ECUtil.cc
index 1f3b45857da8a533524d267d56baa07c767fc801..efc57b5e8a4988b64c6ebd2ce3d6a250874875a4 100644 (file)
--- a/src/osd/ECUtil.cc
+++ b/src/osd/ECUtil.cc
@@ -56,7 +56,8 @@ int ECUtil::decode(
   for (map<int, bufferlist>::iterator i = to_decode.begin();
        i != to_decode.end();
        ++i) {
-    assert(i->second.length() == total_chunk_size);
+    if (i->second.length() != total_chunk_size)
+      return -EINVAL;
   }
 
   if (total_chunk_size == 0)