os/bluestore: pin onodes as they are added to the cache

When onodes are added to the cache, they may have more than one
reference already. Due to the implicit nature of boost::intrusive, in
the common case of creating a new Onode, they have 3 references at
this point.

Since they haven't been added to the cache yet, Onode::get() could not
pin them when they went from 1->2 references. Instead, take care of it
when they do get added to the cache. The pinning in get() is still
needed in case they get unpinned and then referenced again while still
cached.

This lack of pinning meant that newly created onodes were immediately
removed from the cache, since the last step of adding a new onode is
trimming the cache. In tests that read the object just after queueing
a write to it, the read got ENOENT, when it should have read the onode
from cache.

Fixes: https://tracker.ceph.com/issues/42495
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

diff --git a/src/os/bluestore/BlueStore.cc b/src/os/bluestore/BlueStore.cc
index 8998142b43244d4dccb16c3cc21f8986bf793ec8..9c8832a6498abeccd89ee5858965f202a97e55a0 100644 (file)
--- a/src/os/bluestore/BlueStore.cc
+++ b/src/os/bluestore/BlueStore.cc
@@ -855,8 +855,15 @@ struct LruOnodeCacheShard : public BlueStore::OnodeCacheShard {
 
   void _add(BlueStore::OnodeRef& o, int level) override
   {
-    (level > 0) ? lru.push_front(*o) : lru.push_back(*o);
+    ceph_assert(o->s == nullptr);
     o->s = this;
+    if (o->nref > 1) {
+      pin_list.push_front(*o);
+      o->pinned = true;
+      num_pinned = pin_list.size();
+    } else {
+      (level > 0) ? lru.push_front(*o) : lru.push_back(*o);
+    }
     num = lru.size();
   }
   void _rm(BlueStore::OnodeRef& o) override