mgr/volumes: access volume in lockless mode when fetching async job

Saw a deadlock when deleting lot of subvolumes -- purge threads were
stuck in accessing global lock for volume access. This can happen
when there is a concurrent remove (which renames and signals the
purge threads) and a purge thread is just about to scan the trash
directory for entries.

For the fix, purge threads fetches entries by accessing the volume
in lockless mode. This is safe from functionality point-of-view as
the rename and directory scan is correctly handled by the filesystem.
Worst case the purge thread would pick up the trash entry on next
scan, never leaving a stale trash entry.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 808a1ce1f96f6dfd3472156ce5087372da4c1314)

diff --git a/src/pybind/mgr/volumes/fs/async_cloner.py b/src/pybind/mgr/volumes/fs/async_cloner.py
index 6554a1c34ca810cb7039e474ac47ce841c9c4df7..88dac61b74be51da75bce1c1edb3dece3b4d367d 100644 (file)
--- a/src/pybind/mgr/volumes/fs/async_cloner.py
+++ b/src/pybind/mgr/volumes/fs/async_cloner.py
@@ -24,7 +24,7 @@ def get_next_clone_entry(volume_client, volname, running_jobs):
     log.debug("fetching clone entry for volume '{0}'".format(volname))
 
     try:
-        with open_volume(volume_client, volname) as fs_handle:
+        with open_volume_lockless(volume_client, volname) as fs_handle:
             try:
                 with open_clone_index(fs_handle, volume_client.volspec) as clone_index:
                     job = clone_index.get_oldest_clone_entry(running_jobs)

diff --git a/src/pybind/mgr/volumes/fs/purge_queue.py b/src/pybind/mgr/volumes/fs/purge_queue.py
index 922c8f0027688a323f60a3323baaa72e12c6999f..41052ba303d658999b4ce67b85d2d050b7236eb2 100644 (file)
--- a/src/pybind/mgr/volumes/fs/purge_queue.py
+++ b/src/pybind/mgr/volumes/fs/purge_queue.py
@@ -13,7 +13,7 @@ def get_trash_entry_for_volume(volume_client, volname, running_jobs):
     log.debug("fetching trash entry for volume '{0}'".format(volname))
 
     try:
-        with open_volume(volume_client, volname) as fs_handle:
+        with open_volume_lockless(volume_client, volname) as fs_handle:
             try:
                 with open_trashcan(fs_handle, volume_client.volspec) as trashcan:
                     path = trashcan.get_trash_entry(running_jobs)