blk/KernelDevice: Fix several issues with stopping discard threads
1. In _discard_stop(), the wait for !discard_threads.empty() was there
from a prior implementation where there could theoretically be a race
between _discard_start() and _discard_stop(). If that race does
exist, this check won't help; not only is _discard_stop() not called
if discard_threads is empty, but discard_threads won't be populated
until _discard_start() runs and thus this won't detect such a race.
2. Calling _discard_stop() from handle_conf_change() is a guaranteed
deadlock because discard_lock is already held, so don't do that. Use
the same flow whether we're stopping a subset of threads or all
threads.
3. Asking a subset of discard threads to stop was not guaranteed to take
effect, since if they continued to find contents in discard_queue
then they would continue to run indefinitely. Add additional logic to
_discard_thread() to have threads stop if they have been requested to
stop and other threads exist to continue draining discard_queue.
4. Make the flow of _discard_stop() and handle_conf_change() more
similar.
Fixes: https://tracker.ceph.com/issues/66817
Signed-off-by: Joshua Baergen <jbaergen@digitalocean.com>
(cherry picked from commit
3d4a899d6c5f0c3c4c663d5d6dbb78285fbd7672)