osd/scrub: extracting scrub scheduling code from OSD.cc
A separate object (ScrubQueue) now manages the collection of ScrubJob-s
that are to be scrubbed.
Notable changes:
- ScrubQueue keeps track of the CPU load (instead of the OSD main)
- ScrubJob-s are part of each PG's scrubber. They are registered with the
ScrubQueue when eligible for scrubbing (i.e. - active Primary), and removed
from it otherwise.
- "a PG is trying to secure Replica resources" was always treated as an OSD-wide
state that delays new scrubs from starting. Here - it is also implemented as such
(instead of per-PG state that must be queried)
Internally, the SQ maintains two groups of jobs: the regular ones, waiting to be
scrubbed, and the "penalized" - those PGs that failed in securing their
replicas' scrub resources (@dzafman changes, slightly modified to fit the
new environment)
Also included: minor fixes to the scrub state-machine and events priorities.
Signed-off-by: Ronen Friedman <rfriedma@redhat.com> Co-authored-by: David Zafman <dzafman@redhat.com>