osd/scrub: modify deep scrub interval randomization

osd/scrub: modify deep scrub interval randomization

The interaction between the various configuration parameters controlling
the scheduling of deep scrubs is not clearly defined nor clearly
documented.
The existing set of parameters creates unnecessary code complexity, is
surprising to the operators, and does not provide the level of control desired by
Ceph users.

This is a proposed change to the deep scrub interval randomization:

Pre this PR, deep scrubs scheduling is controlled by the following set
of parameters:
The desired interval between deep scrubs is determined by osd_deep_scrub_interval.
To prevent a "thundering herd" problem if multiple PGs were created at
the same time, a randomization effect was added: at a configurable frequency, a shallow
scrub is "upgraded" to a deep scrub.

As mentioned above, the interaction between these parameters isn't
always clear to the operators. But the main issue is its effect on code complexity
and design choices (as it is never known in advance whether the next scrub will be
deep or shallow).

Here we change the randomization method, decoupling it from shallow
scrubs scheduling. In the new method, deep scrubs are scheduled at the desired
interval - in average. The actual time is randomized to a normal distribution
with a CV of osd_deep_scrub_interval_cv (clamped to reasonable values).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>