tests: tiering health report reworked

tests: tiering health report reworked

Instead of

* setting limit
* populate the cache
* check the health warnings

do the following

* populate the cache
* set limits below the content of the cache
* check the health warnings

The problem with the former approach is that the limits stored by the
OSD internally do not exactly match the one set by the user: they are
converted in ratios and there may be rounding errors.

Also replace the busy loop waiting for pg stats to flush with

  ceph tell osd.* flush_pg_stats || true

for simplicity.

Signed-off-by: Loic Dachary <ldachary@redhat.com>