teuthology/misc: make the Valgrind's early exit configurable.

teuthology/misc: make the Valgrind's early exit configurable.

This commit is a follow-up to a98eb3e1405c8ca8f6933eb0356c03955e4e2e83
where Valgrind has been configured to exit on first-seen error as it
was (wrongly!) assumed that all components are green when it comes
to the Valgrind verification.
This assumption turned out to be broken for RGW which got a few issues
over the course as a result of having the Valgrind checks knocked out
as a side effect of the python3 transition [1]. In the consequence,
multiple problems accumulated and introducing a mechanism to disable
the early exit to e.g. develop a list of these issues looks desirable.

[1]: https://github.com/ceph/teuthology/pull/1503#issuecomment-762837504

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>