]> git.apps.os.sepia.ceph.com Git - teuthology.git/commit
exporter: Restart every 24h 1858/head
authorZack Cerza <zack@redhat.com>
Tue, 13 Jun 2023 23:49:48 +0000 (17:49 -0600)
committerZack Cerza <zack@redhat.com>
Wed, 21 Jun 2023 21:40:39 +0000 (15:40 -0600)
commit35fc1b03dee1f5d7a8d1c25b9a9beabc48ad1665
tree60d3e2dbb546b9f81e980f17caeb6eb7d4263fed
parentb580191f840dede6fc19423f098cbe32f5ed42bf
exporter: Restart every 24h

A design limitation of prometheus-client's multiprocessing mode is that
each process creates files to store its own metrics; the exporter then
has to read each file, even if the process which created it is dead.

This results in request latency growing over time, to the point of
multiple seconds when the file count gets into the thousands. This
eventually results in prometheus failing to fetch, leaving gaps in our
data.

We can work around this by restarting at a regular interval; 24h seems
like a fine place to start.

Signed-off-by: Zack Cerza <zack@redhat.com>
teuthology/dispatcher/__init__.py
teuthology/exporter.py