Instead of directly modifying the store whenever we want to trim our Paxos
state, we should do it through Paxos, proposing the trim to the quorum and
commit it once accepted.
This enforces three major invariants that we will be able to leverage later
on during the store synchronization:
1) The Leader will set the pace for trimming across the system. No one
will trim their state unless they are committing the value proposed by
the Leader;
2) Following (1), the monitors in the quorum will trim at the same time.
There will be no diverging states due to trimming on different monitors.
3) Each trim will be kept as a transaction in the Paxos' store allowing
us to obtain a consistent state during synchronization, by shipping
the Paxos versions to the other monitor and applying them. We could
incur in an inconsistent state if the trim happened without
constraints, without being logged; by going through Paxos this concern
is no longer relevant.
The trimming itself may be triggered each time a proposal finishes, which
is the time at which we know we have committed a new version on the store.
It shall be triggered iff we are sure we have enough versions on the store
to fill the gap of any monitor that might become alive and still hasn't
drifted enough to require synchronization. Roughly speaking, we will check
if the number of available versions is higher than 'paxos_max_join_drift'.
Furthermore, we added a new option, 'paxos_trim_tolerance', so we are able
to avoid trimming every single time the above condition is met -- which
would happen every time we trimmed a version, and then proposed a new one,
and then we would trim it again, etc. So, just tolerate a couple of commits
before trimming again.
Finally, we added support to enable/disable trimming, which will be
essential during the store synchronization process.
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>