messages: fix out of range assertion

messages: fix out of range assertion

When clang uses an 8 bit type for the enum, it
complains (out of range) if comparing <256,
and complains (tautological) if comparing <=256.

Avoid this by explicitly making the enum an
uint8_t, and just asserting that that it has
that size at the point that we assume so for
the encoding (in case someone modified the
type definition without checking how it was used).

Signed-off-by: John Spray <john.spray@redhat.com>