osd: Introduce new PGOpQueueable class for recovery push/reply messages.

osd: Introduce new PGOpQueueable class for recovery push/reply messages.

Introduce a new PGOpQueueable class called PGRecoveryMsg to be used for
background recovery specific messages - MOSDPGPush and MOSDPGPushReply.
Earlier the above was categorized into the PGOpItem class leading to the
scheduler (for e.g. dmclock) giving higher importance to recovery IO
compared to client IO regardless of the weightage given to client IO.
This resulted in the poor performance of client IO.

This new class helps the existing and future (e.g. dmclock) IO schedulers
to schedule either client or recovery specific IO operations based on the
scheduling algorithm being employed and try to meet the QoS requirements.

Messages with CEPH_MSG_PRIO_HIGH or higher priority are given higher
importance.

Fixes: https://tracker.ceph.com/issues/47344
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>