rgw: multisite metadata sync fairness

rgw: multisite metadata sync fairness

    multisite metadata sync fairness

    The approach of this commit is to allow multiple RGWs to participate in the multisite metadata sync.
    Before this commit only single RGW has caught the all the sync locks.
    This feature is using bidding algorithm.
    For each lock, RGW is randomizing a number from 0 to shard count and for each shard is picking randomally one number and giving it as the bid_amount.
    each one of those vectors each RGW handles are being sent using watch notify (based on RADOS watch notify).
    Each time the RGW tries to lock it will compare its bid for the lock and the bids of other rgws, if the current RGW has the highest bid it will try to acquire the lock.

    Important configs:

    rgw_sync_work_period - For how long the RGW will sync until it will send unlock (very important in the beggining, because in the beginning only single RGW holds the locks)
    rgw_sync_lease_period - not new to this commit but affecting it, For how many seconds the RGW will request from the RADOS to keep the lock, mainly important in case of failure, so automatically the RGW will lose a lock if it's down

Fixes: https://tracker.ceph.com/issues/41230
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Signed-off-by: Or Friedmann <ofriedma@ibm.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>