mgr/DaemonServer: Implement ok-to-upgrade command

mgr/DaemonServer: Implement ok-to-upgrade command

Implement a new Mgr command called 'ok-to-upgrade' that returns a set of OSDs
within the provided CRUSH bucket that are safe to upgrade without reducing
immediate data availability.

The command accepts the following as input:
 - CRUSH bucket name (required)
   - The CRUSH bucket type is limited to 'rack', 'chassis', 'host' and 'osd'.
     This is to prevent users from specifying a bucket type higher up the tree
     which could result in performance issues if the number of OSDs in the
     bucket is very high.
 - The new Ceph version to check against. The format accepted is the short
   form of the Ceph version, for e.g. 20.3.0-3803-g63ca1ffb5a2. (required)
 - The maximum number of OSDs to consider if specified. (optional)

Implementation Details:

After sanity checks on the provided parameters, the following steps are
performed:

1. The set of OSDs within the CRUSH bucket is first determined.
2. From the main set of OSDs, a filtered set of OSDs not yet running the new
   Ceph version is created.
   - For this purpose, the OSD's 'ceph_version_short' string is read from
     the metadata. For this purpose a new method called
     DaemonServer::get_osd_metadata() is used. The information is determined
     from the DaemonStatePtr maintained within the DaemonServer.
3. If all OSDs are already running the new Ceph version, a success report is
   generated and returned.
4. If OSDs are not running the new Ceph version, a new set (to_upgrade) is
   created.
5. If the current version cannot be determined, an error is logged and the
   output report with 'bad_no_version' field populated with the OSD in question
   is generated.
6. On the new set (to_upgrade), the existing logic in _check_offline_pgs() is
   executed to see if stopping any or all OSDs in the set as part of the upgrade
   can reduce immediate data availability.
   - If data availability is impacted, then the number of OSDs in the filtered
     set is reduced by a factor defined by a new config option called
     'mgr_osd_upgrade_check_convergence_factor' which is set to 0.8 by default.
   - The logic in _check_offline_pgs() is repeated for the new set.
   - The above is repeated until a safe subset of OSDs that can be stopped for
     upgrade is found. Each iteration reduces the number of OSDs to check by
     the convergence factor mentioned above.
7. It must be noted that the default value of
   'mgr_osd_upgrade_check_convergence_factor' is on the higher side in order to
   help determine an optimal set of OSDs to upgrade. In other words, a higher
   convergence factor would help maximize the number of OSDs to upgrade. In this
   case, the number of iterations and therefore the time taken to determine the
   OSDs to upgrade is proportional to the number of OSDs in the CRUSH bucket.
   The converse is true if a lower convergence factor is used.
8. If the number of OSDs determined is lower than the 'max' specified, then an
   additional loop is executed to determine if other children of the CRUSH
   bucket can be added to the existing set.
9. Once a viable set is determined, an output report similar to the following is
   generated:

A standalone test is introduced that exercises the logic for both replicated
and erasure-coded pools by manipulating the min_size for a pool and check for
upgradability. The tests also performs other basic sanity checks and error
conditions.

The output shown below is for a cluster running on a single node with 10 OSDs
and with replicated pool configuration:

$ ceph osd ok-to-upgrade incerta06 01.00.00-gversion-test --format=json
{"ok_to_upgrade":true,"all_osds_upgraded":false,\
 "osds_in_crush_bucket":[0,1,2,3,4,5,6,7,8,9],\
 "osds_ok_to_upgrade":[0],"osds_upgraded":[],"bad_no_version":[]}

The following report is shown if all OSDs are running the desired Ceph version:

$ ceph osd ok-to-upgrade --crush_bucket  localrack \
  --ceph_version 20.3.0-3803-g63ca1ffb5a2
{"ok_to_upgrade":false,"all_osds_upgraded":true,\
 "osds_in_crush_bucket":[0,1,2,3,4,5,6,7,8,9],"osds_ok_to_upgrade":[],\
"osds_upgraded":[0,1,2,3,4,5,6,7,8,9],"bad_no_version":[]}'

Fixes: https://tracker.ceph.com/issues/73031
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>