osd: Implement Context based completion for mon cmd to set a config option

osd: Implement Context based completion for mon cmd to set a config option

The method, OSD::mon_cmd_set_config() currently sets a config option
related to mClock during OSD boot-up. The method used to wait on a
condition variable until the mon ack'ed the command. This was generally
not a problem.

But there could be scenarios where monitor could be slow to respond, or
due to a flaky network, response could be delayed. The OSD could therefore
be blocked from booting-up. To avoid this, the conditional wait is
replaced with an async Context completion.

Moreover, persisting this in the monitor store is not very critical. An
existing fallback mechanism stores this value in the in-memory "values"
map of the config subsystem. This can be read by the OSD at any point
during its operation.

The issue of the OSDs being blocked from booting-up properly was
observed when running tests with failure injections during OSD boot-up.

Following are the changes:

The changes to mon_cmd_set_config() are generic and any osd specific
option may be set in the config monitor store using this method.

- Implement Context based completion tracking of the mon command using
  MonCmdSetConfigOnFinish which is derived from the base Context class.
  In case of failures, the finish() method is overriden to save the config
  option in the config subsystem's "values" map at CONF_DEFAULT level
  using set_val_default() method. This allows users to modify this at a
  later point using "ceph config set" cli command.

- Additionally, if requested, finish() also checks if the config option
  needs to be applied on each op shard and calls update_scheduler_config()
  on each shard. This is required for some config options related to the
  mClock scheduler.

Fixes: https://tracker.ceph.com/issues/57040
Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>
(cherry picked from commit 3c0603cb87a75b7bfb1766e8b9edd1c4a8e43028)