osd/scheduler: Classify EC subOp reads according to op priority for mClock

The change brings MSG_OSD_EC_READ into the fold of mClock scheduler. This
improves the scheduling of client and other classes of operation as they
are no longer unnecessarily preempted by the 'immediate' queue.
EC SubOps are now handled as follows:

 - EC SubOp reads generated during recovery will either go into the
   'background_recovery' or 'background_best_effort' class based on
   the recovery priority set for the op. EC SubOp reads generated due
   to client will continue to be classified as 'immediate'.

 - EC SubOp writes generated as a result of client operations will
   continue to be classified as 'immediate'.

 - EC SubOp replies are considered high priority and therefore
   continue to be classed as 'immediate'.

Fixes: https://tracker.ceph.com/issues/71655
Signed-off-by: Sridhar Seshasayee <sridhar.seshasayee@ibm.com>
(cherry picked from commit a0464c3d82825beddb5598b15a8752e567160a05)

 Conflicts:
         src/osd/scheduler/OpSchedulerItem.h
- enum SchedulerClass doesn't exist in tentacle.Therefore, the original
  enum op_scheduler_class is used.

  enum SchedulerClass was introduced when mclock_common.h/cc was created
  as a refactor in later releases to share common bits between classic and
  crimson OSDs. Introduced in PR: https://github.com/ceph/ceph/pull/63516.

diff --git a/src/osd/scheduler/OpSchedulerItem.h b/src/osd/scheduler/OpSchedulerItem.h
index d0281cf84e7f3ca6cae6f069edddf78381137d58..490307d04f24e21e297e34adc7a7c5aacfa7d17a 100644 (file)
--- a/src/osd/scheduler/OpSchedulerItem.h
+++ b/src/osd/scheduler/OpSchedulerItem.h
@@ -241,11 +241,46 @@ public:
   }
 
   op_scheduler_class get_scheduler_class() const final {
-    auto type = op->get_req()->get_type();
-    if (type == CEPH_MSG_OSD_OP ||
-       type == CEPH_MSG_OSD_BACKOFF) {
+    switch (op->get_req()->get_type()) {
+    case CEPH_MSG_OSD_OP:
+    case CEPH_MSG_OSD_BACKOFF:
       return op_scheduler_class::client;
-    } else {
+    /**
+     * EC SubOp reads for mClock are now classed based on their priority.
+     * The primary reason is to prevent subOps from overwhelming the
+     * 'immediate' queue during OSD events like failures, removal,
+     * reweight and any operation that trigger recovery/backfills. A sudden
+     * and long enough sustained burst of subOps in the 'immediate' could
+     * result in slow ops since client ops are preempted due to ops in the
+     * higher priority 'immediate' queue.
+     *
+     * The new classification described below improves the scheduling
+     * of client and other classes of operation during recovery/backfill as
+     * they are no longer preempted by recovery EC subOps in the 'immediate'
+     * queue. EC SubOps are now handled as follows:
+     *
+     *  - EC SubOp reads generated during recovery will either go into the
+     *    'background_recovery' or 'background_best_effort' class based on
+     *    the recovery priority set for the op. EC SubOp reads generated due
+     *    to client will continue to be classified as 'immediate'.
+     *
+     *  - EC SubOp writes generated as a result of client operations will
+     *    continue to be classified as 'immediate'.
+     *
+     *  - EC SubOp replies are considered high priority and therefore
+     *    continue to be classed as 'immediate'.
+     *
+     *  Note: The 'cost' for EC subOp read operation is set according to
+     *  the amount of data to be read/written.
+     */
+    case MSG_OSD_EC_READ: {
+      auto prio = op->get_req()->get_priority();
+      if (prio <= PeeringState::recovery_msg_priority_t::FORCED) {
+        return priority_to_scheduler_class(prio);
+      }
+      [[fallthrough]];
+    }
+    default:
       return op_scheduler_class::immediate;
     }
   }