monclient: try to resend the mon commands to the same monitor if available

When we have a socket failure or connection issue, we may send a mon command
and never check if it completed. If we resend the command to another monitor,
the resent command may complete before the first sent command. This can cause
users to send the command twice, which can lead to issues in automated
environments. For example:

We have 2 monitors: mon.a and mon.b

1. Send command to delete pool - monclient targets mon.a
2. A socket failure occurs, and mon.a has a delay in response
3. Monclient hunts for another monitor to resend the delete pool command
   and finds mon.b
4. Mon.b removes the pool and sends an acknowledgment
5. The user script now sends a create pool command, but mon.a now sends the
   acknowledgment for the pool delete from step #1

We end up without a pool, as mon.a deleted it.

The mon_client_hunt_on_resent configuration was added to control the behavior of
retrying commands on monitor connection failures.
By default, this option is enabled to prevent situations where a command is retried
on the same monitor, potentially missing better monitor candidates.
Clients experiencing specific conditions that require retrying on the same monitor
can disable this feature by setting the configuration to false.

Fixes: https://tracker.ceph.com/issues/63789
Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>

diff --git a/src/common/options/global.yaml.in b/src/common/options/global.yaml.in
index b4ac2840da9409453ed63dd40076ff9c861ad2fe..6dac20dd4b9a9a86a52d262c21656fcdc9ebc5ee 100644 (file)
--- a/src/common/options/global.yaml.in
+++ b/src/common/options/global.yaml.in
@@ -2325,6 +2325,14 @@ options:
   level: advanced
   default: 10
   with_legacy: true
+- name: mon_client_hunt_on_resend
+  type: bool
+  level: advanced
+  default: true
+  fmt_desc: On commands connection failure, hunt for any monitor.
+    If false, try to resend the command to the same monitor to prevent
+    command race conditions.
+  with_legacy: true
 - name: mon_client_max_log_entries_per_message
   type: int
   level: advanced

diff --git a/src/mon/MonClient.cc b/src/mon/MonClient.cc
index 0667c078b630d7d8952df4ad67119aa76394f876..761b615bb83a3a50718a0bea3e752c41cb9c9c98 100644 (file)
--- a/src/mon/MonClient.cc
+++ b/src/mon/MonClient.cc
@@ -1249,6 +1249,7 @@ void MonClient::_send_command(MonCommand *r)
   }
 
   // normal CLI command
+  r->sent_name = monmap.get_name(active_con ? (active_con->get_con()->get_peer_addr()) : "");
   ldout(cct, 10) << __func__ << " " << r->tid << " " << r->cmd << dendl;
   auto m = ceph::make_message<MMonCommand>(monmap.fsid);
   m->set_tid(r->tid);
@@ -1286,6 +1287,25 @@ void MonClient::_resend_mon_commands()
       // starting with octopus, tell commands use their own connetion and need no
       // special resend when we finish hunting.
     } else {
+      if (!cct->_conf->mon_client_hunt_on_resend) {
+        // Ensure the target name of the resend mon command matches the rank
+        // of the current active connection.
+        ldout(cct, 20) << __func__ << " " << cmd->tid << " " << cmd->cmd
+                       << " last sent_name " << cmd->sent_name << dendl;
+        if (!monmap.contains(cmd->sent_name)) {
+          ldout(cct, 20) << __func__ << " " << cmd->tid << " " << cmd->cmd
+                        << " sent_name " << cmd->sent_name << " not in monmap using mon:" <<
+                        monmap.get_name(active_con->get_con()->get_peer_addr()) << dendl;
+        } else if (active_con && cmd->sent_name.length() &&
+                   cmd->sent_name != monmap.get_name(active_con->get_con()->get_peer_addr()) &&
+                   monmap.contains(cmd->sent_name)) {
+          ldout(cct, 20) << __func__ << " " << cmd->tid << " " << cmd->cmd
+                         << " wants mon " << cmd->sent_name
+                         << " current connection is " << monmap.get_name(active_con->get_con()->get_peer_addr())
+                         << ", reopening session" << dendl;
+          _reopen_session(monmap.get_rank(cmd->sent_name));
+        }
+      }
       _send_command(cmd); // might remove cmd from mon_commands
     }
   }

diff --git a/src/mon/MonClient.h b/src/mon/MonClient.h
index 803c74eb7f62945a5841aad665846a42e82f8d0f..00e1401ef6ca191ca9fa9147d4ea35a0774f88d8 100644 (file)
--- a/src/mon/MonClient.h
+++ b/src/mon/MonClient.h
@@ -559,6 +559,7 @@ private:
   struct MonCommand {
     // for tell only
     std::string target_name;
+    std::string sent_name;
     int target_rank = -1;
     ConnectionRef target_con;
     std::unique_ptr<MonConnection> target_session;