mds: reverse MDSMap encoding of max_xattr_size/bal_rank_mask

Commit e134c890 adds the bal_rank_mask with encoded (ev) version 17.  This was
merged into main Oct 2022 and made it into the reef release normally.

Commit 7b8def5c adds the max_xattr_size also with encoded (ev) version 17 but
places it before bal_rank_mask. This is problematic as there were no plans to
backport e134c890 to quincy or pacific so piggybacking on the ev 17 bump would
not work and otherwise would require the backports to be done as a set to
ensure consistency (including with the kernel client).

However, the real issue is that 7b8def5c was not merged until after reef was
already cut. This required 7b8def5c to be backported separately in [1] which
was not merged until after v18.2.1 (current reef HEAD as of this commit).
Ultimately, this means that there are reef versions (v18.2.[01]) in the wild
which expect bal_rank_mask to be encoded at ev17 and not (max_xattr_size,
bal_rank_mask). Adding to the complications, the kernel client has already
merged code [2] expecting max_xattr_size for ev17.

It was decided in a github discussion [3] to move bal_rank_mask to ev18 to
avoid updating the kernel client which was done in the main branch via 36ee8e7e
and update the reef max_xattr_size backport with the same change (d8cebd67).

Unfortunately, this breaks upgrades v18.2.[01] to newer reef versions or to
main.  The reason is that monitors will encode v17 with bal_rank_mask
(max_xattr_size is not merged yet) and send that to upgraded mgrs (which are
upgraded first). The mgr will attempt to decode bal_rank_mask as a uint64_t
(max_xattr_size) but fail because an empty (by default) bal_rank_mask is simply
encoded as a signed 32-bit integer. Consequently, the mgr will fail decoding
with:

    failed to decode message of type 45 v1: End of buffer [buffer:2]

Of course the problem does not stop there, even if the mgr were able to handle
this, the monitors/mds/clients would fail in similar fashion.

So the only choice left is to fix max_xattr_size to be encoded at ev18.
Fortunately, v18.2.2 has not been released nor has any max_xattr_size backport
to quincy/pacific been merged. The main downside will be that kernels will
wrongly decode ev17 (which is already true for ceph clusters running
v18.2.[01]). A follow-up kernel fix will be required.

[1] https://tracker.ceph.com/issues/59405
[2] linux.git d93231a6bc8a452323d5fef16cca7107ce483a27
[3] https://github.com/ceph/ceph/pull/53340#discussion_r1399255031

Fixes: https://tracker.ceph.com/issues/64440
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 78abfeaff27fee343fb664db633de5b221699a73)

diff --git a/src/mds/MDSMap.cc b/src/mds/MDSMap.cc
index 95d208881a0e0d504a67c82ff8253bc744686d1c..2377147aefff75da10734885af6ebcf1e62fb1c7 100644 (file)
--- a/src/mds/MDSMap.cc
+++ b/src/mds/MDSMap.cc
@@ -798,8 +798,8 @@ void MDSMap::encode(bufferlist& bl, uint64_t features) const
     encode(min_compat_client, bl);
   }
   encode(required_client_features, bl);
-  encode(max_xattr_size, bl);
   encode(bal_rank_mask, bl);
+  encode(max_xattr_size, bl);
   ENCODE_FINISH(bl);
 }
 
@@ -948,11 +948,11 @@ void MDSMap::decode(bufferlist::const_iterator& p)
   }
 
   if (ev >= 17) {
-    decode(max_xattr_size, p);
+    decode(bal_rank_mask, p);
   }
 
   if (ev >= 18) {
-    decode(bal_rank_mask, p);
+    decode(max_xattr_size, p);
   }
 
   /* All MDS since at least v14.0.0 understand INLINE */