rgw: fix chunked-encoding for chunks >1MiB

For HTTP responses sent with chunked-encoding, and greater than 1MiB in
size, the chunk-size field was being printed wrong.

Specifically, the chunk-size field was being sent with a mangled or
missing trailer of '\r\n'.

This bug manifested as HTTP clients being unable to read the response:
Chrome generates ERR_INCOMPLETE_CHUNKED_ENCODING
Python/boto generates httplib.LineTooLong: got more than 65536 bytes when reading chunk size

The wrong variable was being used to determine the size of the buffer
used for the chunk-size field.

Fix it by using the correct variable, and rename the variables to
clearly reflect their purpose.

Prior to PR#23940, this would only have been seen in some Swift
operations. PR#23940 changed some S3 operations to also use chunked
encoding to get responses sent faster, and made the bug easier to
detect. It was initially reported for a ListBucket call with a high
max-keys argument.

Backport: luminous, mimic
Reference: https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
Reference: https://github.com/ceph/ceph/pull/23940
Fixes: http://tracker.ceph.com/issues/35990
Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>

diff --git a/src/rgw/rgw_client_io_filters.h b/src/rgw/rgw_client_io_filters.h
index 82163cf8d746f63284189b12087cbe0a568aca41..9ce83a936634c62866fe2776c71f4748276b7164 100644 (file)
--- a/src/rgw/rgw_client_io_filters.h
+++ b/src/rgw/rgw_client_io_filters.h
@@ -286,11 +286,15 @@ public:
       return DecoratedRestfulClient<T>::send_body(buf, len);
     } else {
       static constexpr char HEADER_END[] = "\r\n";
-      char sizebuf[32];
-      const auto slen = snprintf(sizebuf, sizeof(buf), "%" PRIx64 "\r\n", len);
+      /* https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1 */
+      // TODO: we have no support for sending chunked-encoding
+      // extensions/trailing headers.
+      char chunk_size[32];
+      const auto chunk_size_len = snprintf(chunk_size, sizeof(chunk_size),
+                                           "%" PRIx64 "\r\n", len);
       size_t sent = 0;
 
-      sent += DecoratedRestfulClient<T>::send_body(sizebuf, slen);
+      sent += DecoratedRestfulClient<T>::send_body(chunk_size, chunk_size_len);
       sent += DecoratedRestfulClient<T>::send_body(buf, len);
       sent += DecoratedRestfulClient<T>::send_body(HEADER_END,
                                                    sizeof(HEADER_END) - 1);