osd: OSD: limit the value of 'size' and 'count' on 'osd bench'

Otherwise, a high enough 'count' value will trigger all sorts of timeouts
on the OSD; a low enough 'size' value will have the same effect for a
high enough value of 'count' (even the default value may have ill effects
on the osd's behaviour).  Limiting these values do not fix how 'osd bench'
should behave, but avoid someone from inadvertently bork an OSD.

Four options have been added and the user may adjust them if he so
desires to play with the OSD's fate:

 - 'osd_bench_small_size_max_iops' [default: 100] defines the amount of
   expected IOPS for a small block size (i.e., <1MB).
 - 'osd_bench_large_size_max_throughput' [default: 100<<20] defines
   the expected throughput in B/s.  We assume 100MB/s.
 - 'osd_bench_max_block_size' [default: 64 << 20] caps the block size
   allowed.  We have defined 64 MB.
 - 'osd_bench_duration' [default: 30] caps the expected duration.  This
   values is used when calculating the maximum allowed 'count', and is
   not enforced as the maximum duration of the operation.  If other IO
   is undergoing, or 'osd bench' is somehow slowed down, 'osd bench' may
   go over this duration.  Adjusting this option does however allow the
   user to specify higher 'count' values for (e.g.) a small block size,
   as the operation is assumed to perform the operation over a longer
   time span.

These options attempt to avoid combinations of dangerous parameters.  For
instance, we limit the block size to 64 MB (by default) so that there is
no temptation to specify a large enough block size, along with a very small
'count', such that the end result is similar to specifying a big count with
a sane block size.

Fixes: 7248
Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>

diff --git a/src/common/config_opts.h b/src/common/config_opts.h
index 3549e5465ecdadaca48c90cee5c039f78e2a9943..1f7645d7bee83beffdde63aacc73bbd5a61e7bb5 100644 (file)
--- a/src/common/config_opts.h
+++ b/src/common/config_opts.h
@@ -573,6 +573,11 @@ OPTION(osd_objectstore, OPT_STR, "filestore")  // ObjectStore backend type
 // Set to true for testing.  Users should NOT set this.
 OPTION(osd_debug_override_acting_compat, OPT_BOOL, false)
 
+OPTION(osd_bench_small_size_max_iops, OPT_U32, 100) // 100 IOPS
+OPTION(osd_bench_large_size_max_throughput, OPT_U64, 100 << 20) // 100 MB/s
+OPTION(osd_bench_max_block_size, OPT_U64, 64 << 20) // cap the block size at 64MB
+OPTION(osd_bench_duration, OPT_U32, 30) // duration of 'osd bench', capped at 30s to avoid triggering timeouts
+
 OPTION(filestore_debug_disable_sharded_check, OPT_BOOL, false)
 
 /// filestore wb throttle limits

diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc
index ac6102903046fc40b20502e055b51ca42800c983..53e33ca67973703af6672da747afa5c19bcfb85f 100644 (file)
--- a/src/osd/OSD.cc
+++ b/src/osd/OSD.cc
@@ -4263,6 +4263,59 @@ void OSD::do_command(Connection *con, tid_t tid, vector<string>& cmd, bufferlist
     cmd_getval(cct, cmdmap, "count", count, (int64_t)1 << 30);
     cmd_getval(cct, cmdmap, "size", bsize, (int64_t)4 << 20);
 
+    uint32_t duration = g_conf->osd_bench_duration;
+
+    if (bsize > (int64_t) g_conf->osd_bench_max_block_size) {
+      // let us limit the block size because the next checks rely on it
+      // having a sane value.  If we allow any block size to be set things
+      // can still go sideways.
+      ss << "block 'size' values are capped at "
+         << prettybyte_t(g_conf->osd_bench_max_block_size) << ". If you wish to use"
+         << " a higher value, please adjust 'osd_bench_max_block_size'";
+      r = -EINVAL;
+      goto out;
+    } else if (bsize < (int64_t) (1 << 20)) {
+      // entering the realm of small block sizes.
+      // limit the count to a sane value, assuming a configurable amount of
+      // IOPS and duration, so that the OSD doesn't get hung up on this,
+      // preventing timeouts from going off
+      int64_t max_count =
+        bsize * duration * g_conf->osd_bench_small_size_max_iops;
+      if (count > max_count) {
+        ss << "'count' values greater than " << max_count
+           << " for a block size of " << prettybyte_t(bsize) << ", assuming "
+           << g_conf->osd_bench_small_size_max_iops << " IOPS,"
+           << " for " << duration << " seconds,"
+           << " can cause ill effects on osd. "
+           << " Please adjust 'osd_bench_small_size_max_iops' with a higher"
+           << " value if you wish to use a higher 'count'.";
+        r = -EINVAL;
+        goto out;
+      }
+    } else {
+      // 1MB block sizes are big enough so that we get more stuff done.
+      // However, to avoid the osd from getting hung on this and having
+      // timers being triggered, we are going to limit the count assuming
+      // a configurable throughput and duration.
+      int64_t total_throughput =
+        g_conf->osd_bench_large_size_max_throughput * duration;
+      int64_t max_count = (int64_t) (total_throughput / bsize);
+      if (count > max_count) {
+        ss << "'count' values greater than " << max_count
+           << " for a block size of " << prettybyte_t(bsize) << ", assuming "
+           << prettybyte_t(g_conf->osd_bench_large_size_max_throughput) << "/s,"
+           << " for " << duration << " seconds,"
+           << " can cause ill effects on osd. "
+           << " Please adjust 'osd_bench_large_size_max_throughput'"
+           << " with a higher value if you wish to use a higher 'count'.";
+        r = -EINVAL;
+        goto out;
+      }
+    }
+
+    dout(1) << " bench count " << count
+            << " bsize " << prettybyte_t(bsize) << dendl;
+
     bufferlist bl;
     bufferptr bp(bsize);
     bp.zero();