Creation of ceph_test_radios_io_sequence tool.
Application creates IO sequences using seeded random numbers.
IO sequences are varied in IO offset, length, IO order and data patterns to try stress erasure coding specifically and detect defects earlier in the development cycle.
We feel the creation of this as a new tool is benificial for several reasons:
* Existing test tools either generate completely random I/O (with different offsets and lengths) or test very uniform I/O (either read/write whole object or read/writes with fixed length). This is very inefficient (relies on brute force) for testing boundary conditions. The I/O sequence test tool generates sequences of I/Os to specifically test boundary conditions
* Existing test tools only generate RADOS requests that contain a single read or write request. Recent regressions found in EC show that clients (e.g. RBD with LUKs) can generate RADOS requests that contain multiple reads or writes. This tool will sometimes generate requests with 2 or 3 I/O operations
* Quality of data patterns when writing to objects and validation of data for reading is not as great as desired in existing test tools. This means that code bugs where the wrong part of an object is read/written may not be detected. This tool creates more sophisticated data patterns and keeps a model of the object to track the expected contents of the object.
* For erasure coding there are many choices of erasure code profile, this tool can create pools and exercise multiple different profiles in parallel to speed up testing.
The test tool is located in src/test/osd, as the new tool is used explicitly for testing the erasure coding implementation of OSDs.
A lot of the logic for generating IO Sequences has been stored in a new location, io_exerciser, located in src/common. The thinking behind storing the code here is we feel it is useful to share between other applications.
It is abstracted to a level it shouldn't be necissairy to have to be only used with only the new ceph_test_rados_io_sequence application and we would like to be able to integrate with other tools such as ceph_test_rados or possibly rados bench as well in future PRs to enhance testing done with those tools.
Signed-off-by: Jon Bailey <jonathan.bailey1@ibm.com>
add_library(dlfcn_win32 STATIC win32/dlfcn.cc win32/errno.cc)
endif()
+add_subdirectory(io_exerciser)
add_subdirectory(options)
set(common_srcs
--- /dev/null
+add_library(object_io_exerciser STATIC
+ DataGenerator.cc
+ IoOp.cc
+ IoSequence.cc
+ Model.cc
+ ObjectModel.cc
+ RadosIo.cc
+)
+
+target_link_libraries(object_io_exerciser
+ librados
+ global
+)
\ No newline at end of file
--- /dev/null
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab
+#include "DataGenerator.h"
+
+#include "ObjectModel.h"
+
+#include "common/debug.h"
+#include "common/dout.h"
+
+#include <chrono>
+#include <iostream>
+#include <stdexcept>
+
+#define dout_subsys ceph_subsys_rados
+#define dout_context g_ceph_context
+
+std::mutex ceph::io_exerciser::data_generation::DataGenerator
+ ::DataGenerationSingleton::m_mutex;
+ceph::io_exerciser::data_generation::DataGenerator::DataGenerationSingleton
+ ceph::io_exerciser::data_generation::DataGenerator::DataGenerationSingleton::m_singletonInstance =
+ ceph::io_exerciser::data_generation::DataGenerator::DataGenerationSingleton();
+
+std::unique_ptr<ceph::io_exerciser::data_generation::DataGenerator>
+ ceph::io_exerciser::data_generation::DataGenerator::create_generator(
+ ceph::io_exerciser::data_generation::GenerationType generationType,
+ const ObjectModel& model)
+{
+ switch(generationType)
+ {
+ case GenerationType::SeededRandom:
+ return std::make_unique<SeededRandomGenerator>(model);
+ case GenerationType::HeaderedSeededRandom:
+ return std::make_unique<HeaderedSeededRandomGenerator>(model);
+ default:
+ throw std::invalid_argument("Not yet implemented");
+ }
+
+ return nullptr;
+}
+
+ceph::io_exerciser::data_generation::DataGenerator
+ ::DataGenerationSingleton::DataGenerationSingleton()
+{
+ m_created = false;
+}
+
+ceph::io_exerciser::data_generation::DataGenerator
+ ::DataGenerationSingleton::DataGenerationSingleton(uint64_t unique_id)
+{
+ m_uniqueId = unique_id;
+ m_created = true;
+}
+
+const ceph::io_exerciser::data_generation
+ ::DataGenerator::DataGenerationSingleton&
+ ceph::io_exerciser::data_generation::DataGenerator
+ ::DataGenerationSingleton::createSpecificInstance(uint64_t unique_id)
+{
+ std::scoped_lock lock(m_mutex);
+ ceph_assert(!m_singletonInstance.m_created);
+ m_singletonInstance = DataGenerationSingleton(unique_id);
+
+ return m_singletonInstance;
+}
+
+const ceph::io_exerciser::data_generation::DataGenerator::DataGenerationSingleton&
+ ceph::io_exerciser::data_generation::DataGenerator
+ ::DataGenerationSingleton::getInstance()
+{
+ if (!m_singletonInstance.m_created)
+ {
+ std::scoped_lock lock(m_mutex);
+ if (!m_singletonInstance.m_created)
+ {
+ std::mt19937_64 random_generator =
+ std::mt19937_64(duration_cast<std::chrono::milliseconds>(
+ std::chrono::system_clock::now().time_since_epoch()).count());
+ m_singletonInstance = DataGenerator::DataGenerationSingleton(random_generator());
+ }
+ }
+ return m_singletonInstance;
+}
+
+const uint64_t ceph::io_exerciser::data_generation::DataGenerator
+ ::DataGenerationSingleton::getUniqueId()
+{
+ return getInstance().m_uniqueId;
+}
+
+void ceph::io_exerciser::data_generation
+ ::DataGenerator::generate_wrong_data(uint64_t offset, uint64_t length,
+ ceph::bufferlist& retlist)
+{
+ uint64_t block_size = m_model.get_block_size();
+ char buffer[block_size];
+ for (uint64_t block_offset = offset;
+ block_offset < offset + length;
+ block_offset++)
+ {
+ std::memset(buffer, 0, block_size);
+ retlist.append(ceph::bufferptr(buffer, block_size));
+ }
+}
+
+bool ceph::io_exerciser::data_generation
+ ::DataGenerator::validate(ceph::bufferlist& bufferlist, uint64_t offset, uint64_t length)
+{
+ ceph::bufferlist comparison_list;
+ generate_data(offset, length, comparison_list);
+ return bufferlist.contents_equal(comparison_list);
+}
+
+#include <bitset>
+
+ceph::bufferptr ceph::io_exerciser::data_generation
+ ::SeededRandomGenerator::generate_block(uint64_t block_offset)
+{
+ uint64_t block_size = m_model.get_block_size();
+ char buffer[block_size];
+
+ std::mt19937_64 random_generator(m_model.get_seed(block_offset));
+ uint64_t rand1 = random_generator();
+ uint64_t rand2 = random_generator();
+
+ constexpr size_t generation_length = sizeof(uint64_t);
+
+ for (uint64_t i = 0; i < block_size; i+=(2*generation_length), rand1++, rand2--)
+ {
+ std::memcpy(buffer + i, &rand1, generation_length);
+ std::memcpy(buffer + i + generation_length, &rand2, generation_length);
+ }
+
+ size_t remainingBytes = block_size % (generation_length * 2);
+ if (remainingBytes > generation_length)
+ {
+ size_t remainingBytes2 = remainingBytes - generation_length;
+ std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes);
+ std::memcpy(buffer + block_size - remainingBytes2, &rand2, remainingBytes2);
+ }
+ else if (remainingBytes > 0)
+ {
+ std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes);
+ }
+
+ return ceph::bufferptr(buffer, block_size);
+}
+
+void ceph::io_exerciser::data_generation
+ ::SeededRandomGenerator::generate_data(uint64_t offset, uint64_t length, ceph::bufferlist& retlist)
+{
+ ceph_assert(retlist.length() == 0);
+
+ for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++)
+ {
+ retlist.append(generate_block(block_offset));
+ }
+}
+
+ceph::bufferptr ceph::io_exerciser::data_generation
+ ::HeaderedSeededRandomGenerator::generate_block(uint64_t block_offset)
+{
+ UniqueIdBytes unique_run_id = DataGenerator::DataGenerationSingleton::getUniqueId();
+ SeedBytes seed = m_model.get_seed(block_offset);
+ TimeBytes current_time = duration_cast<std::chrono::milliseconds>(
+ std::chrono::system_clock::now().time_since_epoch()).count();
+
+ ceph::bufferptr bufferptr = SeededRandomGenerator::generate_block(block_offset);
+
+ std::memcpy(bufferptr.c_str(), &unique_run_id + uniqueIdStart(), uniqueIdLength());
+ std::memcpy(bufferptr.c_str() + seedStart(), &seed, seedLength());
+ std::memcpy(bufferptr.c_str() + timeStart(), ¤t_time, timeLength());
+
+ return bufferptr;
+}
+
+void ceph::io_exerciser::data_generation
+ ::HeaderedSeededRandomGenerator::generate_wrong_data(uint64_t offset,
+ uint64_t length,
+ ceph::bufferlist& retlist)
+{
+ ceph_assert(retlist.length() == 0);
+ ceph_assert(m_model.get_block_size() >= headerLength());
+
+ ceph::bufferptr bufferptr;
+ for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++)
+ {
+ UniqueIdBytes unique_run_id = DataGenerator::DataGenerationSingleton::getUniqueId();
+ SeedBytes seed = m_model.get_seed(block_offset-1);
+ TimeBytes current_time = duration_cast<std::chrono::milliseconds>(
+ std::chrono::system_clock::now().time_since_epoch()).count();
+
+ bufferptr = generate_block(block_offset-1);
+
+ std::memcpy(bufferptr.c_str(), &unique_run_id + uniqueIdStart(), uniqueIdLength());
+ std::memcpy(bufferptr.c_str() + seedStart(), &seed, seedLength());
+ std::memcpy(bufferptr.c_str() + timeStart(), ¤t_time, timeLength());
+
+ retlist.append(std::move(bufferptr));
+ }
+}
+
+const ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator::UniqueIdBytes
+ ceph::io_exerciser::data_generation
+ ::HeaderedSeededRandomGenerator::readUniqueRunId(uint64_t block_offset,
+ const ceph::bufferlist& bufferlist)
+{
+ UniqueIdBytes read_unique_run_id = 0;
+ std::memcpy(&read_unique_run_id,
+ &bufferlist[(block_offset * m_model.get_block_size()) + uniqueIdStart()],
+ uniqueIdLength());
+ return read_unique_run_id;
+}
+
+const ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator::SeedBytes
+ ceph::io_exerciser::data_generation
+ ::HeaderedSeededRandomGenerator::readSeed(uint64_t block_offset,
+ const ceph::bufferlist& bufferlist)
+{
+ SeedBytes read_seed = 0;
+ std::memcpy(&read_seed,
+ &bufferlist[(block_offset * m_model.get_block_size()) + seedStart()],
+ seedLength());
+ return read_seed;
+}
+
+const ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator::TimeBytes
+ ceph::io_exerciser::data_generation
+ ::HeaderedSeededRandomGenerator::readDateTime(uint64_t block_offset,
+ const ceph::bufferlist& bufferlist)
+{
+ TimeBytes read_time = 0;
+ std::memcpy(&read_time,
+ &bufferlist[(block_offset * m_model.get_block_size()) + timeStart()],
+ timeLength());
+ return read_time;
+}
+
+bool ceph::io_exerciser::data_generation
+ ::HeaderedSeededRandomGenerator::validate(ceph::bufferlist& bufferlist,
+ uint64_t offset, uint64_t length)
+{
+ std::vector<uint64_t> invalid_block_offsets;
+
+ for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++)
+ {
+ bool valid_block
+ = validate_block(block_offset,
+ (bufferlist.c_str() + ((block_offset - offset) *
+ m_model.get_block_size())));
+ if (!valid_block)
+ {
+ invalid_block_offsets.push_back(block_offset);
+ }
+ }
+
+ if (!invalid_block_offsets.empty())
+ {
+ printDebugInformationForOffsets(offset, invalid_block_offsets, bufferlist);
+ }
+
+ return invalid_block_offsets.empty();
+}
+
+bool ceph::io_exerciser::data_generation
+ ::HeaderedSeededRandomGenerator::validate_block(uint64_t block_offset,
+ const char* buffer_start)
+{
+ // We validate the block matches what we generate byte for byte, however we ignore the time section of the header
+ ceph::bufferptr bufferptr = generate_block(block_offset);
+ bool valid = strncmp(bufferptr.c_str(), buffer_start, timeStart()) == 0;
+ valid = valid ? strncmp(bufferptr.c_str() + timeEnd(),
+ buffer_start + timeEnd(),
+ m_model.get_block_size() - timeEnd()) == 0 : valid;
+ return valid;
+}
+
+const ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator::ErrorType
+ ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::getErrorTypeForBlock(uint64_t read_offset, uint64_t block_offset,
+ const ceph::bufferlist& bufferlist)
+{
+ try
+ {
+ UniqueIdBytes read_unique_run_id = readUniqueRunId(block_offset - read_offset,
+ bufferlist);
+ if (DataGenerationSingleton::getUniqueId() != read_unique_run_id)
+ {
+ return ErrorType::RUN_ID_MISMATCH;
+ }
+
+ SeedBytes read_seed = readSeed(block_offset - read_offset, bufferlist);
+ if (m_model.get_seed(block_offset) != read_seed)
+ {
+ return ErrorType::SEED_MISMATCH;
+ }
+
+ if (std::strncmp(&bufferlist[((block_offset - read_offset) *
+ m_model.get_block_size()) + bodyStart()],
+ generate_block(block_offset).c_str() + bodyStart(),
+ m_model.get_block_size() - bodyStart()) != 0)
+ {
+ return ErrorType::DATA_MISMATCH;
+ }
+ }
+ catch(const std::exception& e)
+ {
+ return ErrorType::DATA_NOT_FOUND;
+ }
+
+ return ErrorType::UNKNOWN;
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::printDebugInformationForBlock(uint64_t read_offset, uint64_t block_offset,
+ const ceph::bufferlist& bufferlist)
+{
+ ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset, bufferlist);
+
+ TimeBytes read_time = 0;
+ std::time_t ttp;
+
+ char read_bytes[m_model.get_block_size()];
+ char generated_bytes[m_model.get_block_size()];
+
+ if (blockError == ErrorType::DATA_MISMATCH || blockError == ErrorType::UNKNOWN)
+ {
+ read_time = readDateTime(block_offset, bufferlist);
+ std::chrono::system_clock::time_point time_point{std::chrono::milliseconds{read_time}};
+ ttp = std::chrono::system_clock::to_time_t(time_point);
+
+ std::memcpy(&read_bytes,
+ &bufferlist[((block_offset - read_offset) * m_model.get_block_size())],
+ m_model.get_block_size() - bodyStart());
+ std::memcpy(&generated_bytes,
+ generate_block(block_offset).c_str(),
+ m_model.get_block_size() - bodyStart());
+ }
+ std::stringstream ss;
+ switch(blockError)
+ {
+ case ErrorType::RUN_ID_MISMATCH:
+ {
+ UniqueIdBytes read_unique_run_id = readUniqueRunId((block_offset - read_offset),
+ bufferlist);
+ ss << "Header (Run ID) mismatch detected at block " << block_offset
+ << " (byte offset " << block_offset * m_model.get_block_size() << ")."
+ << " Header expected run id " << DataGenerationSingleton::getUniqueId()
+ << " but found id " << read_unique_run_id
+ << ". Block data corrupt or not written from this instance of this application.";
+ }
+ break;
+
+ case ErrorType::SEED_MISMATCH:
+ {
+ SeedBytes read_seed = readSeed((block_offset - read_offset), bufferlist);
+
+ if (m_model.get_seed_offsets(read_seed).size() == 0)
+ {
+ ss << "Data (Seed) mismatch detected at block " << block_offset
+ << " (byte offset " << block_offset * m_model.get_block_size() << ")."
+ << " Header expected seed " << m_model.get_seed(block_offset)
+ << " but found seed " << read_seed
+ << ". Read data was not from any other recognised block in the object.";
+ }
+ else
+ {
+ ss << "Data (Seed) mismatch detected at block " << block_offset
+ << " (byte offset " << block_offset * m_model.get_block_size() << ")."
+ << " Header expected seed " << m_model.get_seed(block_offset)
+ << " but found seed " << read_seed
+ << ". Read data was from a different block(s): "
+ << m_model.get_seed_offsets(read_seed);
+ }
+ }
+ break;
+
+ case ErrorType::DATA_MISMATCH:
+ {
+ ss << "Data (Body) mismatch detected at block " << block_offset
+ << " (byte offset " << block_offset * m_model.get_block_size() << ")."
+ << " Header data matches, data body does not."
+ << " Data written at " << std::ctime(&ttp)
+ << "\nExpected data: " << std::endl;
+ ss << std::hex << std::setw(2) << std::setfill('0');
+ for (uint64_t i = headerLength();
+ i < m_model.get_block_size() - headerLength(); i++)
+ {
+ ss << static_cast<int>(generated_bytes[i]);
+ }
+ dout(0) << dendl;
+ ss << " Read data: " << std::endl;
+ for (uint64_t i = headerLength();
+ i < m_model.get_block_size() - headerLength(); i++)
+ {
+ ss << static_cast<int>(read_bytes[i]);
+ }
+ }
+ break;
+
+ case ErrorType::DATA_NOT_FOUND:
+ {
+ uint64_t bufferlist_length = bufferlist.to_str().size();
+ ss << "Data (Body) could not be read at block " << block_offset
+ << " (byte offset " << block_offset * m_model.get_block_size() << ")"
+ << " offset in bufferlist returned from read: " << (block_offset - read_offset)
+ << " (" << (block_offset - read_offset) * m_model.get_block_size() << " bytes)."
+ << " Returned bufferlist length: " << bufferlist_length;
+ }
+ break;
+
+ case ErrorType::UNKNOWN:
+ [[ fallthrough ]];
+
+ default:
+ {
+ ss << "Data mismatch detected at block " << block_offset
+ << " (byte offset " << block_offset * m_model.get_block_size() << ")."
+ << "\nExpected data: " << std::endl;
+ ss << std::hex << std::setw(2) << std::setfill('0');
+ for (uint64_t i = 0; i < m_model.get_block_size(); i++)
+ {
+ ss << static_cast<int>(generated_bytes[i]);
+ }
+ ss << std::endl << "Read data: " << std::endl;
+ for (uint64_t i = 0; i < m_model.get_block_size(); i++)
+ {
+ ss << static_cast<int>(read_bytes[i]);
+ }
+ }
+ break;
+ }
+ dout(0) << ss.str() << dendl;
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::printDebugInformationForRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ ErrorType rangeError,
+ const ceph::bufferlist& bufferlist)
+{
+ switch(rangeError)
+ {
+ case ErrorType::RUN_ID_MISMATCH:
+ printDebugInformationForRunIdMismatchRange(read_offset, start_block_offset,
+ range_length_in_blocks, bufferlist);
+ break;
+ case ErrorType::SEED_MISMATCH:
+ printDebugInformationForSeedMismatchRange(read_offset, start_block_offset,
+ range_length_in_blocks, bufferlist);
+ break;
+ case ErrorType::DATA_MISMATCH:
+ printDebugInformationDataBodyMismatchRange(read_offset, start_block_offset,
+ range_length_in_blocks, bufferlist);
+ break;
+ case ErrorType::DATA_NOT_FOUND:
+ printDebugInformationDataNotFoundRange(read_offset, start_block_offset,
+ range_length_in_blocks, bufferlist);
+ break;
+ case ErrorType::UNKNOWN:
+ [[ fallthrough ]];
+ default:
+ printDebugInformationCorruptRange(read_offset, start_block_offset,
+ range_length_in_blocks, bufferlist);
+ break;
+ }
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::printDebugInformationForRunIdMismatchRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const ceph::bufferlist& bufferlist)
+{
+ uint64_t range_start = start_block_offset;
+ uint64_t range_length = 0;
+ UniqueIdBytes initial_read_unique_run_id = readUniqueRunId(start_block_offset,
+ bufferlist);
+ for (uint64_t i = start_block_offset;
+ i < start_block_offset + range_length_in_blocks; i++)
+ {
+ ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist)
+ == ErrorType::RUN_ID_MISMATCH);
+
+ UniqueIdBytes read_unique_run_id = readUniqueRunId(i, bufferlist);
+ if (initial_read_unique_run_id != read_unique_run_id ||
+ i == (start_block_offset + range_length_in_blocks - 1))
+ {
+ if (range_length == 1)
+ {
+ printDebugInformationForBlock(read_offset, i, bufferlist);
+ }
+ else if (range_length > 1)
+ {
+ dout(0) << "Data (Run ID) Mismatch detected from block " << range_start
+ << " (" << range_start * m_model.get_block_size() << " bytes)"
+ << " and spanning a range of " << range_length << " blocks"
+ << "(" << range_length * m_model.get_block_size() << " bytes). "
+ << "Expected run id " << DataGenerationSingleton::getUniqueId()
+ << " for range but found id " << initial_read_unique_run_id
+ << " for all blocks in range. "
+ << "Block data corrupt or not written from this instance of this application."
+ << dendl;
+ }
+
+ range_start = i;
+ range_length = 1;
+ initial_read_unique_run_id = read_unique_run_id;
+ }
+ else
+ {
+ range_length++;
+ }
+ }
+
+ if (range_length == 1)
+ {
+ printDebugInformationForBlock(read_offset,
+ start_block_offset + range_length_in_blocks - 1,
+ bufferlist);
+ }
+ else if (range_length > 1)
+ {
+ dout(0) << "Data (Run ID) Mismatch detected from block " << range_start
+ << " (" << range_start * m_model.get_block_size() << " bytes) "
+ << "and spanning a range of " << range_length << " blocks "
+ << "(" << range_length * m_model.get_block_size() << " bytes). "
+ << "Expected run id " << DataGenerationSingleton::getUniqueId()
+ << " for range but found id " << initial_read_unique_run_id
+ << " for all blocks in range. "
+ << "Block data corrupt or not written from this instance of this application."
+ << dendl;
+ }
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::printDebugInformationForSeedMismatchRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const ceph::bufferlist& bufferlist)
+{
+ uint64_t range_start = start_block_offset;
+ uint64_t range_length = 0;
+
+ // Assert here if needed, as we can't support values
+ // that can't be converted to a signed integer.
+ ceph_assert(m_model.get_block_size() < (std::numeric_limits<uint64_t>::max() / 2));
+ int64_t range_offset = 0;
+
+ for (uint64_t i = start_block_offset;
+ i < start_block_offset + range_length_in_blocks; i++)
+ {
+ ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist)
+ == ErrorType::SEED_MISMATCH);
+ SeedBytes read_seed = readSeed(i, bufferlist);
+
+ std::vector<int> seed_found_offsets = m_model.get_seed_offsets(read_seed);
+
+ if ((seed_found_offsets.size() == 1 &&
+ (static_cast<int64_t>(seed_found_offsets.front() - i) == range_offset)) ||
+ range_length == 0)
+ {
+ if (range_length == 0)
+ {
+ range_start = i;
+ range_offset = seed_found_offsets.front() - i;
+ }
+ range_length++;
+ }
+ else
+ {
+ if (range_length == 1)
+ {
+ printDebugInformationForBlock(read_offset, i - 1, bufferlist);
+ }
+ else if (range_length > 1)
+ {
+ dout(0) << "Data (Seed) Mismatch detected from block " << range_start
+ << " (" << range_start * m_model.get_block_size() << " bytes) "
+ << "and spanning a range of " << range_length << " blocks "
+ << "(" << range_length * m_model.get_block_size() << " bytes). "
+ << "Returned data located starting from block "
+ << static_cast<uint64_t>(range_offset) + range_start
+ << " (" << (static_cast<uint64_t>(range_offset) + range_start)
+ * m_model.get_block_size() << " bytes) "
+ << "and spanning a range of " << range_length << " blocks "
+ << "(" << range_length * m_model.get_block_size() << " bytes)."
+ << dendl;
+ }
+ range_length = 1;
+ range_start = i;
+ range_offset = seed_found_offsets.front() - i;
+ }
+ }
+
+ if (range_length == 1)
+ {
+ printDebugInformationForBlock(read_offset,
+ start_block_offset + range_length_in_blocks - 1,
+ bufferlist);
+ }
+ else if (range_length > 1)
+ {
+ dout(0) << "Data (Seed) Mismatch detected from block " << range_start
+ << " (" << range_start * m_model.get_block_size() << " bytes) "
+ << "and spanning a range of " << range_length << " blocks "
+ << "(" << range_length * m_model.get_block_size() << " bytes). "
+ << "Returned data located starting from block "
+ << range_offset + range_start
+ << " (" << (range_offset + range_start) * m_model.get_block_size()
+ << " bytes) and spanning a range of " << range_length << " blocks "
+ << "(" << range_length * m_model.get_block_size() << " bytes)."
+ << dendl;
+ }
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+::printDebugInformationDataBodyMismatchRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const ceph::bufferlist& bufferlist)
+{
+ dout(0) << "Data Mismatch detected in blocks "
+ << "from " << start_block_offset
+ << " to " << start_block_offset + range_length_in_blocks - 1 << ". "
+ << "Headers look as expected for range, "
+ << "but generated data body does not match. "
+ << "More information given for individual blocks below." << dendl;
+
+ for (uint64_t i = start_block_offset;
+ i < start_block_offset + range_length_in_blocks; i++)
+ {
+ printDebugInformationForBlock(read_offset, i, bufferlist);
+ }
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::printDebugInformationCorruptRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const ceph::bufferlist& bufferlist)
+{
+ dout(0) << "Data Mismatch detected in blocks "
+ << "from " << start_block_offset
+ << " to " << start_block_offset + range_length_in_blocks - 1 << ". "
+ << "Headers look as expected for range, but generated data body does not match."
+ << " More information given for individual blocks below." << dendl;
+
+ for (uint64_t i = start_block_offset;
+ i < start_block_offset + range_length_in_blocks; i++)
+ {
+ printDebugInformationForBlock(read_offset, i, bufferlist);
+ }
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::printDebugInformationDataNotFoundRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const ceph::bufferlist& bufferlist)
+{
+ dout(0) << "Data not found for blocks "
+ << "from " << start_block_offset
+ << " to " << start_block_offset + range_length_in_blocks - 1 << ". "
+ << "More information given for individual blocks below." << dendl;
+
+ for (uint64_t i = start_block_offset; i < start_block_offset + range_length_in_blocks; i++)
+ {
+ printDebugInformationForBlock(read_offset, i, bufferlist);
+ }
+}
+
+void ceph::io_exerciser::data_generation::HeaderedSeededRandomGenerator
+ ::printDebugInformationForOffsets(uint64_t read_offset,
+ std::vector<uint64_t> offsets,
+ const ceph::bufferlist& bufferlist)
+{
+ uint64_t range_start = 0;
+ uint64_t range_length = 0;
+ ErrorType rangeError = ErrorType::UNKNOWN;
+
+ for (const uint64_t& block_offset : offsets)
+ {
+ ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset,
+ bufferlist);
+
+ if (range_start == 0 && range_length == 0)
+ {
+ range_start = block_offset;
+ range_length = 1;
+ rangeError = blockError;
+ }
+ else if (blockError == rangeError &&
+ range_start + range_length == block_offset)
+{
+ range_length++;
+ }
+ else
+ {
+ if (range_length == 1)
+ {
+ printDebugInformationForBlock(read_offset, range_start, bufferlist);
+ }
+ else if (range_length > 1)
+ {
+ printDebugInformationForRange(read_offset, range_start, range_length,
+ rangeError, bufferlist);
+ }
+
+ range_start = block_offset;
+ range_length = 1;
+ rangeError = blockError;
+ }
+ }
+
+ if (range_length == 1)
+ {
+ printDebugInformationForBlock(read_offset, range_start, bufferlist);
+ }
+ else if (range_length > 1)
+ {
+ printDebugInformationForRange(read_offset, range_start, range_length,
+ rangeError, bufferlist);
+ }
+}
\ No newline at end of file
--- /dev/null
+#pragma once
+
+#include <memory>
+#include <random>
+
+#include "include/buffer.h"
+#include "ObjectModel.h"
+
+/* Overview
+ *
+ * class DataGenerator
+ * Generates data buffers for write I/Os using state queried
+ * from ObjectModel. Validates data buffers for read I/Os
+ * against the state in the ObjectModel. If a data miscompare
+ * is detected provide debug information about the state of the
+ * object, the buffer that was read and the expected buffer.
+ *
+ *
+ * class DataGenerator::DataGenerationSingleton
+ * A singleton object created on first use to create and store
+ * a unique run identifier. Can either be created supplying an id
+ * or it can be generated at first use. Id cannot and should not be
+ * changed after being created.
+ *
+ *
+ * class SeededRandomGenerator
+ * Inherits from DataGenerator. Generates entirely random patterns
+ * based on the seed retrieved by the model.
+ *
+ *
+ * class HeaderedSeededRandomGenerator
+ * Inherits from SeededDataGenerator. Generates entirely random patterns
+ * based on the seed retrieved by the model, however also appends a
+ * header to the start of each block. This generator also provides
+ * a range of verbose debug options to help disagnose a miscompare
+ * whenever it detects unexpected data.
+ */
+
+namespace ceph {
+ namespace io_exerciser {
+ namespace data_generation {
+ enum class GenerationType {
+ SeededRandom,
+ HeaderedSeededRandom
+ // CompressedGenerator
+ // MixedGenerator
+ };
+
+ class DataGenerator {
+ public:
+ virtual ~DataGenerator() = default;
+ static std::unique_ptr<DataGenerator>
+ create_generator(GenerationType generatorType,
+ const ObjectModel& model);
+ virtual void generate_data(uint64_t length, uint64_t offset,
+ bufferlist& retlist)=0;
+ virtual bool validate(bufferlist& bufferlist, uint64_t offset,
+ uint64_t length);
+
+ // Used for testing debug outputs from data generation
+ virtual void generate_wrong_data(uint64_t offset, uint64_t length,
+ bufferlist& retlist);
+
+ protected:
+ const ObjectModel& m_model;
+
+ DataGenerator(const ObjectModel& model) : m_model(model) {}
+
+ class DataGenerationSingleton {
+ private:
+ DataGenerationSingleton();
+ DataGenerationSingleton(uint64_t unique_id);
+
+ bool m_created = false;
+ uint64_t m_uniqueId = 0;
+
+ static DataGenerationSingleton m_singletonInstance;
+ static std::mutex m_mutex;
+
+ public:
+ static const DataGenerationSingleton&
+ createSpecificInstance(uint64_t unique_id);
+ static const DataGenerationSingleton& getInstance();
+ static const uint64_t getUniqueId();
+ };
+ };
+
+ class SeededRandomGenerator : public DataGenerator
+ {
+ public:
+ SeededRandomGenerator(const ObjectModel& model)
+ : DataGenerator(model) {}
+
+ virtual bufferptr generate_block(uint64_t offset);
+ virtual void generate_data(uint64_t length, uint64_t offset,
+ bufferlist& retlist) override;
+ };
+
+ class HeaderedSeededRandomGenerator : public SeededRandomGenerator
+ {
+ public:
+ HeaderedSeededRandomGenerator(const ObjectModel& model)
+ : SeededRandomGenerator(model) {}
+
+ bufferptr generate_block(uint64_t offset) override;
+ void generate_wrong_data(uint64_t offset, uint64_t length,
+ bufferlist& retlist) override;
+ bool validate(bufferlist& bufferlist, uint64_t offset,
+ uint64_t length) override;
+
+ private:
+ using UniqueIdBytes = uint64_t;
+ using SeedBytes = int;
+ using TimeBytes = uint64_t;
+
+ constexpr uint8_t headerStart() const
+ { return 0; };
+ constexpr uint8_t uniqueIdStart() const
+ { return headerStart(); };
+ constexpr uint8_t uniqueIdLength() const
+ { return sizeof(UniqueIdBytes); };
+ constexpr uint8_t seedStart() const
+ { return uniqueIdStart() + uniqueIdLength(); };
+ constexpr uint8_t seedLength() const
+ { return sizeof(SeedBytes); };
+ constexpr uint8_t timeStart() const
+ { return seedStart() + seedLength(); };
+ constexpr uint8_t timeLength() const
+ { return sizeof(TimeBytes); };
+ constexpr uint8_t timeEnd() const
+ { return timeStart() + timeLength(); };
+ constexpr uint8_t headerLength() const
+ { return uniqueIdLength() + seedLength() + timeLength(); };
+ constexpr uint8_t bodyStart() const
+ { return headerStart() + headerLength(); };
+
+ const UniqueIdBytes readUniqueRunId(uint64_t block_offset,
+ const bufferlist& bufferlist);
+ const SeedBytes readSeed(uint64_t block_offset,
+ const bufferlist& bufferlist);
+ const TimeBytes readDateTime(uint64_t block_offset,
+ const bufferlist& bufferlist);
+
+ bool validate_block(uint64_t block_offset, const char* buffer_start);
+
+ enum class ErrorType {
+ RUN_ID_MISMATCH,
+ SEED_MISMATCH,
+ DATA_MISMATCH,
+ DATA_NOT_FOUND,
+ UNKNOWN
+ };
+
+ const ErrorType getErrorTypeForBlock(uint64_t read_offset,
+ uint64_t block_offset,
+ const bufferlist& bufferlist);
+
+ void printDebugInformationForBlock(uint64_t read_offset,
+ uint64_t block_offset,
+ const bufferlist& bufferlist);
+ void printDebugInformationForRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ ErrorType rangeError,
+ const bufferlist& bufferlist);
+
+ void printDebugInformationForRunIdMismatchRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const bufferlist& bufferlist);
+ void printDebugInformationForSeedMismatchRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const bufferlist& bufferlist);
+ void printDebugInformationDataBodyMismatchRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const bufferlist& bufferlist);
+ void printDebugInformationDataNotFoundRange(uint64_t ßread_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const bufferlist& bufferlist);
+ void printDebugInformationCorruptRange(uint64_t read_offset,
+ uint64_t start_block_offset,
+ uint64_t range_length_in_blocks,
+ const bufferlist& bufferlist);
+
+ void printDebugInformationForOffsets(uint64_t read_offset,
+ std::vector<uint64_t> offsets,
+ const bufferlist& bufferlist);
+ };
+ }
+ }
+}
--- /dev/null
+#include "IoOp.h"
+
+ceph::io_exerciser::IoOp::IoOp( OpType op,
+ uint64_t offset1, uint64_t length1,
+ uint64_t offset2, uint64_t length2,
+ uint64_t offset3, uint64_t length3) :
+ op(op),
+ offset1(offset1), length1(length1),
+ offset2(offset2), length2(length2),
+ offset3(offset3), length3(length3)
+{
+
+}
+
+std::string ceph::io_exerciser::IoOp::value_to_string(uint64_t v) const
+{
+ if (v < 1024 || (v % 1024) != 0) {
+ return std::to_string(v);
+ }else if (v < 1024*1024 || (v % (1024 * 1024)) != 0 ) {
+ return std::to_string(v / 1024) + "K";
+ }else{
+ return std::to_string(v / 1024 / 1024) + "M";
+ }
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_done() {
+
+ return std::make_unique<IoOp>(OpType::Done);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_barrier() {
+
+ return std::make_unique<IoOp>(OpType::BARRIER);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_create(uint64_t size) {
+
+ return std::make_unique<IoOp>(OpType::CREATE,0,size);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_remove() {
+
+ return std::make_unique<IoOp>(OpType::REMOVE);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_read(uint64_t offset, uint64_t length) {
+
+ return std::make_unique<IoOp>(OpType::READ, offset, length);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_read2(uint64_t offset1, uint64_t length1,
+ uint64_t offset2, uint64_t length2) {
+
+ if (offset1 < offset2) {
+ ceph_assert( offset1 + length1 <= offset2 );
+ } else {
+ ceph_assert( offset2 + length2 <= offset1 );
+ }
+
+ return std::make_unique<IoOp>(OpType::READ2,
+ offset1, length1,
+ offset2, length2);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_read3(uint64_t offset1, uint64_t length1,
+ uint64_t offset2, uint64_t length2,
+ uint64_t offset3, uint64_t length3) {
+
+ if (offset1 < offset2) {
+ ceph_assert( offset1 + length1 <= offset2 );
+ } else {
+ ceph_assert( offset2 + length2 <= offset1 );
+ }
+ if (offset1 < offset3) {
+ ceph_assert( offset1 + length1 <= offset3 );
+ } else {
+ ceph_assert( offset3 + length3 <= offset1 );
+ }
+ if (offset2 < offset3) {
+ ceph_assert( offset2 + length2 <= offset3 );
+ } else {
+ ceph_assert( offset3 + length3 <= offset2 );
+ }
+ return std::make_unique<IoOp>(OpType::READ3,
+ offset1, length1,
+ offset2, length2,
+ offset3, length3);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_write(uint64_t offset, uint64_t length) {
+
+ return std::make_unique<IoOp>(OpType::WRITE, offset, length);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_write2(uint64_t offset1, uint64_t length1,
+ uint64_t offset2, uint64_t length2) {
+
+ if (offset1 < offset2) {
+ ceph_assert( offset1 + length1 <= offset2 );
+ } else {
+ ceph_assert( offset2 + length2 <= offset1 );
+ }
+ return std::make_unique<IoOp>(OpType::WRITE2,
+ offset1, length1,
+ offset2, length2);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoOp
+ ::generate_write3(uint64_t offset1, uint64_t length1,
+ uint64_t offset2, uint64_t length2,
+ uint64_t offset3, uint64_t length3) {
+
+ if (offset1 < offset2) {
+ ceph_assert( offset1 + length1 <= offset2 );
+ } else {
+ ceph_assert( offset2 + length2 <= offset1 );
+ }
+ if (offset1 < offset3) {
+ ceph_assert( offset1 + length1 <= offset3 );
+ } else {
+ ceph_assert( offset3 + length3 <= offset1 );
+ }
+ if (offset2 < offset3) {
+ ceph_assert( offset2 + length2 <= offset3 );
+ } else {
+ ceph_assert( offset3 + length3 <= offset2 );
+ }
+ return std::make_unique<IoOp>(OpType::WRITE3,
+ offset1, length1,
+ offset2, length2,
+ offset3, length3);
+}
+
+bool ceph::io_exerciser::IoOp::done() {
+ return (op == OpType::Done);
+}
+
+std::string ceph::io_exerciser::IoOp::to_string(uint64_t block_size) const
+{
+ switch (op) {
+ case OpType::Done:
+ return "Done";
+ case OpType::BARRIER:
+ return "Barrier";
+ case OpType::CREATE:
+ return "Create (size=" + value_to_string(length1 * block_size) + ")";
+ case OpType::REMOVE:
+ return "Remove";
+ case OpType::READ:
+ return "Read (offset=" + value_to_string(offset1 * block_size) +
+ ",length=" + value_to_string(length1 * block_size) + ")";
+ case OpType::READ2:
+ return "Read2 (offset1=" + value_to_string(offset1 * block_size) +
+ ",length1=" + value_to_string(length1 * block_size) +
+ ",offset2=" + value_to_string(offset2 * block_size) +
+ ",length2=" + value_to_string(length2 * block_size) + ")";
+ case OpType::READ3:
+ return "Read3 (offset1=" + value_to_string(offset1 * block_size) +
+ ",length1=" + value_to_string(length1 * block_size) +
+ ",offset2=" + value_to_string(offset2 * block_size) +
+ ",length2=" + value_to_string(length2 * block_size) +
+ ",offset3=" + value_to_string(offset3 * block_size) +
+ ",length3=" + value_to_string(length3 * block_size) + ")";
+ case OpType::WRITE:
+ return "Write (offset=" + value_to_string(offset1 * block_size) +
+ ",length=" + value_to_string(length1 * block_size) + ")";
+ case OpType::WRITE2:
+ return "Write2 (offset1=" + value_to_string(offset1 * block_size) +
+ ",length1=" + value_to_string(length1 * block_size) +
+ ",offset2=" + value_to_string(offset2 * block_size) +
+ ",length2=" + value_to_string(length2 * block_size) + ")";
+ case OpType::WRITE3:
+ return "Write3 (offset1=" + value_to_string(offset1 * block_size) +
+ ",length1=" + value_to_string(length1 * block_size) +
+ ",offset2=" + value_to_string(offset2 * block_size) +
+ ",length2=" + value_to_string(length2 * block_size) +
+ ",offset3=" + value_to_string(offset3 * block_size) +
+ ",length3=" + value_to_string(length3 * block_size) + ")";
+ default:
+ break;
+ }
+ return "Unknown";
+}
\ No newline at end of file
--- /dev/null
+#pragma once
+
+#include <string>
+#include <memory>
+#include "include/ceph_assert.h"
+
+/* Overview
+ *
+ * enum OpType
+ * Enumeration of different types of I/O operation
+ *
+ * class IoOp
+ * Stores details for an I/O operation. Generated by IoSequences
+ * and applied by IoExerciser's
+ */
+
+namespace ceph {
+ namespace io_exerciser {
+
+ enum OpType {
+ Done, // End of I/O sequence
+ BARRIER, // Barrier - all prior I/Os must complete
+ CREATE, // Create object and pattern with data
+ REMOVE, // Remove object
+ READ, // Read
+ READ2, // 2 Reads in one op
+ READ3, // 3 Reads in one op
+ WRITE, // Write
+ WRITE2, // 2 Writes in one op
+ WRITE3 // 3 Writes in one op
+ };
+
+ class IoOp {
+ protected:
+ std::string value_to_string(uint64_t v) const;
+
+ public:
+ OpType op;
+ uint64_t offset1;
+ uint64_t length1;
+ uint64_t offset2;
+ uint64_t length2;
+ uint64_t offset3;
+ uint64_t length3;
+
+ IoOp( OpType op,
+ uint64_t offset1 = 0, uint64_t length1 = 0,
+ uint64_t offset2 = 0, uint64_t length2 = 0,
+ uint64_t offset3 = 0, uint64_t length3 = 0 );
+
+ static std::unique_ptr<IoOp> generate_done();
+
+ static std::unique_ptr<IoOp> generate_barrier();
+
+ static std::unique_ptr<IoOp> generate_create(uint64_t size);
+
+ static std::unique_ptr<IoOp> generate_remove();
+
+ static std::unique_ptr<IoOp> generate_read(uint64_t offset,
+ uint64_t length);
+
+ static std::unique_ptr<IoOp> generate_read2(uint64_t offset1,
+ uint64_t length1,
+ uint64_t offset2,
+ uint64_t length2);
+
+ static std::unique_ptr<IoOp> generate_read3(uint64_t offset1,
+ uint64_t length1,
+ uint64_t offset2,
+ uint64_t length2,
+ uint64_t offset3,
+ uint64_t length3);
+
+ static std::unique_ptr<IoOp> generate_write(uint64_t offset,
+ uint64_t length);
+
+ static std::unique_ptr<IoOp> generate_write2(uint64_t offset1,
+ uint64_t length1,
+ uint64_t offset2,
+ uint64_t length2);
+
+ static std::unique_ptr<IoOp> generate_write3(uint64_t offset1,
+ uint64_t length1,
+ uint64_t offset2,
+ uint64_t length2,
+ uint64_t offset3,
+ uint64_t length3);
+
+ bool done();
+
+ std::string to_string(uint64_t block_size) const;
+ };
+ }
+}
\ No newline at end of file
--- /dev/null
+#include "IoSequence.h"
+
+ceph::io_exerciser::IoSequence::IoSequence(std::pair<int,int> obj_size_range,
+ int seed) :
+ min_obj_size(obj_size_range.first), max_obj_size(obj_size_range.second),
+ create(true), barrier(false), done(false), remove(false),
+ obj_size(min_obj_size), step(-1), seed(seed)
+{
+ rng.seed(seed);
+}
+
+std::unique_ptr<ceph::io_exerciser::IoSequence>
+ ceph::io_exerciser::IoSequence::generate_sequence
+ (
+ Sequence s,
+ std::pair<int,int> obj_size_range,
+ int seed
+ )
+{
+ switch (s) {
+ case SEQUENCE_SEQ0:
+ return std::make_unique<Seq0>(obj_size_range, seed);
+ case SEQUENCE_SEQ1:
+ return std::make_unique<Seq1>(obj_size_range, seed);
+ case SEQUENCE_SEQ2:
+ return std::make_unique<Seq2>(obj_size_range, seed);
+ case SEQUENCE_SEQ3:
+ return std::make_unique<Seq3>(obj_size_range, seed);
+ case SEQUENCE_SEQ4:
+ return std::make_unique<Seq4>(obj_size_range, seed);
+ case SEQUENCE_SEQ5:
+ return std::make_unique<Seq5>(obj_size_range, seed);
+ case SEQUENCE_SEQ6:
+ return std::make_unique<Seq6>(obj_size_range, seed);
+ case SEQUENCE_SEQ7:
+ return std::make_unique<Seq7>(obj_size_range, seed);
+ case SEQUENCE_SEQ8:
+ return std::make_unique<Seq8>(obj_size_range, seed);
+ case SEQUENCE_SEQ9:
+ return std::make_unique<Seq9>(obj_size_range, seed);
+ default:
+ break;
+ }
+ return nullptr;
+}
+
+int ceph::io_exerciser::IoSequence::get_step() const
+{
+ return step;
+}
+
+int ceph::io_exerciser::IoSequence::get_seed() const
+{
+ return seed;
+}
+
+void ceph::io_exerciser::IoSequence::set_min_object_size(uint64_t size)
+{
+ min_obj_size = size;
+ if (obj_size < size) {
+ obj_size = size;
+ if (obj_size > max_obj_size) {
+ done = true;
+ }
+ }
+}
+
+void ceph::io_exerciser::IoSequence::set_max_object_size(uint64_t size)
+{
+ max_obj_size = size;
+ if (obj_size > size) {
+ done = true;
+ }
+}
+
+void ceph::io_exerciser::IoSequence::select_random_object_size()
+{
+ if (max_obj_size != min_obj_size) {
+ obj_size = min_obj_size + rng(max_obj_size - min_obj_size);
+ }
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp>
+ ceph::io_exerciser::IoSequence::increment_object_size()
+{
+ obj_size++;
+ if (obj_size > max_obj_size) {
+ done = true;
+ }
+ create = true;
+ barrier = true;
+ remove = true;
+ return IoOp::generate_barrier();
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::IoSequence::next()
+{
+ step++;
+ if (remove) {
+ remove = false;
+ return IoOp::generate_remove();
+ }
+ if (barrier) {
+ barrier = false;
+ return IoOp::generate_barrier();
+ }
+ if (done) {
+ return IoOp::generate_done();
+ }
+ if (create) {
+ create = false;
+ barrier = true;
+ return IoOp::generate_create(obj_size);
+ }
+ return _next();
+}
+
+
+
+ceph::io_exerciser::Seq0::Seq0(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset(0)
+{
+ select_random_object_size();
+ length = 1 + rng(obj_size - 1);
+}
+
+std::string ceph::io_exerciser::Seq0::get_name() const
+{
+ return "Sequential reads of length " + std::to_string(length) +
+ " with queue depth 1 (seqseed " + std::to_string(get_seed()) + ")";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq0::_next()
+{
+ std::unique_ptr<IoOp> r;
+ if (offset >= obj_size) {
+ done = true;
+ barrier = true;
+ remove = true;
+ return IoOp::generate_barrier();
+ }
+ if (offset + length > obj_size) {
+ r = IoOp::generate_read(offset, obj_size - offset);
+ } else {
+ r = IoOp::generate_read(offset, length);
+ }
+ offset += length;
+ return r;
+}
+
+
+
+ceph::io_exerciser::Seq1::Seq1(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed)
+{
+ select_random_object_size();
+ count = 3 * obj_size;
+}
+
+std::string ceph::io_exerciser::Seq1::get_name() const
+{
+ return "Random offset, random length read/write I/O with queue depth 1 (seqseed "
+ + std::to_string(get_seed()) + ")";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq1::_next()
+{
+ barrier = true;
+ if (count-- == 0) {
+ done = true;
+ remove = true;
+ return IoOp::generate_barrier();
+ }
+
+ uint64_t offset = rng(obj_size - 1);
+ uint64_t length = 1 + rng(obj_size - 1 - offset);
+ return (rng(2) != 0) ? IoOp::generate_write(offset, length) :
+ IoOp::generate_read(offset, length);
+}
+
+
+
+ceph::io_exerciser::Seq2::Seq2(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset(0), length(0) {}
+
+std::string ceph::io_exerciser::Seq2::get_name() const
+{
+ return "Permutations of offset and length read I/O";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq2::_next()
+{
+ length++;
+ if (length > obj_size - offset) {
+ length = 1;
+ offset++;
+ if (offset >= obj_size) {
+ offset = 0;
+ length = 0;
+ return increment_object_size();
+ }
+ }
+ return IoOp::generate_read(offset, length);
+}
+
+
+
+ceph::io_exerciser::Seq3::Seq3(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset1(0), offset2(0)
+{
+ set_min_object_size(2);
+}
+
+std::string ceph::io_exerciser::Seq3::get_name() const
+{
+ return "Permutations of offset 2-region 1-block read I/O";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq3::_next()
+{
+ offset2++;
+ if (offset2 >= obj_size - offset1) {
+ offset2 = 1;
+ offset1++;
+ if (offset1 + 1 >= obj_size) {
+ offset1 = 0;
+ offset2 = 0;
+ return increment_object_size();
+ }
+ }
+ return IoOp::generate_read2(offset1, 1, offset1 + offset2, 1);
+}
+
+
+
+ceph::io_exerciser::Seq4::Seq4(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset1(0), offset2(1)
+{
+ set_min_object_size(3);
+}
+
+std::string ceph::io_exerciser::Seq4::get_name() const
+{
+ return "Permutations of offset 3-region 1-block read I/O";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq4::_next()
+{
+ offset2++;
+ if (offset2 >= obj_size - offset1) {
+ offset2 = 2;
+ offset1++;
+ if (offset1 + 2 >= obj_size) {
+ offset1 = 0;
+ offset2 = 1;
+ return increment_object_size();
+ }
+ }
+ return IoOp::generate_read3(offset1, 1,
+ offset1 + offset2, 1,
+ (offset1 * 2 + offset2)/2, 1);
+}
+
+
+
+ceph::io_exerciser::Seq5::Seq5(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset(0), length(1),
+ doneread(false), donebarrier(false) {}
+
+std::string ceph::io_exerciser::Seq5::get_name() const
+{
+ return "Permutation of length sequential writes";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq5::_next()
+{
+ if (offset >= obj_size) {
+ if (!doneread) {
+ if (!donebarrier) {
+ donebarrier = true;
+ return IoOp::generate_barrier();
+ }
+ doneread = true;
+ barrier = true;
+ return IoOp::generate_read(0, obj_size);
+ }
+ doneread = false;
+ donebarrier = false;
+ offset = 0;
+ length++;
+ if (length > obj_size) {
+ length = 1;
+ return increment_object_size();
+ }
+ }
+ uint64_t io_len = (offset + length > obj_size) ? (obj_size - offset) : length;
+ std::unique_ptr<IoOp> r = IoOp::generate_write(offset, io_len);
+ offset += io_len;
+ return r;
+}
+
+
+
+ceph::io_exerciser::Seq6::Seq6(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset(0), length(1),
+ doneread(false), donebarrier(false) {}
+
+std::string ceph::io_exerciser::Seq6::get_name() const
+{
+ return "Permutation of length sequential writes, different alignment";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq6::_next()
+{
+ if (offset >= obj_size) {
+ if (!doneread) {
+ if (!donebarrier) {
+ donebarrier = true;
+ return IoOp::generate_barrier();
+ }
+ doneread = true;
+ barrier = true;
+ return IoOp::generate_read(0, obj_size);
+ }
+ doneread = false;
+ donebarrier = false;
+ offset = 0;
+ length++;
+ if (length > obj_size) {
+ length = 1;
+ return increment_object_size();
+ }
+ }
+ uint64_t io_len = (offset == 0) ? (obj_size % length) : length;
+ if (io_len == 0) {
+ io_len = length;
+ }
+ std::unique_ptr<IoOp> r = IoOp::generate_write(offset, io_len);
+ offset += io_len;
+ return r;
+}
+
+
+
+ceph::io_exerciser::Seq7::Seq7(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed)
+{
+ set_min_object_size(2);
+ offset = obj_size;
+}
+
+std::string ceph::io_exerciser::Seq7::get_name() const
+{
+ return "Permutations of offset 2-region 1-block writes";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq7::_next()
+{
+ if (!doneread) {
+ if (!donebarrier) {
+ donebarrier = true;
+ return IoOp::generate_barrier();
+ }
+ doneread = true;
+ barrier = true;
+ return IoOp::generate_read(0, obj_size);
+ }
+ if (offset == 0) {
+ doneread = false;
+ donebarrier = false;
+ offset = obj_size+1;
+ return increment_object_size();
+ }
+ offset--;
+ if (offset == obj_size/2) {
+ return _next();
+ }
+ doneread = false;
+ donebarrier = false;
+ return IoOp::generate_write2(offset, 1, obj_size/2, 1);
+}
+
+
+
+ceph::io_exerciser::Seq8::Seq8(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset1(0), offset2(1)
+{
+ set_min_object_size(3);
+}
+
+std::string ceph::io_exerciser::Seq8::get_name() const
+{
+ return "Permutations of offset 3-region 1-block write I/O";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq8::_next()
+{
+ if (!doneread) {
+ if (!donebarrier) {
+ donebarrier = true;
+ return IoOp::generate_barrier();
+ }
+ doneread = true;
+ barrier = true;
+ return IoOp::generate_read(0, obj_size);
+ }
+ offset2++;
+ if (offset2 >= obj_size - offset1) {
+ offset2 = 2;
+ offset1++;
+ if (offset1 + 2 >= obj_size) {
+ offset1 = 0;
+ offset2 = 1;
+ return increment_object_size();
+ }
+ }
+ doneread = false;
+ donebarrier = false;
+ return IoOp::generate_write3(offset1, 1,
+ offset1 + offset2, 1,
+ (offset1 * 2 + offset2)/2, 1);
+}
+
+
+
+ceph::io_exerciser::Seq9::Seq9(std::pair<int,int> obj_size_range, int seed) :
+ IoSequence(obj_size_range, seed), offset(0), length(0)
+{
+
+}
+
+std::string ceph::io_exerciser::Seq9::get_name() const
+{
+ return "Permutations of offset and length write I/O";
+}
+
+std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq9::_next()
+{
+ if (!doneread) {
+ if (!donebarrier) {
+ donebarrier = true;
+ return IoOp::generate_barrier();
+ }
+ doneread = true;
+ barrier = true;
+ return IoOp::generate_read(0, obj_size);
+ }
+ length++;
+ if (length > obj_size - offset) {
+ length = 1;
+ offset++;
+ if (offset >= obj_size) {
+ offset = 0;
+ length = 0;
+ return increment_object_size();
+ }
+ }
+ doneread = false;
+ donebarrier = false;
+ return IoOp::generate_write(offset, length);
+}
\ No newline at end of file
--- /dev/null
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab
+#pragma once
+
+#include "IoOp.h"
+
+#include "include/random.h"
+
+/* Overview
+ *
+ * enum Sequence
+ * Enumeration of the different sequences
+ *
+ * class IoSequence
+ * Virtual class. IoSequences generate a stream of IoOPs.
+ * Sequences typically exhastively test permutations of
+ * offset and length to allow validation of code such as
+ * Erasure Coding. An IoSequence does not determine
+ * whether I/Os are issued sequentially or in parallel,
+ * it must generate barrier I/Os where operations must
+ * be serialized.
+ *
+ * class Seq*
+ * Implementations of IoSequence. Each class generates
+ * a different sequence of I/O.
+ *
+ * generate_sequence
+ * Create an IoSequence
+ */
+
+namespace ceph {
+ namespace io_exerciser {
+
+ enum Sequence {
+ SEQUENCE_SEQ0,
+ SEQUENCE_SEQ1,
+ SEQUENCE_SEQ2,
+ SEQUENCE_SEQ3,
+ SEQUENCE_SEQ4,
+ SEQUENCE_SEQ5,
+ SEQUENCE_SEQ6,
+ SEQUENCE_SEQ7,
+ SEQUENCE_SEQ8,
+ SEQUENCE_SEQ9,
+ //
+ SEQUENCE_END,
+ SEQUENCE_BEGIN = SEQUENCE_SEQ0
+ };
+ inline Sequence operator++( Sequence& s )
+ {
+ return s = (Sequence)(((int)(s) + 1));
+ }
+
+ /* I/O Sequences */
+
+ class IoSequence {
+ public:
+ virtual ~IoSequence() = default;
+
+ virtual std::string get_name() const = 0;
+ int get_step() const;
+ int get_seed() const;
+
+ std::unique_ptr<IoOp> next();
+
+ static std::unique_ptr<IoSequence>
+ generate_sequence(Sequence s, std::pair<int,int> obj_size_range, int seed );
+
+ protected:
+ uint64_t min_obj_size;
+ uint64_t max_obj_size;
+ bool create;
+ bool barrier;
+ bool done;
+ bool remove;
+ uint64_t obj_size;
+ int step;
+ int seed;
+ ceph::util::random_number_generator<int> rng =
+ ceph::util::random_number_generator<int>();
+
+ IoSequence(std::pair<int,int> obj_size_range, int seed);
+
+ virtual std::unique_ptr<IoOp> _next() = 0;
+
+ void set_min_object_size(uint64_t size);
+ void set_max_object_size(uint64_t size);
+ void select_random_object_size();
+ std::unique_ptr<IoOp> increment_object_size();
+
+ };
+
+ class Seq0: public IoSequence {
+ public:
+ Seq0(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+
+ private:
+ uint64_t offset;
+ uint64_t length;
+ };
+
+ class Seq1: public IoSequence {
+ public:
+ Seq1(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next();
+
+ private:
+ int count;
+ };
+
+ class Seq2: public IoSequence {
+ public:
+ Seq2(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+
+ private:
+ uint64_t offset;
+ uint64_t length;
+ };
+
+ class Seq3: public IoSequence {
+ public:
+ Seq3(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+ private:
+ uint64_t offset1;
+ uint64_t offset2;
+ };
+
+ class Seq4: public IoSequence {
+ public:
+ Seq4(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+
+ private:
+ uint64_t offset1;
+ uint64_t offset2;
+ };
+
+ class Seq5: public IoSequence {
+ public:
+ Seq5(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+
+ private:
+ uint64_t offset;
+ uint64_t length;
+ bool doneread;
+ bool donebarrier;
+ };
+
+ class Seq6: public IoSequence {
+ public:
+ Seq6(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+
+ private:
+ uint64_t offset;
+ uint64_t length;
+ bool doneread;
+ bool donebarrier;
+ };
+
+ class Seq7: public IoSequence {
+ public:
+ Seq7(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+
+ private:
+ uint64_t offset;
+ bool doneread = true;
+ bool donebarrier = false;
+ };
+
+ class Seq8: public IoSequence {
+ public:
+ Seq8(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+ std::unique_ptr<IoOp> _next() override;
+ private:
+ uint64_t offset1;
+ uint64_t offset2;
+ bool doneread = true;
+ bool donebarrier = false;
+ };
+
+ class Seq9: public IoSequence {
+ private:
+ uint64_t offset;
+ uint64_t length;
+ bool doneread = true;
+ bool donebarrier = false;
+
+ public:
+ Seq9(std::pair<int,int> obj_size_range, int seed);
+
+ std::string get_name() const override;
+
+ std::unique_ptr<IoOp> _next() override;
+ };
+ }
+}
\ No newline at end of file
--- /dev/null
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab
+#include "Model.h"
+
+ceph::io_exerciser::Model::Model(const std::string oid, uint64_t block_size) :
+num_io(0),
+oid(oid),
+block_size(block_size)
+{
+
+}
+
+const uint64_t ceph::io_exerciser::Model::get_block_size() const
+{
+ return block_size;
+}
+
+const std::string& ceph::io_exerciser::Model::get_oid() const
+{
+ return oid;
+}
+
+int ceph::io_exerciser::Model::get_num_io() const
+{
+ return num_io;
+}
\ No newline at end of file
--- /dev/null
+#pragma once
+
+#include "IoOp.h"
+
+#include <boost/asio/io_context.hpp>
+
+#include "librados/librados_asio.h"
+
+#include "include/interval_set.h"
+#include "global/global_init.h"
+#include "global/global_context.h"
+#include "common/Thread.h"
+
+/* Overview
+ *
+ * class Model
+ * Virtual class. Models apply IoOps generated by an
+ * IoSequence, they can choose how many I/Os to execute in
+ * parallel and scale up the size of I/Os by the blocksize
+ *
+ */
+
+namespace ceph {
+ namespace io_exerciser {
+
+ class Model
+ {
+ protected:
+ int num_io;
+ const std::string oid;
+ uint64_t block_size;
+
+ public:
+ Model(const std::string oid, uint64_t block_size);
+ virtual ~Model() = default;
+
+ virtual bool readyForIoOp(IoOp& op) = 0;
+ virtual void applyIoOp(IoOp& op) = 0;
+
+ const std::string& get_oid() const;
+ const uint64_t get_block_size() const;
+ int get_num_io() const;
+ };
+
+ /* Simple RADOS I/O generator */
+
+
+ }
+}
\ No newline at end of file
--- /dev/null
+#include "ObjectModel.h"
+
+ceph::io_exerciser::ObjectModel::ObjectModel(const std::string oid, uint64_t block_size, int seed) :
+ Model(oid, block_size), created(false)
+{
+ rng.seed(seed);
+}
+
+int ceph::io_exerciser::ObjectModel::get_seed(uint64_t offset) const
+{
+ ceph_assert(offset < contents.size());
+ return contents[offset];
+}
+
+std::vector<int> ceph::io_exerciser::ObjectModel::get_seed_offsets(int seed) const
+{
+ std::vector<int> offsets;
+ for (size_t i = 0; i < contents.size(); i++)
+ {
+ if (contents[i] == seed)
+ {
+ offsets.push_back(i);
+ }
+ }
+
+ return offsets;
+}
+
+std::string ceph::io_exerciser::ObjectModel::to_string(int mask) const
+{
+ if (!created) {
+ return "Object does not exist";
+ }
+ std::string result = "{";
+ for (uint64_t i = 0; i < contents.size(); i++) {
+ if (i != 0) {
+ result += ",";
+ }
+ result += std::to_string(contents[i] & mask);
+ }
+ result += "}";
+ return result;
+}
+
+bool ceph::io_exerciser::ObjectModel::readyForIoOp(IoOp& op)
+{
+ return true;
+}
+
+void ceph::io_exerciser::ObjectModel::applyIoOp(IoOp& op)
+{
+ switch (op.op) {
+ case BARRIER:
+ reads.clear();
+ writes.clear();
+ break;
+
+ case CREATE:
+ ceph_assert(!created);
+ ceph_assert(reads.empty());
+ ceph_assert(writes.empty());
+ created = true;
+ contents.resize(op.length1);
+ for (uint64_t i = 0; i < contents.size(); i++) {
+ contents[i] = rng();
+ }
+ break;
+
+ case REMOVE:
+ ceph_assert(created);
+ ceph_assert(reads.empty());
+ ceph_assert(writes.empty());
+ created = false;
+ contents.resize(0);
+ break;
+
+ case READ3:
+ ceph_assert(created);
+ ceph_assert(op.offset3 + op.length3 <= contents.size());
+ // Not allowed: read overlapping with parallel write
+ ceph_assert(!writes.intersects(op.offset3, op.length3));
+ reads.union_insert(op.offset3, op.length3);
+ [[fallthrough]];
+
+ case READ2:
+ ceph_assert(created);
+ ceph_assert(op.offset2 + op.length2 <= contents.size());
+ // Not allowed: read overlapping with parallel write
+ ceph_assert(!writes.intersects(op.offset2, op.length2));
+ reads.union_insert(op.offset2, op.length2);
+ [[fallthrough]];
+
+ case READ:
+ ceph_assert(created);
+ ceph_assert(op.offset1 + op.length1 <= contents.size());
+ // Not allowed: read overlapping with parallel write
+ ceph_assert(!writes.intersects(op.offset1, op.length1));
+ reads.union_insert(op.offset1, op.length1);
+ num_io++;
+ break;
+
+ case WRITE3:
+ ceph_assert(created);
+ // Not allowed: write overlapping with parallel read or write
+ ceph_assert(!reads.intersects(op.offset3, op.length3));
+ ceph_assert(!writes.intersects(op.offset3, op.length3));
+ writes.union_insert(op.offset3, op.length3);
+ for (uint64_t i = op.offset3; i < op.offset3 + op.length3; i++) {
+ ceph_assert(i < contents.size());
+ contents[i] = rng();
+ }
+ [[fallthrough]];
+
+ case WRITE2:
+ ceph_assert(created);
+ // Not allowed: write overlapping with parallel read or write
+ ceph_assert(!reads.intersects(op.offset2, op.length2));
+ ceph_assert(!writes.intersects(op.offset2, op.length2));
+ writes.union_insert(op.offset2, op.length2);
+ for (uint64_t i = op.offset2; i < op.offset2 + op.length2; i++) {
+ ceph_assert(i < contents.size());
+ contents[i] = rng();
+ }
+ [[fallthrough]];
+
+ case WRITE:
+ ceph_assert(created);
+ // Not allowed: write overlapping with parallel read or write
+ ceph_assert(!reads.intersects(op.offset1, op.length1));
+ ceph_assert(!writes.intersects(op.offset1, op.length1));
+ writes.union_insert(op.offset1, op.length1);
+ for (uint64_t i = op.offset1; i < op.offset1 + op.length1; i++) {
+ ceph_assert(i < contents.size());
+ contents[i] = rng();
+ }
+ num_io++;
+ break;
+ default:
+ break;
+ }
+}
+
+void ceph::io_exerciser::ObjectModel::encode(ceph::buffer::list& bl) const {
+ ENCODE_START(1, 1, bl);
+ encode(created, bl);
+ if (created) {
+ encode(contents, bl);
+ }
+ ENCODE_FINISH(bl);
+}
+
+void ceph::io_exerciser::ObjectModel::decode(ceph::buffer::list::const_iterator& bl) {
+ DECODE_START(1, bl);
+ DECODE_OLDEST(1);
+ decode(created, bl);
+ if (created) {
+ decode(contents, bl);
+ } else {
+ contents.resize(0);
+ }
+ DECODE_FINISH(bl);
+}
--- /dev/null
+#pragma once
+
+#include "Model.h"
+
+/* Overview
+ *
+ * class ObjectModel
+ * An IoExerciser. Tracks the data stored in an object, applies
+ * IoOp's to update the model. Polices that I/Os that are
+ * permitted to run in parallel do not break rules. Provides
+ * interface to query state of object. State can be encoded
+ * and decoded
+ *
+ */
+
+namespace ceph {
+ namespace io_exerciser {
+ /* Model of an object to track its data contents */
+
+ class ObjectModel : public Model {
+ private:
+ bool created;
+ std::vector<int> contents;
+ ceph::util::random_number_generator<int> rng =
+ ceph::util::random_number_generator<int>();
+
+ // Track read and write I/Os that can be submitted in
+ // parallel to detect violations:
+ //
+ // * Read may not overlap with a parallel write
+ // * Write may not overlap with a parallel read or write
+ // * Create / remove may not be in parallel with read or write
+ //
+ // Fix broken test cases by adding barrier ops to restrict
+ // I/O exercisers from issuing conflicting ops in parallel
+ interval_set<uint64_t> reads;
+ interval_set<uint64_t> writes;
+ public:
+ ObjectModel(const std::string oid, uint64_t block_size, int seed);
+
+ int get_seed(uint64_t offset) const;
+ std::vector<int> get_seed_offsets(int seed) const;
+
+ std::string to_string(int mask = -1) const;
+
+ bool readyForIoOp(IoOp& op);
+ void applyIoOp(IoOp& op);
+
+ void encode(ceph::buffer::list& bl) const;
+ void decode(ceph::buffer::list::const_iterator& bl);
+ };
+ }
+}
\ No newline at end of file
--- /dev/null
+#include "RadosIo.h"
+
+#include "DataGenerator.h"
+
+ceph::io_exerciser::RadosIo::RadosIo(librados::Rados& rados,
+ boost::asio::io_context& asio,
+ const std::string pool,
+ const std::string oid,
+ uint64_t block_size,
+ int seed,
+ int threads,
+ ceph::mutex& lock,
+ ceph::condition_variable& cond) :
+ Model(oid, block_size),
+ rados(rados),
+ asio(asio),
+ om(std::make_unique<ObjectModel>(oid, block_size, seed)),
+ db(data_generation::DataGenerator::create_generator(
+ data_generation::GenerationType::HeaderedSeededRandom, *om)),
+ pool(pool),
+ threads(threads),
+ lock(lock),
+ cond(cond),
+ outstanding_io(0)
+{
+ int rc;
+ rc = rados.ioctx_create(pool.c_str(), io);
+ ceph_assert(rc == 0);
+ allow_ec_overwrites(true);
+}
+
+ceph::io_exerciser::RadosIo::~RadosIo()
+{
+}
+
+void ceph::io_exerciser::RadosIo::start_io()
+{
+ std::lock_guard l(lock);
+ outstanding_io++;
+}
+
+void ceph::io_exerciser::RadosIo::finish_io()
+{
+ std::lock_guard l(lock);
+ ceph_assert(outstanding_io > 0);
+ outstanding_io--;
+ cond.notify_all();
+}
+
+void ceph::io_exerciser::RadosIo::wait_for_io(int count)
+{
+ std::unique_lock l(lock);
+ while (outstanding_io > count) {
+ cond.wait(l);
+ }
+}
+
+void ceph::io_exerciser::RadosIo::allow_ec_overwrites(bool allow)
+{
+ int rc;
+ bufferlist inbl, outbl;
+ std::string cmdstr =
+ "{\"prefix\": \"osd pool set\", \"pool\": \"" + pool + "\", \
+ \"var\": \"allow_ec_overwrites\", \"val\": \"" +
+ (allow ? "true" : "false") + "\"}";
+ rc = rados.mon_command(cmdstr, inbl, &outbl, nullptr);
+ ceph_assert(rc == 0);
+}
+
+ceph::io_exerciser::RadosIo::AsyncOpInfo::AsyncOpInfo(
+ uint64_t offset1, uint64_t length1,
+ uint64_t offset2, uint64_t length2,
+ uint64_t offset3, uint64_t length3 ) :
+ offset1(offset1), length1(length1),
+ offset2(offset2), length2(length2),
+ offset3(offset3), length3(length3)
+{
+
+}
+
+bool ceph::io_exerciser::RadosIo::readyForIoOp(IoOp &op)
+{
+ ceph_assert(lock.is_locked_by_me()); //Must be called with lock held
+ if (!om->readyForIoOp(op)) {
+ return false;
+ }
+ switch (op.op) {
+ case Done:
+ case BARRIER:
+ return outstanding_io == 0;
+ default:
+ return outstanding_io < threads;
+ }
+}
+
+void ceph::io_exerciser::RadosIo::applyIoOp(IoOp &op)
+{
+ std::shared_ptr<AsyncOpInfo> op_info;
+
+ om->applyIoOp(op);
+
+ // If there are thread concurrent I/Os in flight then wait for
+ // at least one I/O to complete
+ wait_for_io(threads-1);
+
+ switch (op.op) {
+ case Done:
+ [[ fallthrough ]];
+ case BARRIER:
+ // Wait for all outstanding I/O to complete
+ wait_for_io(0);
+ break;
+
+ case CREATE:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>(0, op.length1);
+ db->generate_data(0, op.length1, op_info->bl1);
+ op_info->wop.write_full(op_info->bl1);
+ auto create_cb = [this] (boost::system::error_code ec) {
+ ceph_assert(ec == boost::system::errc::success);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->wop, 0, nullptr, create_cb);
+ }
+ break;
+
+ case REMOVE:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>();
+ op_info->wop.remove();
+ auto remove_cb = [this] (boost::system::error_code ec) {
+ ceph_assert(ec == boost::system::errc::success);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->wop, 0, nullptr, remove_cb);
+ }
+ break;
+
+ case READ:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1);
+ op_info->rop.read(op.offset1 * block_size,
+ op.length1 * block_size,
+ &op_info->bl1, nullptr);
+ auto read_cb = [this, op_info] (boost::system::error_code ec, bufferlist bl) {
+ ceph_assert(ec == boost::system::errc::success);
+ db->validate(op_info->bl1, op_info->offset1, op_info->length1);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->rop, 0, nullptr, read_cb);
+ num_io++;
+ }
+ break;
+
+ case READ2:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>(op.offset1,
+ op.length1,
+ op.offset2,
+ op.length2);
+
+ op_info->rop.read(op.offset1 * block_size,
+ op.length1 * block_size,
+ &op_info->bl1, nullptr);
+ op_info->rop.read(op.offset2 * block_size,
+ op.length2 * block_size,
+ &op_info->bl2, nullptr);
+ auto read2_cb = [this, op_info] (boost::system::error_code ec,
+ bufferlist bl) {
+ ceph_assert(ec == boost::system::errc::success);
+ db->validate(op_info->bl1, op_info->offset1, op_info->length1);
+ db->validate(op_info->bl2, op_info->offset2, op_info->length2);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->rop, 0, nullptr, read2_cb);
+ num_io++;
+ }
+ break;
+
+ case READ3:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1,
+ op.offset2, op.length2,
+ op.offset3, op.length3);
+ op_info->rop.read(op.offset1 * block_size,
+ op.length1 * block_size,
+ &op_info->bl1, nullptr);
+ op_info->rop.read(op.offset2 * block_size,
+ op.length2 * block_size,
+ &op_info->bl2, nullptr);
+ op_info->rop.read(op.offset3 * block_size,
+ op.length3 * block_size,
+ &op_info->bl3, nullptr);
+ auto read3_cb = [this, op_info] (boost::system::error_code ec,
+ bufferlist bl) {
+ ceph_assert(ec == boost::system::errc::success);
+ db->validate(op_info->bl1, op_info->offset1, op_info->length1);
+ db->validate(op_info->bl2, op_info->offset2, op_info->length2);
+ db->validate(op_info->bl3, op_info->offset3, op_info->length3);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->rop, 0, nullptr, read3_cb);
+ num_io++;
+ }
+ break;
+
+ case WRITE:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1);
+ db->generate_data(op.offset1, op.length1, op_info->bl1);
+
+ op_info->wop.write(op.offset1 * block_size, op_info->bl1);
+ auto write_cb = [this] (boost::system::error_code ec) {
+ ceph_assert(ec == boost::system::errc::success);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->wop, 0, nullptr, write_cb);
+ num_io++;
+ }
+ break;
+
+ case WRITE2:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1,
+ op.offset2, op.length2);
+ db->generate_data(op.offset1, op.length1, op_info->bl1);
+ db->generate_data(op.offset2, op.length2, op_info->bl2);
+ op_info->wop.write(op.offset1 * block_size, op_info->bl1);
+ op_info->wop.write(op.offset2 * block_size, op_info->bl2);
+ auto write2_cb = [this] (boost::system::error_code ec) {
+ ceph_assert(ec == boost::system::errc::success);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->wop, 0, nullptr, write2_cb);
+ num_io++;
+ }
+ break;
+
+ case WRITE3:
+ {
+ start_io();
+ op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1,
+ op.offset2, op.length2,
+ op.offset3, op.length3);
+ db->generate_data(op.offset1, op.length1, op_info->bl1);
+ db->generate_data(op.offset2, op.length2, op_info->bl2);
+ db->generate_data(op.offset3, op.length3, op_info->bl3);
+ op_info->wop.write(op.offset1 * block_size, op_info->bl1);
+ op_info->wop.write(op.offset2 * block_size, op_info->bl2);
+ op_info->wop.write(op.offset3 * block_size, op_info->bl3);
+ auto write3_cb = [this] (boost::system::error_code ec) {
+ ceph_assert(ec == boost::system::errc::success);
+ finish_io();
+ };
+ librados::async_operate(asio, io, oid,
+ &op_info->wop, 0, nullptr, write3_cb);
+ num_io++;
+ }
+ break;
+
+ default:
+ break;
+ }
+}
--- /dev/null
+#pragma once
+
+#include "ObjectModel.h"
+
+/* Overview
+ *
+ * class RadosIo
+ * An IoExerciser. A simple RADOS client that generates I/Os
+ * from IoOps. Uses an ObjectModel to track the data stored
+ * in the object. Uses DataBuffer to create and validate
+ * data buffers. When there are not barrier I/Os this may
+ * issue multiple async I/Os in parallel.
+ *
+ */
+
+namespace ceph {
+ namespace io_exerciser {
+ namespace data_generation {
+ class DataGenerator;
+ }
+
+ class RadosIo: public Model {
+ protected:
+ librados::Rados& rados;
+ boost::asio::io_context& asio;
+ std::unique_ptr<ObjectModel> om;
+ std::unique_ptr<ceph::io_exerciser::data_generation::DataGenerator> db;
+ const std::string pool;
+ int threads;
+ ceph::mutex& lock;
+ ceph::condition_variable& cond;
+ librados::IoCtx io;
+ int outstanding_io;
+
+ void start_io();
+ void finish_io();
+ void wait_for_io(int count);
+
+ public:
+ RadosIo(librados::Rados& rados,
+ boost::asio::io_context& asio,
+ const std::string pool,
+ const std::string oid,
+ uint64_t block_size,
+ int seed,
+ int threads,
+ ceph::mutex& lock,
+ ceph::condition_variable& cond);
+
+ ~RadosIo();
+
+ void allow_ec_overwrites(bool allow);
+
+ class AsyncOpInfo {
+ public:
+ librados::ObjectReadOperation rop;
+ librados::ObjectWriteOperation wop;
+ ceph::buffer::list bl1;
+ ceph::buffer::list bl2;
+ ceph::buffer::list bl3;
+ uint64_t offset1;
+ uint64_t length1;
+ uint64_t offset2;
+ uint64_t length2;
+ uint64_t offset3;
+ uint64_t length3;
+
+ AsyncOpInfo(uint64_t offset1 = 0, uint64_t length1 = 0,
+ uint64_t offset2 = 0, uint64_t length2 = 0,
+ uint64_t offset3 = 0, uint64_t length3 = 0 );
+ ~AsyncOpInfo() = default;
+ };
+
+ // Must be called with lock held
+ bool readyForIoOp(IoOp& op);
+
+ void applyIoOp(IoOp& op);
+ };
+ }
+}
\ No newline at end of file
ceph_test_rados
DESTINATION ${CMAKE_INSTALL_BINDIR})
+add_executable(ceph_test_rados_io_sequence
+ ${CMAKE_CURRENT_SOURCE_DIR}/ceph_test_rados_io_sequence.cc)
+target_link_libraries(ceph_test_rados_io_sequence
+ librados global object_io_exerciser)
+install(TARGETS
+ ceph_test_rados_io_sequence
+ DESTINATION ${CMAKE_INSTALL_BINDIR})
+
# test_stale_read
add_executable(ceph_test_osd_stale_read
ceph_test_osd_stale_read.cc
--- /dev/null
+#include "ceph_test_rados_io_sequence.h"
+
+#include <iostream>
+#include <vector>
+
+#include <boost/asio/io_context.hpp>
+
+#include "include/random.h"
+
+#include "librados/librados_asio.h"
+#include "common/ceph_argparse.h"
+#include "include/interval_set.h"
+#include "global/global_init.h"
+#include "global/global_context.h"
+#include "common/Thread.h"
+#include "common/debug.h"
+#include "common/dout.h"
+#include "common/split.h"
+
+#include "common/io_exerciser/DataGenerator.h"
+#include "common/io_exerciser/Model.h"
+#include "common/io_exerciser/ObjectModel.h"
+#include "common/io_exerciser/RadosIo.h"
+#include "common/io_exerciser/IoOp.h"
+#include "common/io_exerciser/IoSequence.h"
+
+#define dout_subsys ceph_subsys_rados
+#define dout_context g_ceph_context
+
+template <typename T, int N, const std::array<T, N>& Ts>
+ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>
+ ::ProgramOptionSelector(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm,
+ const std::string& option_name,
+ bool set_forced,
+ bool select_first)
+ : rng(rng),
+ // choices(choices),
+ option_name(option_name) {
+ if (set_forced && vm.count(option_name)) {
+ force_value = vm[option_name].as<T>();
+ }
+ if (select_first) {
+ ceph_assert(choices.size() > 0);
+ first_value = choices[0];
+ }
+}
+
+template <typename T, int N, const std::array<T, N>& Ts>
+bool ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::isForced()
+{
+ return force_value.has_value();
+}
+
+template <typename T, int N, const std::array<T, N>& Ts>
+const T ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::choose()
+{
+ if (force_value.has_value()) {
+ return *force_value;
+ } else if (first_value.has_value()) {
+ return *std::exchange(first_value, std::nullopt);
+ } else {
+ return choices[rng(N-1)];
+ }
+}
+
+
+
+ceph::io_sequence::tester::SelectObjectSize::SelectObjectSize(
+ ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm)
+ : ProgramOptionSelector(rng, vm, "objectsize", true, true)
+{
+}
+
+
+
+ceph::io_sequence::tester::SelectBlockSize::SelectBlockSize(
+ ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm)
+ : ProgramOptionSelector(rng, vm, "blocksize", true, true)
+{
+}
+
+
+
+ceph::io_sequence::tester::SelectNumThreads::SelectNumThreads(
+ ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm)
+ : ProgramOptionSelector(rng, vm, "threads", true, true)
+{
+}
+
+
+
+ceph::io_sequence::tester::SelectSeqRange::SelectSeqRange(
+ ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm)
+ : ProgramOptionSelector(rng, vm, "sequence", false, false)
+{
+ if (vm.count(option_name)) {
+ ceph::io_exerciser::Sequence s =
+ static_cast<ceph::io_exerciser::Sequence>(vm["sequence"].as<int>());
+ if (s < ceph::io_exerciser::Sequence::SEQUENCE_BEGIN ||
+ s >= ceph::io_exerciser::Sequence::SEQUENCE_END) {
+ dout(0) << "Sequence argument out of range" << dendl;
+ throw po::validation_error(po::validation_error::invalid_option_value);
+ }
+ ceph::io_exerciser::Sequence e = s;
+ force_value = std::make_optional<std::pair<ceph::io_exerciser::Sequence,
+ ceph::io_exerciser::Sequence>>(
+ std::make_pair(s, ++e));
+ }
+}
+
+const std::pair<ceph::io_exerciser::Sequence,ceph::io_exerciser::Sequence>
+ ceph::io_sequence::tester::SelectSeqRange::choose() {
+ if (force_value.has_value())
+ {
+ return *force_value;
+ } else {
+ return std::make_pair(ceph::io_exerciser::Sequence::SEQUENCE_BEGIN,
+ ceph::io_exerciser::Sequence::SEQUENCE_END);
+ }
+}
+
+
+
+ceph::io_sequence::tester::SelectErasureKM::SelectErasureKM(
+ ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm)
+ : ProgramOptionSelector(rng, vm, "km", true, true)
+{
+}
+
+
+
+ceph::io_sequence::tester::SelectErasurePlugin::SelectErasurePlugin(
+ ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm)
+ : ProgramOptionSelector(rng, vm, "plugin", true, false)
+{
+}
+
+
+
+ceph::io_sequence::tester::SelectErasureChunkSize::SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, po::variables_map vm)
+ : ProgramOptionSelector(rng, vm, "stripe_unit", true, false)
+{
+}
+
+
+
+ceph::io_sequence::tester::SelectECPool::SelectECPool(
+ ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm,
+ librados::Rados& rados,
+ bool dry_run)
+ : ProgramOptionSelector(rng, vm, "pool", false, false),
+ rados(rados),
+ dry_run(dry_run),
+ skm(SelectErasureKM(rng, vm)),
+ spl(SelectErasurePlugin(rng, vm)),
+ scs(SelectErasureChunkSize(rng, vm))
+{
+ if (!skm.isForced()) {
+ if (vm.count("pool")) {
+ force_value = vm["pool"].as<std::string>();
+ }
+ }
+}
+
+const std::string ceph::io_sequence::tester::SelectECPool::choose()
+{
+ std::pair<int,int> value;
+ if (!skm.isForced() && force_value.has_value()) {
+ return *force_value;
+ } else {
+ value = skm.choose();
+ }
+ int k = value.first;
+ int m = value.second;
+
+ const std::string plugin = std::string(spl.choose());
+ const uint64_t chunk_size = scs.choose();
+
+ std::string pool_name = "ec_" + plugin +
+ "_cs" + std::to_string(chunk_size) +
+ "_k" + std::to_string(k) +
+ "_m" + std::to_string(m);
+ if (!dry_run)
+ {
+ create_pool(rados, pool_name, plugin, chunk_size, k, m);
+ }
+ return pool_name;
+}
+
+void ceph::io_sequence::tester::SelectECPool::create_pool(
+ librados::Rados& rados,
+ const std::string& pool_name,
+ const std::string& plugin,
+ uint64_t chunk_size,
+ int k, int m)
+{
+ int rc;
+ bufferlist inbl, outbl;
+ std::string profile_create =
+ "{\"prefix\": \"osd erasure-code-profile set\", \
+ \"name\": \"testprofile-" + pool_name + "\", \
+ \"profile\": [ \"plugin=" + plugin + "\", \
+ \"k=" + std::to_string(k) + "\", \
+ \"m=" + std::to_string(m) + "\", \
+ \"stripe_unit=" + std::to_string(chunk_size) + "\", \
+ \"crush-failure-domain=osd\"]}";
+ rc = rados.mon_command(profile_create, inbl, &outbl, nullptr);
+ ceph_assert(rc == 0);
+ std::string cmdstr =
+ "{\"prefix\": \"osd pool create\", \
+ \"pool\": \"" + pool_name + "\", \
+ \"pool_type\": \"erasure\", \
+ \"pg_num\": 8, \
+ \"pgp_num\": 8, \
+ \"erasure_code_profile\": \"testprofile-" + pool_name + "\"}";
+ rc = rados.mon_command(cmdstr, inbl, &outbl, nullptr);
+ ceph_assert(rc == 0);
+}
+
+
+
+ceph::io_sequence::tester::TestObject::TestObject( const std::string oid,
+ librados::Rados& rados,
+ boost::asio::io_context& asio,
+ SelectBlockSize& sbs,
+ SelectECPool& spo,
+ SelectObjectSize& sos,
+ SelectNumThreads& snt,
+ SelectSeqRange & ssr,
+ ceph::util::random_number_generator<int>& rng,
+ ceph::mutex& lock,
+ ceph::condition_variable& cond,
+ bool dryrun,
+ bool verbose,
+ bool has_seqseed,
+ int seqseed) :
+ rng(rng), verbose(verbose), has_seqseed(has_seqseed), seqseed(seqseed)
+{
+ if (dryrun) {
+ verbose = true;
+ exerciser_model = std::make_unique<ceph::io_exerciser::ObjectModel>(oid,
+ sbs.choose(),
+ rng());
+ } else {
+ const std::string pool = spo.choose();
+ int threads = snt.choose();
+ exerciser_model = std::make_unique<ceph::io_exerciser::RadosIo>(rados,
+ asio,
+ pool,
+ oid,
+ sbs.choose(),
+ rng(),
+ threads,
+ lock,
+ cond);
+ dout(0) << "= " << oid << " pool=" << pool
+ << " threads=" << threads
+ << " blocksize=" << exerciser_model->get_block_size()
+ << " =" << dendl;
+ }
+ obj_size_range = sos.choose();
+ seq_range = ssr.choose();
+ curseq = seq_range.first;
+ seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq,
+ obj_size_range,
+ has_seqseed ?
+ seqseed :
+ rng());
+ op = seq->next();
+ done = false;
+ dout(0) << "== " << exerciser_model->get_oid() << " "
+ << curseq << " "
+ << seq->get_name()
+ << " ==" <<dendl;
+}
+
+bool ceph::io_sequence::tester::TestObject::readyForIo()
+{
+ return exerciser_model->readyForIoOp(*op);
+}
+
+bool ceph::io_sequence::tester::TestObject::next()
+{
+ if (!done) {
+ if (verbose) {
+ dout(0) << exerciser_model->get_oid()
+ << " Step " << seq->get_step() << ": "
+ << op->to_string(exerciser_model->get_block_size()) << dendl;
+ } else {
+ dout(5) << exerciser_model->get_oid()
+ << " Step " << seq->get_step() << ": "
+ << op->to_string(exerciser_model->get_block_size()) << dendl;
+ }
+ exerciser_model->applyIoOp(*op);
+ if (op->done()) {
+ ++curseq;
+ if (curseq == seq_range.second) {
+ done = true;
+ dout(0) << exerciser_model->get_oid()
+ << " Number of IOs = " << exerciser_model->get_num_io()
+ << dendl;
+ } else {
+ seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq,
+ obj_size_range,
+ has_seqseed ?
+ seqseed :
+ rng());
+ dout(0) << "== " << exerciser_model->get_oid() << " "
+ << curseq << " " << seq->get_name()
+ << " ==" <<dendl;
+ op = seq->next();
+ }
+ } else {
+ op = seq->next();
+ }
+ }
+ return done;
+}
+
+bool ceph::io_sequence::tester::TestObject::finished()
+{
+ return done;
+}
+
+int ceph::io_sequence::tester::TestObject::get_num_io()
+{
+ return exerciser_model->get_num_io();
+}
+
+struct Size {};
+void validate(boost::any& v, const std::vector<std::string>& values,
+ Size *target_type, int) {
+ po::validators::check_first_occurrence(v);
+ const std::string &s = po::validators::get_single_string(values);
+
+ std::string parse_error;
+ uint64_t size = strict_iecstrtoll(s, &parse_error);
+ if (!parse_error.empty()) {
+ throw po::validation_error(po::validation_error::invalid_option_value);
+ }
+ v = boost::any(size);
+}
+
+struct Pair {};
+void validate(boost::any& v, const std::vector<std::string>& values,
+ Pair *target_type, int) {
+ po::validators::check_first_occurrence(v);
+ const std::string &s = po::validators::get_single_string(values);
+ auto part = ceph::split(s).begin();
+ std::string parse_error;
+ int first = strict_iecstrtoll(*part++, &parse_error);
+ int second = strict_iecstrtoll(*part, &parse_error);
+ if (!parse_error.empty()) {
+ throw po::validation_error(po::validation_error::invalid_option_value);
+ }
+ v = boost::any(std::pair<int,int>{first,second});
+}
+
+struct PluginString {};
+void validate(boost::any& v, const std::vector<std::string>& values,
+ PluginString *target_type, int) {
+ po::validators::check_first_occurrence(v);
+ const std::string &s = po::validators::get_single_string(values);
+
+ const std::string_view* pluginIt = std::find(
+ ceph::io_sequence::tester::pluginChoices.begin(),
+ ceph::io_sequence::tester::pluginChoices.end(),
+ s
+ );
+ if(ceph::io_sequence::tester::pluginChoices.end() == pluginIt)
+ {
+ throw po::validation_error(po::validation_error::invalid_option_value);
+ }
+
+ v = boost::any(*pluginIt);
+}
+
+int parse_io_seq_options(
+ po::variables_map& vm,
+ const po::options_description& desc,
+ int argc,
+ char** argv)
+{
+ std::vector<std::string> unrecognized_options;
+ try {
+ auto parsed = po::command_line_parser(argc, argv)
+ .options(desc)
+ .allow_unregistered()
+ .run();
+ po::store(parsed, vm);
+ po::notify(vm);
+ unrecognized_options = po::collect_unrecognized(parsed.options,
+ po::include_positional);
+
+ if (!unrecognized_options.empty())
+ {
+ std::stringstream ss;
+ ss << "Unrecognised command options supplied: ";
+ while (unrecognized_options.size() > 1)
+ {
+ ss << unrecognized_options.back().c_str() << ", ";
+ unrecognized_options.pop_back();
+ }
+ ss << unrecognized_options.back();
+ dout(0) << ss.str() << dendl;
+ return 1;
+ }
+ } catch(const po::error& e) {
+ std::cerr << "error: " << e.what() << std::endl;
+ return 1;
+ }
+
+ return 0;
+}
+
+void run_test(const std::vector<
+ std::shared_ptr<ceph::io_sequence::tester::TestObject>
+ >& test_objects,
+ ceph::mutex& lock)
+{
+ // Main loop of test - while not all test objects have finished
+ // check to see if any are able to start a new I/O. If all test
+ // objects are waiting for I/O to complete then wait on a cond
+ // that is signalled each time an I/O completes
+
+ bool started_io = true;
+ bool need_wait = true;
+ while (started_io || need_wait) {
+ started_io = false;
+ need_wait = false;
+ for (auto obj = test_objects.begin(); obj != test_objects.end(); ++obj) {
+ std::shared_ptr<ceph::io_sequence::tester::TestObject> to = *obj;
+ if (!to->finished()) {
+ lock.lock();
+ bool ready = to->readyForIo();
+ lock.unlock();
+ if (ready)
+ {
+ to->next();
+ started_io = true;
+ } else {
+ need_wait = true;
+ }
+ }
+ }
+ if (!started_io && need_wait) {
+ std::unique_lock l(lock);
+ // Recheck with lock incase anything has changed
+ for (auto obj = test_objects.begin(); obj != test_objects.end(); ++obj) {
+ std::shared_ptr<ceph::io_sequence::tester::TestObject> to = *obj;
+ if (!to->finished()) {
+ need_wait = !to->readyForIo();
+ if (!need_wait)
+ {
+ break;
+ }
+ }
+ }
+ need_wait = true;
+ }
+ }
+
+ int total_io = 0;
+ for (auto obj = test_objects.begin(); obj != test_objects.end(); ++obj) {
+ std::shared_ptr<ceph::io_sequence::tester::TestObject> to = *obj;
+ total_io += to->get_num_io();
+ ceph_assert(to->finished());
+ }
+ dout(0) << "Total number of IOs = " << total_io << dendl;
+}
+
+namespace {
+ constexpr std::string_view usage[] = {
+ "Basic usage:",
+ "",
+ "ceph_test_rados_io_sequence",
+ "\t Test I/O to a single object using default settings. Good for",
+ "\t testing boundary conditions",
+ "",
+ "ceph_test_rados_io_sequence --parallel <n>",
+ "\t Run parallel test to multiple objects. First object is tested with",
+ "\t default settings, other objects are tested with random settings",
+ "",
+ "Advanced usage:",
+ "",
+ "ceph_test_rados_io_sequence --blocksize <b> --km <k,m> --plugin <p>",
+ " --objectsize <min,max> --threads <t>",
+ "ceph_test_rados_io_sequence --blocksize <b> --pool <p> --object <oid>",
+ " --objectsize <min,max> --threads <t>",
+ "\tCustomize the test, if a pool is specified then it defines the",
+ "\tReplica/EC configuration",
+ "",
+ "ceph_test_rados_io_sequence --listsequence",
+ "\t Display list of supported I/O sequences",
+ "",
+ "ceph_test_rados_io_sequence --dryrun --sequence <n>",
+ "\t Show I/O that will be generated for a sequence, validate",
+ "\t seqeunce has correct I/O barriers to restrict concurrency",
+ "",
+ "ceph_test_rados_io_sequence --seed <seed>",
+ "\t Repeat a previous test with the same random numbers (seed is",
+ "\t displayed at start of test), if threads = 1 then this will produce",
+ "\t the exact same sequence of I/O, if threads > 1 then I/Os are issued",
+ "\t in parallel so ordering might be slightly different",
+ "",
+ "ceph_test_rados_io_sequence --sequence <n> --seqseed <n>",
+ "\t Repeat a sequence from a previous test with the same random",
+ "\t numbers (seqseed is displayed at start of sequence)",
+ "",
+ "ceph_test_rados_io_sequence --pool <p> --object <oid> --interactive",
+ "\t Execute sequence of I/O commands from stdin. Offset and length",
+ "\t are specified with unit of blocksize. Supported commands:",
+ "\t\t create <len>",
+ "\t\t remove",
+ "\t\t read|write <off> <len>",
+ "\t\t read2|write2 <off> <len> <off> <len>",
+ "\t\t read3|write3 <off> <len> <off> <len> <off> <len>",
+ "\t\t done"
+ };
+}
+
+int main(int argc, char **argv)
+{
+ auto args = argv_to_vec(argc, argv);
+ env_to_vec(args);
+ auto cct = global_init(NULL, args, CEPH_ENTITY_TYPE_CLIENT,
+ CODE_ENVIRONMENT_UTILITY, 0);
+ common_init_finish(cct.get());
+
+ librados::Rados rados;
+ boost::asio::io_context asio;
+ std::thread thread;
+ std::optional<boost::asio::executor_work_guard<
+ boost::asio::io_context::executor_type>> guard;
+ ceph::mutex lock = ceph::make_mutex("RadosIo::lock");
+ ceph::condition_variable cond;
+
+ po::options_description desc("ceph_test_rados_io options");
+
+ desc.add_options()
+ ("help,h",
+ "show help message")
+ ("listsequence,l",
+ "show list of sequences")
+ ("dryrun,d",
+ "test sequence, do not issue any I/O")
+ ("verbose",
+ "more verbose output during test")
+ ("sequence,s", po::value<int>(),
+ "test specified sequence")
+ ("seed", po::value<int>(),
+ "seed for whole test")
+ ("seqseed", po::value<int>(),
+ "seed for sequence")
+ ("blocksize,b", po::value<Size>(),
+ "block size (default 2048)")
+ ("chunksize,c", po::value<Size>(),
+ "chunk size (default 4096)")
+ ("pool,p", po::value<std::string>(),
+ "pool name")
+ ("km", po::value<Pair>(),
+ "k,m EC pool profile (default 2,2)")
+ ("plugin", po::value<PluginString>(),
+ "EC plugin (isa or jerasure)")
+ ("objectsize", po::value<Pair>(),
+ "min,max object size in blocks (default 1,32)")
+ ("threads,t", po::value<int>(),
+ "number of threads of I/O per object (default 1)")
+ ("objects,o", po::value<int>()->default_value(1),
+ "number of objects to exercise in parallel");
+
+ po::variables_map vm;
+ int rc = parse_io_seq_options(vm, desc, argc, argv);
+ if (rc != 0)
+ {
+ return rc;
+ }
+
+ if (vm.count("help")) {
+ std::cout << desc << std::endl;
+ for (auto line : usage) {
+ std::cout << line << std::endl;
+ }
+ return 0;
+ }
+
+ // Seed
+ int seed = time(nullptr);
+ if (vm.count("seed")) {
+ seed = vm["seed"].as<int>();
+ }
+ dout(0) << "Test using seed " << seed << dendl;
+ auto rng = ceph::util::random_number_generator<int>(seed);
+
+ bool verbose = vm.count("verbose");
+ bool dryrun = vm.count("dryrun");
+ bool has_seqseed = vm.count("seqseed");
+ int seqseed = 0;
+ if (has_seqseed) {
+ seqseed = vm["seqseed"].as<int>();
+ }
+ int num_objects = vm["objects"].as<int>();
+
+ if (!dryrun) {
+ rc = rados.init_with_context(g_ceph_context);
+ ceph_assert(rc == 0);
+ rc = rados.connect();
+ ceph_assert(rc == 0);
+
+ guard.emplace(boost::asio::make_work_guard(asio));
+ thread = make_named_thread("io_thread",[&asio] { asio.run(); });
+ }
+
+ // Select block size
+ std::unique_ptr<ceph::io_sequence::tester::SelectBlockSize> sbs
+ = std::make_unique<ceph::io_sequence::tester::SelectBlockSize>(rng, vm);
+
+ // Select pool
+ std::unique_ptr<ceph::io_sequence::tester::SelectECPool> spo
+ = std::make_unique<ceph::io_sequence::tester::SelectECPool>(rng, vm,
+ rados,
+ dryrun);
+
+ // Select object size range
+ std::unique_ptr<ceph::io_sequence::tester::SelectObjectSize> sos
+ = std::make_unique<ceph::io_sequence::tester::SelectObjectSize>(rng,
+ vm);
+
+ // Select number of threads
+ std::unique_ptr<ceph::io_sequence::tester::SelectNumThreads> snt =
+ std::make_unique<ceph::io_sequence::tester::SelectNumThreads>(rng, vm);
+
+ // Select range of sequences
+ std::unique_ptr<ceph::io_sequence::tester::SelectSeqRange> ssr;
+ try {
+ ssr = std::make_unique<ceph::io_sequence::tester::SelectSeqRange>(rng, vm);
+ } catch(const po::error& e) {
+ return 1;
+ }
+
+ // List seqeunces
+ if (vm.count("listsequence")) {
+ std::pair<int,int> obj_size_range = sos->choose();
+ for (ceph::io_exerciser::Sequence s
+ = ceph::io_exerciser::Sequence::SEQUENCE_BEGIN;
+ s < ceph::io_exerciser::Sequence::SEQUENCE_END; ++s) {
+ std::unique_ptr<ceph::io_exerciser::IoSequence> seq =
+ ceph::io_exerciser::IoSequence::generate_sequence(s,
+ obj_size_range,
+ has_seqseed ?
+ seqseed :
+ rng());
+ dout(0) << s << " " << seq->get_name() << dendl;
+ }
+ return 0;
+ }
+
+ // Create a test for each object
+ std::vector<std::shared_ptr<
+ ceph::io_sequence::tester::TestObject>> test_objects;
+
+ for (int obj = 0; obj < num_objects; obj++) {
+ test_objects.push_back(
+ std::make_shared<ceph::io_sequence::tester::TestObject>(
+ "test" + std::to_string(obj),
+ rados, asio,
+ *sbs, *spo, *sos, *snt, *ssr,
+ rng, lock, cond,
+ dryrun, verbose,
+ has_seqseed, seqseed
+ )
+ );
+ }
+ if (!dryrun) {
+ rados.wait_for_latest_osdmap();
+ }
+
+ run_test(test_objects, lock);
+
+ if (!dryrun) {
+ guard = std::nullopt;
+ asio.stop();
+ thread.join();
+ rados.shutdown();
+ }
+ return 0;
+}
--- /dev/null
+#include <utility>
+
+#include "global/global_init.h"
+#include "global/global_context.h"
+
+#include "common/io_exerciser/IoOp.h"
+#include "common/io_exerciser/IoSequence.h"
+#include "common/io_exerciser/Model.h"
+
+#include "librados/librados_asio.h"
+
+#include <boost/program_options.hpp>
+
+/* Overview
+ *
+ * class ProgramOptionSelector
+ * Base class for selector objects below with common code for
+ * selecting options
+ *
+ * class SelectObjectSize
+ * Selects min and max object sizes for a test
+ *
+ * class SelectErasureKM
+ * Selects an EC k and m value for a test
+ *
+ * class SelectErasurePlugin
+ * Selects an plugin for a test
+ *
+ * class SelectECPool
+ * Selects an EC pool (plugin,k and m) for a test. Also creates the
+ * pool as well.
+ *
+ * class SelectBlockSize
+ * Selects a block size for a test
+ *
+ * class SelectNumThreads
+ * Selects number of threads for a test
+ *
+ * class SelectSeqRange
+ * Selects a sequence or range of sequences for a test
+ *
+ * class TestObject
+ * Runs a test against an object, generating IOSequence
+ * and applying them to an IoExerciser
+ *
+ * main
+ * Run sequences of I/O with data integrity checking to
+ * one or more objects in parallel. Without arguments
+ * runs a default configuration against one object.
+ * Command arguments can select alternative
+ * configurations. Alternatively running against
+ * multiple objects with --objects <n> will select a
+ * random configuration for all but the first object.
+ */
+
+namespace po = boost::program_options;
+
+namespace ceph
+{
+ namespace io_sequence::tester
+ {
+ // Choices for min and max object size
+ inline constexpr size_t objectSizeSize = 10;
+ inline constexpr std::array<std::pair<int,int>,objectSizeSize>
+ objectSizeChoices = {{
+ {1,32}, // Default - best for boundary checking
+ {12,14},
+ {28,30},
+ {36,38},
+ {42,44},
+ {52,54},
+ {66,68},
+ {72,74},
+ {83,83},
+ {97,97}
+ }};
+
+ // Choices for block size
+ inline constexpr int blockSizeSize = 5;
+ inline constexpr std::array<uint64_t, blockSizeSize> blockSizeChoices = {{
+ 2048, // Default - test boundaries for EC 4K chunk size
+ 512,
+ 3767,
+ 4096,
+ 32768
+ }};
+
+ // Choices for number of threads
+ inline constexpr int threadArraySize = 4;
+ inline constexpr std::array<int, threadArraySize> threadCountChoices = {{
+ 1, // Default
+ 2,
+ 4,
+ 8
+ }};
+
+ // Choices for EC k+m profile
+ inline constexpr int kmSize = 6;
+ inline constexpr std::array<std::pair<int,int>, kmSize> kmChoices = {{
+ {2,2}, // Default - reasonable coverage
+ {2,1},
+ {2,3},
+ {3,2},
+ {4,2},
+ {5,1}
+ }};
+
+ // Choices for EC chunk size
+ inline constexpr int chunkSizeSize = 3;
+ inline constexpr std::array<uint64_t, chunkSizeSize> chunkSizeChoices = {{
+ 4*1024,
+ 64*1024,
+ 256*1024
+ }};
+
+ // Choices for plugin
+ inline constexpr int pluginListSize = 2;
+ inline constexpr std::array<std::string_view,
+ pluginListSize> pluginChoices = {{
+ "jerasure",
+ "isa"
+ }};
+
+ inline constexpr std::array<std::pair<ceph::io_exerciser::Sequence,
+ ceph::io_exerciser::Sequence>,
+ 0> sequencePairs = {{}};
+
+ inline constexpr std::array<std::string, 0> poolChoices = {{}};
+
+ template <typename T, int N, const std::array<T, N>& Ts>
+ class ProgramOptionSelector
+ {
+ public:
+ ProgramOptionSelector(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm,
+ const std::string& option_name,
+ bool set_forced,
+ bool select_first
+ );
+ virtual ~ProgramOptionSelector() = default;
+ bool isForced();
+ virtual const T choose();
+
+ protected:
+ ceph::util::random_number_generator<int>& rng;
+ static constexpr std::array<T, N> choices = Ts;
+
+ std::optional<T> force_value;
+ std::optional<T> first_value;
+
+ std::string option_name;
+ };
+
+ class SelectObjectSize
+ : public ProgramOptionSelector<std::pair<int, int>,
+ io_sequence::tester::objectSizeSize,
+ io_sequence::tester::objectSizeChoices>
+ {
+ public:
+ SelectObjectSize(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm);
+ };
+
+ class SelectBlockSize
+ : public ProgramOptionSelector<uint64_t,
+ io_sequence::tester::blockSizeSize,
+ io_sequence::tester::blockSizeChoices>
+ {
+ public:
+ SelectBlockSize(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm);
+ };
+
+ class SelectNumThreads
+ : public ProgramOptionSelector<int,
+ io_sequence::tester::threadArraySize,
+ io_sequence::tester::threadCountChoices>
+ {
+ public:
+ SelectNumThreads(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm);
+ };
+
+ class SelectSeqRange
+ : public ProgramOptionSelector<std::pair<ceph::io_exerciser::Sequence,
+ ceph::io_exerciser::Sequence>,
+ 0, io_sequence::tester::sequencePairs>
+ {
+ public:
+ SelectSeqRange(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm);
+
+ const std::pair<ceph::io_exerciser::Sequence,
+ ceph::io_exerciser::Sequence> choose() override;
+ };
+
+ class SelectErasureKM
+ : public ProgramOptionSelector<std::pair<int,int>,
+ io_sequence::tester::kmSize,
+ io_sequence::tester::kmChoices>
+ {
+ public:
+ SelectErasureKM(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm);
+ };
+
+ class SelectErasurePlugin
+ : public ProgramOptionSelector<std::string_view,
+ io_sequence::tester::pluginListSize,
+ io_sequence::tester::pluginChoices>
+ {
+ public:
+ SelectErasurePlugin(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm);
+ };
+
+ class SelectErasureChunkSize
+ : public ProgramOptionSelector<uint64_t,
+ io_sequence::tester::chunkSizeSize,
+ io_sequence::tester::chunkSizeChoices>
+ {
+ public:
+ SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, po::variables_map vm);
+ };
+
+ class SelectECPool
+ : public ProgramOptionSelector<std::string,
+ 0,
+ io_sequence::tester::poolChoices>
+ {
+ public:
+ SelectECPool(ceph::util::random_number_generator<int>& rng,
+ po::variables_map vm,
+ librados::Rados& rados,
+ bool dry_run);
+ const std::string choose() override;
+
+ private:
+ void create_pool(librados::Rados& rados,
+ const std::string& pool_name,
+ const std::string& plugin,
+ uint64_t chunk_size,
+ int k, int m);
+
+ protected:
+ librados::Rados& rados;
+ bool dry_run;
+
+ SelectErasureKM skm;
+ SelectErasurePlugin spl;
+ SelectErasureChunkSize scs;
+ };
+
+ class TestObject
+ {
+ public:
+ TestObject( const std::string oid,
+ librados::Rados& rados,
+ boost::asio::io_context& asio,
+ ceph::io_sequence::tester::SelectBlockSize& sbs,
+ ceph::io_sequence::tester::SelectECPool& spl,
+ ceph::io_sequence::tester::SelectObjectSize& sos,
+ ceph::io_sequence::tester::SelectNumThreads& snt,
+ ceph::io_sequence::tester::SelectSeqRange& ssr,
+ ceph::util::random_number_generator<int>& rng,
+ ceph::mutex& lock,
+ ceph::condition_variable& cond,
+ bool dryrun,
+ bool verbose,
+ bool has_seqseed,
+ int seqseed);
+
+ int get_num_io();
+ bool readyForIo();
+ bool next();
+ bool finished();
+
+ protected:
+ std::unique_ptr<ceph::io_exerciser::Model> exerciser_model;
+ std::pair<int,int> obj_size_range;
+ std::pair<ceph::io_exerciser::Sequence,
+ ceph::io_exerciser::Sequence> seq_range;
+ ceph::io_exerciser::Sequence curseq;
+ std::unique_ptr<ceph::io_exerciser::IoSequence> seq;
+ std::unique_ptr<ceph::io_exerciser::IoOp> op;
+ bool done;
+ ceph::util::random_number_generator<int>& rng;
+ bool verbose;
+ bool has_seqseed;
+ int seqseed;
+ };
+ }
+}
\ No newline at end of file
projects / ceph.git / commitdiff