From: benhanokh Date: Tue, 12 May 2026 12:44:48 +0000 (+0300) Subject: rgw/dedup: add Admin OPS REST API for dedup commands X-Git-Tag: v21.0.1~71^2~1 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=df378ec523460b194827c587ac1b92060b339b71;p=ceph.git rgw/dedup: add Admin OPS REST API for dedup commands Tracker: https://ibm-ceph.atlassian.net/browse/ISCE-4600 Expose the existing radosgw-admin dedup commands (stats, estimate, exec, abort, pause, resume, throttle) as HTTP Admin OPS endpoints under /{admin}/dedup, following the same pattern used by ratelimit, usage, and other admin REST APIs. New files: - rgw_rest_dedup.h: RGWHandler_Dedup and RGWRESTMgr_Dedup - rgw_rest_dedup.cc: REST op classes calling the same cluster:: backend functions as radosgw-admin API summary: - GET /dedup?op=stats - collect and display dedup statistics - GET /dedup?op=throttle - display throttle settings - POST /dedup?op=estimate - start dedup estimate session - POST /dedup?op=exec - start full dedup (requires yes-i-really-mean-it) - POST /dedup?op=abort - abort active dedup session - POST /dedup?op=pause - pause active dedup session - POST /dedup?op=resume - resume paused dedup session - POST /dedup?op=throttle - set throttle limits Documentation added to doc/radosgw/adminops.rst with cross-reference from doc/radosgw/s3_objects_dedup.rst. Signed-off-by: benhanokh --- diff --git a/doc/radosgw/adminops.rst b/doc/radosgw/adminops.rst index 674b7b4a728..7c670265ba0 100644 --- a/doc/radosgw/adminops.rst +++ b/doc/radosgw/adminops.rst @@ -2935,6 +2935,180 @@ permission. :: +Dedup +===== + +The Admin Operations API can be used to manage RGW object deduplication. +See `Full RGW Object Dedup`_ for additional details on the dedup feature and +CLI commands. + +.. _Full RGW Object Dedup: ../s3_objects_dedup + +To view dedup status, the user must have ``dedup=read`` capability. To +control dedup operations, the user must have ``dedup=write`` capability. +See the `Admin Guide`_ for details. + +Get Dedup Stats +~~~~~~~~~~~~~~~ + +Collects and displays last dedup statistics. + +:caps: dedup=read + +Syntax +^^^^^^ + +:: + + GET /{admin}/dedup?op=stats HTTP/1.1 + Host: {fqdn} + + +Get Dedup Throttle +~~~~~~~~~~~~~~~~~~ + +Displays current dedup throttle settings. + +:caps: dedup=read + +Syntax +^^^^^^ + +:: + + GET /{admin}/dedup?op=throttle HTTP/1.1 + Host: {fqdn} + + +Start Dedup Estimate +~~~~~~~~~~~~~~~~~~~~ + +Starts a new dedup estimate session (aborting first any existing session). +No changes are made to the existing system. Only statistics will be +collected and reported. + +:caps: dedup=write + +Syntax +^^^^^^ + +:: + + POST /{admin}/dedup?op=estimate HTTP/1.1 + Host: {fqdn} + + +Start Dedup Exec +~~~~~~~~~~~~~~~~ + +Starts a new dedup session (aborting first any existing session). +Performs a full dedup, finding duplicated tail objects and removing them. + +.. warning:: This operation can lead to data loss and should not be used on + production data. + +:caps: dedup=write + +Syntax +^^^^^^ + +:: + + POST /{admin}/dedup?op=exec&yes-i-really-mean-it HTTP/1.1 + Host: {fqdn} + +Request Parameters +^^^^^^^^^^^^^^^^^^ + +``yes-i-really-mean-it`` + +:Description: Confirmation flag required to execute full dedup. +:Type: Boolean +:Required: Yes + + +Abort Dedup +~~~~~~~~~~~ + +Aborts an active dedup session, releasing all resources used by it. + +:caps: dedup=write + +Syntax +^^^^^^ + +:: + + POST /{admin}/dedup?op=abort HTTP/1.1 + Host: {fqdn} + + +Pause Dedup +~~~~~~~~~~~ + +Pauses an active dedup session (dedup resources are not released). + +:caps: dedup=write + +Syntax +^^^^^^ + +:: + + POST /{admin}/dedup?op=pause HTTP/1.1 + Host: {fqdn} + + +Resume Dedup +~~~~~~~~~~~~ + +Resumes a paused dedup session. + +:caps: dedup=write + +Syntax +^^^^^^ + +:: + + POST /{admin}/dedup?op=resume HTTP/1.1 + Host: {fqdn} + + +Set Dedup Throttle +~~~~~~~~~~~~~~~~~~ + +Specifies maximum allowed operations per second for a single RGW server +during dedup. ``0`` means unlimited. At least one of ``max-bucket-index-ops`` +or ``max-metadata-ops`` must be specified. + +:caps: dedup=write + +Syntax +^^^^^^ + +:: + + POST /{admin}/dedup?op=throttle<[&max-bucket-index-ops=][&max-metadata-ops=]> HTTP/1.1 + Host: {fqdn} + +Request Parameters +^^^^^^^^^^^^^^^^^^ + +``max-bucket-index-ops`` + +:Description: Maximum bucket index read requests per second per RGW during dedup. ``0`` means unlimited. +:Type: Integer +:Required: No (but at least one of ``max-bucket-index-ops`` or ``max-metadata-ops`` is required) + +``max-metadata-ops`` + +:Description: Maximum metadata requests per second per RGW during dedup. ``0`` means unlimited. +:Type: Integer +:Required: No (but at least one of ``max-bucket-index-ops`` or ``max-metadata-ops`` is required) + + + Standard Error Responses ======================== diff --git a/doc/radosgw/s3_objects_dedup.rst b/doc/radosgw/s3_objects_dedup.rst index 249f611e065..132a02319b7 100644 --- a/doc/radosgw/s3_objects_dedup.rst +++ b/doc/radosgw/s3_objects_dedup.rst @@ -5,6 +5,9 @@ Full RGW Object Dedup Full RGW object deduplication adds ``radosgw-admin`` commands to remove duplicated RGW tail objects and to collect and report dedup statistics. +These operations are also available through the `Admin Ops API <../radosgw/adminops/#dedup>`_ +under ``/{admin}/dedup``. + Admin Commands ============== diff --git a/src/rgw/CMakeLists.txt b/src/rgw/CMakeLists.txt index f445184d318..0e9d2035e70 100644 --- a/src/rgw/CMakeLists.txt +++ b/src/rgw/CMakeLists.txt @@ -96,6 +96,7 @@ set(librgw_common_srcs rgw_rest_client.cc rgw_rest_config.cc rgw_rest_conn.cc + rgw_rest_dedup.cc rgw_rest_metadata.cc rgw_rest_ratelimit.cc rgw_rest_role.cc @@ -428,6 +429,7 @@ set(rgw_a_srcs rgw_realm_reloader.cc rgw_rest_config.cc rgw_rest_info.cc + rgw_rest_dedup.cc rgw_rest_metadata.cc rgw_rest_ratelimit.cc rgw_rest_sts.cc diff --git a/src/rgw/driver/rados/rgw_dedup_cluster.cc b/src/rgw/driver/rados/rgw_dedup_cluster.cc index d3c40036b83..92972ebab7c 100644 --- a/src/rgw/driver/rados/rgw_dedup_cluster.cc +++ b/src/rgw/driver/rados/rgw_dedup_cluster.cc @@ -1212,12 +1212,31 @@ namespace rgw::dedup { } } + static void report_throttle_state(const struct rgw::dedup::control_t &ctl, + Formatter *fmt) + { + Formatter::ObjectSection section{*fmt, "throttle"}; + fmt->dump_bool("bucket_index_throttle_enabled", + !ctl.bucket_index_throttle.is_disabled()); + if (!ctl.bucket_index_throttle.is_disabled()) { + fmt->dump_unsigned("bucket_index_throttle", + ctl.bucket_index_throttle.get_max_calls_per_second()); + } + fmt->dump_bool("metadata_throttle_enabled", + !ctl.metadata_access_throttle.is_disabled()); + if (!ctl.metadata_access_throttle.is_disabled()) { + fmt->dump_unsigned("metadata_throttle", + ctl.metadata_access_throttle.get_max_calls_per_second()); + } + } + //--------------------------------------------------------------------------- // command-line called from radosgw-admin.cc int cluster::dedup_control_bl(rgw::sal::RadosStore *store, const DoutPrefixProvider *dpp, urgent_msg_t urgent_msg, - bufferlist urgent_msg_bl) + bufferlist urgent_msg_bl, + Formatter *fmt) { librados::IoCtx ctl_ioctx; int ret = get_control_ioctx(store, dpp, ctl_ioctx); @@ -1244,6 +1263,7 @@ namespace rgw::dedup { return -EAGAIN; } + bool throttle_reported = false; for (auto& ack : acks) { try { ldpp_dout(dpp, 20) << __func__ << "::ACK: notifier_id=" << ack.notifier_id @@ -1253,8 +1273,14 @@ namespace rgw::dedup { struct rgw::dedup::control_t ctl; decode(ctl, iter); ldpp_dout(dpp, 10) << __func__ << "::++ACK::ctl=" << ctl << "::ret=" << ret << dendl; - if (urgent_msg == URGENT_MSG_THROTTLE) { - report_throttle_state(ctl); + if (urgent_msg == URGENT_MSG_THROTTLE && !throttle_reported) { + // report only once + if (fmt) { + report_throttle_state(ctl, fmt); + } else { + report_throttle_state(ctl); + } + throttle_reported = true; } } catch (buffer::error& err) { ldpp_dout(dpp, 1) << __func__ << "::failed decoding notify acks" << dendl; diff --git a/src/rgw/driver/rados/rgw_dedup_cluster.h b/src/rgw/driver/rados/rgw_dedup_cluster.h index da64b1fd90f..446a39f221a 100644 --- a/src/rgw/driver/rados/rgw_dedup_cluster.h +++ b/src/rgw/driver/rados/rgw_dedup_cluster.h @@ -96,7 +96,8 @@ namespace rgw::dedup { static int dedup_control_bl(rgw::sal::RadosStore *store, const DoutPrefixProvider *dpp, urgent_msg_t urgent_msg, - bufferlist urgent_msg_bl); + bufferlist urgent_msg_bl, + Formatter *fmt = nullptr); static int dedup_control(rgw::sal::RadosStore *store, const DoutPrefixProvider *dpp, urgent_msg_t urgent_msg); diff --git a/src/rgw/driver/rados/rgw_sal_rados.cc b/src/rgw/driver/rados/rgw_sal_rados.cc index 0cfddd55987..34576223d8a 100644 --- a/src/rgw/driver/rados/rgw_sal_rados.cc +++ b/src/rgw/driver/rados/rgw_sal_rados.cc @@ -54,6 +54,7 @@ #include "rgw_rest_conn.h" #include "rgw_rest_log.h" #include "rgw_rest_metadata.h" +#include "rgw_rest_dedup.h" #include "rgw_rest_ratelimit.h" #include "rgw_rest_realm.h" #include "rgw_rest_user.h" @@ -2577,6 +2578,7 @@ void RadosStore::register_admin_apis(RGWRESTMgr* mgr) mgr->register_resource("config", new RGWRESTMgr_Config); mgr->register_resource("realm", new RGWRESTMgr_Realm); mgr->register_resource("ratelimit", new RGWRESTMgr_Ratelimit); + mgr->register_resource("dedup", new RGWRESTMgr_Dedup); } std::unique_ptr RadosStore::get_lua_manager(const std::string& luarocks_path) diff --git a/src/rgw/rgw_common.cc b/src/rgw/rgw_common.cc index 54a272df781..a42ce5384e7 100644 --- a/src/rgw/rgw_common.cc +++ b/src/rgw/rgw_common.cc @@ -2166,7 +2166,8 @@ bool RGWUserCaps::is_valid_cap_type(const string& tp) "oidc-provider", "user-info-without-keys", "ratelimit", - "accounts"}; + "accounts", + "dedup"}; for (unsigned int i = 0; i < sizeof(cap_type) / sizeof(char *); ++i) { if (tp.compare(cap_type[i]) == 0) { diff --git a/src/rgw/rgw_rest_dedup.cc b/src/rgw/rgw_rest_dedup.cc new file mode 100644 index 00000000000..fe25b132ceb --- /dev/null +++ b/src/rgw/rgw_rest_dedup.cc @@ -0,0 +1,232 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:nil -*- +// vim: ts=8 sw=2 sts=2 expandtab ft=cpp + +#include "rgw_rest_dedup.h" +#include "rgw_op.h" +#include "rgw_sal.h" +#include "rgw_sal_rados.h" +#include "rgw_dedup_cluster.h" +#include "rgw_dedup_utils.h" + +#define dout_subsys ceph_subsys_rgw + +using namespace std; + +static rgw::sal::RadosStore* get_rados_store(rgw::sal::Driver* driver) +{ + return dynamic_cast(driver); +} + +// GET /admin/dedup?op=stats +class RGWOp_Dedup_Stats : public RGWRESTOp { +public: + int check_caps(const RGWUserCaps& caps) override { + return caps.check_cap("dedup", RGW_CAP_READ); + } + + void execute(optional_yield y) override { + auto store = get_rados_store(driver); + if (!store) { + op_ret = -EPERM; + return; + } + + op_ret = rgw::dedup::cluster::collect_all_shard_stats( + store, s->formatter, this); + } + + const char* name() const override { return "get_dedup_stats"; } +}; + +// GET /admin/dedup?op=throttle +class RGWOp_Dedup_Throttle_Get : public RGWRESTOp { +public: + int check_caps(const RGWUserCaps& caps) override { + return caps.check_cap("dedup", RGW_CAP_READ); + } + + void execute(optional_yield y) override { + using namespace rgw::dedup; + auto store = get_rados_store(driver); + if (!store) { + op_ret = -EPERM; + return; + } + + bufferlist urgent_msg_bl; + urgent_msg_t urgent_msg = URGENT_MSG_THROTTLE; + ceph::encode(urgent_msg, urgent_msg_bl); + throttle_msg_t throttle_msg; + encode(throttle_msg, urgent_msg_bl); + + op_ret = cluster::dedup_control_bl(store, this, urgent_msg, urgent_msg_bl, + s->formatter); + } + + const char* name() const override { return "get_dedup_throttle"; } +}; + +// POST /admin/dedup?op=estimate|exec +class RGWOp_Dedup_Scan : public RGWRESTOp { + rgw::dedup::dedup_req_type_t dedup_type; +public: + RGWOp_Dedup_Scan(rgw::dedup::dedup_req_type_t dedup_type) + : dedup_type(dedup_type) {} + + int check_caps(const RGWUserCaps& caps) override { + return caps.check_cap("dedup", RGW_CAP_WRITE); + } + + void execute(optional_yield y) override { + auto store = get_rados_store(driver); + if (!store) { + op_ret = -EPERM; + return; + } + + if (dedup_type == rgw::dedup::dedup_req_type_t::DEDUP_TYPE_EXEC) { + bool confirmed = false; + RESTArgs::get_bool(s, "yes-i-really-mean-it", false, &confirmed); + if (!confirmed) { + op_ret = -EINVAL; + return; + } +#ifndef FULL_DEDUP_SUPPORT + op_ret = -EPERM; + return; +#endif + } + + op_ret = rgw::dedup::cluster::dedup_restart_scan(store, dedup_type, this); + } + + const char* name() const override { + return dedup_type == rgw::dedup::dedup_req_type_t::DEDUP_TYPE_EXEC + ? "run dedup_exec" : "run dedup_estimate"; + } +}; + +// POST /admin/dedup?op=abort|pause|resume +class RGWOp_Dedup_Control : public RGWRESTOp { + rgw::dedup::urgent_msg_t msg; +public: + RGWOp_Dedup_Control(rgw::dedup::urgent_msg_t msg) : msg(msg) {} + + int check_caps(const RGWUserCaps& caps) override { + return caps.check_cap("dedup", RGW_CAP_WRITE); + } + + void execute(optional_yield y) override { + auto store = get_rados_store(driver); + if (!store) { + op_ret = -EPERM; + return; + } + + op_ret = rgw::dedup::cluster::dedup_control(store, this, msg); + } + + const char* name() const override { + switch (msg) { + case rgw::dedup::URGENT_MSG_ABORT: return "abort dedup"; + case rgw::dedup::URGENT_MSG_PASUE: return "pause dedup"; + case rgw::dedup::URGENT_MSG_RESUME: return "resume dedup"; + default: return "dedup control"; + } + } +}; + +// POST /admin/dedup?op=throttle&max-bucket-index-ops=N&max-metadata-ops=M +class RGWOp_Dedup_Throttle_Set : public RGWRESTOp { +public: + int check_caps(const RGWUserCaps& caps) override { + return caps.check_cap("dedup", RGW_CAP_WRITE); + } + + void execute(optional_yield y) override { + using namespace rgw::dedup; + auto store = get_rados_store(driver); + if (!store) { + op_ret = -EPERM; + return; + } + + bufferlist urgent_msg_bl; + urgent_msg_t urgent_msg = URGENT_MSG_THROTTLE; + ceph::encode(urgent_msg, urgent_msg_bl); + throttle_msg_t throttle_msg; + string err; + + auto parse_limit = [&](const char* param, op_type_t op_type) { + string val; + RESTArgs::get_string(s, param, "", &val); + if (!val.empty()) { + int64_t limit = strict_strtoll(val.c_str(), 10, &err); + if (!err.empty()) { + return -EINVAL; + } + throttle_msg.vec.push_back({ .op_type = op_type, + .limit = (uint32_t)limit }); + } + return 0; + }; + + op_ret = parse_limit("max-bucket-index-ops", BUCKET_INDEX_OP); + if (op_ret) return; + op_ret = parse_limit("max-metadata-ops", METADATA_ACCESS_OP); + if (op_ret) return; + + if (throttle_msg.vec.empty()) { + op_ret = -EINVAL; + return; + } + + encode(throttle_msg, urgent_msg_bl); + op_ret = cluster::dedup_control_bl(store, this, urgent_msg, urgent_msg_bl, + s->formatter); + } + + const char* name() const override { return "set_dedup_throttle"; } +}; + +RGWOp *RGWHandler_Dedup::op_get() +{ + string op; + RESTArgs::get_string(s, "op", "", &op); + + if (op == "stats") { + return new RGWOp_Dedup_Stats; + } + if (op == "throttle") { + return new RGWOp_Dedup_Throttle_Get; + } + + return nullptr; +} + +RGWOp *RGWHandler_Dedup::op_post() +{ + string op; + RESTArgs::get_string(s, "op", "", &op); + + if (op == "estimate") { + return new RGWOp_Dedup_Scan(rgw::dedup::dedup_req_type_t::DEDUP_TYPE_ESTIMATE); + } + if (op == "exec") { + return new RGWOp_Dedup_Scan(rgw::dedup::dedup_req_type_t::DEDUP_TYPE_EXEC); + } + if (op == "abort") { + return new RGWOp_Dedup_Control(rgw::dedup::URGENT_MSG_ABORT); + } + if (op == "pause") { + return new RGWOp_Dedup_Control(rgw::dedup::URGENT_MSG_PASUE); + } + if (op == "resume") { + return new RGWOp_Dedup_Control(rgw::dedup::URGENT_MSG_RESUME); + } + if (op == "throttle") { + return new RGWOp_Dedup_Throttle_Set; + } + + return nullptr; +} diff --git a/src/rgw/rgw_rest_dedup.h b/src/rgw/rgw_rest_dedup.h new file mode 100644 index 00000000000..e55567672a9 --- /dev/null +++ b/src/rgw/rgw_rest_dedup.h @@ -0,0 +1,33 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:nil -*- +// vim: ts=8 sw=2 sts=2 expandtab ft=cpp + +#pragma once + +#include "rgw_rest.h" +#include "rgw_rest_s3.h" + +class RGWHandler_Dedup : public RGWHandler_Auth_S3 { +protected: + RGWOp *op_get() override; + RGWOp *op_post() override; +public: + using RGWHandler_Auth_S3::RGWHandler_Auth_S3; + ~RGWHandler_Dedup() override = default; + + int read_permissions(RGWOp*, optional_yield) override { + return 0; + } +}; + +class RGWRESTMgr_Dedup : public RGWRESTMgr { +public: + RGWRESTMgr_Dedup() = default; + ~RGWRESTMgr_Dedup() override = default; + + RGWHandler_REST *get_handler(rgw::sal::Driver* driver, + req_state*, + const rgw::auth::StrategyRegistry& auth_registry, + const std::string&) override { + return new RGWHandler_Dedup(auth_registry); + } +}; diff --git a/src/test/rgw/dedup/test_dedup.py b/src/test/rgw/dedup/test_dedup.py index b09d5f58f15..6728a47e92a 100644 --- a/src/test/rgw/dedup/test_dedup.py +++ b/src/test/rgw/dedup/test_dedup.py @@ -4,7 +4,6 @@ import random import math import time import subprocess -import urllib.request import hashlib from multiprocessing import Process import filecmp @@ -17,6 +16,12 @@ from collections import namedtuple import boto3 from boto3.s3.transfer import TransferConfig from dataclasses import dataclass +import urllib.parse +import urllib.request +import urllib.error +from botocore.auth import HmacV1Auth +from botocore.credentials import Credentials +from botocore.awsrequest import AWSRequest from . import( configfile, @@ -90,6 +95,84 @@ def rados(args, **kwargs): cmd = [test_path + 'test-rgw-call.sh', 'call_rgw_rados', 'noname'] + args return bash(cmd, **kwargs) +#------------------------------------------------------------------ +# Rest API helper functions +#------------------------------------------------------------------ + +_dedup_caps_granted = False +#------------------------------------------------------------------------ +def _ensure_dedup_caps(): + """Grant 'dedup=*' caps to the test user (once) so REST calls pass + the RGWUserCaps check.""" + global _dedup_caps_granted + if _dedup_caps_granted: + return + access_key = get_access_key() + result = admin(['user', 'info', '--access-key', access_key]) + assert result[1] == 0, "failed to look up test user" + info = json.loads(result[0]) + uid = info['user_id'] + tenant = info.get('tenant', '') + if tenant: + uid = tenant + '$' + uid + result = admin(['caps', 'add', '--uid', uid, '--caps', 'dedup=*']) + assert result[1] == 0, "failed to add dedup caps" + log.debug("granted dedup=* caps to uid=%s", uid) + _dedup_caps_granted = True + +#------------------------------------------------------------------------- +def _admin_rest_url(): + hostname = get_config_host() + port_no = get_config_port() + scheme = 'https' if port_no in (443, 8443) else 'http' + return f'{scheme}://{hostname}:{port_no}/admin/dedup' + +#-------------------------------------------------------------------------- +def admin_rest(method, params): + """Send a signed GET/POST to /admin/dedup and return + (body, returncode) matching the tuple that ``admin()`` returns.""" + _ensure_dedup_caps() + url = f'{_admin_rest_url()}?{urllib.parse.urlencode(params, doseq=True)}' + + creds = Credentials(get_access_key(), get_secret_key()) + aws_req = AWSRequest(method=method, url=url) + HmacV1Auth(creds).add_auth(aws_req) + + req = urllib.request.Request(url, method=method, + headers=dict(aws_req.headers)) + try: + resp = urllib.request.urlopen(req, timeout=120) + body = resp.read().decode('utf-8') + return (body, 0) + except urllib.error.HTTPError as e: + body = e.read().decode('utf-8', errors='replace') + log.error("admin_rest %s [params=%s] HTTP %d: %s", + method, params, e.code, body) + return (body, 1) + +#-------------------------------------------------------------- +def dedup_admin(subcmd, **kwargs): + """Invoke a dedup admin operation via REST API.""" + is_read = subcmd in ('stats',) or (subcmd == 'throttle' and kwargs.pop('stat', False)) + method = 'GET' if is_read else 'POST' + params = {'op': subcmd} + if subcmd == 'exec': + params['yes-i-really-mean-it'] = '' + for k, v in kwargs.items(): + params[k.replace('_', '-')] = str(v) + log.debug("dedup_admin [REST %s]: params=%s", method, params) + return admin_rest(method, params) + +#-------------------------------------------------------------- +def dedup_admin_cli(subcmd, *args): + """Invoke a dedup admin operation via radosgw-admin CLI.""" + cli_args = ['dedup', subcmd] + if subcmd == 'exec': + cli_args.append('--yes-i-really-mean-it') + cli_args += list(args) + log.debug("dedup_admin_cli: args=%s", cli_args) + return admin(cli_args) + #----------------------------------------------- def gen_bucket_name(): global num_buckets @@ -1371,7 +1454,7 @@ def read_dedup_stats(dry_run): dedup_ratio_estimate=Dedup_Ratio() dedup_ratio_actual=Dedup_Ratio() - result = admin(['dedup', 'stats']) + result = dedup_admin('stats') assert result[1] == 0 jstats=json.loads(result[0]) @@ -1414,8 +1497,7 @@ def read_dedup_stats(dry_run): #------------------------------------------------------------------------------- def set_bucket_index_throttling(limit): - cmd = ['dedup', 'throttle', '--max-bucket-index-ops', str(limit)] - result = admin(cmd) + result = dedup_admin('throttle', max_bucket_index_ops=limit) assert result[1] == 0 log.debug(result[0]) @@ -1427,10 +1509,10 @@ def exec_dedup_internal(expected_dedup_stats, dry_run, max_dedup_time): log.debug("sending exec_dedup request: dry_run=%d", dry_run) if dry_run: - result = admin(['dedup', 'estimate']) + result = dedup_admin('estimate') reset_full_dedup_stats(expected_dedup_stats) else: - result = admin(['dedup', 'exec', '--yes-i-really-mean-it']) + result = dedup_admin('exec') assert result[1] == 0 log.debug("wait for dedup to complete") @@ -1657,11 +1739,11 @@ def check_full_dedup_state(): global full_dedup_state_was_checked global full_dedup_state_disabled log.debug("check_full_dedup_state:: sending FULL Dedup request") - result = admin(['dedup', 'exec', '--yes-i-really-mean-it']) + result = dedup_admin('exec') if result[1] == 0: log.debug("full dedup is enabled!") full_dedup_state_disabled = False - result = admin(['dedup', 'abort']) + result = dedup_admin('abort') assert result[1] == 0 else: log.debug("full dedup is disabled, skip all full dedup tests") @@ -1987,8 +2069,8 @@ def corrupt_etag(key, corruption, expected_dedup_stats): names=result[0].split() for name in names: log.debug("name=%s", name) - if key in name: - log.debug("key=%s is a substring of name=%s", key, name); + if name.endswith(key): + log.debug("key=%s is a suffix of name=%s", key, name); rados_name = name break; @@ -3488,7 +3570,8 @@ def test_dedup_dry_large_scale_with_tenants(): try: threads_simple_dedup_with_tenants(files, conns, bucket_names, config, True) except Exception: - log.warning("test_dedup_dry_large_scale: failed!!") + log.warning("test_dedup_dry_large_scale_with_tenants: failed!!") + assert 0, "abort test_dedup_dry_large_scale_with_tenants " finally: # cleanup must be executed even after a failure cleanup_all_buckets(bucket_names, conns) @@ -3520,6 +3603,167 @@ def test_dedup_dry_large_scale(): cleanup(bucket_name, conn) +#------------------------------------------------------------------------------- +@pytest.mark.basic_test +def test_dedup_cli_operations(): + """Exercise all dedup CLI subcommands: estimate, stats, exec, pause, resume, + abort, throttle.""" + if full_dedup_is_disabled(): + return + + prepare_test() + bucket_name = gen_bucket_name() + conn = get_single_connection() + try: + files = [] + gen_files(files, 16*KB, 3) + bucket = conn.create_bucket(Bucket=bucket_name) + indices = [0] * len(files) + upload_objects(bucket_name, files, indices, conn, default_config, True) + + log.info("Test radosgw-admin dedup estimate"); + result = dedup_admin_cli('estimate') + assert result[1] == 0, "CLI estimate failed" + + dedup_time = 0 + dedup_timeout = 3 + max_dedup_time = 30 + while True: + assert dedup_time < max_dedup_time + time.sleep(dedup_timeout) + dedup_time += dedup_timeout + ret = read_dedup_stats(dry_run=True) + if ret[0]: + break + + + log.info("Test radosgw-admin dedup stats"); + result = dedup_admin_cli('stats') + assert result[1] == 0, "CLI stats after estimate failed" + + log.info("Test radosgw-admin dedup exec"); + result = dedup_admin_cli('exec') + assert result[1] == 0, "CLI exec failed" + + log.info("Test radosgw-admin dedup throttle"); + result = dedup_admin_cli('throttle', '--max-bucket-index-ops', '100') + assert result[1] == 0, "CLI throttle failed" + + log.info("Test radosgw-admin dedup throttle stat"); + result = dedup_admin_cli('throttle', '--stat') + assert result[1] == 0, "CLI throttle failed" + + log.info("Test radosgw-admin dedup pause"); + result = dedup_admin_cli('pause') + assert result[1] == 0, "CLI pause failed" + + log.info("Test radosgw-admin dedup resume"); + result = dedup_admin_cli('resume') + assert result[1] == 0, "CLI resume failed" + + log.info("Test radosgw-admin dedup abort"); + result = dedup_admin_cli('abort') + assert result[1] == 0, "CLI abort failed" + + log.info("Test radosgw-admin dedup stats"); + result = dedup_admin_cli('stats') + assert result[1] == 0, "CLI stats after abort failed" + finally: + cleanup(bucket_name, conn) + + +#------------------------------------------------------------------------------- +@pytest.mark.basic_test +def test_dedup_rest_pause_resume(): + """Exercise pause and resume via REST API.""" + if full_dedup_is_disabled(): + return + + prepare_test() + bucket_name = gen_bucket_name() + conn = get_single_connection() + try: + files = [] + gen_files(files, 16*KB, 3) + bucket = conn.create_bucket(Bucket=bucket_name) + indices = [0] * len(files) + upload_objects(bucket_name, files, indices, conn, default_config, True) + + result = dedup_admin('exec') + assert result[1] == 0, "REST exec failed" + + result = dedup_admin('pause') + assert result[1] == 0, "REST pause failed" + + result = dedup_admin('throttle', stat=True) + assert result[1] == 0, "REST throttle stat failed" + + result = dedup_admin('resume') + assert result[1] == 0, "REST resume failed" + + result = dedup_admin('abort') + assert result[1] == 0, "REST abort failed" + + result = dedup_admin('stats') + assert result[1] == 0, "REST stats after pause/resume failed" + finally: + cleanup(bucket_name, conn) + + +#------------------------------------------------------------------------------- +@pytest.mark.basic_test +def test_dedup_rest_throttle(): + """Verify REST throttle set/get preserves unmodified values.""" + def parse_throttle(result): + raw = json.loads(result[0]) if result[0].strip() else {} + return raw.get('throttle', raw) + + result = dedup_admin('throttle', stat=True) + assert result[1] == 0, "REST throttle initial stat failed" + orig = parse_throttle(result) + orig_bucket = orig.get('bucket_index_throttle', 0) + orig_metadata = orig.get('metadata_throttle', 0) + log.info("throttle initial: bucket_index=%s, metadata=%s", + orig_bucket, orig_metadata) + + new_bucket=orig_bucket+17 + new_metadata=orig_metadata+17 + result = dedup_admin('throttle', max_bucket_index_ops=new_bucket) + assert result[1] == 0, "REST throttle set bucket-index failed" + body = parse_throttle(result) + log.info("throttle after set bucket_index=%d:", + body.get('bucket_index_throttle', 0)) + assert body.get('bucket_index_throttle') == new_bucket + assert body.get('metadata_throttle', 0) == orig_metadata + + result = dedup_admin('throttle', max_metadata_ops=new_metadata) + assert result[1] == 0, "REST throttle set metadata failed" + body = parse_throttle(result) + log.info("throttle after set metadata=%d", + body.get('metadata_throttle', 0)) + assert body.get('bucket_index_throttle') == new_bucket + assert body.get('metadata_throttle') == new_metadata + + result = dedup_admin('throttle', stat=True) + assert result[1] == 0, "REST throttle final stat failed" + body = parse_throttle(result) + assert body.get('bucket_index_throttle') == new_bucket + assert body.get('metadata_throttle') == new_metadata + + kwargs = {} + kwargs['max_bucket_index_ops'] = orig_bucket + kwargs['max_metadata_ops'] = orig_metadata + result = dedup_admin('throttle', **kwargs) + assert result[1] == 0, "REST throttle restore failed" + body = parse_throttle(result) + log.info("throttle after restore: bucket_index_throttle=%d, metadata=%d", + body.get('bucket_index_throttle', 0), + body.get('metadata_throttle', 0)) + + log.info("throttle restored to: bucket_index=%s, metadata=%s", + orig_bucket, orig_metadata) + + #------------------------------------------------------------------------------- @pytest.mark.basic_test def test_cleanup(): @@ -3756,7 +4000,7 @@ def test_dedup_filter_storage_class_list_parsing(): #------------------------------------------------------------------------------- def read_filter_skip_stats(): """Read ingress_skip_filtered_bucket/storage_class from dedup stats JSON.""" - result = admin(['dedup', 'stats']) + result = dedup_admin('stats') assert result[1] == 0 jstats = json.loads(result[0]) worker_stats = jstats['worker_stats']