1)Add functionality to restore cloud-transitioned objects on demand.
Current commit has below -
* Given <bucket,object>, fetch the object from the cloud endpoint.
* if days provided and > 0, the restore is marked temporary with expiry date.
* Without <days>, it is marked as permanent restore.
2)Use ObjectExpirer/delete_at attr to delete temp objects
For temporarily restored objects, set delete_at attr to the expiration time.
This will add those objects to ObjectExpirer list. Use LC worker thread to
scan that list and delete expired objects. By delete here, it means to delete
restored object data and reset HEAD object as Cloud-transitioned object as it
was before restore.
In addition below changes are done -
* If temporary, object is still marked RGWObj::CloudTiered and mtime is set same as
transition time.
* If permanent, object is marked RGWObj::Main and mtime is set to restore time (now()).
* rgw_restore_debug_interval option added to set configure restore Days (similar to rgw_lc_debug_interval)
There is an issue with ObjectExpirer code where in if an object is added
to ObjectExpirer list and is re-written, it is not deleted from the expirer list
and hence the new object may get deleted. Fixed the same and also addressed
minor review comments.
3)Design doc added
4) ObjCategory should be set to CloudTiered only for cloud-transitioned
objects and temporarily restored objects. Permanent copies are to be
treated as regular objects.
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
services:
- rgw
with_legacy: true
+- name: rgw_restore_debug_interval
+ type: int
+ level: dev
+ desc: The number of seconds that simulate one "day" in order to debug RGW CloudRestore.
+ Do *not* modify for a production cluster.
+ long_desc: For debugging RGW Cloud Restore, the number of seconds that are equivalent to
+ one simulated "day". Values less than 1 are ignored and do not change Restore behavior.
+ For example, during debugging if one wanted every 10 minutes to be equivalent to one day,
+ then this would be set to 600, the number of seconds in 10 minutes.
+ default: -1
+ services:
+ - rgw
+ with_legacy: true
- name: rgw_mp_lock_max_time
type: int
level: advanced
--- /dev/null
+# cloud-restore
+
+## Introduction
+
+[`cloud-transition`](https://docs.ceph.com/en/latest/radosgw/cloud-transition) feature enables data transition to a remote cloud service as part of Lifecycle Configuration via Storage Classes. However the transition is unidirectional; data cannot be transitioned back from the remote zone.
+
+The `cloud-restore` feature enables restoration of those transitioned objects from the remote cloud S3 endpoints back into RGW.
+
+The objects can be restored either by using S3 `restore-object` CLI or via `read-through`. The restored copies can be either temporary or permanent.
+
+## S3 restore-object CLI
+The goal here is to implement minimal functionality of [`S3RestoreObject`](https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html) API so that users can restore the cloud transitioned objects.
+
+```sh
+aws s3api restore-object \
+ --bucket <value> \
+ --key <value> ( can be object name or * for Bulk restore) \
+ [--version-id <value>] \
+ --restore-request (structure) {
+ // for temporary restore
+ { "Days": integer, }
+ // if Days not provided, it will be considered as permanent copy
+ }
+```
+This CLI may be extended in future to include custom parameters (like target-bucket/storage-class etc) specific to RGW.
+
+
+## read-through
+As per the cloud-transition feature functionality, the cloud-transitioned objects cannot be read. `GET` on those objects fails with ‘InvalidObjectState’ error.
+
+But using this restore feature, transitioned objects can be restored and read. New tier-config options `allow_read_through` and `read_through_restore_days` are added for the same. Only when `allow_read_through` is enabled, `GET` on the transitioned objects will restore the objects from the S3 endpoint.
+
+Note: The object copy restored via `readthrough` is temporary and is retained only for the duration of `read_through_restore_days`.
+
+## Design
+
+* Similar to cloud-transition feature, this feature currently works for **only s3 compatible cloud endpoint**.
+* This feature works for only **cloud-transitioned objects**. In order to validate this, `retain_head_object` option should be set to true so that the object’s `HEAD` object can be verified before restoring the object.
+
+* **Request flow:**
+ * Once the `HEAD` object is verified, its cloudtier storage class config details are fetched.
+Note: Incase the cloudtier storage-class is deleted/updated, the object may not be restored.
+ * RestoreStatus for the `HEAD` object is marked `RestoreAlreadyInProgress`
+ * Object Restore is done asynchronously by issuing either S3 `GET` or S3 `RESTORE` request to the remote endpoint.
+ * Once the object is restored, RestoreStaus is updated as `CloudRestored` and RestoreType is set to either `Temporary` or `Permanent`.
+ * Incase the operation fails, RestoreStatus is marked as `RestoreFailed`.
+
+* **New attrs:** Below are the new attrs being added
+ * `user.rgw.restore-status`: <Restore operation Status>
+ * `user.rgw.restore-type`: <Type of Restore>
+ * `user.rgw.restored-at`: <Restoration Time>
+ * `user.rgw.restore-expiry-date`: <Expiration time incase of temporary copies>
+ * `user.rgw.cloudtier_storage_class`: <CloudTier storage class used in case of temporarily restored copies>
+ ```sh
+ enum RGWRestoreStatus : uint8_t {
+ None = 0,
+ RestoreAlreadyInProgress = 1,
+ CloudRestored = 2,
+ RestoreFailed = 3
+ };
+ enum class RGWRestoreType : uint8_t {
+ None = 0,
+ Temporary = 1,
+ Permanent = 2
+ };
+ ```
+
+* **Response:**
+* `S3 restore-object CLI` returns SUCCESS - either the 200 OK or 202 Accepted status code.
+ * If the object is not previously restored, then RGW returns 202 Accepted in the response.
+ * If the object is previously restored, RGW returns 200 OK in the response.
+ * Special errors:
+ Code: RestoreAlreadyInProgress ( Cause: Object restore is already in progress.)
+ Code: ObjectNotFound (if Object is not found in cloud endpoint)
+ Code: I/O error (for any other I/O errors during restore)
+* `GET request` continues to return an ‘InvalidObjectState’ error till the object is successfully restored.
+ * S3 head-object can be used to verify if the restore is still in progress.
+ * Once the object is restored, GET will return the object data.
+
+
+* **StorageClass**: By default, the objects are restored to `STANDARD` storage class. However, as per [AWS S3 Restore](https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html) the storage-class remains the same for restored objects. Hence for the temporary copies, the `x-amz-storage-class` returned contains original cloudtier storage-class.
+ * Note: A new tier-config option may be added to select the storage-class to restore the objects to.
+
+* **mtime**: If the restored object is temporary, object is still marked `RGWObj::CloudTiered` and mtime is not changed i.e, still set to transition time. But in case the object is permanent copy, it is marked `RGWObj::Main` and mtime is updated to the restore time (now()).
+
+* **Lifecycle**:
+ * `Temporary` copies are not subjected to any further transition to the cloud. However (as is the case with cloud-transitioned objects) they can be deleted via regular LC expiration rules or via external S3 Delete request.
+ * `Permanent` copies are treated as any regular objects and are subjected to any LC rules applicable.
+
+* **Replication**: The restored objects (both temporary and permanent) are also replicated like regular objects and will be deleted across the zones post expiration.
+
+* **VersionedObjects** : In case of versioning, if any object is cloud-transitioned, it would have been non-current. Post restore too, the same non-current object will be updated with the downloaded data and its HEAD object will be updated accordingly as the case with regular objects.
+
+* **Temporary Object Expiry**: This is done via Object Expirer
+ * When the object is restored as temporary, `user.rgw.expiry-date` is set accordingly and `delete_at` attr is also updated with the same value.
+ * This object is then added to the list used by `ObjectExpirer`.
+ * `LC` worker thread is used to scan through that list and post expiry, resets the objects back to cloud-transitioned state i.e,
+ * HEAD object with size=0
+ * new attrs removed
+ * `delete_at` reset
+ * Note: A new RGW option `rgw_restore_debug_interval` is added, which when set will be considered as `Days` value (similar to `rgw_lc_debug_interval`).
+
+* **FAILED Restore**: In case the restore operation fails,
+ * The HEAD object will be updated accordingly.. i.e, Storage-class is reset to the original cloud-tier storage class
+ * All the new attrs added will be removed , except for `user.rgw.restore-status` which will be updated as `RestoreFailed`
+
+* **Check Restore Progress**: Users can issue S3 `head-object` request to check if the restore is done or still in progress for any object.
+
+* **RGW down/restarts** - Since the restore operation is asynchronous, we need to keep track of the objects being restored. In case RGW is down/restarts, this data will be used to retrigger on-going restore requests or do appropriate cleanup for the failed requests.
+
+* **Compression** - If the placement-target to which the objects are being restored to has compression enabled, the data will be compressed accordingly (bug2294512)
+
+* **Encryption** - If the restored object is encrypted, the old sse-related xattrs/keys from the HEAD stub will be copied back into object metadata (bug2294512)
+
+* **Delete cloud object post restore** - Once the object is successfully restored, the object at the remote endpoint is still retained. However we could choose to delete it for permanent restored copies by adding new tier-config option.
+
+
+## Future work
+
+* **Bulk Restore**: In the case of BulkRestore, some of the objects may not be restored. User needs to manually cross-check the objects to check the objects restored or InProgress.
+
+* **Admin CLIs**: Admin debug commands will be provided to start, check the status and cancel the restore operations.
+
+* **Admin Ops**
+
+* **Restore Notifications**
+
return DAOS_NOT_IMPLEMENTED_LOG(dpp);
}
+int DaosObject::restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags)
+{
+ return DAOS_NOT_IMPLEMENTED_LOG(dpp);
+}
+
bool DaosObject::placement_rules_match(rgw_placement_rule& r1,
rgw_placement_rule& r2) {
/* XXX: support single default zone and zonegroup for now */
CephContext* cct, bool update_object,
const DoutPrefixProvider* dpp,
optional_yield y) override;
+ virtual int restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags) override;
virtual bool placement_rules_match(rgw_placement_rule& r1,
rgw_placement_rule& r2) override;
virtual int dump_obj_layout(const DoutPrefixProvider* dpp, optional_yield y,
return -ERR_NOT_IMPLEMENTED;
}
+int POSIXObject::restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags)
+{
+ return -ERR_NOT_IMPLEMENTED;
+}
+
bool POSIXObject::placement_rules_match(rgw_placement_rule& r1, rgw_placement_rule& r2)
{
return (r1 == r2);
bool update_object,
const DoutPrefixProvider* dpp,
optional_yield y) override;
+ virtual int restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags) override;
virtual bool placement_rules_match(rgw_placement_rule& r1, rgw_placement_rule& r2) override;
virtual int dump_obj_layout(const DoutPrefixProvider *dpp, optional_yield y, Formatter* f) override;
virtual int swift_versioning_restore(const ACLOwner& owner, const rgw_user& remote_user, bool& restored,
#include "rgw_common.h"
#include "rgw_rest.h"
#include "svc_zone.h"
+#include "rgw_rados.h"
#include <boost/algorithm/string/split.hpp>
#include <boost/algorithm/string.hpp>
}
}
-/* Read object or just head from remote endpoint. For now initializes only headers,
- * but can be extended to fetch etag, mtime etc if needed.
+struct generic_attr {
+ const char *http_header;
+ const char *rgw_attr;
+};
+
+/*
+ * mapping between http env fields and rgw object attrs
+ */
+static const struct generic_attr generic_attrs[] = {
+ { "CONTENT_TYPE", RGW_ATTR_CONTENT_TYPE },
+ { "HTTP_CONTENT_LANGUAGE", RGW_ATTR_CONTENT_LANG },
+ { "HTTP_EXPIRES", RGW_ATTR_EXPIRES },
+ { "HTTP_CACHE_CONTROL", RGW_ATTR_CACHE_CONTROL },
+ { "HTTP_CONTENT_DISPOSITION", RGW_ATTR_CONTENT_DISP },
+ { "HTTP_CONTENT_ENCODING", RGW_ATTR_CONTENT_ENC },
+ { "HTTP_X_ROBOTS_TAG", RGW_ATTR_X_ROBOTS_TAG },
+ { "ETAG", RGW_ATTR_ETAG },
+};
+
+/* Read object or just head from remote endpoint.
*/
-static int cloud_tier_get_object(RGWLCCloudTierCtx& tier_ctx, bool head,
- std::map<std::string, std::string>& headers) {
+int rgw_cloud_tier_get_object(RGWLCCloudTierCtx& tier_ctx, bool head,
+ std::map<std::string, std::string>& headers,
+ real_time* pset_mtime, std::string& etag,
+ uint64_t& accounted_size, rgw::sal::Attrs& attrs,
+ void* cb) {
RGWRESTConn::get_obj_params req_params;
std::string target_obj_name;
int ret = 0;
rgw_lc_obj_properties obj_properties(tier_ctx.o.meta.mtime, tier_ctx.o.meta.etag,
tier_ctx.o.versioned_epoch, tier_ctx.acl_mappings,
tier_ctx.target_storage_class);
- std::string etag;
RGWRESTStreamRWRequest *in_req;
rgw_bucket dest_bucket;
req_params.rgwx_stat = true;
req_params.sync_manifest = true;
req_params.skip_decrypt = true;
+ req_params.cb = (RGWHTTPStreamRWRequest::ReceiveCB *)cb;
- ret = tier_ctx.conn.get_obj(tier_ctx.dpp, dest_obj, req_params, true /* send */, &in_req);
- if (ret < 0) {
- ldpp_dout(tier_ctx.dpp, 0) << "ERROR: " << __func__ << "(): conn.get_obj() returned ret=" << ret << dendl;
- return ret;
+ ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): fetching object from cloud bucket:" << dest_bucket << ", object: " << target_obj_name << dendl;
+
+ static constexpr int NUM_ENPOINT_IOERROR_RETRIES = 20;
+ for (int tries = 0; tries < NUM_ENPOINT_IOERROR_RETRIES; tries++) {
+ ret = tier_ctx.conn.get_obj(tier_ctx.dpp, dest_obj, req_params, true /* send */, &in_req);
+ if (ret < 0) {
+ ldpp_dout(tier_ctx.dpp, 0) << "ERROR: " << __func__ << "(): conn.get_obj() returned ret=" << ret << dendl;
+ return ret;
+ }
+
+ /* fetch headers */
+ // accounted_size in complete_request() reads from RGWX_OBJECT_SIZE which is set
+ // only for internal ops/sync. So instead read from headers[CONTENT_LEN].
+ // Same goes for pattrs.
+ ret = tier_ctx.conn.complete_request(tier_ctx.dpp, in_req, &etag, pset_mtime, nullptr, nullptr, &headers, null_yield);
+ if (ret < 0) {
+ if (ret == -EIO && tries < NUM_ENPOINT_IOERROR_RETRIES - 1) {
+ ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): failed to fetch object from remote. retries=" << tries << dendl;
+ continue;
+ }
+ return ret;
+ }
+ break;
}
- /* fetch headers */
- ret = tier_ctx.conn.complete_request(tier_ctx.dpp, in_req, nullptr, nullptr, nullptr, nullptr, &headers, null_yield);
- if (ret < 0 && ret != -ENOENT) {
- ldpp_dout(tier_ctx.dpp, 20) << "ERROR: " << __func__ << "(): conn.complete_request() returned ret=" << ret << dendl;
- return ret;
+ static map<string, string> generic_attrs_map;
+ for (const auto& http2rgw : generic_attrs) {
+ generic_attrs_map[http2rgw.http_header] = http2rgw.rgw_attr;
}
- return 0;
+
+ for (auto header: headers) {
+ const char* name = header.first.c_str();
+ const string& val = header.second;
+ bufferlist bl;
+ bl.append(val.c_str(), val.size());
+
+ const auto aiter = generic_attrs_map.find(name);
+ if (aiter != std::end(generic_attrs_map)) {
+ ldpp_dout(tier_ctx.dpp, 20) << __func__ << " Received attrs aiter->first = " << aiter->first << ", aiter->second = " << aiter->second << ret << dendl;
+ attrs[aiter->second] = bl;
+ }
+
+ if (header.first == "CONTENT_LENGTH") {
+ accounted_size = atoi(val.c_str());
+ }
+ }
+
+ ldpp_dout(tier_ctx.dpp, 20) << __func__ << "(): Sucessfully fetched object from cloud bucket:" << dest_bucket << ", object: " << target_obj_name << dendl;
+ return ret;
}
static bool is_already_tiered(const DoutPrefixProvider *dpp,
static int cloud_tier_check_object(RGWLCCloudTierCtx& tier_ctx, bool& already_tiered) {
int ret;
std::map<std::string, std::string> headers;
+ std::string etag;
+ uint64_t accounted_size;
+ rgw::sal::Attrs attrs;
/* Fetch Head object */
- ret = cloud_tier_get_object(tier_ctx, true, headers);
+ ret = rgw_cloud_tier_get_object(tier_ctx, true, headers, nullptr, etag, accounted_size, attrs, nullptr);
if (ret < 0) {
ldpp_dout(tier_ctx.dpp, 0) << "ERROR: failed to fetch HEAD from cloud for obj=" << tier_ctx.obj << " , ret = " << ret << dendl;
/* Transition object to cloud endpoint */
int rgw_cloud_tier_transfer_object(RGWLCCloudTierCtx& tier_ctx, std::set<std::string>& cloud_targets);
+
+int rgw_cloud_tier_get_object(RGWLCCloudTierCtx& tier_ctx, bool head,
+ std::map<std::string, std::string>& headers,
+ real_time* pset_mtime, std::string& etag,
+ uint64_t& accounted_size, rgw::sal::Attrs& attrs,
+ void* cb);
}
rgw_obj_key key = hint.obj_key;
- if (key.instance.empty()) {
- key.instance = "null";
- }
std::unique_ptr<rgw::sal::Object> obj = bucket->get_object(key);
- obj->set_atomic();
- ret = obj->delete_object(dpp, null_yield, rgw::sal::FLAG_LOG_OP, nullptr, nullptr);
+ ret = static_cast<rgw::sal::RadosObject*>(obj.get())->handle_obj_expiry(dpp, null_yield);
return ret;
}
#include "rgw_cr_rest.h"
#include "rgw_datalog.h"
#include "rgw_putobj_processor.h"
+#include "rgw_lc_tier.h"
#include "cls/rgw/cls_rgw_ops.h"
#include "cls/rgw/cls_rgw_client.h"
return 0;
}
+int RGWRados::restore_obj_from_cloud(RGWLCCloudTierCtx& tier_ctx,
+ RGWObjectCtx& obj_ctx,
+ RGWBucketInfo& dest_bucket_info,
+ const rgw_obj& dest_obj,
+ rgw_placement_rule& dest_placement,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool log_op){
+
+ //XXX: read below from attrs .. check transition_obj()
+ ACLOwner owner;
+ rgw::sal::Attrs attrs;
+ const req_context rctx{dpp, y, nullptr};
+ int ret = 0;
+ bufferlist t, t_tier;
+ string tag;
+ append_rand_alpha(cct, tag, tag, 32);
+ auto aio = rgw::make_throttle(cct->_conf->rgw_put_obj_min_window_size, y);
+ using namespace rgw::putobj;
+ jspan_context no_trace{false, false};
+ rgw::putobj::AtomicObjectProcessor processor(aio.get(), this, dest_bucket_info, nullptr,
+ owner, obj_ctx, dest_obj, olh_epoch, tag, dpp, y, no_trace);
+
+ void (*progress_cb)(off_t, void *) = NULL;
+ void *progress_data = NULL;
+ bool cb_processed = false;
+ RGWFetchObjFilter *filter;
+ RGWFetchObjFilter_Default source_filter;
+ if (!filter) {
+ filter = &source_filter;
+ }
+ boost::optional<RGWPutObj_Compress> compressor;
+ CompressorRef plugin;
+ RGWRadosPutObj cb(dpp, cct, plugin, compressor, &processor, progress_cb, progress_data,
+ [&](map<string, bufferlist> obj_attrs) {
+ // XXX: do we need filter() like in fetch_remote_obj() cb
+ dest_placement.inherit_from(dest_bucket_info.placement_rule);
+ /* For now we always restore to STANDARD storage-class.
+ * Later we will add support to take restore-target-storage-class
+ * for permanent restore
+ */
+ dest_placement.storage_class = RGW_STORAGE_CLASS_STANDARD;
+
+ processor.set_tail_placement(dest_placement);
+
+ ret = processor.prepare(rctx.y);
+ if (ret < 0) {
+ return ret;
+ }
+ cb_processed = true;
+ return 0;
+ });
+
+ uint64_t accounted_size = 0;
+ string etag;
+ real_time set_mtime;
+ std::map<std::string, std::string> headers;
+ ldpp_dout(dpp, 20) << "Fetching from cloud, object:" << dest_obj << dendl;
+ ret = rgw_cloud_tier_get_object(tier_ctx, false, headers,
+ &set_mtime, etag, accounted_size,
+ attrs, &cb);
+
+ if (ret < 0) {
+ ldpp_dout(dpp, 20) << "Fetching from cloud failed, object:" << dest_obj << dendl;
+ return ret;
+ }
+
+ if (!cb_processed) {
+ ldpp_dout(dpp, 20) << "Callback not processed, object:" << dest_obj << dendl;
+ return -EIO;
+ }
+
+ ret = cb.flush();
+ if (ret < 0) {
+ return ret;
+ }
+
+ if (cb.get_data_len() != accounted_size) {
+ ret = -EIO;
+ ldpp_dout(dpp, -1) << "ERROR: object truncated during fetching, expected "
+ << accounted_size << " bytes but received " << cb.get_data_len() << dendl;
+ return ret;
+ }
+
+ {
+ bufferlist bl;
+ encode(rgw::sal::RGWRestoreStatus::CloudRestored, bl);
+ attrs[RGW_ATTR_RESTORE_STATUS] = std::move(bl);
+ }
+
+ ceph::real_time restore_time = real_clock::now();
+ {
+ char buf[32];
+ utime_t ut(restore_time);
+ snprintf(buf, sizeof(buf), "%lld.%09lld",
+ (long long)ut.sec(),
+ (long long)ut.nsec());
+ bufferlist bl;
+ bl.append(buf, 32);
+ encode(restore_time, bl);
+ attrs[RGW_ATTR_RESTORE_TIME] = std::move(bl);
+ }
+
+ real_time delete_at = real_time();
+ if (days) { //temp copy; do not change mtime and set expiry date
+ int expiry_days = days.value();
+ constexpr int32_t secs_in_a_day = 24 * 60 * 60;
+ ceph::real_time expiration_date ;
+
+ if (cct->_conf->rgw_restore_debug_interval > 0) {
+ expiration_date = restore_time + make_timespan(double(expiry_days)*cct->_conf->rgw_restore_debug_interval);
+ ldpp_dout(dpp, 20) << "Setting expiration time to rgw_restore_debug_interval: " << double(expiry_days)*cct->_conf->rgw_restore_debug_interval << ", days:" << expiry_days << dendl;
+ } else {
+ expiration_date = restore_time + make_timespan(double(expiry_days) * secs_in_a_day);
+ }
+ delete_at = expiration_date;
+
+ {
+ char buf[32];
+ utime_t ut(expiration_date);
+ snprintf(buf, sizeof(buf), "%lld.%09lld",
+ (long long)ut.sec(),
+ (long long)ut.nsec());
+ bufferlist bl;
+ bl.append(buf, 32);
+ encode(expiration_date, bl);
+ attrs[RGW_ATTR_RESTORE_EXPIRY_DATE] = std::move(bl);
+ }
+ {
+ bufferlist bl;
+ bl.clear();
+ using ceph::encode;
+ encode(rgw::sal::RGWRestoreType::Temporary, bl);
+ attrs[RGW_ATTR_RESTORE_TYPE] = std::move(bl);
+ ldpp_dout(dpp, 20) << "Temporary restore, object:" << dest_obj << dendl;
+ }
+ {
+ string sc = tier_ctx.storage_class;
+ bufferlist bl;
+ bl.append(sc.c_str(), sc.size());
+ attrs[RGW_ATTR_CLOUDTIER_STORAGE_CLASS] = std::move(bl);
+ ldpp_dout(dpp, 20) << "Setting RGW_ATTR_CLOUDTIER_STORAGE_CLASS: " << tier_ctx.storage_class << dendl;
+ }
+ //set same old mtime as that of transition time
+ set_mtime = mtime;
+
+ // set tier-config only for temp restored objects, as
+ // permanent copies will be treated as regular objects
+ {
+ t.append("cloud-s3");
+ encode(tier_config, t_tier);
+ attrs[RGW_ATTR_CLOUD_TIER_TYPE] = t;
+ attrs[RGW_ATTR_CLOUD_TIER_CONFIG] = t_tier;
+ }
+
+ } else { // permanent restore
+ {
+ bufferlist bl;
+ bl.clear();
+ using ceph::encode;
+ encode(rgw::sal::RGWRestoreType::Permanent, bl);
+ attrs[RGW_ATTR_RESTORE_TYPE] = std::move(bl);
+ ldpp_dout(dpp, 20) << "Permanent restore, object:" << dest_obj << dendl;
+ }
+ //set mtime to now()
+ set_mtime = real_clock::now();
+ }
+
+ {
+ string sc = dest_placement.get_storage_class(); //"STANDARD";
+ bufferlist bl;
+ bl.append(sc.c_str(), sc.size());
+ attrs[RGW_ATTR_STORAGE_CLASS] = std::move(bl);
+ }
+
+ // XXX: handle COMPLETE_RETRY like in fetch_remote_obj
+ bool canceled = false;
+ rgw_zone_set zone_set{};
+ ret = processor.complete(accounted_size, etag, &mtime, set_mtime,
+ attrs, rgw::cksum::no_cksum, delete_at , nullptr, nullptr, nullptr,
+ (rgw_zone_set *)&zone_set, &canceled, rctx, log_op ? rgw::sal::FLAG_LOG_OP : 0);
+ if (ret < 0) {
+ return ret;
+ }
+
+ // XXX: handle olh_epoch for versioned objects like in fetch_remote_obj
+ return ret;
+}
+
int RGWRados::check_bucket_empty(const DoutPrefixProvider *dpp, RGWBucketInfo& bucket_info, optional_yield y)
{
constexpr uint NUM_ENTRIES = 1000u;
}
uint64_t epoch = ioctx.get_last_version();
int64_t poolid = ioctx.get_id();
+
+ // Retain Object category as CloudTiered while restore is in
+ // progress or failed
+ RGWObjCategory category = RGWObjCategory::Main;
+ auto r_iter = attrs.find(RGW_ATTR_RESTORE_STATUS);
+ if (r_iter != attrs.end()) {
+ rgw::sal::RGWRestoreStatus st = rgw::sal::RGWRestoreStatus::None;
+ auto iter = r_iter->second.cbegin();
+
+ try {
+ using ceph::decode;
+ decode(st, iter);
+
+ if (st != rgw::sal::RGWRestoreStatus::CloudRestored) {
+ category = RGWObjCategory::CloudTiered;
+ }
+ } catch (buffer::error& err) {
+ }
+ }
r = index_op.complete(dpp, poolid, epoch, state->size, state->accounted_size,
mtime, etag, content_type, storage_class, owner,
- RGWObjCategory::Main, nullptr, y, nullptr, false, log_op);
+ category, nullptr, y, nullptr, false, log_op);
} else {
int ret = index_op.cancel(dpp, nullptr, y, log_op);
if (ret < 0) {
#include "rgw_tools.h"
struct D3nDataCache;
+struct RGWLCCloudTierCtx;
class RGWWatcher;
class ACLOwner;
const DoutPrefixProvider *dpp,
optional_yield y,
bool log_op = true);
+int restore_obj_from_cloud(RGWLCCloudTierCtx& tier_ctx,
+ RGWObjectCtx& obj_ctx,
+ RGWBucketInfo& dest_bucket_info,
+ const rgw_obj& dest_obj,
+ rgw_placement_rule& dest_placement,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool log_op = true);
int check_bucket_empty(const DoutPrefixProvider *dpp, RGWBucketInfo& bucket_info, optional_yield y);
#include "rgw_rest_ratelimit.h"
#include "rgw_rest_realm.h"
#include "rgw_rest_user.h"
+#include "rgw_lc_tier.h"
#include "services/svc_sys_obj.h"
#include "services/svc_mdlog.h"
#include "services/svc_cls.h"
mtime, olh_epoch, dpp, y, flags & FLAG_LOG_OP);
}
+int RadosObject::restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags)
+{
+ /* init */
+ rgw::sal::RadosPlacementTier* rtier = static_cast<rgw::sal::RadosPlacementTier*>(tier);
+ string id = "cloudid";
+ string endpoint = rtier->get_rt().t.s3.endpoint;
+ RGWAccessKey key = rtier->get_rt().t.s3.key;
+ string region = rtier->get_rt().t.s3.region;
+ HostStyle host_style = rtier->get_rt().t.s3.host_style;
+ string bucket_name = rtier->get_rt().t.s3.target_path;
+ const rgw::sal::ZoneGroup& zonegroup = store->get_zone()->get_zonegroup();
+ int ret = 0;
+ string src_storage_class = o.meta.storage_class; // or take src_placement also as input
+
+ // fetch mtime of the object
+ std::unique_ptr<rgw::sal::Object::ReadOp> read_op(get_read_op());
+ read_op->params.lastmod = &mtime;
+
+ ret = read_op->prepare(y, dpp);
+ if (ret < 0) {
+ ldpp_dout(dpp, 0) << "Restoring object(" << o.key << "): read_op failed ret=" << ret << dendl;
+ return ret;
+ }
+
+ if (bucket_name.empty()) {
+ bucket_name = "rgwx-" + zonegroup.get_name() + "-" + tier->get_storage_class() +
+ "-cloud-bucket";
+ boost::algorithm::to_lower(bucket_name);
+ }
+ /* Create RGW REST connection */
+ S3RESTConn conn(cct, id, { endpoint }, key, zonegroup.get_id(), region, host_style);
+
+ // save source cloudtier storage class
+ RGWLCCloudTierCtx tier_ctx(cct, dpp, o, store, bucket->get_info(),
+ this, conn, bucket_name,
+ rtier->get_rt().t.s3.target_storage_class);
+ tier_ctx.acl_mappings = rtier->get_rt().t.s3.acl_mappings;
+ tier_ctx.multipart_min_part_size = rtier->get_rt().t.s3.multipart_min_part_size;
+ tier_ctx.multipart_sync_threshold = rtier->get_rt().t.s3.multipart_sync_threshold;
+ tier_ctx.storage_class = tier->get_storage_class();
+
+ ldpp_dout(dpp, 20) << "Restoring object(" << o.key << ") from the cloud endpoint(" << endpoint << ")" << dendl;
+
+ if (days && days == 0) {
+ ldpp_dout(dpp, 0) << "Days = 0 not valid; Not restoring object (" << o.key << ") from the cloud endpoint(" << endpoint << ")" << dendl;
+ return 0;
+ }
+
+ // Note: For non-versioned objects, below should have already been set by the callers-
+ // o.current should be false; this(obj)->instance should have version-id.
+
+ // set restore_status as RESTORE_ALREADY_IN_PROGRESS
+ ret = set_cloud_restore_status(dpp, y, RGWRestoreStatus::RestoreAlreadyInProgress);
+ if (ret < 0) {
+ ldpp_dout(dpp, 0) << " Setting cloud restore status to RESTORE_ALREADY_IN_PROGRESS for the object(" << o.key << ") from the cloud endpoint(" << endpoint << ") failed, ret=" << ret << dendl;
+ return ret;
+ }
+
+ /* Restore object from the cloud endpoint.
+ * All restore related status and attrs are set as part of object download to
+ * avoid any races */
+ ret = store->getRados()->restore_obj_from_cloud(tier_ctx, *rados_ctx,
+ bucket->get_info(), get_obj(), placement_rule,
+ tier_config,
+ mtime, olh_epoch, days, dpp, y, flags & FLAG_LOG_OP);
+
+ if (ret < 0) { //failed to restore
+ ldpp_dout(dpp, 0) << "Restoring object(" << o.key << ") from the cloud endpoint(" << endpoint << ") failed, ret=" << ret << dendl;
+ auto reset_ret = set_cloud_restore_status(dpp, y, RGWRestoreStatus::RestoreFailed);
+
+ rgw_placement_rule target_placement;
+ target_placement.inherit_from(tier_ctx.bucket_info.placement_rule);
+ target_placement.storage_class = tier->get_storage_class();
+
+ /* Reset HEAD object as CloudTiered */
+ reset_ret = write_cloud_tier(dpp, y, tier_ctx.o.versioned_epoch,
+ tier, tier_ctx.is_multipart_upload,
+ target_placement, tier_ctx.obj);
+
+ if (reset_ret < 0) {
+ ldpp_dout(dpp, 0) << " Reset to cloud_tier of object(" << o.key << ") from the cloud endpoint(" << endpoint << ") failed, ret=" << reset_ret << dendl;
+ }
+ return ret;
+ }
+
+ ldpp_dout(dpp, 20) << "Sucessfully restored object(" << o.key << ") from the cloud endpoint(" << endpoint << ")" << dendl;
+
+ return ret;
+}
+
int RadosObject::transition_to_cloud(Bucket* bucket,
rgw::sal::PlacementTier* tier,
rgw_bucket_dir_entry& o,
return ret;
}
+int RadosObject::set_cloud_restore_status(const DoutPrefixProvider* dpp,
+ optional_yield y,
+ rgw::sal::RGWRestoreStatus restore_status)
+{
+ int ret = 0;
+ set_atomic();
+
+ bufferlist bl;
+ using ceph::encode;
+ encode(restore_status, bl);
+
+ ret = modify_obj_attrs(RGW_ATTR_RESTORE_STATUS, bl, y, dpp);
+
+ return ret;
+}
+
+/*
+ * If the object is restored temporarily and is expired, delete the data and
+ * reset the HEAD object as cloud-transitioned.
+ */
+int RadosObject::handle_obj_expiry(const DoutPrefixProvider* dpp, optional_yield y) {
+ int ret = 0;
+ real_time read_mtime;
+ std::unique_ptr<rgw::sal::Object::ReadOp> read_op(get_read_op());
+ read_op->params.lastmod = &read_mtime;
+ ldpp_dout(dpp, 20) << "Entering handle_obj_expiry Obj:" << get_key() << dendl;
+
+ ret = read_op->prepare(y, dpp);
+ if (ret < 0) {
+ ldpp_dout(dpp, -1) << "handle_obj_expiry Obj:" << get_key() <<
+ ", read_op failed ret=" << ret << dendl;
+ return ret;
+ }
+
+ set_atomic();
+ map<string, bufferlist> attrs = get_attrs();
+ RGWRados::Object op_target(store->getRados(), bucket->get_info(), *rados_ctx, get_obj());
+ RGWRados::Object::Write obj_op(&op_target);
+ Object* obj = (Object*)this;
+
+ bufferlist bl;
+ auto attr_iter = attrs.find(RGW_ATTR_RESTORE_TYPE);
+ if (attr_iter != attrs.end()) {
+ using ceph::decode;
+ rgw::sal::RGWRestoreType restore_type;
+ decode(restore_type, attr_iter->second);
+ if (restore_type == rgw::sal::RGWRestoreType::Temporary) {
+ ldpp_dout(dpp, 10) << "Expiring temporary restored Obj:" << get_key() << dendl;
+
+ attr_iter = attrs.find(RGW_ATTR_MANIFEST);
+ if (attr_iter != attrs.end()) {
+ RGWObjManifest m;
+ try {
+ using ceph::decode;
+ decode(m, attr_iter->second);
+ obj_op.meta.modify_tail = true;
+ obj_op.meta.flags = PUT_OBJ_CREATE;
+ obj_op.meta.category = RGWObjCategory::CloudTiered;
+ obj_op.meta.delete_at = real_time();
+ bufferlist blo;
+ obj_op.meta.data = &blo;
+ obj_op.meta.if_match = NULL;
+ obj_op.meta.if_nomatch = NULL;
+ obj_op.meta.user_data = NULL;
+ obj_op.meta.zones_trace = NULL;
+ obj_op.meta.set_mtime = read_mtime;
+
+ RGWObjManifest *pmanifest;
+ pmanifest = &m;
+
+ Object* head_obj = (Object*)this;
+ RGWObjTier tier_config;
+ m.get_tier_config(&tier_config);
+
+ rgw_placement_rule target_placement(pmanifest->get_head_placement_rule(), tier_config.name);
+
+ pmanifest->set_head(target_placement, head_obj->get_obj(), 0);
+ pmanifest->set_tail_placement(target_placement, head_obj->get_obj().bucket);
+ pmanifest->set_obj_size(0);
+ obj_op.meta.manifest = pmanifest;
+
+ // erase restore attrs
+ attrs.erase(RGW_ATTR_RESTORE_STATUS);
+ attrs.erase(RGW_ATTR_RESTORE_TYPE);
+ attrs.erase(RGW_ATTR_RESTORE_TIME);
+ attrs.erase(RGW_ATTR_RESTORE_EXPIRY_DATE);
+ attrs.erase(RGW_ATTR_CLOUDTIER_STORAGE_CLASS);
+
+ bufferlist bl;
+ bl.append(tier_config.name);
+ attrs[RGW_ATTR_STORAGE_CLASS] = bl;
+
+ const req_context rctx{dpp, y, nullptr};
+ return obj_op.write_meta(0, 0, attrs, rctx, head_obj->get_trace());
+ } catch (const buffer::end_of_buffer&) {
+ // ignore empty manifest; it's not cloud-tiered
+ } catch (const std::exception& e) {
+ }
+ }
+ return 0;
+ }
+ }
+ // object is not restored/temporary; go for regular deletion
+ // ensure object is not overwritten and is really expired
+ if (is_expired()) {
+ ldpp_dout(dpp, 10) << "Deleting expired obj:" << get_key() << dendl;
+
+ ret = obj->delete_object(dpp, null_yield, rgw::sal::FLAG_LOG_OP, nullptr, nullptr);
+ }
+
+ return ret;
+}
int RadosObject::write_cloud_tier(const DoutPrefixProvider* dpp,
optional_yield y,
uint64_t olh_epoch,
obj_op.meta.if_nomatch = NULL;
obj_op.meta.user_data = NULL;
obj_op.meta.zones_trace = NULL;
- obj_op.meta.delete_at = real_time();
obj_op.meta.olh_epoch = olh_epoch;
RGWObjManifest *pmanifest;
attrs.erase(RGW_ATTR_ID_TAG);
attrs.erase(RGW_ATTR_TAIL_TAG);
+ // erase restore attrs
+ attrs.erase(RGW_ATTR_RESTORE_STATUS);
+ attrs.erase(RGW_ATTR_RESTORE_TYPE);
+ attrs.erase(RGW_ATTR_RESTORE_TIME);
+ attrs.erase(RGW_ATTR_RESTORE_EXPIRY_DATE);
+ attrs.erase(RGW_ATTR_CLOUDTIER_STORAGE_CLASS);
+
const req_context rctx{dpp, y, nullptr};
return obj_op.write_meta(0, 0, attrs, rctx, head_obj->get_trace());
}
bool update_object,
const DoutPrefixProvider* dpp,
optional_yield y) override;
+ virtual int restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags) override;
virtual bool placement_rules_match(rgw_placement_rule& r1, rgw_placement_rule& r2) override;
virtual int dump_obj_layout(const DoutPrefixProvider *dpp, optional_yield y, Formatter* f) override;
bool is_multipart_upload,
rgw_placement_rule& target_placement,
Object* head_obj);
+ int handle_obj_expiry(const DoutPrefixProvider* dpp, optional_yield y);
+ int set_cloud_restore_status(const DoutPrefixProvider* dpp,
+ optional_yield y,
+ RGWRestoreStatus restore_status);
RGWObjManifest* get_manifest() { return manifest; }
RGWObjectCtx& get_ctx() { return *rados_ctx; }
#define RGW_ATTR_SOURCE_ZONE RGW_ATTR_PREFIX "source_zone"
#define RGW_ATTR_TAGS RGW_ATTR_PREFIX RGW_AMZ_PREFIX "tagging"
+#define RGW_ATTR_CLOUDTIER_STORAGE_CLASS RGW_ATTR_PREFIX "cloudtier_storage_class"
+#define RGW_ATTR_RESTORE_STATUS RGW_ATTR_PREFIX "restore-status"
+#define RGW_ATTR_RESTORE_TYPE RGW_ATTR_PREFIX "restore-type"
+#define RGW_ATTR_RESTORE_TIME RGW_ATTR_PREFIX "restored-at"
+#define RGW_ATTR_RESTORE_EXPIRY_DATE RGW_ATTR_PREFIX "restore-expiry-date"
+
#define RGW_ATTR_TEMPURL_KEY1 RGW_ATTR_META_PREFIX "temp-url-key"
#define RGW_ATTR_TEMPURL_KEY2 RGW_ATTR_META_PREFIX "temp-url-key-2"
}
}
+ ret = static_cast<rgw::sal::RadosStore*>(driver)->getRados()->process_expire_objects(this, null_yield);
+ if (ret < 0) {
+ ldpp_dout(this, 5) << "RGWLC::process_expire_objects: failed, "
+ << " worker ix: " << worker->ix << dendl;
+ }
+
return 0;
}
}
}
} /* checksum_mode */
+ auto attr_iter = attrs.find(RGW_ATTR_RESTORE_TYPE);
+ if (attr_iter != attrs.end()) {
+ rgw::sal::RGWRestoreType rt;
+ bufferlist bl = attr_iter->second;
+ auto iter = bl.cbegin();
+ decode(rt, iter);
+
+ if (rt == rgw::sal::RGWRestoreType::Temporary) {
+ // temporary restore; set storage-class to cloudtier storage class
+ auto c_iter = attrs.find(RGW_ATTR_CLOUDTIER_STORAGE_CLASS);
+
+ if (c_iter != attrs.end()) {
+ attrs[RGW_ATTR_STORAGE_CLASS] = c_iter->second;
+ }
+ }
+ }
for (struct response_attr_param *p = resp_attr_params; p->param; p++) {
bool exists;
static constexpr uint32_t FLAG_LOG_OP = 0x0001;
static constexpr uint32_t FLAG_PREVENT_VERSIONING = 0x0002;
+enum RGWRestoreStatus : uint8_t {
+ None = 0,
+ RestoreAlreadyInProgress = 1,
+ CloudRestored = 2,
+ RestoreFailed = 3
+};
+
+
+enum class RGWRestoreType : uint8_t {
+ None = 0,
+ Temporary = 1,
+ Permanent = 2
+};
+
+
// a simple streaming data processing abstraction
/**
* @brief A simple streaming data processing abstraction
bool update_object,
const DoutPrefixProvider* dpp,
optional_yield y) = 0;
+ virtual int restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags) = 0;
/** Check to see if two placement rules match */
virtual bool placement_rules_match(rgw_placement_rule& r1, rgw_placement_rule& r2) = 0;
/** Dump driver-specific object layout info in JSON */
o, cloud_targets, cct, update_object, dpp, y);
}
+int FilterObject::restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags)
+{
+ return next->restore_obj_from_cloud(nextBucket(bucket), nextPlacementTier(tier),
+ placement_rule, o, cct, tier_config, mtime, olh_epoch, days, dpp, y, flags);
+}
+
bool FilterObject::placement_rules_match(rgw_placement_rule& r1, rgw_placement_rule& r2)
{
return next->placement_rules_match(r1, r2);
bool update_object,
const DoutPrefixProvider* dpp,
optional_yield y) override;
+ virtual int restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags) override;
virtual bool placement_rules_match(rgw_placement_rule& r1, rgw_placement_rule& r2) override;
virtual int dump_obj_layout(const DoutPrefixProvider *dpp, optional_yield y,
Formatter* f) override;
* work with lifecycle */
return -1;
}
+ virtual int restore_obj_from_cloud(Bucket* bucket,
+ rgw::sal::PlacementTier* tier,
+ rgw_placement_rule& placement_rule,
+ rgw_bucket_dir_entry& o,
+ CephContext* cct,
+ RGWObjTier& tier_config,
+ real_time& mtime,
+ uint64_t olh_epoch,
+ std::optional<uint64_t> days,
+ const DoutPrefixProvider* dpp,
+ optional_yield y,
+ uint32_t flags) override {
+ return -1;
+ }
jspan_context& get_trace() override { return trace_ctx; }
void set_trace (jspan_context&& _trace_ctx) override { trace_ctx = std::move(_trace_ctx); }
projects / ceph.git / commitdiff