--- /dev/null
+====================
+Bucket Logging
+====================
+
+.. versionadded:: T
+
+.. contents::
+
+Bucket logging provides a mechanism for logging all access to a bucket. The
+log data can be used to monitor bucket activity, detect unauthorized
+access, get insights into the bucket usage and use the logs as a journal for bucket changes.
+The log records are stored in objects in a separate bucket and can be analyzed later.
+Logging configuration is done at the bucket level and can be enabled or disabled at any time.
+The log bucket can accumulate logs from multiple buckets. It is recommended to configured
+a different "prefix" for each bucket, so that the logs of different buckets will be stored
+in different objects in the log bucket.
+
+
+.. toctree::
+ :maxdepth: 1
+
+Logging Reliability
+-------------------
+For performance reasons, even though the log records are written to persistent storage, the log object will
+appear in the log bucket only after some configurable amount of time (or if the maximum object size of 128MB is reached).
+This time (in seconds) could be set per source bucket via a Ceph extension to the REST API,
+or globally via the `rgw_bucket_logging_obj_roll_time` configuration option. If not set, the default time is 5 minutes.
+Adding a log object to the log bucket is done "lazily", meaning, that if no more records are written to the object, it may
+remain outside of the log bucket even after the configured time has passed.
+
+Standard
+````````
+If logging type is set to "Standard" (the default) the log records are written to the log bucket after the bucket operation is completed.
+This means that there are the logging operation may fail, with no indication to he client.
+
+Journal
+```````
+If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed.
+This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client.
+An exception to the above are "multi/delete" log records: if writing these log records fail, the operation continues and may still be successful.
+Note that it may happen that the log records were successfully written, but the bucket operation failed, since the logs are written
+before such a failure, there will be no indication for that in the log records.
+
+
+Bucket Logging REST API
+-----------------------
+Detailed under: `Bucket Operations`_.
+
+
+Log Objects Key Format
+----------------------
+
+Simple
+``````
+has the following format:
+
+::
+
+ <prefix><year-month-day-hour-minute-second>-<16 bytes unique-id>
+
+For example:
+
+::
+
+ fish/2024-08-06-09-40-09-TI9ROKN05DD4HPQF
+
+Partitioned
+```````````
+has the following format:
+
+::
+
+ <prefix><bucket owner>/<source region>/<bucket name>/<year>/<month>/<day>/<year-month-day-hour-minute-second>-<16 bytes unique-id>
+
+For example:
+
+::
+
+ fish/testid//all-log/2024/08/06/2024-08-06-10-11-18-1HMU3UMWOJKNQJ0X
+
+Log Records
+~~~~~~~~~~~
+
+The log records are space separated string columns and have the following possible formats:
+
+Journal
+```````
+minimum amount of data used for journaling bucket changes (this is a Ceph extension).
+
+ - bucket owner (or dash if empty)
+ - bucket name (or dash if empty)
+ - time in the following format: ``[day/month/year:hour:minute:second timezone]``
+ - object key (or dash if empty)
+ - operation in the following format: ``WEBSITE/REST.<HTTP method>.<resource>``
+ - object size (or dash if empty)
+ - version id (dash if empty or question mark if unknown)
+ - eTag
+
+For example:
+
+::
+
+ testid fish [06/Aug/2024:09:40:09 +0000] myfile - REST.PUT.OBJECT 4cfdfc1f58e762d3e116787cb92fac60
+ testid fish [06/Aug/2024:09:40:28 +0000] myfile REST.DELETE.OBJECT 4cfdfc1f58e762d3e116787cb92fac60
+
+
+Standard
+````````
+based on `AWS Logging Record Format`_.
+
+ - bucket owner (or dash if empty)
+ - bucket name (or dash if empty)
+ - time
+ - remote IP (not supported, always a dash)
+ - user or account (or dash if empty)
+ - request ID
+ - operation in the following format: ``WEBSITE/REST.<HTTP method>.<resource>``
+ - object key (or dash if empty)
+ - request URI in the following format: ``"<HTTP method> <URI> <HTTP version>"``
+ - HTTP status (or dash if zero). Note that in most cases log is written before the status is known
+ - error code (or dash if empty)
+ - bytes sent (or dash if zero)
+ - object size (or dash if zero)
+ - total time (not supported, always a dash)
+ - turnaround time (not supported, always a dash)
+ - referrer (not supported, always a dash)
+ - user agent (not supported, always a dash)
+ - version id (or dash if empty)
+ - host id taken from "x-amz-id-2" (or dash if empty)
+ - signature version (not supported, always a dash)
+ - cipher suite (not supported, always a dash)
+ - authentication type (not supported, always a dash)
+ - host header (or dash if empty)
+ - TLS version (not supported, always a dash)
+ - access point ARN (not supported, always a dash)
+ - ACL flag ("Yes" if the request is an ACL operation, otherwise dash)
+
+For example:
+
+::
+
+ testid fish [06/Aug/2024:09:30:25 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.15085270386962380710 REST.PUT.OBJECT myfile "PUT /fish/myfile HTTP/1.1" 200 - 512 512 - - - - - - - - - localhost - -
+ testid fish [06/Aug/2024:09:30:51 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.7046073853138417766 REST.GET.OBJECT myfile "GET /fish/myfile HTTP/1.1" 200 - - 512 - - - - - - - - - localhost - -
+ testid fish [06/Aug/2024:09:30:56 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.10723158448701085570 REST.DELETE.OBJECT myfile "DELETE /fish/myfile1 HTTP/1.1" 200 - - 512 - - - - - - - - - localhost - -
+
+
+.. _AWS Logging Record Format: https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html
+.. _Bucket Operations: ../s3/bucketops
Cloud Transition <cloud-transition>
Metrics <metrics>
UADK Acceleration for Compression <uadk-accel>
+ Bucket Logging <bucket_logging>
+
+---------------------------------+-----------------+----------------------------------------+
| **Storage Class** | Supported | See :ref:`storage_classes` |
+---------------------------------+-----------------+----------------------------------------+
+| **Bucket Logging** | Supported | |
++---------------------------------+-----------------+----------------------------------------+
Unsupported Header Fields
-------------------------
+---------------+-----------------------+----------------------------------------------------------+
.. _S3 Notification Compatibility: ../../s3-notification-compatibility
+
+Enable Bucket Logging
+---------------------
+
+Enable logging for a bucket.
+
+Syntax
+~~~~~~
+
+::
+
+ PUT /{bucket}?logging HTTP/1.1
+
+
+Request Entities
+~~~~~~~~~~~~~~~~
+
+Parameters are XML encoded in the body of the request, in the following format:
+
+::
+
+ <BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+ <LoggingEnabled>
+ <TargetBucket>string</TargetBucket>
+ <TargetGrants>
+ <Grant>
+ <Grantee>
+ <DisplayName>string</DisplayName>
+ <EmailAddress>string</EmailAddress>
+ <ID>string</ID>
+ <xsi:type>string</xsi:type>
+ <URI>string</URI>
+ </Grantee>
+ <Permission>string</Permission>
+ </Grant>
+ </TargetGrants>
+ <TargetObjectKeyFormat>
+ <PartitionedPrefix>
+ <PartitionDateSource>DeliveryTime|EventTime</PartitionDateSource>
+ </PartitionedPrefix>
+ <SimplePrefix>
+ </SimplePrefix>
+ </TargetObjectKeyFormat>
+ <TargetPrefix>string</TargetPrefix>
+ <LoggingType>Standard|Journal</LoggingType>
+ <ObjectRollTime>integer</ObjectRollTime>
+ </LoggingEnabled>
+ </BucketLoggingStatus>
+
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| Name | Type | Description | Required |
++===============================+===========+======================================================================================+==========+
+| ``BucketLoggingStatus`` | Container | Enabling/Disabling logging configuration for the bucket. | Yes |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``LoggingEnabled`` | Container | Holding the logging configuration for the bucket. | Yes |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``TargetBucket`` | String | The bucket where the logs are stored. The log bucket cannot have bucket logging | Yes |
+| | | enabled. | |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``TargetGrants`` | Container | Not supported. The owner of the log bucket is the owner of the log objects. | No |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``TargetObjectKeyFormat`` | Container | The format of the log object key. Contains either ``PartitionedPrefix`` or | No |
+| | | ``SimplePrefix`` entities. | |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``PartitionedPrefix`` | Container | Indicates a partitioned log object key format. Note that ``PartitionDateSource`` | No |
+| | | is ignored and hardcoded as ``DeliveryTime`` | |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``SimplePrefix`` | Container | Indicates a simple log object key format (default format) | No |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``TargetPrefix`` | String | The prefix for the log objects. Used in both formats. May be used to distinguish | No |
+| | | between different source buckets writing log records to the same log bucket. | |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``LoggingType`` | String | The type of logging. Valid values are: | No |
+| | | ``Standard`` (default) all bucket operations are logged after being perfomed. | |
+| | | The log record will contain all fields. | |
+| | | ``Journal`` only PUT, COPY, MULTI/DELETE and MPU operations are logged. | |
+| | | Will record the minimum subset of fields in the log record that is needed | |
+| | | for journaling. | |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+| ``ObjectRollTime`` | Integer | The time in seconds after which a new log object is created, and the previous log | No |
+| | | object added to the log bucket. Default is 3600 seconds (1 hour). | |
++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+
+
+
+HTTP Response
+~~~~~~~~~~~~~
+
++---------------+-----------------------+----------------------------------------------------------+
+| HTTP Status | Status Code | Description |
++===============+=======================+==========================================================+
+| ``400`` | MalformedXML | The XML is not well-formed |
++---------------+-----------------------+----------------------------------------------------------+
+| ``400`` | InvalidArgument | Missing mandatory value or invalid value |
++---------------+-----------------------+----------------------------------------------------------+
+| ``404`` | NoSuchBucket | The bucket does not exist |
++---------------+-----------------------+----------------------------------------------------------+
+
+
+Disable Bucket Logging
+----------------------
+
+Disable bucket logging from a bucket.
+
+Syntax
+~~~~~~
+
+::
+
+ PUT /{bucket}?logging HTTP/1.1
+
+
+Request Entities
+~~~~~~~~~~~~~~~~
+
+Parameters are XML encoded in the body of the request, in the following format:
+
+::
+
+ <BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+ </BucketLoggingStatus>
+
+
+HTTP Response
+~~~~~~~~~~~~~
+
++---------------+-----------------------+----------------------------------------------------------+
+| HTTP Status | Status Code | Description |
++===============+=======================+==========================================================+
+| ``404`` | NoSuchBucket | The bucket does not exist |
++---------------+-----------------------+----------------------------------------------------------+
+
+Get Bucket Logging
+------------------
+
+Get logging configured on a bucket.
+
+Syntax
+~~~~~~
+
+::
+
+ GET /{bucket}?logging HTTP/1.1
+
+
+Response Entities
+~~~~~~~~~~~~~~~~~
+
+Response is XML encoded in the body of the request, in the following format:
+
+::
+
+ <BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+ <LoggingEnabled>
+ <TargetBucket>string</TargetBucket>
+ <TargetGrants>
+ <Grant>
+ <Grantee>
+ <DisplayName>string</DisplayName>
+ <EmailAddress>string</EmailAddress>
+ <ID>string</ID>
+ <xsi:type>string</xsi:type>
+ <URI>string</URI>
+ </Grantee>
+ <Permission>string</Permission>
+ </Grant>
+ </TargetGrants>
+ <TargetObjectKeyFormat>
+ <PartitionedPrefix>
+ <PartitionDateSource>DeliveryTime|EventTime</PartitionDateSource>
+ </PartitionedPrefix>
+ <SimplePrefix>
+ </SimplePrefix>
+ </TargetObjectKeyFormat>
+ <TargetPrefix>string</TargetPrefix>
+ <LoggingType>Standard|Journal</LoggingType>
+ <ObjectRollTime>integer</ObjectRollTime>
+ </LoggingEnabled>
+ </BucketLoggingStatus>
+
+
+HTTP Response
+~~~~~~~~~~~~~
+
++---------------+-----------------------+----------------------------------------------------------+
+| HTTP Status | Status Code | Description |
++===============+=======================+==========================================================+
+| ``404`` | NoSuchBucket | The bucket does not exist |
++---------------+-----------------------+----------------------------------------------------------+
+
--- /dev/null
+#!/usr/bin/python
+
+import boto3
+import sys
+
+if len(sys.argv) != 3:
+ print('Usage: ' + sys.argv[0] + ' <bucket> <target bucket>')
+ sys.exit(1)
+
+# bucket name as first argument
+bucket = sys.argv[1]
+# target bucket name as the 2nd argument
+target_bucket = sys.argv[2]
+
+# endpoint and keys from vstart
+endpoint = 'http://127.0.0.1:8000'
+access_key='0555b35654ad1656d804'
+secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==' # notsecret
+
+client = boto3.client('s3',
+ endpoint_url=endpoint,
+ aws_access_key_id=access_key,
+ aws_secret_access_key=secret_key)
+
+
+# create the source bucket
+response = client.create_bucket(Bucket=bucket)
+print(response)
+
+# create the target bucket
+response = client.create_bucket(Bucket=target_bucket)
+print(response)
+
+bucket_logging_conf = {'LoggingEnabled': {
+ 'TargetBucket': target_bucket,
+ 'TargetPrefix': 'log/',
+ 'TargetObjectKeyFormat': {
+ 'SimplePrefix': {}
+ },
+ 'ObjectRollTime': 60,
+ 'LoggingType': 'Journal',
+ }
+}
+
+response = client.put_bucket_logging(Bucket=bucket, BucketLoggingStatus=bucket_logging_conf)
+print(response)
+
+response = client.get_bucket_logging(Bucket=bucket)
+print(response)
+
"UsageStatsSummary": {
"type": "structure",
"members": {
- "QuotaMaxBytes":{"shape":"QuotaMaxBytes"},
- "QuotaMaxBuckets":{"shape": "QuotaMaxBuckets"},
- "QuotaMaxObjCount":{"shape":"QuotaMaxObjCount"},
- "QuotaMaxBytesPerBucket":{"shape":"QuotaMaxBytesPerBucket"},
+ "QuotaMaxBytes":{"shape":"QuotaMaxBytes"},
+ "QuotaMaxBuckets":{"shape": "QuotaMaxBuckets"},
+ "QuotaMaxObjCount":{"shape":"QuotaMaxObjCount"},
+ "QuotaMaxBytesPerBucket":{"shape":"QuotaMaxBytesPerBucket"},
"QuotaMaxObjCountPerBucket":{"shape":"QuotaMaxObjCountPerBucket"},
- "TotalBytes":{"shape":"TotalBytes"},
+ "TotalBytes":{"shape":"TotalBytes"},
"TotalBytesRounded":{"shape":"TotalBytesRounded"},
"TotalEntries":{"shape":"TotalEntries"}
}
},
"QuotaMaxBytes":{"type":"integer"},
- "QuotaMaxBuckets":{"type": "integer"},
- "QuotaMaxObjCount":{"type":"integer"},
- "QuotaMaxBytesPerBucket":{"type":"integer"},
- "QuotaMaxObjCountPerBucket":{"type":"integer"},
- "TotalBytesRounded":{"type":"integer"},
+ "QuotaMaxBuckets":{"type": "integer"},
+ "QuotaMaxObjCount":{"type":"integer"},
+ "QuotaMaxBytesPerBucket":{"type":"integer"},
+ "QuotaMaxObjCountPerBucket":{"type":"integer"},
+ "TotalBytesRounded":{"type":"integer"},
"TotalBytes":{"type":"integer"},
- "TotalEntries":{"type":"integer"}
+ "TotalEntries":{"type":"integer"},
+ "LoggingEnabled":{
+ "type":"structure",
+ "required":[
+ "TargetBucket",
+ "TargetPrefix"
+ ],
+ "members":{
+ "TargetBucket":{
+ "shape":"TargetBucket",
+ "documentation":"<p>Specifies the bucket where you want to store server access logs. You can have your logs delivered to any bucket that you own. You can also configure multiple buckets to deliver their logs to the same target bucket. In this case, you should choose a different <code>TargetPrefix</code> for each source bucket so that the delivered log files can be distinguished by key.</p>"
+ },
+ "TargetGrants":{
+ "shape":"TargetGrants",
+ "documentation":"<p>Container for granting information.</p> <p>Should be used when the write permissions to the tagert bucket should eb different than the permissions of the user performing the operation thta needs to be logged. This is usually used in cased of batched logging. see: <code>RecordBatchSize</code>.</p>"
+ },
+ "TargetPrefix":{
+ "shape":"TargetPrefix",
+ "documentation":"<p>A prefix for all log object keys. If you store log files from multiple buckets in a single bucket, you can use a prefix to distinguish which log files came from which bucket.</p>"
+ },
+ "TargetObjectKeyFormat":{
+ "shape":"TargetObjectKeyFormat",
+ "documentation":"<p>key format for log objects.</p>"
+ },
+ "ObjectRollTime":{
+ "shape":"ObjectRollTime",
+ "documentation":"<p>time in seconds to move the log object to the target bucket and start another log object.</p>"
+ },
+ "LoggingType":{
+ "shape":"LoggingType",
+ "documentation":"<p>use Standard log type to log all bucket operations i nthe standard format. use Journal log type to log only creations and deletion of objects in more compact format.</p>"
+ },
+ "RecordsBatchSize":{
+ "shape":"RecordsBatchSize",
+ "documentation":"indicates how many records to batch in memory before writing to the object. if set to zero, records are written syncronously to the object. if <code>ObjectRollTime</code>e is reached, the batch of records will be written to the object regardless of the number of records. </p>"
+ }
+ },
+ "documentation":"<p>Describes where logs are stored the prefix assigned to all log object keys for a bucket, and their format. also, the level the delivery guarantee of the records.</p>"
+ },
+ "TargetObjectKeyFormat":{
+ "type":"structure",
+ "members":{
+ "SimplePrefix":{
+ "shape":"SimplePrefix",
+ "documentation":"<p>To use the simple format for S3 keys for log objects. To specify SimplePrefix format, set SimplePrefix to {}.</p>",
+ "locationName":"SimplePrefix"
+ },
+ "PartitionedPrefix":{
+ "shape":"PartitionedPrefix",
+ "documentation":"<p>Partitioned S3 key for log objects.</p>",
+ "locationName":"PartitionedPrefix"
+ }
+ },
+ "documentation":"<p>Key format for log objects. Only one format, PartitionedPrefix or SimplePrefix, is allowed.</p>"
+ },
+ "SimplePrefix":{
+ "type":"structure",
+ "members":{
+ },
+ "documentation":"<p>To use simple format for S3 keys for log objects, set SimplePrefix to an empty object.</p> <p> <code>[DestinationPrefix][YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString]</code> </p>",
+ "locationName":"SimplePrefix"
+ },
+ "PartitionDateSource":{
+ "type":"string",
+ "enum":[
+ "EventTime",
+ "DeliveryTime"
+ ]
+ },
+ "PartitionedPrefix":{
+ "type":"structure",
+ "members":{
+ "PartitionDateSource":{
+ "shape":"PartitionDateSource",
+ "documentation":"<p>Specifies the partition date source for the partitioned prefix. PartitionDateSource can be EventTime or DeliveryTime.</p>"
+ }
+ },
+ "documentation":"<p>Amazon S3 keys for log objects are partitioned in the following format:</p> <p> <code>[DestinationPrefix][SourceAccountId]/[SourceRegion]/[SourceBucket]/[YYYY]/[MM]/[DD]/[YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString]</code> </p> <p>PartitionedPrefix defaults to EventTime delivery when server access logs are delivered.</p>",
+ "locationName":"PartitionedPrefix"
+ },
+ "ObjectRollTime":{"type":"integer"},
+ "RecordsBatchSize":{"type":"integer"},
+ "LoggingType":{
+ "type":"string",
+ "enum": [
+ "Standard",
+ "Journal"
+ ]
+ }
},
"documentation":"<p/>"
}
flags:
- startup
with_legacy: true
+- name: rgw_bucket_logging_obj_roll_time
+ type: uint
+ level: advanced
+ desc: Default time in seconds for the bucket logging object to roll
+ long_desc: Object roll time can be provided in the bucket logging configuration.
+ If not provided, this value will be used.
+ default: 300
+ services:
+ - rgw
+ with_legacy: true
rgw_data_access.cc
driver/rados/account.cc
driver/rados/buckets.cc
+ rgw_bucket_logging.cc
+ rgw_rest_bucket_logging.cc
driver/rados/cls_fifo_legacy.cc
driver/rados/group.cc
driver/rados/groups.cc
*
*/
+#include <asm-generic/errno-base.h>
#include <errno.h>
+#include <fmt/core.h>
#include <stdlib.h>
+#include <string>
#include <system_error>
#include <filesystem>
#include <unistd.h>
#include "include/function2.hpp"
#include "common/Clock.h"
+#include "common/ceph_time.h"
#include "common/errno.h"
#include "role.h"
+#include "rgw_obj_types.h"
+#include "rgw_rados.h"
#include "rgw_sal.h"
#include "rgw_sal_rados.h"
#include "rgw_bucket.h"
#include "rgw_rest_realm.h"
#include "rgw_rest_user.h"
#include "rgw_lc_tier.h"
+#include "rgw_bucket_logging.h"
#include "services/svc_sys_obj.h"
#include "services/svc_mdlog.h"
#include "services/svc_cls.h"
objv_tracker, y);
}
+int RadosBucket::get_logging_object_name(std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ RGWObjVersionTracker* objv_tracker) {
+ rgw_pool data_pool;
+ const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix);
+ if (!store->getRados()->get_obj_data_pool(get_placement_rule(), rgw_obj{get_key(), obj_name_oid}, &data_pool)) {
+ ldpp_dout(dpp, 1) << "failed to get data pool for bucket '" << get_name() <<
+ "' when getting logging object name" << dendl;
+ return -EIO;
+ }
+ bufferlist bl;
+ const int ret = rgw_get_system_obj(store->svc()->sysobj,
+ data_pool,
+ obj_name_oid,
+ bl,
+ objv_tracker,
+ nullptr,
+ y,
+ dpp,
+ nullptr,
+ nullptr);
+ if (ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to get logging object name from '" << obj_name_oid << "'. ret = " << ret << dendl;
+ return ret;
+ }
+ obj_name = bl.to_str();
+ return 0;
+}
+
+int RadosBucket::set_logging_object_name(const std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ bool new_obj,
+ RGWObjVersionTracker* objv_tracker) {
+ rgw_pool data_pool;
+ const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix);
+ if (!store->getRados()->get_obj_data_pool(get_placement_rule(), rgw_obj{get_key(), obj_name_oid}, &data_pool)) {
+ ldpp_dout(dpp, 1) << "failed to get data pool for bucket '" << get_name() <<
+ "' when setting logging object name" << dendl;
+ return -EIO;
+ }
+ bufferlist bl;
+ bl.append(obj_name);
+ const int ret = rgw_put_system_obj(dpp, store->svc()->sysobj,
+ data_pool,
+ obj_name_oid,
+ bl,
+ new_obj,
+ objv_tracker,
+ ceph::real_time::clock::now(),
+ y,
+ nullptr);
+ if (ret == -EEXIST) {
+ ldpp_dout(dpp, 20) << "race detected in initializing '" << obj_name_oid << "' with logging object name:'" << obj_name << "'. ret = " << ret << dendl;
+ } else if (ret == -ECANCELED) {
+ ldpp_dout(dpp, 20) << "race detected in updating logging object name '" << obj_name << "' at '" << obj_name_oid << "'. ret = " << ret << dendl;
+ } else if (ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to set logging object name '" << obj_name << "' at '" << obj_name_oid << "'. ret = " << ret << dendl;
+ }
+ return ret;
+}
+
+std::string to_temp_object_name(const rgw::sal::Bucket* bucket, const std::string& obj_name) {
+ return fmt::format("{}__shadow_{}0",
+ bucket->get_bucket_id(),
+ obj_name);
+}
+
+int RadosBucket::commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) {
+ rgw_pool data_pool;
+ const rgw_obj head_obj{get_key(), obj_name};
+ const auto placement_rule = get_placement_rule();
+
+ if (!store->getRados()->get_obj_data_pool(placement_rule, head_obj, &data_pool)) {
+ ldpp_dout(dpp, 1) << "failed to get data pool for bucket '" << get_name() <<
+ "' when comitting logging object" << dendl;
+ return -EIO;
+ }
+
+ const auto temp_obj_name = to_temp_object_name(this, obj_name);
+ std::map<string, bufferlist> obj_attrs;
+ ceph::real_time mtime;
+ bufferlist bl_data;
+ // TODO: this is needed only for etag calculation
+ if (const auto ret = rgw_get_system_obj(store->svc()->sysobj,
+ data_pool,
+ temp_obj_name,
+ bl_data,
+ nullptr,
+ &mtime,
+ y,
+ dpp,
+ &obj_attrs,
+ nullptr); ret < 0) {
+ ldpp_dout(dpp, 1) << "faild to read logging data when comitting to object '" << temp_obj_name
+ << ". error: " << ret << dendl;
+ return ret;
+ }
+
+ uint64_t size = bl_data.length();
+ const uint64_t max_obj_size = store->ctx()->_conf->osd_max_object_size;
+ RGWObjManifest manifest;
+ manifest.set_prefix(obj_name);
+ manifest.set_trivial_rule(0, max_obj_size);
+ RGWObjManifest::generator manifest_gen;
+ if (const auto ret = manifest_gen.create_begin(store->ctx(), &manifest,
+ placement_rule,
+ nullptr, // no special placment for tail
+ get_key(),
+ head_obj); ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to create manifest when comitting logging object. error: " <<
+ ret << dendl;
+ return ret;
+ }
+
+ if (const auto ret = manifest_gen.create_next(size); ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to add object to manifest when comitting logging object. error: " <<
+ ret << dendl;
+ return ret;
+ }
+
+ if (const auto expected_temp_obj = manifest_gen.get_cur_obj(store->getRados());
+ temp_obj_name != expected_temp_obj.oid) {
+ // TODO: cleanup temporary object, commit would never succeed
+ ldpp_dout(dpp, 1) << "temporary logging object name mismatch: '" <<
+ temp_obj_name << "' != '" << expected_temp_obj.oid << "'" << dendl;
+ return -EINVAL;
+ }
+
+ RGWObjectCtx obj_ctx(store);
+ obj_ctx.set_atomic(head_obj);
+ const auto& bucket_info = get_info();
+ RGWRados::Object rgw_head_obj(store->getRados(),
+ bucket_info,
+ obj_ctx,
+ head_obj);
+ // disable versioning on the logging objects
+ rgw_head_obj.set_versioning_disabled(true);
+ RGWRados::Object::Write head_obj_wop(&rgw_head_obj);
+ head_obj_wop.meta.manifest = &manifest;
+ head_obj_wop.meta.bucket_owner = bucket_info.owner;
+ head_obj_wop.meta.flags = PUT_OBJ_CREATE;
+ head_obj_wop.meta.mtime = &mtime;
+ // TODO: head_obj_wop.meta.ptag
+ // the owner of the logging object is the bucket owner
+ // not the user that wrote the log that triggered the commit
+ const ACLOwner owner{bucket_info.owner, ""}; // TODO: missing display name
+ head_obj_wop.meta.owner = owner;
+ const auto etag = TOPNSPC::crypto::digest<TOPNSPC::crypto::MD5>(bl_data).to_str();
+ bufferlist bl_etag;
+ bl_etag.append(etag.c_str());
+ obj_attrs.emplace(RGW_ATTR_ETAG, std::move(bl_etag));
+ const req_context rctx{dpp, y, nullptr};
+ jspan_context trace{false, false};
+ if (const auto ret = head_obj_wop.write_meta(0, size, obj_attrs, rctx, trace); ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to commit logging object '" << temp_obj_name <<
+ "' to bucket id '" << get_bucket_id() <<"'. error: " << ret << dendl;
+ return ret;
+ }
+ ldpp_dout(dpp, 20) << "committed logging object '" << temp_obj_name <<
+ "' with size of " << size << " bytes, to bucket '" << get_key() << "' as '" <<
+ obj_name << "'" << dendl;
+ return 0;
+}
+
+struct BucketLoggingCompleteArg {
+ BucketLoggingCompleteArg(const std::string& _obj_name, size_t _size, CephContext* _cct)
+ : obj_name{_obj_name}, size{_size}, cct{_cct} {}
+ const std::string obj_name;
+ const size_t size;
+ CephContext* cct;
+};
+
+void bucket_logging_completion(rados_completion_t completion, void* args) {
+ auto* aio_comp = reinterpret_cast<librados::AioCompletionImpl*>(completion);
+ std::unique_ptr<BucketLoggingCompleteArg> logging_args(reinterpret_cast<BucketLoggingCompleteArg*>(args));
+ if (aio_comp->get_return_value() < 0) {
+ ldout(logging_args->cct, 1) << "failed to complete append to logging object '" << logging_args->obj_name <<
+ "'. ret = " << aio_comp->get_return_value() << dendl;
+ } else {
+ ldout(logging_args->cct, 20) << "wrote " << logging_args->size << " bytes to logging object '" <<
+ logging_args->obj_name << "'" << dendl;
+ }
+}
+
+int RadosBucket::write_logging_object(const std::string& obj_name,
+ const std::string& record,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ bool async_completion) {
+ const auto temp_obj_name = to_temp_object_name(this, obj_name);
+ rgw_pool data_pool;
+ rgw_obj obj{get_key(), obj_name};
+ if (!store->getRados()->get_obj_data_pool(get_placement_rule(), obj, &data_pool)) {
+ ldpp_dout(dpp, 1) << "failed to get data pool for bucket '" << get_name() <<
+ "' when writing logging object" << dendl;
+ return -EIO;
+ }
+ librados::IoCtx io_ctx;
+ if (const auto ret = rgw_init_ioctx(dpp, store->getRados()->get_rados_handle(), data_pool, io_ctx); ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to get IO context for logging object from data pool:" << data_pool.to_str() << dendl;
+ return -EIO;
+ }
+ bufferlist bl;
+ bl.append(record);
+ bl.append("\n");
+ // append the record to the temporary object
+ // if this is the first record, the object will be created
+ librados::ObjectWriteOperation op;
+ op.append(bl);
+ if (async_completion) {
+ aio_completion_ptr completion{librados::Rados::aio_create_completion()};
+ auto arg = std::make_unique<BucketLoggingCompleteArg>(temp_obj_name, record.length(), store->ctx());
+ completion->set_complete_callback(arg.get(), bucket_logging_completion);
+ if (const auto ret = io_ctx.aio_operate(temp_obj_name, completion.get(), &op); ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to append to logging object '" << temp_obj_name <<
+ "'. ret = " << ret << dendl;
+ return ret;
+ }
+ std::ignore = arg.release();
+ std::ignore = completion.release();
+ return 0;
+ }
+ if (const auto ret = rgw_rados_operate(dpp, io_ctx, temp_obj_name, &op, y); ret < 0) {
+ ldpp_dout(dpp, 1) << "failed to append to logging object '" << temp_obj_name <<
+ "'. ret = " << ret << dendl;
+ return ret;
+ }
+ ldpp_dout(dpp, 20) << "wrote " << record.length() << " bytes to logging object '" <<
+ temp_obj_name << "'" << dendl;
+ return 0;
+}
+
std::unique_ptr<User> RadosStore::get_user(const rgw_user &u)
{
return std::make_unique<RadosUser>(this, u);
}
int RadosStore::stat_topics_v1(const std::string& tenant, optional_yield y, const DoutPrefixProvider *dpp) {
- return rgw_stat_system_obj(dpp, svc()->sysobj, svc()->zone->get_zone_params().log_pool, topics_oid(tenant), nullptr, nullptr, y, nullptr);
+ return rgw_stat_system_obj(dpp, svc()->sysobj, svc()->zone->get_zone_params().log_pool, topics_oid(tenant), nullptr, nullptr, nullptr, y, nullptr);
}
int RadosStore::write_topics(const std::string& tenant, const rgw_pubsub_topics& topics, RGWObjVersionTracker* objv_tracker,
multipart_upload_info upload_info;
upload_info.dest_placement = dest_placement;
upload_info.cksum_type = cksum_type;
-
+
if (obj_legal_hold) {
upload_info.obj_legal_hold_exist = true;
upload_info.obj_legal_hold = (*obj_legal_hold);
#ifdef WITH_RADOSGW_LUA_PACKAGES
rgw::lua::packages_t failed_packages;
std::string install_dir;
- auto r = rgw::lua::install_packages(dpp, store,
- y, store->ctx()->_conf.get_val<std::string>("rgw_luarocks_location"),
+ auto r = rgw::lua::install_packages(dpp, store,
+ y, store->ctx()->_conf.get_val<std::string>("rgw_luarocks_location"),
failed_packages, install_dir);
if (r < 0) {
ldpp_dout(dpp, 1) << "WARNING: failed to install Lua packages from allowlist. error code: " << r
ldpp_dout(dpp, 5) << "WARNING: failed to install Lua package: " << p
<< " from allowlist" << dendl;
}
-#else
+#else
const int r = 0;
-#endif
+#endif
ack_reload(dpp, notify_id, cookie, r);
}
<< ". error: " << cpp_strerror(r) << dendl;
return r;
}
-
+
std::vector<librados::notify_ack_t> acks;
std::vector<librados::notify_timeout_t> timeouts;
ioctx.decode_notify_response(reply_bl, &acks, &timeouts);
auto iter = ack.payload_bl.cbegin();
ceph::decode(r, iter);
} catch (buffer::error& err) {
- ldpp_dout(dpp, 1) << "ERROR: couldn't decode Lua packages reload status. error: " <<
+ ldpp_dout(dpp, 1) << "ERROR: couldn't decode Lua packages reload status. error: " <<
err.what() << dendl;
return -EINVAL;
}
optional_yield y, const DoutPrefixProvider *dpp) override;
int remove_topics(RGWObjVersionTracker* objv_tracker,
optional_yield y, const DoutPrefixProvider *dpp) override;
+ int get_logging_object_name(std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ RGWObjVersionTracker* objv_tracker) override;
+ int set_logging_object_name(const std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ bool new_obj,
+ RGWObjVersionTracker* objv_tracker) override;
+ int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override;
+ int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override;
private:
int link(const DoutPrefixProvider* dpp, const rgw_owner& new_owner, optional_yield y, bool update_entrypoint = true, RGWObjVersionTracker* objv = nullptr);
int rgw_stat_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj* svc_sysobj,
const rgw_pool& pool, const std::string& key,
RGWObjVersionTracker *objv_tracker,
- real_time *pmtime, optional_yield y,
+ real_time *pmtime, uint64_t *psize, optional_yield y,
std::map<std::string, bufferlist> *pattrs)
{
rgw_raw_obj obj(pool, key);
return sysobj.rop()
.set_attrs(pattrs)
.set_last_mod(pmtime)
+ .set_obj_size(psize)
.stat(y, dpp);
}
.read(dpp, &bl, y);
}
-int rgw_delete_system_obj(const DoutPrefixProvider *dpp,
+int rgw_delete_system_obj(const DoutPrefixProvider *dpp,
RGWSI_SysObj *sysobj_svc, const rgw_pool& pool, const string& oid,
RGWObjVersionTracker *objv_tracker, optional_yield y)
{
rgw_cache_entry_info *cache_info = nullptr,
boost::optional<obj_version> refresh_version = boost::none,
bool raw_attrs=false);
-int rgw_delete_system_obj(const DoutPrefixProvider *dpp,
+int rgw_delete_system_obj(const DoutPrefixProvider *dpp,
RGWSI_SysObj *sysobj_svc, const rgw_pool& pool, const std::string& oid,
RGWObjVersionTracker *objv_tracker, optional_yield y);
int rgw_stat_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj* svc_sysobj,
const rgw_pool& pool, const std::string& key,
RGWObjVersionTracker *objv_tracker,
- real_time *pmtime, optional_yield y,
+ real_time *pmtime, uint64_t *psize, optional_yield y,
std::map<std::string, bufferlist> *pattrs = nullptr);
const char *rgw_find_mime_by_ext(std::string& ext);
#include "rgw_sal_config.h"
#include "rgw_data_access.h"
#include "rgw_account.h"
+#include "rgw_bucket_logging.h"
#include "services/svc_sync_modules.h"
#include "services/svc_cls.h"
cout << " bucket sync disable disable bucket sync\n";
cout << " bucket sync enable enable bucket sync\n";
cout << " bucket radoslist list rados objects backing bucket's objects\n";
+ cout << " bucket logging flush flush pending log records object of source bucket to the log bucket to bucket\n";
cout << " bi get retrieve bucket index object entries\n";
cout << " bi put store bucket index object entries\n";
cout << " bi list list raw bucket index entries\n";
BUCKET_SHARD_OBJECTS,
BUCKET_OBJECT_SHARD,
BUCKET_RESYNC_ENCRYPTED_MULTIPART,
+ BUCKET_LOGGING_FLUSH,
POLICY,
LOG_LIST,
LOG_SHOW,
{ "bucket shard object", OPT::BUCKET_SHARD_OBJECTS },
{ "bucket object shard", OPT::BUCKET_OBJECT_SHARD },
{ "bucket resync encrypted multipart", OPT::BUCKET_RESYNC_ENCRYPTED_MULTIPART },
+ { "bucket logging flush", OPT::BUCKET_LOGGING_FLUSH },
{ "policy", OPT::POLICY },
{ "log list", OPT::LOG_LIST },
{ "log show", OPT::LOG_SHOW },
}
}
+ if (opt_cmd == OPT::BUCKET_LOGGING_FLUSH) {
+ if (bucket_name.empty()) {
+ cerr << "ERROR: bucket not specified" << std::endl;
+ return EINVAL;
+ }
+ int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket);
+ if (ret < 0) {
+ return -ret;
+ }
+ const auto& bucket_attrs = bucket->get_attrs();
+ auto iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING);
+ if (iter == bucket_attrs.end()) {
+ cerr << "WARNING: no logging configured on bucket" << std::endl;
+ return 0;
+ }
+ rgw::bucketlogging::configuration configuration;
+ try {
+ configuration.enabled = true;
+ decode(configuration, iter->second);
+ } catch (buffer::error& err) {
+ cerr << "ERROR: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING
+ << "'. error: " << err.what() << std::endl;
+ return EINVAL;
+ }
+ std::unique_ptr<rgw::sal::Bucket> target_bucket;
+ ret = init_bucket(tenant, configuration.target_bucket, "", &target_bucket);
+ if (ret < 0) {
+ cerr << "ERROR: failed to get target logging bucket '" << configuration.target_bucket << "'" << std::endl;
+ return -ret;
+ }
+ std::string obj_name;
+ RGWObjVersionTracker objv_tracker;
+ ret = target_bucket->get_logging_object_name(obj_name, configuration.target_prefix, null_yield, dpp(), &objv_tracker);
+ if (ret < 0) {
+ cerr << "ERROR: failed to get pending logging object name from target bucket '" << configuration.target_bucket << "'" << std::endl;
+ return -ret;
+ }
+ ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, dpp(), null_yield, true, &objv_tracker);
+ if (ret < 0) {
+ cerr << "ERROR: failed to flush pending logging object '" << obj_name
+ << "' to target bucket '" << configuration.target_bucket << "'" << std::endl;
+ return -ret;
+ }
+ cerr << "flushed pending logging object '" << obj_name
+ << "' to target bucket '" << configuration.target_bucket << "'" << std::endl;
+ return 0;
+ }
+
if (opt_cmd == OPT::LOG_LIST) {
// filter by date?
if (date.size() && date.size() != 10) {
--- /dev/null
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab ft=cpp
+
+#include <time.h>
+#include <random>
+#include "common/ceph_time.h"
+#include "rgw_bucket_logging.h"
+#include "rgw_xml.h"
+#include "rgw_sal.h"
+#include "rgw_op.h"
+
+#define dout_subsys ceph_subsys_rgw
+
+namespace rgw::bucketlogging {
+
+bool configuration::decode_xml(XMLObj* obj) {
+ const auto throw_if_missing = true;
+ enabled = false;
+ XMLObjIter iter = obj->find("LoggingEnabled");
+ XMLObj* const o = iter.get_next();
+ if (o) {
+ enabled = true;
+ RGWXMLDecoder::decode_xml("TargetBucket", target_bucket, o, throw_if_missing);
+ RGWXMLDecoder::decode_xml("TargetPrefix", target_prefix, o);
+ // TODO: decode grant
+ RGWXMLDecoder::decode_xml("ObjectRollTime", obj_roll_time, default_obj_roll_time, o);
+ std::string default_type{"Standard"};
+ std::string type;
+ RGWXMLDecoder::decode_xml("LoggingType", type, default_type, o);
+ if (type == "Standard") {
+ logging_type = LoggingType::Standard;
+ } else if (type == "Journal") {
+ logging_type = LoggingType::Journal;
+ } else {
+ // we don't allow for type "Any" in the configuration
+ throw RGWXMLDecoder::err("invalid bucket logging record type: '" + type + "'");
+ }
+ RGWXMLDecoder::decode_xml("RecordsBatchSize", records_batch_size, o);
+ if (iter = o->find("TargetObjectKeyFormat"); XMLObj* const oo = iter.get_next()) {
+ if (iter = oo->find("PartitionedPrefix"); XMLObj* const ooo = iter.get_next()) {
+ obj_key_format = KeyFormat::Partitioned;
+ default_type = "DeliveryTime";
+ RGWXMLDecoder::decode_xml("PartitionDateSource", type, default_type, ooo);
+ if (type == "DeliveryTime") {
+ date_source = PartitionDateSource::DeliveryTime;
+ } else if (type == "EventTime") {
+ date_source = PartitionDateSource::EventTime;
+ } else {
+ throw RGWXMLDecoder::err("invalid bucket logging partition date source: '" + type + "'");
+ }
+ } else if (iter = oo->find("SimplePrefix"); iter.get_next()) {
+ obj_key_format = KeyFormat::Simple;
+ } else {
+ throw RGWXMLDecoder::err("TargetObjectKeyFormat must contain a format tag");
+ }
+ }
+ }
+
+ return true;
+}
+
+void configuration::dump_xml(Formatter *f) const {
+ if (!enabled) {
+ return;
+ }
+ f->open_object_section("LoggingEnabled");
+ ::encode_xml("TargetBucket", target_bucket, f);
+ ::encode_xml("TargetPrefix", target_prefix, f);
+ ::encode_xml("ObjectRollTime", obj_roll_time, f);
+ switch (logging_type) {
+ case LoggingType::Standard:
+ ::encode_xml("LoggingType", "Standard", f);
+ break;
+ case LoggingType::Journal:
+ ::encode_xml("LoggingType", "Journal", f);
+ break;
+ case LoggingType::Any:
+ ::encode_xml("LoggingType", "", f);
+ break;
+ }
+ ::encode_xml("RecordsBatchSize", records_batch_size, f);
+ f->open_object_section("TargetObjectKeyFormat");
+ switch (obj_key_format) {
+ case KeyFormat::Partitioned:
+ f->open_object_section("PartitionedPrefix");
+ switch (date_source) {
+ case PartitionDateSource::DeliveryTime:
+ ::encode_xml("PartitionDateSource", "DeliveryTime", f);
+ break;
+ case PartitionDateSource::EventTime:
+ ::encode_xml("PartitionDateSource", "EventTime", f);
+ break;
+ }
+ f->close_section(); // PartitionedPrefix
+ break;
+ case KeyFormat::Simple:
+ f->open_object_section("SimplePrefix"); // empty section
+ f->close_section();
+ break;
+ }
+ f->close_section(); // TargetObjectKeyFormat
+ f->close_section(); // LoggingEnabled
+}
+
+void configuration::dump(Formatter *f) const {
+ Formatter::ObjectSection s(*f, "bucketLoggingStatus");
+ if (!enabled) {
+ return;
+ }
+ {
+ Formatter::ObjectSection s(*f, "loggingEnabled");
+ encode_json("targetBucket", target_bucket, f);
+ encode_json("targetPrefix", target_prefix, f);
+ encode_json("objectRollTime", obj_roll_time, f);
+ switch (logging_type) {
+ case LoggingType::Standard:
+ encode_json("loggingType", "Standard", f);
+ break;
+ case LoggingType::Journal:
+ encode_json("loggingType", "Journal", f);
+ break;
+ case LoggingType::Any:
+ encode_json("loggingType", "", f);
+ break;
+ }
+ encode_json("recordsBatchSize", records_batch_size, f);
+ {
+ Formatter::ObjectSection s(*f, "targetObjectKeyFormat");
+ switch (obj_key_format) {
+ case KeyFormat::Partitioned:
+ {
+ Formatter::ObjectSection s(*f, "partitionedPrefix");
+ switch (date_source) {
+ case PartitionDateSource::DeliveryTime:
+ encode_json("partitionDateSource", "DeliveryTime", f);
+ break;
+ case PartitionDateSource::EventTime:
+ encode_json("partitionDateSource", "EventTime", f);
+ break;
+ }
+ }
+ break;
+ case KeyFormat::Simple:
+ {
+ Formatter::ObjectSection s(*f, "simplePrefix");
+ }
+ break;
+ }
+ }
+ }
+}
+
+std::string configuration::to_json_str() const {
+ JSONFormatter f;
+ dump(&f);
+ std::stringstream ss;
+ f.flush(ss);
+ return ss.str();
+}
+
+template<size_t N>
+std::string unique_string() {
+ static const std::string possible_characters{"0123456789ABCDEFGHIJKLMNOPQRSTUVWXY"};
+ static const auto max_possible_value = possible_characters.length() - 1;
+ std::random_device rd;
+ std::mt19937 engine(rd());
+ std::uniform_int_distribution<> dist(0, max_possible_value);
+ std::string str(N, '\0');
+ std::generate_n(str.begin(), N, [&](){return possible_characters[dist(engine)];});
+ return str;
+}
+
+constexpr size_t UniqueStringLength = 16;
+
+ceph::coarse_real_time time_from_name(const std::string& obj_name, const DoutPrefixProvider *dpp) {
+ static const auto time_format_length = std::string{"YYYY-MM-DD-hh-mm-ss"}.length();
+ const auto obj_name_length = obj_name.length();
+ ceph::coarse_real_time extracted_time;
+ if (obj_name_length < time_format_length + UniqueStringLength + 1) {
+ ldpp_dout(dpp, 1) << "ERROR: logging object name too short: " << obj_name << dendl;
+ return extracted_time;
+ }
+ const auto time_start_pos = obj_name_length - (time_format_length + UniqueStringLength + 1);
+ // note: +1 is for the dash between the timestamp and the unique string
+ std::string time_str = obj_name.substr(time_start_pos, time_format_length);
+
+ std::tm t = {};
+ if (const auto ret = strptime(time_str.c_str(), "%Y-%m-%d-%H-%M-%S", &t); ret == nullptr || *ret != '\0') {
+ ldpp_dout(dpp, 1) << "ERROR: invalid time format: '" << time_str << "' in logging object name: " << obj_name << dendl;
+ return extracted_time;
+ }
+ extracted_time = ceph::coarse_real_time::clock::from_time_t(mktime(&t));
+ ldpp_dout(dpp, 20) << "INFO: time '" << extracted_time << "' extracted from logging object name: " << obj_name << dendl;
+ return extracted_time;
+}
+
+int new_logging_object(const configuration& conf,
+ const std::unique_ptr<rgw::sal::Bucket>& bucket,
+ std::string& obj_name,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool init_obj,
+ RGWObjVersionTracker* objv_tracker) {
+ const auto tt = ceph::coarse_real_time::clock::to_time_t(ceph::coarse_real_time::clock::now());
+ std::tm t{};
+ localtime_r(&tt, &t);
+
+ const auto unique = unique_string<UniqueStringLength>();
+ const auto old_name = obj_name;
+
+ switch (conf.obj_key_format) {
+ case KeyFormat::Simple:
+ obj_name = fmt::format("{}{:%Y-%m-%d-%H-%M-%S}-{}",
+ conf.target_prefix,
+ t,
+ unique);
+ break;
+ case KeyFormat::Partitioned:
+ {
+ // TODO: use date_source
+ const auto source_region = ""; // TODO
+ obj_name = fmt::format("{}{}/{}/{}/{:%Y/%m/%d}/{:%Y-%m-%d-%H-%M-%S}-{}",
+ conf.target_prefix,
+ to_string(bucket->get_owner()),
+ source_region,
+ bucket->get_name(),
+ t,
+ t,
+ unique);
+ }
+ break;
+ }
+
+ int ret = bucket->set_logging_object_name(obj_name, conf.target_prefix, y, dpp, init_obj, objv_tracker);
+ if (ret == -EEXIST || ret == -ECANCELED) {
+ if (ret = bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, nullptr); ret < 0) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" <<
+ conf.target_bucket << "'. ret = " << ret << dendl;
+ return ret;
+ }
+ ldpp_dout(dpp, 20) << "INFO: name already set. got name of logging object '" << obj_name << "' of bucket '" <<
+ conf.target_bucket << "'" << dendl;
+ return -ECANCELED;
+ } else if (ret < 0) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to write name of logging object '" << obj_name << "' of bucket '" <<
+ conf.target_bucket << "'. ret = " << ret << dendl;
+ return ret;
+ }
+ ldpp_dout(dpp, 20) << "INFO: wrote name of logging object '" << obj_name << "' of bucket '" <<
+ conf.target_bucket << "'" << dendl;
+ return 0;
+}
+
+int rollover_logging_object(const configuration& conf,
+ const std::unique_ptr<rgw::sal::Bucket>& bucket,
+ std::string& obj_name,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool must_commit,
+ RGWObjVersionTracker* objv_tracker) {
+ if (conf.target_bucket != bucket->get_name()) {
+ ldpp_dout(dpp, 1) << "ERROR: bucket name mismatch: '" << conf.target_bucket << "' != '" << bucket->get_name() << "'" << dendl;
+ return -EINVAL;
+ }
+ const auto old_obj = obj_name;
+ const auto ret = new_logging_object(conf, bucket, obj_name, dpp, y, false, objv_tracker);
+ if (ret == -ECANCELED) {
+ ldpp_dout(dpp, 20) << "INFO: rollover already performed for '" << old_obj << "' to bucket '" <<
+ conf.target_bucket << "'. ret = " << ret << dendl;
+ return 0;
+ } else if (ret < 0) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to rollover logging object '" << old_obj << "' to bucket '" <<
+ conf.target_bucket << "'. ret = " << ret << dendl;
+ return ret;
+ }
+ if (const auto ret = bucket->commit_logging_object(old_obj, y, dpp); ret < 0) {
+ if (must_commit) {
+ return ret;
+ }
+ ldpp_dout(dpp, 5) << "WARNING: failed to commit logging object '" << old_obj << "' to bucket '" <<
+ conf.target_bucket << "'. ret = " << ret << dendl;
+ // we still want to write the new records to the new object even if commit failed
+ // will try to commit again next time
+ }
+ return 0;
+}
+
+#define dash_if_empty(S) (S).empty() ? "-" : S
+#define dash_if_empty_or_null(P, S) (((P) == nullptr) || (S).empty()) ? "-" : S
+#define dash_if_zero(I) (I) == 0 ? "-" : std::to_string(I)
+#define dash_if_zero_or_null(P, I) (((P) == nullptr) || ((I) == 0)) ? "-" : std::to_string(I)
+
+/* S3 bucket standard log record
+ * based on: https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html
+ - bucket owner
+ - bucket name
+ - The time at which the request was received at UTC time. The format, as follows: [%d/%b/%Y:%H:%M:%S %z]
+ - The apparent IP address of the requester
+ - The canonical user ID of the requester, or a - for unauthenticated requests
+ - Request ID
+ - REST.HTTP_method.resource_type or S3.action.resource_type for Lifecycle and logging
+ - The key (object name) part of the request (source key in case of copy)
+ - The Request-URI part of the HTTP request message
+ - The numeric HTTP status code of the response
+ - The S3 Error code, or - if no error occurred
+ - The number of response bytes sent, excluding HTTP protocol overhead, or - if zero
+ - Object Size
+ - Total time: milliseconds including network transmission time. from first byte received to last byte transmitted
+ - turn around time: milliseconds exluding networks transmission time. from last byte received to first byte transmitted
+ - The value of the HTTP Referer header, if present, or - if not
+ - User Agent
+ - The version ID in the request, or - if the operation doesn't take a versionId parameter
+ - Host ID: x-amz-id-2
+ - SigV2 or SigV4, that was used to authenticate the request or a - for unauthenticated requests
+ - SSL cipher that was negotiated for an HTTPS request or a - for HTTP
+ - The type of request authentication used: AuthHeader, QueryString or a - for unauthenticated requests
+ - Host Header: The RGW endpoint fqdn
+ - TLS version negotiated by the client: TLSv1.1, TLSv1.2, TLSv1.3, or - if TLS wasn't used
+ - ARN of the access point of the request. If the access point ARN is malformed or not used, the field will contain a -
+ - A string that indicates whether the request required an (ACL) for authorization. If ACL is required, the string is Yes. If no ACLs were required, the string is -
+
+S3 bucket short (ceph) log record
+ - bucket owner
+ - bucket name
+ - The time at which the request was received at UTC time. The format, as follows: [%d/%b/%Y:%H:%M:%S %z]
+ - REST.HTTP_method.resource_type or S3.action.resource_type for Lifecycle and logging
+ - The key (object name) part of the request (source key in case of copy)
+ - Object version in case of versioned bucket
+ - Object Size
+ - eTag
+};*/
+
+int log_record(rgw::sal::Driver* driver,
+ const sal::Object* obj,
+ const req_state* s,
+ const std::string& op_name,
+ const std::string& etag,
+ size_t size,
+ const configuration& conf,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool async_completion,
+ bool log_source_bucket) {
+ if (!s->bucket) {
+ ldpp_dout(dpp, 1) << "ERROR: only bucket operations are logged" << dendl;
+ return -EINVAL;
+ }
+ std::unique_ptr<rgw::sal::Bucket> target_bucket;
+ auto ret = driver->load_bucket(dpp, rgw_bucket(s->bucket_tenant, conf.target_bucket),
+ &target_bucket, y);
+ if (ret < 0) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to get target logging bucket '" << conf.target_bucket << "'. ret = " << ret << dendl;
+ return ret;
+ }
+ std::string obj_name;
+ RGWObjVersionTracker objv_tracker;
+ ret = target_bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, &objv_tracker);
+ if (ret == 0) {
+ const auto time_to_commit = time_from_name(obj_name, dpp) + std::chrono::seconds(conf.obj_roll_time);
+ if (ceph::coarse_real_time::clock::now() > time_to_commit) {
+ ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' exceeded its time, will be committed to bucket '" <<
+ conf.target_bucket << "'" << dendl;
+ if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, y, false, &objv_tracker); ret < 0) {
+ return ret;
+ }
+ } else {
+ ldpp_dout(dpp, 20) << "INFO: record will be written to current logging object '" << obj_name << "'. will be comitted at: " << time_to_commit << dendl;
+ }
+ } else if (ret == -ENOENT) {
+ // try to create the temporary log object for the first time
+ ret = new_logging_object(conf, target_bucket, obj_name, dpp, y, true, nullptr);
+ if (ret == 0) {
+ ldpp_dout(dpp, 20) << "INFO: first time logging for bucket '" << conf.target_bucket << "'" << dendl;
+ } else if (ret == -ECANCELED) {
+ ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' already exists for bucket '" << conf.target_bucket << "', will be used" << dendl;
+ } else {
+ ldpp_dout(dpp, 1) << "ERROR: failed to create logging object of bucket '" <<
+ conf.target_bucket << "' for the first time. ret = " << ret << dendl;
+ return ret;
+ }
+ } else {
+ ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" <<
+ conf.target_bucket << "'. ret = " << ret << dendl;
+ return ret;
+ }
+
+ std::string record;
+ const auto tt = ceph::coarse_real_time::clock::to_time_t(s->time);
+ std::tm t{};
+ localtime_r(&tt, &t);
+ auto user_or_account = s->account_name;
+ if (user_or_account.empty()) {
+ s->user->get_id().to_str(user_or_account);
+ }
+ auto fqdn = s->info.host;
+ if (!s->info.domain.empty() && !fqdn.empty()) {
+ fqdn.append(".").append(s->info.domain);
+ }
+
+ std::string bucket_owner;
+ std::string bucket_name;
+ if (log_source_bucket) {
+ if (!s->src_object || !s->src_object->get_bucket()) {
+ ldpp_dout(dpp, 1) << "ERROR: source object or bucket is missing when logging source bucket" << dendl;
+ return -EINVAL;
+ }
+ bucket_owner = to_string(s->src_object->get_bucket()->get_owner());
+ bucket_name = s->src_bucket_name;
+ } else {
+ bucket_owner = to_string( s->bucket->get_owner());
+ bucket_name = s->bucket->get_name();
+ }
+
+ switch (conf.logging_type) {
+ case LoggingType::Standard:
+ record = fmt::format("{} {} [{:%d/%b/%Y:%H:%M:%S %z}] {} {} {} {} {} \"{} {}{}{} HTTP/1.1\" {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {}",
+ dash_if_empty(bucket_owner),
+ dash_if_empty(bucket_name),
+ t,
+ "-", // no requester IP
+ dash_if_empty(user_or_account),
+ dash_if_empty(s->req_id),
+ op_name,
+ dash_if_empty_or_null(obj, obj->get_name()),
+ s->info.method,
+ s->info.request_uri,
+ s->info.request_params.empty() ? "" : "?",
+ s->info.request_params,
+ dash_if_zero(s->err.http_ret),
+ dash_if_empty(s->err.err_code),
+ dash_if_zero(s->content_length),
+ dash_if_zero(size),
+ "-", // no total time when logging record
+ std::chrono::duration_cast<std::chrono::milliseconds>(s->time_elapsed()),
+ "-", // TODO: referer
+ "-", // TODO: user agent
+ dash_if_empty_or_null(obj, obj->get_instance()),
+ s->info.x_meta_map.contains("x-amz-id-2") ? s->info.x_meta_map.at("x-amz-id-2") : "-",
+ "-", // TODO: Signature Version (SigV2 or SigV4)
+ "-", // TODO: SSL cipher. e.g. "ECDHE-RSA-AES128-GCM-SHA256"
+ "-", // TODO: Auth type. e.g. "AuthHeader"
+ dash_if_empty(fqdn),
+ "-", // TODO: TLS version. e.g. "TLSv1.2" or "TLSv1.3"
+ "-", // no access point ARN
+ (s->has_acl_header) ? "Yes" : "-");
+ break;
+ case LoggingType::Journal:
+ record = fmt::format("{} {} [{:%d/%b/%Y:%H:%M:%S %z}] {} {} {} {} {}",
+ dash_if_empty(to_string(s->bucket->get_owner())),
+ dash_if_empty(s->bucket->get_name()),
+ t,
+ op_name,
+ dash_if_empty_or_null(obj, obj->get_name()),
+ dash_if_zero(size),
+ dash_if_empty_or_null(obj, obj->get_instance()),
+ dash_if_empty(etag));
+ break;
+ case LoggingType::Any:
+ ldpp_dout(dpp, 1) << "ERROR: failed to format record when writing to logging object '" <<
+ obj_name << "' due to unsupported logging type" << dendl;
+ return -EINVAL;
+ }
+
+ if (ret = target_bucket->write_logging_object(obj_name,
+ record,
+ y,
+ dpp,
+ async_completion); ret < 0 && ret != -EFBIG) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to write record to logging object '" <<
+ obj_name << "'. ret = " << ret << dendl;
+ return ret;
+ }
+ if (ret == -EFBIG) {
+ ldpp_dout(dpp, 20) << "WARNING: logging object '" << obj_name << "' is full, will be committed to bucket '" <<
+ conf.target_bucket << "'" << dendl;
+ if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, y, true, nullptr); ret < 0 ) {
+ return ret;
+ }
+ if (ret = target_bucket->write_logging_object(obj_name,
+ record,
+ y,
+ dpp,
+ async_completion); ret < 0) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to write record to logging object '" <<
+ obj_name << "'. ret = " << ret << dendl;
+ return ret;
+ }
+ }
+
+ ldpp_dout(dpp, 20) << "INFO: wrote logging record: '" << record
+ << "' to '" << obj_name << "'" << dendl;
+ return 0;
+}
+
+std::string object_name_oid(const rgw::sal::Bucket* bucket, const std::string& prefix) {
+ // TODO: do i need bucket marker in the name?
+ return fmt::format("logging.{}.bucket.{}/{}", bucket->get_tenant(), bucket->get_bucket_id(), prefix);
+}
+
+int log_record(rgw::sal::Driver* driver,
+ LoggingType type,
+ const sal::Object* obj,
+ const req_state* s,
+ const std::string& op_name,
+ const std::string& etag,
+ size_t size,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool async_completion,
+ bool log_source_bucket) {
+ if (!s->bucket) {
+ // logging only bucket operations
+ return 0;
+ }
+ // check if bucket logging is needed
+ const auto& bucket_attrs = s->bucket->get_attrs();
+ auto iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING);
+ if (iter == bucket_attrs.end()) {
+ return 0;
+ }
+ configuration configuration;
+ try {
+ configuration.enabled = true;
+ auto bl_iter = iter->second.cbegin();
+ decode(configuration, bl_iter);
+ if (type != LoggingType::Any && configuration.logging_type != type) {
+ return 0;
+ }
+ ldpp_dout(dpp, 20) << "INFO: found matching logging configuration of bucket '" << s->bucket->get_name() <<
+ "' configuration: " << configuration.to_json_str() << dendl;
+ if (auto ret = log_record(driver, obj, s, op_name, etag, size, configuration, dpp, y, async_completion, log_source_bucket); ret < 0) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to perform logging for bucket '" << s->bucket->get_name() <<
+ "'. ret=" << ret << dendl;
+ return ret;
+ }
+ } catch (buffer::error& err) {
+ ldpp_dout(dpp, 1) << "ERROR: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING
+ << "'. error: " << err.what() << dendl;
+ return -EINVAL;
+ }
+ return 0;
+}
+
+} // namespace rgw::bucketlogging
+
--- /dev/null
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab ft=cpp
+
+#pragma once
+
+#include <string>
+#include <optional>
+#include <cstdint>
+#include "rgw_sal_fwd.h"
+#include "include/buffer.h"
+#include "include/encoding.h"
+#include "common/async/yield_context.h"
+
+class XMLObj;
+namespace ceph { class Formatter; }
+class DoutPrefixProvider;
+struct req_state;
+class RGWObjVersionTracker;
+class RGWOp;
+
+namespace rgw::bucketlogging {
+/* S3 bucket logging configuration
+ * based on: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLogging.html
+ * with ceph extensions
+<BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+ <LoggingEnabled>
+ <TargetBucket>string</TargetBucket>
+ <TargetGrants>
+ <Grant>
+ <Grantee>
+ <DisplayName>string</DisplayName>
+ <EmailAddress>string</EmailAddress>
+ <ID>string</ID>
+ <xsi:type>string</xsi:type>
+ <URI>string</URI>
+ </Grantee>
+ <Permission>string</Permission>
+ </Grant>
+ </TargetGrants>
+ <TargetObjectKeyFormat>
+ <PartitionedPrefix>
+ <PartitionDateSource>DeliveryTime|EventTime</PartitionDateSource>
+ </PartitionedPrefix>
+ <SimplePrefix>
+ </SimplePrefix>
+ </TargetObjectKeyFormat>
+ <TargetPrefix>string</TargetPrefix>
+ <LoggingType>Standard|Journal</LoggingType> <!-- Ceph extension -->
+ <ObjectRollTime>integer</ObjectRollTime> <!-- Ceph extension -->
+ <RecordsBatchSize>integer</RecordsBatchSize> <!-- Ceph extension -->
+ </LoggingEnabled>
+</BucketLoggingStatus>
+*/
+
+enum class KeyFormat {Partitioned, Simple};
+enum class LoggingType {Standard, Journal, Any};
+enum class PartitionDateSource {DeliveryTime, EventTime};
+
+struct configuration {
+ uint32_t default_obj_roll_time = 300;
+ bool enabled = false;
+ std::string target_bucket;
+ KeyFormat obj_key_format = KeyFormat::Simple;
+ // target object key formats:
+ // Partitioned: [DestinationPrefix][SourceAccountId]/[SourceRegion]/[SourceBucket]/[YYYY]/[MM]/[DD]/[YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString]
+ // Simple: [DestinationPrefix][YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString]
+ std::string target_prefix; // a prefix for all log object keys.
+ // useful when multiple bucket log to the same target
+ // or when the target bucket is used for other things than logs
+ uint32_t obj_roll_time; // time in seconds to move object to bucket and start another object
+ LoggingType logging_type = LoggingType::Standard;
+ // in case of "Standard: logging type, all bucket operations are logged
+ // in case of "Journal" logging type only the following operations are logged: PUT, COPY, MULTI/DELETE, Complete MPU
+ uint32_t records_batch_size = 0; // how many records to batch in memory before writing to the object
+ // if set to zero, records are written syncronously to the object.
+ // if obj_roll_time is reached, the batch of records will be written to the object
+ // regardless of the number of records
+ PartitionDateSource date_source = PartitionDateSource::DeliveryTime;
+ // EventTime: use only year, month, and day. The hour, minutes and seconds are set to 00 in the key
+ // DeliveryTime: the time the log object was created
+ bool decode_xml(XMLObj *obj);
+ void dump_xml(Formatter *f) const;
+ void dump(Formatter *f) const; // json
+ std::string to_json_str() const;
+
+ void encode(ceph::bufferlist& bl) const {
+ ENCODE_START(1, 1, bl);
+ encode(target_bucket, bl);
+ encode(static_cast<int>(obj_key_format), bl);
+ encode(target_prefix, bl);
+ encode(obj_roll_time, bl);
+ encode(static_cast<int>(logging_type), bl);
+ encode(records_batch_size, bl);
+ encode(static_cast<int>(date_source), bl);
+ ENCODE_FINISH(bl);
+ }
+
+ void decode(bufferlist::const_iterator& bl) {
+ DECODE_START(1, bl);
+ decode(target_bucket, bl);
+ int type;
+ decode(type, bl);
+ obj_key_format = static_cast<KeyFormat>(type);
+ decode(target_prefix, bl);
+ decode(obj_roll_time, bl);
+ decode(type, bl);
+ logging_type = static_cast<LoggingType>(type);
+ decode(records_batch_size, bl);
+ decode(type, bl);
+ date_source = static_cast<PartitionDateSource>(type);
+ DECODE_FINISH(bl);
+ }
+};
+WRITE_CLASS_ENCODER(configuration)
+
+constexpr unsigned MAX_BUCKET_LOGGING_BUFFER = 1000;
+
+using bucket_logging_records = std::array<std::string, MAX_BUCKET_LOGGING_BUFFER>;
+
+template <typename Records>
+inline std::string to_string(const Records& records) {
+ std::string str_records;
+ for (const auto& record : records) {
+ str_records.append(to_string(record)).append("\n");
+ }
+ return str_records;
+}
+
+// log a bucket logging record according to the configuration
+int log_record(rgw::sal::Driver* driver,
+ const sal::Object* obj,
+ const req_state* s,
+ const std::string& op_name,
+ const std::string& etag,
+ size_t size,
+ const configuration& conf,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool async_completion,
+ bool log_source_bucket);
+
+// commit the pending log objec tto the log bucket
+// and create a new pending log object
+// if "must_commit" is "false" the function will return success even if the pending log object was not committed
+int rollover_logging_object(const configuration& conf,
+ const std::unique_ptr<rgw::sal::Bucket>& bucket,
+ std::string& obj_name,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool must_commit,
+ RGWObjVersionTracker* objv_tracker);
+
+// return the oid of the object holding the name of the temporary logging object
+// bucket - log bucket
+// prefix - logging prefix from configuration. should be used when multiple buckets log into the same log bucket
+std::string object_name_oid(const rgw::sal::Bucket* bucket, const std::string& prefix);
+
+// log a bucket logging record according to type
+// configuration is fetched from bucket attributes
+// if no configuration exists, or if type does not match the function return zero (success)
+int log_record(rgw::sal::Driver* driver,
+ LoggingType type,
+ const sal::Object* obj,
+ const req_state* s,
+ const std::string& op_name,
+ const std::string& etag,
+ size_t size,
+ const DoutPrefixProvider *dpp,
+ optional_yield y,
+ bool async_completion,
+ bool log_source_bucket);
+} // namespace rgw::bucketlogging
+
#define RGW_ATTR_SLO_UINDICATOR RGW_ATTR_META_PREFIX "static-large-object"
#define RGW_ATTR_X_ROBOTS_TAG RGW_ATTR_PREFIX "x-robots-tag"
#define RGW_ATTR_STORAGE_CLASS RGW_ATTR_PREFIX "storage_class"
+#define RGW_ATTR_BUCKET_LOGGING RGW_ATTR_PREFIX "logging"
/* S3 Object Lock*/
#define RGW_ATTR_OBJECT_LOCK RGW_ATTR_PREFIX "object-lock"
#include "rgw_lua.h"
#include "rgw_iam_managed_policy.h"
#include "rgw_bucket_sync.h"
+#include "rgw_bucket_logging.h"
#include "services/svc_zone.h"
#include "services/svc_quota.h"
return 0;
}
-static int rgw_iam_add_buckettags(const DoutPrefixProvider *dpp, req_state* s) {
+int rgw_iam_add_buckettags(const DoutPrefixProvider *dpp, req_state* s) {
return rgw_iam_add_buckettags(dpp, s, s->bucket.get());
}
return make_tuple(has_existing_obj_tag, has_resource_tag);
}
-static std::tuple<bool, bool> rgw_check_policy_condition(const DoutPrefixProvider *dpp, req_state* s, bool check_obj_exist_tag=true) {
+std::tuple<bool, bool> rgw_check_policy_condition(const DoutPrefixProvider *dpp, req_state* s, bool check_obj_exist_tag) {
return rgw_check_policy_condition(dpp, s->iam_policy, s->iam_identity_policies, s->session_policies, check_obj_exist_tag);
}
rgw::op_counters::inc(counters, l_rgw_op_get_obj, 1);
std::unique_ptr<rgw::sal::Object::ReadOp> read_op(s->object->get_read_op());
+ std::string etag;
op_ret = get_params(y);
if (op_ret < 0)
rgw::op_counters::tinc(counters, l_rgw_op_list_obj_lat, s->time_elapsed());
}
-int RGWGetBucketLogging::verify_permission(optional_yield y)
-{
- auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false);
- if (has_s3_resource_tag)
- rgw_iam_add_buckettags(this, s);
-
- if (!verify_bucket_permission(this, s, rgw::IAM::s3GetBucketLogging)) {
- return -EACCES;
- }
-
- return 0;
-}
-
int RGWGetBucketLocation::verify_permission(optional_yield y)
{
auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false);
obj_retention->encode(obj_retention_bl);
emplace_attr(RGW_ATTR_OBJECT_RETENTION, std::move(obj_retention_bl));
}
+
+ if (!multipart) {
+ op_ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), etag, s->object->get_size(), this, y, false, false);
+ if (op_ret < 0) {
+ return;
+ }
+ }
// don't track the individual parts of multipart uploads. they replicate in
// full after CompleteMultipart
}
}
+ if (op_ret == 0) {
+ if (auto ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), etag, obj_size, this, y, false, false); ret < 0) {
+ // don't reply with an error in case of failed delete logging
+ ldpp_dout(this, 5) << "WARNING: DELETE operation ignores bucket logging failure: " << ret << dendl;
+ }
+ }
+
if (op_ret == -ECANCELED) {
op_ret = 0;
}
return;
}
+ etag = s->src_object->get_attrs()[RGW_ATTR_ETAG].to_str();
+ op_ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), etag, obj_size, this, y, false, false);
+ if (op_ret < 0) {
+ return;
+ }
+
op_ret = s->src_object->copy_object(s->owner,
s->user->get_id(),
&s->info,
this,
s->yield);
+ int ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Standard, s->src_object.get(), s, "REST.COPY.OBJECT_GET", etag, obj_size, this, y, true, true);
+ if (ret < 0) {
+ ldpp_dout(this, 5) << "WARNING: COPY operation ignores bucket logging failure of the GET part: " << ret << dendl;
+ }
+
if (op_ret < 0) {
return;
}
// send request to notification manager
- int ret = res->publish_commit(this, obj_size, mtime, etag, s->object->get_instance());
+ ret = res->publish_commit(this, obj_size, mtime, etag, s->object->get_instance());
if (ret < 0) {
ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << ret << dendl;
// too late to rollback operation, hence op_ret is not set here
RGWObjVersionTracker& objv_tracker = meta_obj->get_version_tracker();
using prefix_map_t = rgw::sal::MultipartUpload::prefix_map_t;
- prefix_map_t processed_prefixes;
+ prefix_map_t processed_prefixes;
+
+ // no etag and size before completion
+ op_ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), "", 0, this, y, false, false);
+ if (op_ret < 0) {
+ return;
+ }
op_ret =
upload->complete(this, y, s->cct, parts->parts, remove_objs, accounted_size,
if (op_ret == -ENOENT) {
op_ret = 0;
}
+
+ if (auto ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Any, obj.get(), s, canonical_name(), etag, obj_size, this, y, true, false); ret < 0) {
+ // don't reply with an error in case of failed delete logging
+ ldpp_dout(this, 5) << "WARNING: multi DELETE operation ignores bucket logging failure: " << ret << dendl;
+ }
+
if (op_ret == 0) {
// send request to notification manager
int ret = res->publish_commit(dpp, obj_size, ceph::real_clock::now(), etag, version_id);
RGWAccessControlPolicy& policy,
optional_yield y);
+std::tuple<bool, bool> rgw_check_policy_condition(const DoutPrefixProvider *dpp, req_state* s, bool check_obj_exist_tag=true);
+
+int rgw_iam_add_buckettags(const DoutPrefixProvider *dpp, req_state* s);
+
class RGWHandler {
protected:
rgw::sal::Driver* driver{nullptr};
}
virtual const char* name() const = 0;
virtual RGWOpType get_type() { return RGW_OP_UNKNOWN; }
+ virtual std::string canonical_name() const { return fmt::format("REST.{}.{}", s->info.method, name()); }
virtual uint32_t op_mask() { return 0; }
virtual bool need_container_stats() { return false; }
};
-class RGWGetBucketLogging : public RGWOp {
-public:
- RGWGetBucketLogging() {}
- int verify_permission(optional_yield y) override;
- void execute(optional_yield) override { }
-
- void send_response() override = 0;
- const char* name() const override { return "get_bucket_logging"; }
- RGWOpType get_type() override { return RGW_OP_GET_BUCKET_LOGGING; }
- uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
-};
-
class RGWGetBucketLocation : public RGWOp {
public:
RGWGetBucketLocation() {}
RGW_OP_ATTACH_GROUP_POLICY,
RGW_OP_DETACH_GROUP_POLICY,
RGW_OP_LIST_ATTACHED_GROUP_POLICIES,
+ RGW_OP_PUT_BUCKET_LOGGING,
/* rgw specific */
RGW_OP_ADMIN_SET_METADATA,
RGW_OP_GET_OBJ_LAYOUT,
#include "rgw_lua_request.h"
#include "rgw_tracer.h"
#include "rgw_ratelimit.h"
+#include "rgw_bucket_logging.h"
#include "services/svc_zone_utils.h"
rgw_log_op(rest, s, op, penv.olog);
}
+ if (op) {
+ std::ignore = rgw::bucketlogging::log_record(driver,
+ rgw::bucketlogging::LoggingType::Standard,
+ s->object.get(),
+ s,
+ op->canonical_name(),
+ "",
+ (s->src_object ? s->src_object->get_size() : (s->object ? s->object->get_size() : 0)),
+ op,
+ yield,
+ true,
+ false);
+ }
+
if (http_ret != nullptr) {
*http_ret = s->err.http_ret;
}
}
int get_params(optional_yield y) override;
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
};
class RGWGetObjTags_ObjStore : public RGWGetObjTags {
public:
RGWGetObjTags_ObjStore() {};
~RGWGetObjTags_ObjStore() {};
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
};
class RGWPutObjTags_ObjStore: public RGWPutObjTags {
public:
RGWPutObjTags_ObjStore() {};
~RGWPutObjTags_ObjStore() {};
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); }
};
class RGWGetBucketTags_ObjStore : public RGWGetBucketTags {
public:
RGWGetBucketTags_ObjStore() = default;
virtual ~RGWGetBucketTags_ObjStore() = default;
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_TAGGING", s->info.method); }
};
class RGWPutBucketTags_ObjStore: public RGWPutBucketTags {
public:
RGWPutBucketTags_ObjStore() = default;
virtual ~RGWPutBucketTags_ObjStore() = default;
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_TAGGING", s->info.method); }
};
class RGWGetBucketReplication_ObjStore : public RGWGetBucketReplication {
public:
RGWListBuckets_ObjStore() {}
~RGWListBuckets_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKETS", s->info.method); }
};
class RGWGetUsage_ObjStore : public RGWGetUsage {
public:
RGWGetUsage_ObjStore() {}
~RGWGetUsage_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.USER_USAGE", s->info.method); }
};
class RGWListBucket_ObjStore : public RGWListBucket {
public:
RGWListBucket_ObjStore() {}
~RGWListBucket_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
};
class RGWStatAccount_ObjStore : public RGWStatAccount {
public:
RGWStatAccount_ObjStore() {}
~RGWStatAccount_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACCOUNT_STATUS", s->info.method); }
};
class RGWStatBucket_ObjStore : public RGWStatBucket {
public:
RGWStatBucket_ObjStore() {}
~RGWStatBucket_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_STATUS", s->info.method); }
};
class RGWCreateBucket_ObjStore : public RGWCreateBucket {
public:
RGWCreateBucket_ObjStore() {}
~RGWCreateBucket_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
};
class RGWDeleteBucket_ObjStore : public RGWDeleteBucket {
public:
RGWDeleteBucket_ObjStore() {}
~RGWDeleteBucket_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); }
};
class RGWPutObj_ObjStore : public RGWPutObj
int verify_params() override;
int get_params(optional_yield y) override;
int get_data(bufferlist& bl) override;
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
};
class RGWPostObj_ObjStore : public RGWPostObj
~RGWPostObj_ObjStore() override {}
int verify_params() override;
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
};
public:
RGWPutMetadataAccount_ObjStore() {}
~RGWPutMetadataAccount_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACCOUNT_METADATA", s->info.method); }
};
class RGWPutMetadataBucket_ObjStore : public RGWPutMetadataBucket
public:
RGWPutMetadataBucket_ObjStore() {}
~RGWPutMetadataBucket_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_METADATA", s->info.method); }
};
class RGWPutMetadataObject_ObjStore : public RGWPutMetadataObject
public:
RGWPutMetadataObject_ObjStore() {}
~RGWPutMetadataObject_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_METADATA", s->info.method); }
};
class RGWRestoreObj_ObjStore : public RGWRestoreObj {
public:
RGWRestoreObj_ObjStore() {}
~RGWRestoreObj_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
};
class RGWDeleteObj_ObjStore : public RGWDeleteObj {
public:
RGWDeleteObj_ObjStore() {}
~RGWDeleteObj_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
};
class RGWGetCrossDomainPolicy_ObjStore : public RGWGetCrossDomainPolicy {
public:
RGWCopyObj_ObjStore() {}
~RGWCopyObj_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); }
};
class RGWGetACLs_ObjStore : public RGWGetACLs {
public:
RGWGetACLs_ObjStore() {}
~RGWGetACLs_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); }
};
class RGWPutACLs_ObjStore : public RGWPutACLs {
~RGWPutACLs_ObjStore() override {}
int get_params(optional_yield y) override;
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); }
};
class RGWGetLC_ObjStore : public RGWGetLC {
public:
RGWGetLC_ObjStore() {}
~RGWGetLC_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
};
class RGWPutLC_ObjStore : public RGWPutLC {
~RGWPutLC_ObjStore() override {}
int get_params(optional_yield y) override;
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
};
class RGWDeleteLC_ObjStore : public RGWDeleteLC {
RGWDeleteLC_ObjStore() {}
~RGWDeleteLC_ObjStore() override {}
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); }
};
class RGWGetCORS_ObjStore : public RGWGetCORS {
public:
RGWGetCORS_ObjStore() {}
~RGWGetCORS_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
};
class RGWPutCORS_ObjStore : public RGWPutCORS {
public:
RGWPutCORS_ObjStore() {}
~RGWPutCORS_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
};
class RGWDeleteCORS_ObjStore : public RGWDeleteCORS {
public:
RGWDeleteCORS_ObjStore() {}
~RGWDeleteCORS_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
};
class RGWOptionsCORS_ObjStore : public RGWOptionsCORS {
public:
RGWOptionsCORS_ObjStore() {}
~RGWOptionsCORS_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); }
};
class RGWGetBucketEncryption_ObjStore : public RGWGetBucketEncryption {
public:
RGWInitMultipart_ObjStore() {}
~RGWInitMultipart_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); }
};
class RGWCompleteMultipart_ObjStore : public RGWCompleteMultipart {
RGWCompleteMultipart_ObjStore() {}
~RGWCompleteMultipart_ObjStore() override {}
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
int get_params(optional_yield y) override;
};
public:
RGWAbortMultipart_ObjStore() {}
~RGWAbortMultipart_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
};
class RGWListMultipart_ObjStore : public RGWListMultipart {
RGWListMultipart_ObjStore() {}
~RGWListMultipart_ObjStore() override {}
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); }
int get_params(optional_yield y) override;
};
RGWListBucketMultiparts_ObjStore() {}
~RGWListBucketMultiparts_ObjStore() override {}
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); }
int get_params(optional_yield y) override;
};
public:
RGWBulkDelete_ObjStore() {}
~RGWBulkDelete_ObjStore() override {}
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BULK_DELETE", s->info.method); }
};
class RGWBulkUploadOp_ObjStore : public RGWBulkUploadOp {
public:
RGWBulkUploadOp_ObjStore() = default;
~RGWBulkUploadOp_ObjStore() = default;
+
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BULK_UPLOAD", s->info.method); }
};
class RGWDeleteMultiObj_ObjStore : public RGWDeleteMultiObj {
~RGWDeleteMultiObj_ObjStore() override {}
int get_params(optional_yield y) override;
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.DELETE_MULTI_OBJECT", s->info.method); }
};
class RGWInfo_ObjStore : public RGWInfo {
--- /dev/null
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab
+
+#include "common/dout.h"
+#include "rgw_op.h"
+#include "rgw_rest.h"
+#include "rgw_rest_s3.h"
+#include "rgw_arn.h"
+#include "rgw_auth_s3.h"
+#include "rgw_url.h"
+#include "rgw_bucket_logging.h"
+#include "rgw_rest_bucket_logging.h"
+
+#define dout_context g_ceph_context
+#define dout_subsys ceph_subsys_rgw
+
+namespace {
+ int verify_bucket_logging_params(const DoutPrefixProvider* dpp, const req_state* s) {
+ bool exists;
+ const auto no_value = s->info.args.get("logging", &exists);
+ if (!exists) {
+ ldpp_dout(dpp, 1) << "ERROR: missing required param 'logging'" << dendl;
+ return -EINVAL;
+ }
+ if (no_value.length() > 0) {
+ ldpp_dout(dpp, 1) << "ERROR: param 'logging' should not have any value" << dendl;
+ return -EINVAL;
+ }
+ if (s->bucket_name.empty()) {
+ ldpp_dout(dpp, 1) << "ERROR: logging request must be on a bucket" << dendl;
+ return -EINVAL;
+ }
+ return 0;
+ }
+}
+
+// GET /<bucket name>/?logging
+// reply is XML encoded
+class RGWGetBucketLoggingOp : public RGWOp {
+ rgw::bucketlogging::configuration configuration;
+
+public:
+ int verify_permission(optional_yield y) override {
+ auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false);
+ if (has_s3_resource_tag)
+ rgw_iam_add_buckettags(this, s);
+
+ if (!verify_bucket_permission(this, s, rgw::IAM::s3GetBucketLogging)) {
+ return -EACCES;
+ }
+
+ return 0;
+ }
+
+ void execute(optional_yield y) override {
+ op_ret = verify_bucket_logging_params(this, s);
+ if (op_ret < 0) {
+ return;
+ }
+
+ std::unique_ptr<rgw::sal::Bucket> bucket;
+ op_ret = driver->load_bucket(this, rgw_bucket(s->bucket_tenant, s->bucket_name),
+ &bucket, y);
+ if (op_ret < 0) {
+ ldpp_dout(this, 1) << "ERROR: failed to get bucket '" <<
+ (s->bucket_tenant.empty() ? s->bucket_name : s->bucket_tenant + ":" + s->bucket_name) <<
+ "' info, ret = " << op_ret << dendl;
+ return;
+ }
+ if (auto iter = bucket->get_attrs().find(RGW_ATTR_BUCKET_LOGGING); iter != bucket->get_attrs().end()) {
+ try {
+ configuration.enabled = true;
+ decode(configuration, iter->second);
+ } catch (buffer::error& err) {
+ ldpp_dout(this, 1) << "ERROR: failed to decode attribute '" << RGW_ATTR_BUCKET_LOGGING
+ << "'. error: " << err.what() << dendl;
+ op_ret = -EIO;
+ return;
+ }
+ } else {
+ ldpp_dout(this, 5) << "WARNING: no logging configuration on bucket '" << bucket->get_name() << "'" << dendl;
+ return;
+ }
+ ldpp_dout(this, 20) << "INFO: found logging configuration on bucket '" << bucket->get_name() << "'"
+ << "'. configuration: " << configuration.to_json_str() << dendl;
+ }
+
+ void send_response() override {
+ dump_errno(s);
+ end_header(s, this, to_mime_type(s->format));
+ dump_start(s);
+
+ s->formatter->open_object_section_in_ns("BucketLoggingStatus", XMLNS_AWS_S3);
+ configuration.dump_xml(s->formatter);
+ s->formatter->close_section();
+ rgw_flush_formatter_and_reset(s, s->formatter);
+ }
+ const char* name() const override { return "get_bucket_logging"; }
+ RGWOpType get_type() override { return RGW_OP_GET_BUCKET_LOGGING; }
+ uint32_t op_mask() override { return RGW_OP_TYPE_READ; }
+};
+
+// PUT /<bucket name>/?logging
+// actual configuration is XML encoded in the body of the message
+class RGWPutBucketLoggingOp : public RGWDefaultResponseOp {
+ int verify_permission(optional_yield y) override {
+ auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false);
+ if (has_s3_resource_tag)
+ rgw_iam_add_buckettags(this, s);
+
+ if (!verify_bucket_permission(this, s, rgw::IAM::s3PutBucketLogging)) {
+ return -EACCES;
+ }
+
+ return 0;
+ }
+
+ const char* name() const override { return "put_bucket_logging"; }
+ RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_LOGGING; }
+ uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; }
+
+ void execute(optional_yield y) override {
+ op_ret = verify_bucket_logging_params(this, s);
+ if (op_ret < 0) {
+ return;
+ }
+
+ const auto max_size = s->cct->_conf->rgw_max_put_param_size;
+ bufferlist data;
+ std::tie(op_ret, data) = read_all_input(s, max_size, false);
+ if (op_ret < 0) {
+ ldpp_dout(this, 1) << "ERROR: failed to read XML logging payload, ret = " << op_ret << dendl;
+ return;
+ }
+ if (data.length() == 0) {
+ ldpp_dout(this, 1) << "ERROR: XML logging payload missing" << dendl;
+ op_ret = -EINVAL;
+ return;
+ }
+
+ RGWXMLDecoder::XMLParser parser;
+ if (!parser.init()){
+ ldpp_dout(this, 1) << "ERROR: failed to initialize XML parser" << dendl;
+ op_ret = -EINVAL;
+ return;
+ }
+ if (!parser.parse(data.c_str(), data.length(), 1)) {
+ ldpp_dout(this, 1) << "ERROR: failed to parse XML logging payload" << dendl;
+ op_ret = -ERR_MALFORMED_XML;
+ return;
+ }
+ rgw::bucketlogging::configuration configuration;
+ configuration.default_obj_roll_time = get_cct()->_conf->rgw_bucket_logging_obj_roll_time;
+ try {
+ RGWXMLDecoder::decode_xml("BucketLoggingStatus", configuration, &parser, true);
+ } catch (RGWXMLDecoder::err& err) {
+ ldpp_dout(this, 1) << "ERROR: failed to parse XML logging payload. error: " << err << dendl;
+ op_ret = -ERR_MALFORMED_XML;
+ return;
+ }
+
+ std::unique_ptr<rgw::sal::Bucket> bucket;
+ op_ret = driver->load_bucket(this, rgw_bucket(s->bucket_tenant, s->bucket_name),
+ &bucket, y);
+ if (op_ret < 0) {
+ ldpp_dout(this, 1) << "ERROR: failed to get bucket '" << s->bucket_name << "', ret = " << op_ret << dendl;
+ return;
+ }
+
+
+ auto& attrs = bucket->get_attrs();
+ if (!configuration.enabled) {
+ if (auto iter = attrs.find(RGW_ATTR_BUCKET_LOGGING); iter != attrs.end()) {
+ attrs.erase(iter);
+ }
+ } else {
+ std::unique_ptr<rgw::sal::Bucket> target_bucket;
+ op_ret = driver->load_bucket(this, rgw_bucket(s->bucket_tenant, configuration.target_bucket),
+ &target_bucket, y);
+ if (op_ret < 0) {
+ ldpp_dout(this, 1) << "ERROR: failed to get target bucket '" << configuration.target_bucket << "', ret = " << op_ret << dendl;
+ return;
+ }
+ const auto& target_attrs = target_bucket->get_attrs();
+ if (target_attrs.find(RGW_ATTR_BUCKET_LOGGING) != target_attrs.end()) {
+ // target bucket must not have logging set on it
+ ldpp_dout(this, 1) << "ERROR: logging target bucket '" << configuration.target_bucket << "', is configured with bucket logging" << dendl;
+ op_ret = -EINVAL;
+ return;
+ }
+ // TODO: verify target bucket does not have encryption
+ bufferlist conf_bl;
+ encode(configuration, conf_bl);
+ attrs[RGW_ATTR_BUCKET_LOGGING] = conf_bl;
+ // TODO: should we add attribute to target bucket indicating it is target to bucket logging?
+ // if we do, how do we maintain it when bucket logging changes?
+ }
+ // TODO: use retry_raced_bucket_write from rgw_op.cc
+ op_ret = bucket->merge_and_store_attrs(this, attrs, y);
+ if (op_ret < 0) {
+ ldpp_dout(this, 1) << "ERROR: failed to set logging attribute '" << RGW_ATTR_BUCKET_LOGGING << "' to bucket '" <<
+ bucket->get_name() << "', ret = " << op_ret << dendl;
+ return;
+ }
+
+ ldpp_dout(this, 20) << "INFO: " << (configuration.enabled ? "wrote" : "removed")
+ << " logging configuration. bucket '" << bucket->get_name() << "'. configuration: " <<
+ configuration.to_json_str() << dendl;
+ }
+};
+
+RGWOp* RGWHandler_REST_BucketLogging_S3::create_put_op() {
+ return new RGWPutBucketLoggingOp();
+}
+
+RGWOp* RGWHandler_REST_BucketLogging_S3::create_get_op() {
+ return new RGWGetBucketLoggingOp();
+}
+
--- /dev/null
+// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*-
+// vim: ts=8 sw=2 smarttab
+#pragma once
+
+#include "rgw_rest_s3.h"
+
+// s3 compliant bucket logging handler factory
+class RGWHandler_REST_BucketLogging_S3 : public RGWHandler_REST_S3 {
+protected:
+ int init_permissions(RGWOp* op, optional_yield y) override {return 0;}
+ int read_permissions(RGWOp* op, optional_yield y) override {return 0;}
+ bool supports_quota() override {return false;}
+public:
+ virtual ~RGWHandler_REST_BucketLogging_S3() = default;
+ static RGWOp* create_get_op();
+ static RGWOp* create_put_op();
+};
+
#include "rgw_role.h"
#include "rgw_rest_sts.h"
#include "rgw_rest_iam.h"
+#include "rgw_rest_bucket_logging.h"
#include "rgw_sts.h"
#include "rgw_sal_rados.h"
#include "rgw_cksum_pipe.h"
rgw_flush_formatter_and_reset(s, s->formatter);
}
-void RGWGetBucketLogging_ObjStore_S3::send_response()
-{
- dump_errno(s);
- end_header(s, this, to_mime_type(s->format));
- dump_start(s);
-
- s->formatter->open_object_section_in_ns("BucketLoggingStatus", XMLNS_AWS_S3);
- s->formatter->close_section();
- rgw_flush_formatter_and_reset(s, s->formatter);
-}
void RGWGetBucketLocation_ObjStore_S3::send_response()
{
return nullptr;
if (s->info.args.sub_resource_exists("logging"))
- return new RGWGetBucketLogging_ObjStore_S3;
+ return RGWHandler_REST_BucketLogging_S3::create_get_op();
if (s->info.args.sub_resource_exists("location"))
return new RGWGetBucketLocation_ObjStore_S3;
RGWOp *RGWHandler_REST_Bucket_S3::op_put()
{
- if (s->info.args.sub_resource_exists("logging") ||
- s->info.args.sub_resource_exists("encryption"))
+ if (s->info.args.sub_resource_exists("encryption"))
return nullptr;
+ if (s->info.args.sub_resource_exists("logging"))
+ return RGWHandler_REST_BucketLogging_S3::create_put_op();
if (s->info.args.sub_resource_exists("versioning"))
return new RGWSetBucketVersioning_ObjStore_S3;
if (s->info.args.sub_resource_exists("website")) {
RGWOp *RGWHandler_REST_Bucket_S3::op_delete()
{
- if (s->info.args.sub_resource_exists("logging") ||
- s->info.args.sub_resource_exists("encryption"))
+ if (s->info.args.sub_resource_exists("encryption"))
return nullptr;
if (is_tagging_op()) {
case RGW_OP_GET_BUCKET_PUBLIC_ACCESS_BLOCK:
case RGW_OP_DELETE_BUCKET_PUBLIC_ACCESS_BLOCK:
case RGW_OP_GET_OBJ://s3select its post-method(payload contain the query) , the request is get-object
+ case RGW_OP_PUT_BUCKET_LOGGING:
+ case RGW_OP_GET_BUCKET_LOGGING:
break;
default:
ldpp_dout(s, 10) << "ERROR: AWS4 completion for operation: " << s->op_type << ", NOT IMPLEMENTED" << dendl;
void send_versioned_response();
};
-class RGWGetBucketLogging_ObjStore_S3 : public RGWGetBucketLogging {
-public:
- RGWGetBucketLogging_ObjStore_S3() {}
- ~RGWGetBucketLogging_ObjStore_S3() override {}
-
- void send_response() override;
-};
-
class RGWGetBucketLocation_ObjStore_S3 : public RGWGetBucketLocation {
public:
RGWGetBucketLocation_ObjStore_S3() {}
~RGWGetBucketWebsite_ObjStore_S3() override {}
void send_response() override;
+ virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); }
};
class RGWSetBucketWebsite_ObjStore_S3 : public RGWSetBucketWebsite {
int get_params(optional_yield y) override;
void send_response() override;
+ virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); }
};
class RGWDeleteBucketWebsite_ObjStore_S3 : public RGWDeleteBucketWebsite {
~RGWDeleteBucketWebsite_ObjStore_S3() override {}
void send_response() override;
+ virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); }
};
class RGWStatBucket_ObjStore_S3 : public RGWStatBucket_ObjStore {
public:
RGWConfigBucketMetaSearch_ObjStore_S3() {}
~RGWConfigBucketMetaSearch_ObjStore_S3() {}
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_MDSEARCH", s->info.method); }
int get_params(optional_yield y) override;
void send_response() override;
public:
RGWDelBucketMetaSearch_ObjStore_S3() {}
~RGWDelBucketMetaSearch_ObjStore_S3() {}
+ virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_MDSEARCH", s->info.method); }
void send_response() override;
};
virtual int remove_topics(RGWObjVersionTracker* objv_tracker,
optional_yield y, const DoutPrefixProvider *dpp) = 0;
+ /** Read the name of the pending bucket logging object name */
+ virtual int get_logging_object_name(std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ RGWObjVersionTracker* objv_tracker) = 0;
+ /** Update the name of the pending bucket logging object name */
+ virtual int set_logging_object_name(const std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ bool new_obj,
+ RGWObjVersionTracker* objv_tracker) = 0;
+ /** Move the pending bucket logging object into the bucket */
+ virtual int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) = 0;
+ /** Write a record to the pending bucket logging object */
+ virtual int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) = 0;
+
/* dang - This is temporary, until the API is completed */
virtual rgw_bucket& get_key() = 0;
virtual RGWBucketInfo& get_info() = 0;
optional_yield y, const DoutPrefixProvider *dpp) override {
return next->remove_topics(objv_tracker, y, dpp);
}
+ int get_logging_object_name(std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ RGWObjVersionTracker* objv_tracker) override {
+ return next->get_logging_object_name(obj_name, prefix, y, dpp, objv_tracker);
+ }
+ int set_logging_object_name(const std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ bool new_obj,
+ RGWObjVersionTracker* objv_track) override {
+ return next->set_logging_object_name(obj_name, prefix, y, dpp, new_obj, objv_track);
+ }
+ int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp)override {
+ return next->commit_logging_object(obj_name, y, dpp);
+ }
+ int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override {
+ return next->write_logging_object(obj_name, record, y, dpp, async_completion);
+ }
virtual rgw_bucket& get_key() override { return next->get_key(); }
virtual RGWBucketInfo& get_info() override { return next->get_info(); }
optional_yield y, const DoutPrefixProvider *dpp) override {return 0;}
int remove_topics(RGWObjVersionTracker* objv_tracker,
optional_yield y, const DoutPrefixProvider *dpp) override {return 0;}
+ int get_logging_object_name(std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ RGWObjVersionTracker* objv_tracker) override { return 0; }
+ int set_logging_object_name(const std::string& obj_name,
+ const std::string& prefix,
+ optional_yield y,
+ const DoutPrefixProvider *dpp,
+ bool new_obj,
+ RGWObjVersionTracker* objv_tracker) override { return 0; }
+ int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override { return 0; }
+ int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override {
+ return 0;
+ }
friend class BucketList;
};
bucket sync disable disable bucket sync
bucket sync enable enable bucket sync
bucket radoslist list rados objects backing bucket's objects
+ bucket logging flush flush pending log records object of source bucket to the log bucket to bucket
bi get retrieve bucket index object entries
bi put store bucket index object entries
bi list list raw bucket index entries
projects / ceph.git / commitdiff