From: Yuval Lifshitz Date: Wed, 25 Sep 2019 08:00:50 +0000 (+0300) Subject: rgw/pubsub: backporting pubsub/notifications to nautilus X-Git-Tag: v14.2.5~243^2~2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=91cb3928a916f4c6ad55904f8631bbc85e86bd97;p=ceph.git rgw/pubsub: backporting pubsub/notifications to nautilus Signed-off-by: Yuval Lifshitz --- diff --git a/doc/radosgw/index.rst b/doc/radosgw/index.rst index 453b56ca777..0523caadb09 100644 --- a/doc/radosgw/index.rst +++ b/doc/radosgw/index.rst @@ -61,6 +61,7 @@ you may write data with one API and retrieve it with the other. Dynamic bucket index resharding Multi factor authentication Sync Modules + Bucket Notifications Data Layout in RADOS STS Lite Role diff --git a/doc/radosgw/notifications.rst b/doc/radosgw/notifications.rst new file mode 100644 index 00000000000..152dc03f83c --- /dev/null +++ b/doc/radosgw/notifications.rst @@ -0,0 +1,291 @@ +==================== +Bucket Notifications +==================== + +.. versionadded:: Nautilus + +.. contents:: + +Bucket notifications provide a mechanism for sending information out of the radosgw when certain events are happening on the bucket. +Currently, notifications could be sent to HTTP and AMQP0.9.1 endpoints. + +Note, that if the events should be stored in Ceph, in addition, or instead of being pushed to an endpoint, +the `PubSub Module`_ should be used instead of the bucket notification mechanism. + +A user can create different topics. A topic entity is defined by its user and its name. A +user can only manage its own topics, and can only associate them with buckets it owns. + +In order to send notifications for events for a specific bucket, a notification entity needs to be created. A +notification can be created on a subset of event types, or for all event types (default). +The notification may also filter out events based on preffix/suffix and/or regular expression matching of the keys. As well as, +on the metadata attributes attached to the object. +There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications. + +REST API has been defined to provide configuration and control interfaces for the bucket notification +mechanism. This API is similar to the one defined as S3-compatible API of the pubsub sync module. + +.. toctree:: + :maxdepth: 1 + + S3 Bucket Notification Compatibility + +Notificatios Performance Stats +------------------------------ +Same counters are shared between the pubsub sync module and the bucket notification mechanism. + +- ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them +- ``pubsub_event_lost``: running counter of events that had topics associated with them but that were not pushed to any of the endpoints +- ``pubsub_push_ok``: running counter, for all notifications, of events successfully pushed to their endpoint +- ``pubsub_push_fail``: running counter, for all notifications, of events failed to be pushed to their endpoint +- ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet + +.. note:: + + ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while: + ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per push action on each notification. + +Bucket Notification REST API +---------------------------- + +Topics +~~~~~~ + +Create a Topic +`````````````` + +This will create a new topic. The topic should be provided with push endpoint parameters that would be used later +when a notification is created. +Upon successful request, the response will include the topic ARN that could be later used to reference this topic in the notification request. +To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values. + +.. tip:: Any notification already associated with the topic needs to be re-created for the topic update to take effect + +:: + + POST + Action=CreateTopic + &Name= + &push-endpoint= + [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=] + [&Attributes.entry.2.key=amqp-sck-level&Attributes.entry.2.value=ack-level] + &Attributes.entry.3.key=verify-sll&Attributes.entry.3.value=true|false] + +Request parameters: + +- push-endpoint: URI of endpoint to send push notification to + + - URI schema is: ``http[s]|amqp://[:@][:][/]`` + - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) + - Default values for HTTP/S: no user/password, port 80/443 + - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" + +- verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) +- amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics +- amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist: + + - "none" - message is considered "delivered" if sent to broker + - "broker" message is considered "delivered" if acked by broker + +.. note:: + + - The key/value of a specific parameter does not have to reside in the same line, or in any specific order, but must use the same index + - Attribute indexing does not need to be sequntial or start from any specific value + - `AWS Create Topic`_ has detailed explanation on endpoint attributes format. However, in our case different keys and values are used + +The response will have the following format: + +:: + + + + + + + + + + +The topic ARN in the response will have the following format: + +:: + + arn:aws:sns::: + +Get Topic Information +````````````````````` + +Returns information about specific topic. This includes push-endpoint information, if provided. + +:: + + POST + Action=GetTopic&TopicArn= + +Response will have the following format: + +:: + + + + + + + + + + + + + + + + + + + +- User: name of the user that created the topic +- Name: name of the topic +- EndPoinjtAddress: the push-endpoint URL +- EndPointArgs: the push-endpoint args +- EndpointTopic: the topic name that should be sent to the endpoint (mat be different than the above topic name) +- TopicArn: topic ARN + +Delete Topic +```````````` + +:: + + POST + Action=DeleteTopic&TopicArn= + +Delete the specified topic. Note that deleting a deleted topic should result with no-op and not a failure. + +The response will have the following format: + +:: + + + + + + + +List Topics +``````````` + +List all topics that user defined. + +:: + + POST + Action=ListTopics + +Response will have the following format: + +:: + + + + + + + + + + + + + + + + + + + + + +Notifications +~~~~~~~~~~~~~ + +Detailed under: `Bucket Operations`_. + +.. note:: + + - "Abort Multipart Upload" request does not emit a notification + - "Delete Multiple Objects" request does not emit a notification + - Both "Initiate Multipart Upload" and "POST Object" requests will emit an ``s3:ObjectCreated:Post`` notification + + +Events +~~~~~~ + +The events are in JSON format (regardless of the actual endpoint), and share the same structure as the S3-compatible events +pushed or pulled using the pubsub sync module. + +:: + + {"Records":[ + { + "eventVersion":"2.1" + "eventSource":"aws:s3", + "awsRegion":"", + "eventTime":"", + "eventName":"", + "userIdentity":{ + "principalId":"" + }, + "requestParameters":{ + "sourceIPAddress":"" + }, + "responseElements":{ + "x-amz-request-id":"", + "x-amz-id-2":"" + }, + "s3":{ + "s3SchemaVersion":"1.0", + "configurationId":"", + "bucket":{ + "name":"", + "ownerIdentity":{ + "principalId":"" + }, + "arn":"", + "id:"" + }, + "object":{ + "key":"", + "size":"", + "eTag":"", + "versionId":"", + "sequencer": "", + "metadata":"" + } + }, + "eventId":"", + } + ]} + +- awsRegion: zonegroup +- eventTime: timestamp indicating when the event was triggered +- eventName: for list of supported events see: `S3 Notification Compatibility`_ +- userIdentity.principalId: user that triggered the change +- requestParameters.sourceIPAddress: not supported +- responseElements.x-amz-request-id: request ID of the original change +- responseElements.x_amz_id_2: RGW on which the change was made +- s3.configurationId: notification ID that created the event +- s3.bucket.name: name of the bucket +- s3.bucket.ownerIdentity.principalId: owner of the bucket +- s3.bucket.arn: ARN of the bucket +- s3.bucket.id: Id of the bucket (an extension to the S3 notification API) +- s3.object.key: object key +- s3.object.size: object size +- s3.object.eTag: object etag +- s3.object.version: object version in case of versioned bucket +- s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format) +- s3.object.metadata: any metadata set on the object sent as: ``x-amz-meta-`` (an extension to the S3 notification API) +- s3.eventId: not supported (an extension to the S3 notification API) + +.. _PubSub Module : ../pubsub-module +.. _S3 Notification Compatibility: ../s3-notification-compatibility +.. _AWS Create Topic: https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html +.. _Bucket Operations: ../s3/bucketops diff --git a/doc/radosgw/pubsub-module.rst b/doc/radosgw/pubsub-module.rst index 796b18d3631..9be303b8572 100644 --- a/doc/radosgw/pubsub-module.rst +++ b/doc/radosgw/pubsub-module.rst @@ -1,49 +1,74 @@ -========================= +================== PubSub Sync Module -========================= +================== .. versionadded:: Nautilus +.. contents:: + This sync module provides a publish and subscribe mechanism for the object store modification -events. Events are published into defined topics. Topics can be subscribed to, and events +events. Events are published into predefined topics. Topics can be subscribed to, and events can be pulled from them. Events need to be acked. Also, events will expire and disappear -after a period of time. A push notification mechanism exists too, currently supporting HTTP and -AMQP0.9.1 endpoints. +after a period of time. + +A push notification mechanism exists too, currently supporting HTTP and +AMQP0.9.1 endpoints, on top of storing the events in Ceph. If events should only be pushed to an endpoint +and do not need to be stored in Ceph, the `Bucket Notification`_ mechanism should be used instead of pubsub sync module. A user can create different topics. A topic entity is defined by its user and its name. A user can only manage its own topics, and can only subscribe to events published by buckets it owns. -In order to publish events for specific bucket a notification needs to be created. A -notification can be created only on subset of event types, or for all event types (default). -There can be multiple notifications for any specific topic. +In order to publish events for specific bucket a notification entity needs to be created. A +notification can be created on a subset of event types, or for all event types (default). +There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications. A subscription to a topic can also be defined. There can be multiple subscriptions for any specific topic. -A new REST api has been defined to provide configuration and control interfaces for the pubsub -mechanisms. +REST API has been defined to provide configuration and control interfaces for the pubsub +mechanisms. This API has two flavors, one is S3-compatible and one is not. The two flavors can be used +together, although it is recommended to use the S3-compatible one. +The S3-compatible API is similar to the one used in the bucket notification mechanism. -Events are stored as rgw objects in a special bucket, under a special user. Events cannot -be accessed directly, but need to be pulled and acked using the new REST api. +Events are stored as RGW objects in a special bucket, under a special user. Events cannot +be accessed directly, but need to be pulled and acked using the new REST API. +.. toctree:: + :maxdepth: 1 + S3 Bucket Notification Compatibility + +PubSub Zone Configuration +------------------------- -PubSub Tier Type Configuration -------------------------------------- +The pubsub sync module requires the creation of a new zone in a `Multisite`_ environment. +First, a master zone must exist (see: :ref:`master-zone-label`), +then a secondary zone should be created (see :ref:`secondary-zone-label`). +In the creation of the secondary zone, its tier type must be set to ``pubsub``: :: - { - "tenant": , # default: - "uid": , # default: "pubsub" - "data_bucket_prefix": # default: "pubsub-" - "data_oid_prefix": # + # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --endpoints={http://fqdn}[,{http://fqdn}] \ + --sync-from-all=0 \ + --sync-from={master-zone-name} \ + --tier-type=pubsub - "events_retention_days": # default: 7 - } +PubSub Zone Configuration Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + { + "tenant": , # default: + "uid": , # default: "pubsub" + "data_bucket_prefix": # default: "pubsub-" + "data_oid_prefix": # + "events_retention_days": # default: 7 + } * ``tenant`` (string) @@ -65,82 +90,129 @@ The oid prefix for the stored events. How many days to keep events that weren't acked. -How to Configure -~~~~~~~~~~~~~~~~ - -See `Multisite Configuration`_ for how to multisite config instructions. The pubsub sync module requires a creation of a new zone. The zone -tier type needs to be defined as ``pubsub``: - -:: - - # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ - --rgw-zone={zone-name} \ - --endpoints={http://fqdn}[,{http://fqdn}] - --tier-type=pubsub - +Configuring Parameters via CLI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The tier configuration can be then done using the following command +The tier configuration could be set using the following command: :: - # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ + # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ --rgw-zone={zone-name} \ --tier-config={key}={val}[,{key}={val}] -The ``key`` in the configuration specifies the config variable that needs to be updated, and -the ``val`` specifies its new value. Nested values can be accessed using period. For example: +Where the ``key`` in the configuration specifies the configuration variable that needs to be updated (from the list above), and +the ``val`` specifies its new value. For example, setting the pubsub control user ``uid`` to ``user_ps``: :: - # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ + # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ --rgw-zone={zone-name} \ --tier-config=uid=pubsub - A configuration field can be removed by using ``--tier-config-rm={key}``. PubSub Performance Stats ------------------------- -- **pubsub_event_triggered**: running counter of events with at lease one pubsub topic associated with them -- **pubsub_event_lost**: running counter of events that had pubsub topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions -- **pubsub_store_ok**: running counter, for all subscriptions, of stored pubsub events -- **pubsub_store_fail**: running counter, for all subscriptions, of pubsub events that needed to be stored but failed -- **pubsub_push_ok**: running counter, for all subscriptions, of pubsub events successfully pushed to their endpoint -- **pubsub_push_fail**: running counter, for all subscriptions, of pubsub events failed to be pushed to their endpoint -- **pubsub_push_pending**: gauge value of pubsub events pushed to a endpoined but not acked or nacked yet +Same counters are shared between the pubsub sync module and the notification mechanism. + +- ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them +- ``pubsub_event_lost``: running counter of events that had topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions +- ``pubsub_store_ok``: running counter, for all subscriptions, of stored events +- ``pubsub_store_fail``: running counter, for all subscriptions, of events failed to be stored +- ``pubsub_push_ok``: running counter, for all subscriptions, of events successfully pushed to their endpoint +- ``pubsub_push_fail``: running counter, for all subscriptions, of events failed to be pushed to their endpoint +- ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet -Note that **pubsub_event_triggered** and **pubsub_event_lost** are incremented per event, while: **pubsub_store_ok**, **pubsub_store_fail**, **pubsub_push_ok**, **pubsub_push_fail**, are incremented per store/push action on each subscriptions. +.. note:: + + ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while: + ``pubsub_store_ok``, ``pubsub_store_fail``, ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per store/push action on each subscriptions. PubSub REST API -------------------------- +--------------- +.. tip:: PubSub REST calls, and only them, should be sent to an RGW which belong to a PubSub zone Topics ~~~~~~ - + Create a Topic -`````````````````````````` +`````````````` + +This will create a new topic. Topic creation is needed both for both flavors of the API. +Optionally the topic could be provided with push endpoint parameters that would be used later +when an S3-compatible notification is created. +Upon successful request, the response will include the topic ARN that could be later used to reference this topic in an S3-compatible notification request. +To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values. -This will create a new topic. +.. tip:: Any S3-compatible notification already associated with the topic needs to be re-created for the topic update to take effect :: - PUT /topics/ + PUT /topics/[?push-endpoint=[&amqp-exchange=][&amqp-ack-level=][&verify-ssl=true|false]] + +Request parameters: + +- push-endpoint: URI of endpoint to send push notification to + + - URI schema is: ``http[s]|amqp://[:@][:][/]`` + - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) + - Default values for HTTP/S: no user/password, port 80/443 + - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" + +- verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) +- amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics +- amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist: + + - "none" - message is considered "delivered" if sent to broker + - "broker" message is considered "delivered" if acked by broker + +The topic ARN in the response will have the following format: +:: + + arn:aws:sns::: Get Topic Information -```````````````````````````````` +````````````````````` -Returns information about specific topic. This includes subscriptions to that topic. +Returns information about specific topic. This includes subscriptions to that topic, and push-endpoint information, if provided. :: GET /topics/ +Response will have the following format (JSON): + +:: + { + "topic":{ + "user":"", + "name":"", + "dest":{ + "bucket_name":"", + "oid_prefix":"", + "push_endpoint":"", + "push_endpoint_args":"" + }, + "arn":"" + }, + "subs":[] + } + +- topic.user: name of the user that created the topic +- name: name of the topic +- dest.bucket_name: not used +- dest.oid_prefix: not used +- dest.push_endpoint: in case of S3-compliant notifications, this value will be used as the push-endpoint URL +- dest.push_endpoint_args: in case of S3-compliant notifications, this value will be used as the push-endpoint args +- topic.arn: topic ARN +- subs: list of subscriptions associated with this topic Delete Topic -```````````````````````````````````` +```````````` :: @@ -149,21 +221,35 @@ Delete Topic Delete the specified topic. List Topics -```````````````````````````````````` +``````````` List all topics that user defined. :: GET /topics + +S3-Compliant Notifications +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Detailed under: `Bucket Operations`_. +.. note:: + + - Notification creation will also create a subscription for pushing/pulling events + - The generated subscription's name will have the same as the notification Id, and could be used later to fetch and ack events with the subscription API. + - Notification deletion will deletes all generated subscriptions + - In case that bucket deletion implicitly deletes the notification, + the associated subscription will not be deleted automatically (any events of the deleted bucket could still be access), + and will have to be deleted explicitly with the subscription deletion API + - Filtering based on metadata (which is an extension to S3) is not supported, and such rules will be ignored -Notifications -~~~~~~~~~~~~~ + +Non S3-Compliant Notifications +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a Notification -`````````````````````````` +````````````````````` This will create a publisher for a specific bucket into a topic. @@ -171,15 +257,13 @@ This will create a publisher for a specific bucket into a topic. PUT /notifications/bucket/?topic=[&events=[,]] +Request parameters: -Request Params: - - topic-name: name of topic - - event: event type (string), one of: OBJECT_CREATE, OBJECT_DELETE - - - +- topic-name: name of topic +- event: event type (string), one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE`` + Delete Notification Information -```````````````````````````````` +``````````````````````````````` Delete publisher from a specific bucket into a specific topic. @@ -187,13 +271,47 @@ Delete publisher from a specific bucket into a specific topic. DELETE /notifications/bucket/?topic= -Request Params: - - topic-name: name of topic +Request parameters: + +- topic-name: name of topic + +.. note:: When the bucket is deleted, any notification defined on it is also deleted + +List Notifications +`````````````````` + +List all topics with associated events defined on a bucket. + +:: + + GET /notifications/bucket/ + +Response will have the following format (JSON): + +:: + {"topics":[ + { + "topic":{ + "user":"", + "name":"", + "dest":{ + "bucket_name":"", + "oid_prefix":"", + "push_endpoint":"", + "push_endpoint_args":"" + } + "arn":"" + }, + "events":[] + } + ]} +Subscriptions +~~~~~~~~~~~~~ -Create Subscription -```````````````````````````````````` +Create a Subscription +````````````````````` Creates a new subscription. @@ -201,61 +319,192 @@ Creates a new subscription. PUT /subscriptions/?topic=[&push-endpoint=[&amqp-exchange=][&amqp-ack-level=][&verify-ssl=true|false]] -Request Params: +Request parameters: - - topic-name: name of topic - - push-endpoint: URI of endpoint to send push notification to +- topic-name: name of topic +- push-endpoint: URI of endpoint to send push notification to - - URI schema is: ``http|amqp://[:@][:][/]`` - - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) - - Default values for HTTP: no user/password, port 80 - - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" + - URI schema is: ``http[s]|amqp://[:@][:][/]`` + - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) + - Default values for HTTP/S: no user/password, port 80/443 + - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" - - verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) - - amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics - - amqp-ack-level: 2 ack levels exist: "none" - message is considered "delivered" if sent to broker; - "broker" message is considered "delivered" if acked by broker. - No end2end acking is required, as messages may persist in the broker before delivered into their final destination +- verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) +- amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics +- amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist: -Get Subscription Info -```````````````````````````````````` + - "none": message is considered "delivered" if sent to broker + - "broker": message is considered "delivered" if acked by broker -Returns info about specific subscription +Get Subscription Information +```````````````````````````` + +Returns information about specific subscription. :: GET /subscriptions/ +Response will have the following format (JSON): + +:: + + { + "user":"", + "name":"", + "topic":"", + "dest":{ + "bucket_name":"", + "oid_prefix":"", + "push_endpoint":"", + "push_endpoint_args":"" + } + "s3_id":"" + } + +- user: name of the user that created the subscription +- name: name of the subscription +- topic: name of the topic the subscription is associated with Delete Subscription -````````````````````````````````` +``````````````````` -Removes a subscription +Removes a subscription. :: DELETE /subscriptions/ - Events ~~~~~~ Pull Events -````````````````````````````````` +``````````` -Pull events sent to a specific subscription +Pull events sent to a specific subscription. :: GET /subscriptions/?events[&max-entries=][&marker=] -Request Params: - - marker: pagination marker for list of events, if not specified will start from the oldest - - max-entries: max number of events to return +Request parameters: + +- marker: pagination marker for list of events, if not specified will start from the oldest +- max-entries: max number of events to return + +The response will hold information on the current marker and whether there are more events not fetched: + +:: + + {"next_marker":"","is_truncated":"",...} + + +The actual content of the response is depended with how the subscription was created. +In case that the subscription was created via an S3-compatible notification, +the events will have an S3-compatible record format (JSON): + +:: + + {"Records":[ + { + "eventVersion":"2.1" + "eventSource":"aws:s3", + "awsRegion":"", + "eventTime":"", + "eventName":"", + "userIdentity":{ + "principalId":"" + }, + "requestParameters":{ + "sourceIPAddress":"" + }, + "responseElements":{ + "x-amz-request-id":"", + "x-amz-id-2":"" + }, + "s3":{ + "s3SchemaVersion":"1.0", + "configurationId":"", + "bucket":{ + "name":"", + "ownerIdentity":{ + "principalId":"" + }, + "arn":"", + "id":"" + }, + "object":{ + "key":"", + "size":"0", + "eTag":"", + "versionId":"", + "sequencer":"", + "metadata":"" + } + }, + "eventId":"", + } + ]} + +- awsRegion: zonegroup +- eventTime: timestamp indicating when the event was triggered +- eventName: either ``s3:ObjectCreated:``, or ``s3:ObjectRemoved:`` +- userIdentity: not supported +- requestParameters: not supported +- responseElements: not supported +- s3.configurationId: notification ID that created the subscription for the event +- s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API) +- s3.bucket.name: name of the bucket +- s3.bucket.ownerIdentity.principalId: owner of the bucket +- s3.bucket.arn: ARN of the bucket +- s3.bucket.id: Id of the bucket (an extension to the S3 notification API) +- s3.object.key: object key +- s3.object.size: not supported +- s3.object.eTag: object etag +- s3.object.version: object version in case of versioned bucket +- s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format) +- s3.object.metadata: not supported (an extension to the S3 notification API) +- s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API) + +In case that the subscription was not created via a non S3-compatible notification, +the events will have the following event format (JSON): + +:: + {"events":[ + { + "id":"", + "event":"", + "timestamp":"", + "info":{ + "attrs":{ + "mtime":"" + }, + "bucket":{ + "bucket_id":"", + "name":"", + "tenant":"" + }, + "key":{ + "instance":"", + "name":"" + } + } + } + ]} + +- id: unique ID of the event, that could be used for acking +- event: one of: ``OBJECT_CREATE``, ``OBJECT_DELETE``, ``DELETE_MARKER_CREATE`` +- timestamp: timestamp indicating when the event was sent +- info.attrs.mtime: timestamp indicating when the event was triggered +- info.bucket.bucket_id: id of the bucket +- info.bucket.name: name of the bucket +- info.bucket.tenant: tenant the bucket belongs to +- info.key.instance: object version in case of versioned bucket +- info.key.name: object key Ack Event -````````````````````````````````` +````````` Ack event so that it can be removed from the subscription history. @@ -263,8 +512,10 @@ Ack event so that it can be removed from the subscription history. POST /subscriptions/?ack&event-id= +Request parameters: -Request Params: - - event-id: id of event to be acked +- event-id: id of event to be acked -.. _Multisite Configuration: ./multisite.rst +.. _Multisite : ../multisite +.. _Bucket Notification : ../notifications +.. _Bucket Operations: ../s3/bucketops diff --git a/doc/radosgw/s3-notification-compatibility.rst b/doc/radosgw/s3-notification-compatibility.rst new file mode 100644 index 00000000000..6cc6ac0283f --- /dev/null +++ b/doc/radosgw/s3-notification-compatibility.rst @@ -0,0 +1,122 @@ +===================================== +S3 Bucket Notifications Compatibility +===================================== + +Ceph's `Bucket Notifications`_ and `PubSub Module`_ APIs follow `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below. + + +.. note:: + + Compatibility is different depending on which of the above mechanism is used + +Supported Destination +--------------------- + +AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations). +Currently, we support: **HTTP/S** and **AMQP**. And also support pulling and acking of events stored in Ceph (as an intenal destination). + +We are using the **SNS** ARNs to represent the **HTTP/S** and **AMQP** destinations. + +Notification Configuration XML +------------------------------ + +Following tags (and the tags inside them) are not supported: + ++-----------------------------------+----------------------------------------------+ +| Tag | Remaks | ++===================================+==============================================+ +| ```` | not needed, we treat all destinations as SNS | ++-----------------------------------+----------------------------------------------+ +| ```` | not needed, we treat all destinations as SNS | ++-----------------------------------+----------------------------------------------+ + +REST API Extension +------------------ + +Ceph's bucket notification API has the following extensions: + +- Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb + + - In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket + +- Getting the information on a specific notification (when more than one exists on a bucket) + + - In S3, it is only possible to fetch all notifications on a bucket + +- In addition to filtering based on prefix/suffix of object keys we support: + + - Filtering based on regular expression matching + + - Filtering based on metadata attributes attached to the object + +- Filtering overlapping is allowed, so that same event could be sent as different notification + + +Unsupported Fields in the Event Record +-------------------------------------- + +The records sent for bucket notification follow format described in: `Event Message Structure`_. +However, the following fields may be sent empty, under the different deployment options (Notification/PubSub): + ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| Field | Notification | PubSub | Description | ++========================================+==============+===============+============================================================+ +| ``userIdentity.principalId`` | Supported | Not Supported | The identity of the user that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.sourceIPAddress`` | Not Supported | The IP address of the client that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.x-amz-request-id`` | Supported | Not Supported | The request id that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.x-amz-id-2`` | Supported | Not Supported | The IP address of the RGW on which the event was triggered | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``s3.object.size`` | Supported | Not Supported | The size of the object | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ + +Event Types +----------- + ++----------------------------------------------+-----------------+-------------------------------------------+ +| Event | Notification | PubSub | ++==============================================+=================+===========================================+ +| ``s3:ObjectCreated:*`` | Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:Put`` | Supported | Supported at ``s3:ObjectCreated:*`` level | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:Post`` | Supported | Not Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:Copy`` | Supported | Supported at ``s3:ObjectCreated:*`` level | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectCreated:CompleteMultipartUpload`` | Supported | Supported at ``s3:ObjectCreated:*`` level | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRemoved:*`` | Supported | Supported only the specific events below | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRemoved:Delete`` | Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRemoved:DeleteMarkerCreated`` | Supported | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRestore:Post`` | Not applicable to Ceph | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ObjectRestore:Complete`` | Not applicable to Ceph | ++----------------------------------------------+-----------------+-------------------------------------------+ +| ``s3:ReducedRedundancyLostObject`` | Not applicable to Ceph | ++----------------------------------------------+-----------------+-------------------------------------------+ + +Topic Configuration +------------------- +In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_. +Note that most of the API is not applicable to Ceph, and only the following actions are implemented: + + - ``CreateTopic`` + - ``DeleteTopic`` + - ``ListTopics`` + +We also extend it by: + + - ``GetTopic`` - allowing for fetching a specific topic, instead of all user topics + - In ``CreateTopic`` we allow setting endpoint attributes + +.. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html +.. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html +.. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html +.. _`PubSub Module`: ../pubsub-module +.. _`Bucket Notifications`: ../notifications diff --git a/doc/radosgw/s3.rst b/doc/radosgw/s3.rst index cf6eaba7fa4..36cc898ac44 100644 --- a/doc/radosgw/s3.rst +++ b/doc/radosgw/s3.rst @@ -48,7 +48,7 @@ The following table describes the support status for current Amazon S3 functiona +---------------------------------+-----------------+----------------------------------------+ | **Bucket Location** | Supported | | +---------------------------------+-----------------+----------------------------------------+ -| **Bucket Notification** | Not Supported | | +| **Bucket Notification** | Supported | See `S3 Notification Compatibility`_ | +---------------------------------+-----------------+----------------------------------------+ | **Bucket Object Versions** | Supported | | +---------------------------------+-----------------+----------------------------------------+ @@ -98,3 +98,4 @@ The following common request header fields are not supported: +----------------------------+------------+ .. _Amazon S3 API: http://docs.aws.amazon.com/AmazonS3/latest/API/APIRest.html +.. _S3 Notification Compatibility: ../s3-notification-compatibility diff --git a/doc/radosgw/s3/bucketops.rst b/doc/radosgw/s3/bucketops.rst index 01e5eb4bfdd..4fc23ab133b 100644 --- a/doc/radosgw/s3/bucketops.rst +++ b/doc/radosgw/s3/bucketops.rst @@ -386,3 +386,213 @@ REQUEST ENTITIES +-----------------------------+-----------+---------------------------------------------------------------------------+ | ``Status`` | String | Sets the versioning state of the bucket. Valid Values: Suspended/Enabled | +-----------------------------+-----------+---------------------------------------------------------------------------+ + + +Create Notification +------------------- + +Create a publisher for a specific bucket into a topic. + +Syntax +~~~~~~ + +:: + + PUT /?notification HTTP/1.1 + + +Request Entities +~~~~~~~~~~~~~~~~ + +Parameters are XML encoded in the body of the request, in the following format: + +:: + + + + + + + + + + + + + + + + + + + + + + + ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++===============================+===========+======================================================================================+==========+ +| ``NotificationConfiguration`` | Container | Holding list of ``TopicConfiguration`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TopicConfiguration`` | Container | Holding ``Id``, ``Topic`` and list of ``Event`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Id`` | String | Name of the notification | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Topic`` | String | Topic ARN. Topic must be created beforehand | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Event`` | String | List of supported events see: `S3 Notification Compatibility`_. Multiple ``Event`` | No | +| | | entities can be used. If omitted, all events are handled | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Filter`` | Container | Holding ``S3Key`` and ``S3Metadata`` entities | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Key`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object key. | No | +| | | At most, 3 entities may be in the list, with ``Name`` be ``prefix``, ``suffix`` or | | +| | | ``regex``. All filter rules in the list must match for the filter to match. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Metadata`` | Container | Holding a list of ``FilterRule`` entities, for filtering based on object metadata. | No | +| | | All filter rules in the list must match the ones defined on the object. The object, | | +| | | have other metadata entitied not listed in the filter. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Key.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be: ``prefix``, ``suffix`` | Yes | +| | | or ``regex``. The ``Value`` would hold the key prefix, key suffix or a regular | | +| | | expression for matching the key, accordingly. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``S3Metadata.FilterRule`` | Container | Holding ``Name`` and ``Value`` entities. ``Name`` would be the name of the metadata | Yes | +| | | attribute (e.g. ``x-amz-meta-xxx``). The ``Value`` would be the expected value for | | +| | | this attribute | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``400`` | MalformedXML | The XML is not well-formed | ++---------------+-----------------------+----------------------------------------------------------+ +| ``400`` | InvalidArgument | Missing Id; Missing/Invalid Topic ARN; Invalid Event | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchKey | The topic does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + + +Delete Notification +------------------- + +Delete a specific, or all, notifications from a bucket. + +.. note:: + + - Notification deletion is an extension to the S3 notification API + - When the bucket is deleted, any notification defined on it is also deleted + - Deleting an unkown notification (e.g. double delete) is not considered an error + +Syntax +~~~~~~ + +:: + + DELETE /bucket?notification[=] HTTP/1.1 + + +Parameters +~~~~~~~~~~ + ++------------------------+-----------+----------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+========================================================================================+ +| ``notification-id`` | String | Name of the notification. If not provided, all notifications on the bucket are deleted | ++------------------------+-----------+----------------------------------------------------------------------------------------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + +Get/List Notification +--------------------- + +Get a specific notification, or list all notifications configured on a bucket. + +Syntax +~~~~~~ + +:: + + GET /bucket?notification[=] HTTP/1.1 + + +Parameters +~~~~~~~~~~ + ++------------------------+-----------+----------------------------------------------------------------------------------------+ +| Name | Type | Description | ++========================+===========+========================================================================================+ +| ``notification-id`` | String | Name of the notification. If not provided, all notifications on the bucket are listed | ++------------------------+-----------+----------------------------------------------------------------------------------------+ + +Response Entities +~~~~~~~~~~~~~~~~~ + +Response is XML encoded in the body of the request, in the following format: + +:: + + + + + + + + + + + + + + + + + + + + + + + ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++===============================+===========+======================================================================================+==========+ +| ``NotificationConfiguration`` | Container | Holding list of ``TopicConfiguration`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TopicConfiguration`` | Container | Holding ``Id``, ``Topic`` and list of ``Event`` entities | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Id`` | String | Name of the notification | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Topic`` | String | Topic ARN | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Event`` | String | Handled event. Multiple ``Event`` entities may exist | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``Filter`` | Container | Holding the filters configured for this notification | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchKey | The notification does not exist (if provided) | ++---------------+-----------------------+----------------------------------------------------------+ + +.. _S3 Notification Compatibility: ../s3-notification-compatibility diff --git a/qa/suites/rgw/multisite/realms/three-zone-plus-pubsub.yaml b/qa/suites/rgw/multisite/realms/three-zone-plus-pubsub.yaml new file mode 100644 index 00000000000..e77e5ade409 --- /dev/null +++ b/qa/suites/rgw/multisite/realms/three-zone-plus-pubsub.yaml @@ -0,0 +1,23 @@ +overrides: + rgw-multisite: + realm: + name: test-realm + is default: true + zonegroups: + - name: test-zonegroup + is_master: true + is_default: true + endpoints: [c1.client.0] + zones: + - name: test-zone1 + is_master: true + is_default: true + endpoints: [c1.client.0] + - name: test-zone2 + is_default: true + endpoints: [c2.client.0] + - name: test-zone3 + endpoints: [c1.client.1] + - name: test-zone4 + endpoints: [c2.client.1] + is_pubsub: true diff --git a/qa/tasks/rgw_multisite.py b/qa/tasks/rgw_multisite.py index a41238daa04..9dea39312de 100644 --- a/qa/tasks/rgw_multisite.py +++ b/qa/tasks/rgw_multisite.py @@ -11,6 +11,7 @@ from util.rgw import rgwadmin, wait_for_radosgw from util.rados import create_ec_pool, create_replicated_pool from rgw_multi import multisite from rgw_multi.zone_rados import RadosZone as RadosZone +from rgw_multi.zone_ps import PSZone as PSZone from teuthology.orchestra import run from teuthology import misc @@ -33,6 +34,7 @@ class RGWMultisite(Task): * 'is_master' is passed on the command line as --master * 'is_default' is passed on the command line as --default + * 'is_pubsub' is used to create a zone with tier-type=pubsub * 'endpoints' given as client names are replaced with actual endpoints zonegroups: @@ -78,6 +80,9 @@ class RGWMultisite(Task): - name: test-zone2 is_default: true endpoints: [c2.client.0] + - name: test-zone3 + is_pubsub: true + endpoints: [c1.client.1] """ def __init__(self, ctx, config): @@ -369,7 +374,10 @@ def create_zonegroup(cluster, gateways, period, config): def create_zone(ctx, cluster, gateways, creds, zonegroup, config): """ create a zone with the given configuration """ zone = multisite.Zone(config['name'], zonegroup, cluster) - zone = RadosZone(config['name'], zonegroup, cluster) + if config.pop('is_pubsub', False): + zone = PSZone(config['name'], zonegroup, cluster) + else: + zone = RadosZone(config['name'], zonegroup, cluster) # collect Gateways for the zone's endpoints endpoints = config.get('endpoints') diff --git a/qa/tasks/rgw_multisite_tests.py b/qa/tasks/rgw_multisite_tests.py index 4e6e2b3dff0..dade6e47483 100644 --- a/qa/tasks/rgw_multisite_tests.py +++ b/qa/tasks/rgw_multisite_tests.py @@ -10,10 +10,11 @@ from teuthology.exceptions import ConfigError from teuthology.task import Task from teuthology import misc -from rgw_multi import multisite, tests +from rgw_multi import multisite, tests, tests_ps log = logging.getLogger(__name__) + class RGWMultisiteTests(Task): """ Runs the rgw_multi tests against a multisite configuration created by the @@ -63,9 +64,16 @@ class RGWMultisiteTests(Task): # run nose tests in the rgw_multi.tests module conf = nose.config.Config(stream=get_log_stream(), verbosity=2) + error_msg = '' result = nose.run(defaultTest=tests.__name__, argv=argv, config=conf) if not result: - raise RuntimeError('rgw multisite test failures') + error_msg += 'rgw multisite, ' + result = nose.run(defaultTest=tests_ps.__name__, argv=argv, config=conf) + if not result: + error_msg += 'rgw multisite pubsub, ' + if error_msg: + raise RuntimeError(error_msg + 'test failures') + def get_log_stream(): """ return a log stream for nose output """ @@ -88,4 +96,5 @@ def get_log_stream(): return LogStream() + task = RGWMultisiteTests diff --git a/src/common/options.cc b/src/common/options.cc index 4d42dfedc97..79e25836814 100644 --- a/src/common/options.cc +++ b/src/common/options.cc @@ -5641,7 +5641,7 @@ std::vector