From c345566831c72446c8250f630337825e526847a8 Mon Sep 17 00:00:00 2001 From: Yuval Lifshitz Date: Thu, 22 Aug 2019 16:31:01 +0300 Subject: [PATCH] rgw/pubsub: push notifications documentation Signed-off-by: Yuval Lifshitz --- doc/radosgw/index.rst | 1 + doc/radosgw/notifications.rst | 358 ++++++++++++++++++ doc/radosgw/pubsub-module.rst | 50 ++- doc/radosgw/s3-notification-compatibility.rst | 90 +++-- src/test/rgw/rgw_multi/tests_ps.py | 2 +- 5 files changed, 448 insertions(+), 53 deletions(-) create mode 100644 doc/radosgw/notifications.rst diff --git a/doc/radosgw/index.rst b/doc/radosgw/index.rst index 15701f9ed7d07..06f9df536a9ad 100644 --- a/doc/radosgw/index.rst +++ b/doc/radosgw/index.rst @@ -61,6 +61,7 @@ you may write data with one API and retrieve it with the other. Dynamic bucket index resharding Multi factor authentication Sync Modules + Bucket Notifications Data Layout in RADOS STS Lite Role diff --git a/doc/radosgw/notifications.rst b/doc/radosgw/notifications.rst new file mode 100644 index 0000000000000..9f4f68e5c92b0 --- /dev/null +++ b/doc/radosgw/notifications.rst @@ -0,0 +1,358 @@ +==================== +Bucket Notifications +==================== + +.. versionadded:: Nautilus + +.. contents:: + +Bucket notifications provide a mechanism for sending information out of the radosgw when certain events are happening on the bucket. +Currently, notifications could be sent to HTTP and AMQP0.9.1 endpoints. + +Note, that if the events should be stored in Ceph, in addition, or instead of being pushed to an endpoint, +the `PubSub Module`_ should be used instead of the bucket notification mechanism. + +A user can create different topics. A topic entity is defined by its user and its name. A +user can only manage its own topics, and can only associate them with buckets it owns. + +In order to send notifications for events for a specific bucket, a notification ientity needs to be created. A +notification can be created on a subset of event types, or for all event types (default). +There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications. + +REST API has been defined to provide configuration and control interfaces for the bucket notification +mechanism. This API is similar to the one defined as S3-compatible API of the pubsub sync module. + +.. toctree:: + :maxdepth: 1 + + S3 Bucket Notification Compatibility + +Notificatios Performance Stats +------------------------------ +Same counters are shared between the pubsub sync module and the bucket notification mechanism. + +- ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them +- ``pubsub_event_lost``: running counter of events that had topics associated with them but that were not pushed to any of the endpoints +- ``pubsub_push_ok``: running counter, for all notifications, of events successfully pushed to their endpoint +- ``pubsub_push_fail``: running counter, for all notifications, of events failed to be pushed to their endpoint +- ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet + +.. note:: + + ``pubsub_event_triggered`` and ``pubsub_event_lost`` are incremented per event, while: + ``pubsub_push_ok``, ``pubsub_push_fail``, are incremented per push action on each notification. + +Bucket Notification REST API +---------------------------- + +Topics +~~~~~~ + +Create a Topic +`````````````` + +This will create a new topic. The topic should be provided with push endpoint parameters that would be used later +when a notification is created. +Upon successful request, the response will include the topic ARN that could be later used to reference this topic in the notification request. +To update a topic, use the same command used for topic creation, with the topic name of an existing topic and different endpoint values. + +.. tip:: Any notification already associated with the topic needs to be re-created for the topic update to take effect + +:: + + POST + Action=CreateTopic + &Name= + &push-endpoint= + [&Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=] + [&Attributes.entry.2.key=amqp-sck-level&Attributes.entry.2.value=ack-level] + &Attributes.entry.3.key=verify-sll&Attributes.entry.3.value=true|false] + +Request parameters: + +- push-endpoint: URI of endpoint to send push notification to + + - URI schema is: ``http[s]|amqp://[:@][:][/]`` + - Same schema is used for HTTP and AMQP endpoints (except amqp-vhost which is specific to AMQP) + - Default values for HTTP/S: no user/password, port 80/443 + - Default values for AMQP: user/password=guest/guest, port 5672, amqp-vhost is "/" + +- verify-ssl: can be used with https endpoints (ignored for other endpoints), indicate whether the server certificate is validated or not ("true" by default) +- amqp-exchange: mandatory parameter for AMQP endpoint. The exchanges must exist and be able to route messages based on topics +- amqp-ack-level: No end2end acking is required, as messages may persist in the broker before delivered into their final destination. 2 ack methods exist: + + - "none" - message is considered "delivered" if sent to broker + - "broker" message is considered "delivered" if acked by broker + +.. note:: + + - The key/value of a specific parameter does not have to reside in the same line, or in any specific order, but must use the same index + - Attribute indexing does not need to be sequntial or start from any specific value + - `AWS Create Topic`_ has detailed explanation on endpoint attributes format. However, in our case different keys and values are used + +The response will have the following format: + +:: + + + + + + + + + + +The topic ARN in the response will have the following format: + +:: + + arn:aws:sns::: + +Get Topic Information +````````````````````` + +Returns information about specific topic. This includes push-endpoint information, if provided. + +:: + + POST + Action=GetTopic&TopicArn= + +Response will have the following format: + +:: + + + + + + + + + + + + + + + + + + + +- User: name of the user that created the topic +- Name: name of the topic +- EndPoinjtAddress: the push-endpoint URL +- EndPointArgs: the push-endpoint args +- EndpointTopic: the topic name that should be sent to the endpoint (mat be different than the above topic name) +- TopicArn: topic ARN + +Delete Topic +```````````` + +:: + + POST + Action=DeleteTopic&TopicArn= + +Delete the specified topic. Note that deleting a deleted topic should result with no-op and not a failure. + +The response will have the following format: + +:: + + + + + + + +List Topics +``````````` + +List all topics that user defined. + +:: + + POST + Action=ListTopics + +Response will have the following format: + +:: + + + + + + + + + + + + + + + + + + + + + +Notifications +~~~~~~~~~~~~~ + +Create a Notification +````````````````````` + +This will create a publisher for a specific bucket into a topic. + +:: + + PUT /?notification + +Request parameters are encoded in XML in the body of the request, with the following format: + +:: + + + + + + + + + +- Id: name of the notification +- Topic: topic ARN +- Event: either ``s3:ObjectCreated:*``, or ``s3:ObjectRemoved:*`` (multiple ``Event`` tags may be used) + +Delete Notification +``````````````````` + +Delete a specific, or all, notifications from a bucket. + +:: + + DELETE /bucket?notification[=] + +Request parameters: + +- notification-id: name of the notification (if not provided, all notifications on the bucket are deleted) + +.. note:: + + - Notification deletion is an extension to the S3 notification API + - When the bucket is deleted, any notification defined on it is also deleted + +Get/List Notifications +`````````````````````` + +Get a specific notification, or list all notifications defined on a bucket. + +:: + + GET /bucket?notification[=] + +Request parameters: + +- notification-id: name of the notification (if not provided, all notifications on the bucket are listed) + +Response is XML formatted: + +:: + + + + + + + + + +- Id: name of the notification +- Topic: topic ARN +- Event: for list of supported events see: `S3 Notification Compatibility`_ (to support multiple types, any combination of wildcard events and ``Event`` tags may be used) + + +.. note:: + + - Getting information on a specific notification is an extension to the S3 notification API + - When multiple notifications are fetched from the bucket, multiple ``NotificationConfiguration`` tags will be used + + +Events +~~~~~~ + +The events are in JSON format (regardless of the actual endpoint), and share the same structure as the S3-compatible events +pushed or pulled using the pubsub sync module. + +:: + + {"Records":[ + { + "eventVersion":"2.1" + "eventSource":"aws:s3", + "awsRegion":"", + "eventTime":"", + "eventName":"", + "userIdentity":{ + "principalId":"" + }, + "requestParameters":{ + "sourceIPAddress":"" + }, + "responseElements":{ + "x-amz-request-id":"", + "x-amz-id-2":"" + }, + "s3":{ + "s3SchemaVersion":"1.0", + "configurationId":"", + "bucket":{ + "name":"", + "ownerIdentity":{ + "principalId":"" + }, + "arn":"", + "id:"" + }, + "object":{ + "key":"", + "size":"", + "eTag":"", + "versionId":"", + "sequencer": "", + "metadata":"" + } + }, + "eventId":"", + } + ]} + +- awsRegion: zonegroup +- eventTime: timestamp indicating when the event was triggered +- eventName: for list of supported events see: `S3 Notification Compatibility`_ +- userIdentity.principalId: user that triggered the change +- requestParameters.sourceIPAddress: not supported +- responseElements.x-amz-request-id: request ID of the original change +- responseElements.x_amz_id_2: RGW on which the change was made +- s3.configurationId: notification ID that created the event +- s3.bucket.name: name of the bucket +- s3.bucket.ownerIdentity.principalId: owner of the bucket +- s3.bucket.arn: ARN of the bucket +- s3.bucket.id: Id of the bucket (an extension to the S3 notification API) +- s3.object.key: object key +- s3.object.size: object size +- s3.object.eTag: object etag +- s3.object.version: object version in case of versioned bucket +- s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format) +- s3.object.metadata: any metadata set on the object sent as: ``x-amz-meta-`` (an extension to the S3 notification API) +- s3.eventId: not supported (an extension to the S3 notification API) + +.. _PubSub Module : ../pubsub-module +.. _S3 Notification Compatibility: ../s3-notification-compatibility +.. _AWS Create Topic: https://docs.aws.amazon.com/sns/latest/api/API_CreateTopic.html diff --git a/doc/radosgw/pubsub-module.rst b/doc/radosgw/pubsub-module.rst index ccf6617fe8b66..7dc5b0e913501 100644 --- a/doc/radosgw/pubsub-module.rst +++ b/doc/radosgw/pubsub-module.rst @@ -9,16 +9,19 @@ PubSub Sync Module This sync module provides a publish and subscribe mechanism for the object store modification events. Events are published into predefined topics. Topics can be subscribed to, and events can be pulled from them. Events need to be acked. Also, events will expire and disappear -after a period of time. A push notification mechanism exists too, currently supporting HTTP and -AMQP0.9.1 endpoints. +after a period of time. + +A push notification mechanism exists too, currently supporting HTTP and +AMQP0.9.1 endpoints, on top of storing the events in Ceph. If events should only be pushed to an endpoint +and do not need to be stored in Ceph, the `Bucket Notification`_ mechanism should be used instead of pubsub sync module. A user can create different topics. A topic entity is defined by its user and its name. A user can only manage its own topics, and can only subscribe to events published by buckets it owns. -In order to publish events for specific bucket a notification needs to be created. A -notification can be created only on subset of event types, or for all event types (default). -There can be multiple notifications for any specific topic. +In order to publish events for specific bucket a notification entity needs to be created. A +notification can be created on a subset of event types, or for all event types (default). +There can be multiple notifications for any specific topic, and the same topic could be used for multiple notifications. A subscription to a topic can also be defined. There can be multiple subscriptions for any specific topic. @@ -26,6 +29,7 @@ specific topic. REST API has been defined to provide configuration and control interfaces for the pubsub mechanisms. This API has two flavors, one is S3-compatible and one is not. The two flavors can be used together, although it is recommended to use the S3-compatible one. +The S3-compatible API is similar to the one used in the bucket notification mechanism. Events are stored as RGW objects in a special bucket, under a special user. Events cannot be accessed directly, but need to be pulled and acked using the new REST API. @@ -110,13 +114,15 @@ A configuration field can be removed by using ``--tier-config-rm={key}``. PubSub Performance Stats ------------------------- -- ``pubsub_event_triggered``: running counter of events with at lease one pubsub topic associated with them -- ``pubsub_event_lost``: running counter of events that had pubsub topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions -- ``pubsub_store_ok``: running counter, for all subscriptions, of stored pubsub events -- ``pubsub_store_fail``: running counter, for all subscriptions, of pubsub events that needed to be stored but failed -- ``pubsub_push_ok``: running counter, for all subscriptions, of pubsub events successfully pushed to their endpoint -- ``pubsub_push_fail``: running counter, for all subscriptions, of pubsub events failed to be pushed to their endpoint -- ``pubsub_push_pending``: gauge value of pubsub events pushed to a endpoint but not acked or nacked yet +Same counters are shared between the pubsub sync module and the notification mechanism. + +- ``pubsub_event_triggered``: running counter of events with at lease one topic associated with them +- ``pubsub_event_lost``: running counter of events that had topics and subscriptions associated with them but that were not stored or pushed to any of the subscriptions +- ``pubsub_store_ok``: running counter, for all subscriptions, of stored events +- ``pubsub_store_fail``: running counter, for all subscriptions, of events failed to be stored +- ``pubsub_push_ok``: running counter, for all subscriptions, of events successfully pushed to their endpoint +- ``pubsub_push_fail``: running counter, for all subscriptions, of events failed to be pushed to their endpoint +- ``pubsub_push_pending``: gauge value of events pushed to an endpoint but not acked or nacked yet .. note:: @@ -130,7 +136,7 @@ PubSub REST API Topics ~~~~~~ - + Create a Topic `````````````` @@ -222,7 +228,7 @@ List all topics that user defined. :: GET /topics - + S3-Compliant Notifications ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -325,7 +331,7 @@ Request parameters: - topic-name: name of topic - event: event type (string), one of: OBJECT_CREATE, OBJECT_DELETE - + Delete Notification Information ``````````````````````````````` @@ -369,7 +375,7 @@ Response will have the following format (JSON): }, "events":[] } - ]} + ]} Subscriptions ~~~~~~~~~~~~~ @@ -494,14 +500,16 @@ the events will have an S3-compatible record format (JSON): "ownerIdentity":{ "principalId":"" }, - "arn":"" + "arn":"", + "id":"" }, "object":{ "key":"", - "size": , + "size":"0", "eTag":"", "versionId":"", - "sequencer": "" + "sequencer":"", + "metadata":"" } }, "eventId":"", @@ -519,11 +527,14 @@ the events will have an S3-compatible record format (JSON): - s3.bucket.name: name of the bucket - s3.bucket.ownerIdentity.principalId: owner of the bucket - s3.bucket.arn: ARN of the bucket +- s3.bucket.id: Id of the bucket (an extension to the S3 notification API) - s3.object.key: object key - s3.object.size: not supported - s3.object.eTag: object etag - s3.object.version: object version in case of versioned bucket - s3.object.sequencer: monotonically increasing identifier of the change per object (hexadecimal format) +- s3.object.metadata: not supported (an extension to the S3 notification API) +- s3.eventId: unique ID of the event, that could be used for acking (an extension to the S3 notification API) In case that the subscription was not created via a non S3-compatible notification, the events will have the following event format (JSON): @@ -576,3 +587,4 @@ Request parameters: - event-id: id of event to be acked .. _Multisite : ../multisite +.. _Bucket Notification : ../notifications diff --git a/doc/radosgw/s3-notification-compatibility.rst b/doc/radosgw/s3-notification-compatibility.rst index 57e163c8faba1..fe016a27b9e50 100644 --- a/doc/radosgw/s3-notification-compatibility.rst +++ b/doc/radosgw/s3-notification-compatibility.rst @@ -2,15 +2,20 @@ S3 Bucket Notifications Compatibility ===================================== -Ceph's `PubSub module`_ follows `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below. +Ceph's `Bucket Notifications`_ and `PubSub Module`_ APIs follow `AWS S3 Bucket Notifications API`_. However, some differences exist, as listed below. + + +.. note:: + + Compatibility is different depending on which of the above mechanism is used Supported Destination ----------------------- +--------------------- AWS supports: **SNS**, **SQS** and **Lambda** as possible destinations (AWS internal destinations). Currently, we support: **HTTP/S** and **AMQP**. And also support pulling and acking of events stored in Ceph (as an intenal destination). -We are using the SNS ARNs to represent these destinations. +We are using the **SNS** ARNs to represent the **HTTP/S** and **AMQP** destinations. Notification Configuration XML ------------------------------ @@ -27,66 +32,85 @@ Following tags (and the tags inside them) are not supported: | ```` | object filtering not supported | +-----------------------------------+----------------------------------------------+ -REST API Extension +REST API Extension ------------------ -Ceph's bucket notification API follows has the following extensions: +Ceph's bucket notification API has the following extensions: - Deletion of a specific notification, or all notifications on a bucket, using the ``DELETE`` verb - In S3, all notifications are deleted when the bucket is deleted, or when an empty notification is set on the bucket -- Getting the information on a specific notification (when some exists on a bucket) +- Getting the information on a specific notification (when more than one exists on a bucket) + + - In S3, it is only possible to fetch all notifications on a bucket Unsupported Fields in the Event Record -------------------------------------- The records sent for bucket notification follow format described in: `Event Message Structure`_. -However, the following fields are sent empty: - -+----------------------------------------+-------------------------------------------------------------+ -| Field | Description | -+========================================+=============================================================+ -| ``userIdentity.principalId`` | The identity of the user that triggered the event | -+----------------------------------------+-------------------------------------------------------------+ -| ``requestParameters.sourceIPAddress`` | The IP address of the client that triggered the event | -+----------------------------------------+-------------------------------------------------------------+ -| ``requestParameters.x-amz-request-id`` | The request id that triggered the event | -+----------------------------------------+-------------------------------------------------------------+ -| ``requestParameters.x-amz-id-2`` | The IP address of the RGW on which the event was triggered | -+----------------------------------------+-------------------------------------------------------------+ -| ``s3.object.size`` | The size of the object | -+----------------------------------------+-------------------------------------------------------------+ +However, the following fields may be sent empty, under the different deployment options (Notification/PubSub): + ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| Field | Notification | PubSub | Description | ++========================================+==============+===============+============================================================+ +| ``userIdentity.principalId`` | Supported | Not Supported | The identity of the user that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.sourceIPAddress`` | Not Supported | The IP address of the client that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.x-amz-request-id`` | Supported | Not Supported | The request id that triggered the event | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``requestParameters.x-amz-id-2`` | Supported | Not Supported | The IP address of the RGW on which the event was triggered | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ +| ``s3.object.size`` | Supported | Not Supported | The size of the object | ++----------------------------------------+--------------+---------------+------------------------------------------------------------+ Event Types ----------- +----------------------------------------------+-----------------+-------------------------------------------+ -| Event | Status | Remarks | +| Event | Notification | PubSub | +==============================================+=================+===========================================+ -| ``s3:ObjectCreated:*`` | Supported | | +| ``s3:ObjectCreated:*`` | Supported | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectCreated:Put`` | Supported | supported at ``s3:ObjectCreated:*`` level | +| ``s3:ObjectCreated:Put`` | Supported | Supported at ``s3:ObjectCreated:*`` level | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectCreated:Post`` | Not Supported | start of multi-part upload not supported | +| ``s3:ObjectCreated:Post`` | Supported | Not Supported | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectCreated:Copy`` | Supported | supported at ``s3:ObjectCreated:*`` level | +| ``s3:ObjectCreated:Copy`` | Supported | Supported at ``s3:ObjectCreated:*`` level | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectCreated:CompleteMultipartUpload`` | Supported | supported at ``s3:ObjectCreated:*`` level | +| ``s3:ObjectCreated:CompleteMultipartUpload`` | Not Supported | Supported at ``s3:ObjectCreated:*`` level | +| (an extension to AWS) | | | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectRemoved:*`` | Supported | | +| ``s3:ObjectRemoved:*`` | Supported | +----------------------------------------------+-----------------+-------------------------------------------+ | ``s3:ObjectRemoved:Delete`` | Supported | supported at ``s3:ObjectRemoved:*`` level | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectRemoved:DeleteMarkerCreated`` | Supported | supported at ``s3:ObjectRemoved:*`` level | +| ``s3:ObjectRemoved:DeleteMarkerCreated`` | Supported at ``s3:ObjectRemoved:*`` level | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectRestore:Post`` | Not Supported | not applicable to Ceph | +| ``s3:ObjectRestore:Post`` | Not applicable to Ceph | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ObjectRestore:Complete`` | Not Supported | not applicable to Ceph | +| ``s3:ObjectRestore:Complete`` | Not applicable to Ceph | +----------------------------------------------+-----------------+-------------------------------------------+ -| ``s3:ReducedRedundancyLostObject`` | Not Supported | not applicable to Ceph | +| ``s3:ReducedRedundancyLostObject`` | Not applicable to Ceph | +----------------------------------------------+-----------------+-------------------------------------------+ +Topic Configuration +------------------- +In the case of bucket notifications, the topics management API will be derived from `AWS Simple Notification Service API`_. +Note that most of the API is not applicable to Ceph, and only the following actions are implemented: + + - ``CreateTopic`` + - ``DeleteTopic`` + - ``ListTopics`` + +We also extend it by: + + - ``GetTopic`` - allowing for fetching a specific topic, instead of all user topics + - In ``CreateTopic`` we allow setting endpoint attributes + +.. _AWS Simple Notification Service API: https://docs.aws.amazon.com/sns/latest/api/API_Operations.html .. _AWS S3 Bucket Notifications API: https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html .. _Event Message Structure: https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html -.. _`PubSub module`: ../pubsub-module +.. _`PubSub Module`: ../pubsub-module +.. _`Bucket Notifications`: ../notifications diff --git a/src/test/rgw/rgw_multi/tests_ps.py b/src/test/rgw/rgw_multi/tests_ps.py index 5918efb6f7f14..786a1c6617e76 100644 --- a/src/test/rgw/rgw_multi/tests_ps.py +++ b/src/test/rgw/rgw_multi/tests_ps.py @@ -28,7 +28,7 @@ from nose.tools import assert_not_equal, assert_equal # configure logging for the tests module log = logging.getLogger(__name__) -skip_push_tests = False +skip_push_tests = True #################################### # utility functions for pubsub tests -- 2.39.5