archive sync module requires the creation of a new zone. The zone tier type needs
to be defined as ``archive``:
-::
+.. prompt:: bash #
+
+ radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
+ --rgw-zone={zone-name} \
+ --endpoints={http://fqdn}[,{http://fqdn}] \
+ --tier-type=archive
+
- # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
- --rgw-zone={zone-name} \
- --endpoints={http://fqdn}[,{http://fqdn}]
- --tier-type=archive
.. _Multisite Configuration: ../multisite
-===============
- Cloud Restore
-===============
+=============
+Cloud Restore
+=============
The :doc:`cloud-transition>` feature makes it possible to transition objects to a remote
cloud service. The ``cloud-restore`` feature described below enables restoration
the ``retain_head_object`` option should be set to ``true``
in the ``tier-config`` when configuring the storage class.
-Objects can be restored using the `S3 RestoreObject <https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html>`
-PI . The restored copies will be retained within RGW only for the number
+Objects can be restored using the `S3 RestoreObject <https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html>`_
+API. The restored copies will be retained within RGW only for the number
of ``days`` specified. However if ``days`` is not provided, the restored copies
are considered permanent and will be treated as regular objects.
In addition, by enabling the ``allow_read_through`` option,
-the `S3 GetObject <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html>`
+the `S3 GetObject <https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html>`_
API can be used to restore the object temporarily.
Cloud Storage Class Tier Configuration
--------------------------------------
-The `tier configuration <https://docs.ceph.com/en/latest/radosgw/cloud-transition/#cloud-storage-class-configuration>`
+The `tier configuration <https://docs.ceph.com/en/latest/radosgw/cloud-transition/#cloud-storage-class-configuration>`_
of the cloud storage class configured for data transition is used to restore
-objects as well:
+objects as well::
-```
{
"access_key": <access>,
"secret": <secret>,`
"multipart_min_part_size": {part_size},
"retain_head_object": <true | false>
}
-```
The below options have been added to the tier configuration to facilitate object restoration.
The storage class to which object data is to be restored. Default value is ``STANDARD``.
-read-through specific Configurables:
+Read-through Specific Configurables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* ``allow_read_through`` (``true`` | ``false``)
For example:
-```
- # radosgw-admin zonegroup placement modify --rgw-zonegroup default \
- --placement-id default-placement \
- --storage-class CLOUDTIER \
- --tier-config=endpoint=http://XX.XX.XX.XX:YY,\
- access_key=<access_key>,secret=<secret>, \
- retain_head_object=true, \
- restore_storage_class=COLDTIER, \
- allow_read_through=true, \
- read_through_restore_days=10
-```
+.. prompt:: bash #
+ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
+ --placement-id default-placement \
+ --storage-class CLOUDTIER \
+ --tier-config=endpoint=http://XX.XX.XX.XX:YY,\
+ access_key=<access_key>,secret=<secret>, \
+ retain_head_object=true, \
+ restore_storage_class=COLDTIER, \
+ allow_read_through=true, \
+ read_through_restore_days=10
-S3 Glacier specific Configurables:
+
+
+S3 Glacier Specific Configurables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To restore objects archived in an S3 Glacier or Tape cloud storage class, the
For example:
-```
- # radosgw-admin zonegroup placement add --rgw-zonegroup=default \
- --placement-id=default-placement \
- --storage-class=CLOUDTIER-GLACIER --tier-type=cloud-s3-glacier
-
- # radosgw-admin zonegroup placement modify --rgw-zonegroup default \
- --placement-id default-placement \
- --storage-class CLOUDTIER \
- --tier-config=endpoint=http://XX.XX.XX.XX:YY,\
- access_key=XXXXX,secret=YYYYY, \
- retain_head_object=true, \
- target_storage_class=Glacier, \
- ............
- ............
- restore_storage_class=COLDTIER, \
- glacier_restore_days=2, \
- glacier_restore_tier_type=Expedited
-
-
- [
+.. prompt:: bash #
+
+ radosgw-admin zonegroup placement add --rgw-zonegroup=default \
+ --placement-id=default-placement \
+ --storage-class=CLOUDTIER-GLACIER --tier-type=cloud-s3-glacier
+ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
+ --placement-id default-placement \
+ --storage-class CLOUDTIER \
+ --tier-config=endpoint=http://XX.XX.XX.XX:YY,\
+ access_key=XXXXX,secret=YYYYY, \
+ retain_head_object=true, \
+ target_storage_class=Glacier, \
+ ............ \
+ ............ \
+ restore_storage_class=COLDTIER, \
+ glacier_restore_days=2, \
+ glacier_restore_tier_type=Expedited
+
+::
+
+ [
{
"key": "default-placement",
"val": {
}
}
]
-```
-Examples of Restore Objects:
-----------------------------
+Examples of Restore Objects
+---------------------------
Using the S3 RestoreObject CLI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Th `S3 restore-object <https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html>`
+Th `S3 restore-object <https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html>`_
CLI supports these options:
-```
- $ aws s3api restore-object
- --bucket <value>
- --key <value>
- [--version-id <value>]
- --restore-request (structure) {
- Days=<integer>
- }
-```
+.. prompt:: bash $
+
+ aws s3api restore-object --bucket <value> \
+ --key <value> \
+ [--version-id <value>] \
+ --restore-request (structure) { \
+ Days=<integer> \
+ }
+
Note: ``Days`` is optional and if not provided, the object is restored permanently.
Example 1:
-```
- $ aws s3api restore-object --bucket bucket1 --key doc1.rtf
- [--version-id 3sL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo]
- --restore-request Days=10
+.. prompt:: bash $
+
+ aws s3api restore-object --bucket bucket1 --key doc1.rtf \
+ [--version-id 3sL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo] \
+ --restore-request Days=10 \
....
-```
+
This will restore the object ``doc1.rtf`` at an optional version,
for the duration of 10 days.
Example 2:
-```
- $ aws s3api restore-object --bucket bucket1 --key doc2.rtf --restore-request {} ....
-```
+.. prompt:: bash $
+
+ aws s3api restore-object --bucket bucket1 --key doc2.rtf --restore-request {} ....
This will restore the object ``doc2.rtf`` permanently and it will be treated as regular object.
Example 3:
-```
- $ aws s3api get-object --bucket bucket1 --key doc3.rtf ....
-```
+.. prompt:: bash $
+
+ aws s3api get-object --bucket bucket1 --key doc3.rtf ....
+
This will restore the object `doc3.rtf`` for ``read_through_restore_days`` days.
You can verify the restore status before reissuing the command.
-Verifying the restoration status
+Verifying the Restoration Status
--------------------------------
Verify the status of the restoration by issuing
-an `S3 HeadObject <https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_ResponseSyntax>`
+an `S3 HeadObject <https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_ResponseSyntax>`_
request. The response includes the ``x-amz-restore`` header if object restoration
is in progress or a copy of it is already restored.
Example:
-```
- $ aws s3api head-object --key doc1.rtf --bucket bucket1 ....
-```
+.. prompt:: bash $
+
+ aws s3api head-object --key doc1.rtf --bucket bucket1 ....
+
The ``radosgw-admin`` CLI can be used to check restoration status and other
details.
Example:
-```
- $ radosgw-admin object stat --bucket bucket1 --object doc1.rtf
-```
+.. prompt:: bash $
+
+ radosgw-admin object stat --bucket bucket1 --object doc1.rtf
+
Restored Object Properties
--------------------------
Storage
-~~~~~~
+~~~~~~~
Objects are restored to the storage class configured via ``restore_storage_class``
in the tier-config. However, as
-per `<https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html>`
+per `<https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html>`_
the storage class of restored objects should remain unchanged. Therefore, for
-temporary copies, the ```x-amz-storage-class``` will continue to reflect the
+temporary copies, the ``x-amz-storage-class`` will continue to reflect the
original cloud-tier storage class.
mtime
-~~~~
+~~~~~
The ``mtime`` of the transitioned and restored objects should remain unchanged.
Lifecycle
-~~~~~~~~
+~~~~~~~~~
``Temporary`` copies are not subject to transition to the cloud. However, as is the
case with cloud-transitioned objects, they can be deleted via regular LC (Life Cycle)
expiration rules or an external S3 ``delete`` request.
Replication
-~~~~~~~~~~
+~~~~~~~~~~~
``Temporary`` copies are not replicated and will be retained only by the zone
on which the restore request is initiated.
Versioned Objects
-~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~
For versioned objects, if an object has been cloud-transitioned, it is in a
non-current state. After a restore, the same non-current object will be
updated with the downloaded data, and its ``HEAD`` object will be modified accordingly.
-=========================
+=================
Cloud Sync Module
-=========================
+=================
.. versionadded:: Mimic
Cloud Sync Tier Type Configuration
--------------------------------------
+----------------------------------
-Trivial Configuration:
-~~~~~~~~~~~~~~~~~~~~~~
+Trivial Configuration
+~~~~~~~~~~~~~~~~~~~~~
::
}
-Non Trivial Configuration:
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Non Trivial Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~
::
{
"default": {
"connection": {
-
"access_key": <access>,
- "secret": <secret>,
- "endpoint": <endpoint>,
-
"host_style" <path | virtual>,
+ "access_key": <access>,
+ "secret": <secret>,
+ "endpoint": <endpoint>,
+ "host_style" <path | virtual>,
},
"acls": [
- {
- "type" : <id | email | uri>, # optional, default is id
- "source_id": <id>,
- "dest_id": <id>
- } ... ]
+ {
+ "type" : <id | email | uri>, # optional, default is id
+ "source_id": <id>,
+ "dest_id": <id>
+ } ... ],
"target_path": <path> # optional
},
"connections": [
"type": <id | email | uri>,
"source_id": <id>,
"dest_id": <id>
- } ... ]
+ } ... ],
}
],
"profiles": [
{
- "source_bucket": <source>,
- "connection_id": <connection_id>,
- "acls_id": <mappings_id>,
- "target_path": <dest>, # optional
+ "source_bucket": <source>,
+ "connection_id": <connection_id>,
+ "acls_id": <mappings_id>,
+ "target_path": <dest>, # optional
} ... ],
}
A string that defines how the target path is created. The target path specifies a prefix to which
the source object name is appended. The target path configurable can include any of the following
variables:
-- ``sid``: unique string that represents the sync instance ID
-- ``zonegroup``: the zonegroup name
-- ``zonegroup_id``: the zonegroup ID
-- ``zone``: the zone name
-- ``zone_id``: the zone id
-- ``bucket``: source bucket name
-- ``owner``: source bucket owner ID
+
+* ``sid``: unique string that represents the sync instance ID
+* ``zonegroup``: the zonegroup name
+* ``zonegroup_id``: the zonegroup ID
+* ``zone``: the zone name
+* ``zone_id``: the zone id
+* ``bucket``: source bucket name
+* ``owner``: source bucket owner ID
For example: ``target_path = rgwx-${zone}-${sid}/${owner}/${bucket}``
* ``profiles`` (array)
A list of profiles. Each profile contains the following:
-- ``source_bucket``: either a bucket name, or a bucket prefix (if ends with ``*``) that defines the source bucket(s) for this profile
-- ``target_path``: as defined above
-- ``connection_id``: ID of the connection that will be used for this profile
-- ``acls_id``: ID of ACLs profile that will be used for this profile
+* ``source_bucket``: either a bucket name, or a bucket prefix (if ends with ``*``) that defines the source bucket(s) for this profile
+* ``target_path``: as defined above
+* ``connection_id``: ID of the connection that will be used for this profile
+* ``acls_id``: ID of ACLs profile that will be used for this profile
-S3 Specific Configurables:
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+S3 Specific Configurables
+~~~~~~~~~~~~~~~~~~~~~~~~~
Currently cloud sync will only work with backends that are compatible with AWS S3. There are
a few configurables that can be used to tweak its behavior when accessing these cloud services:
See :ref:`multisite` for how to multisite config instructions. The cloud sync module requires a creation of a new zone. The zone
tier type needs to be defined as ``cloud``:
-::
+.. prompt:: bash #
- # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
- --rgw-zone={zone-name} \
- --endpoints={http://fqdn}[,{http://fqdn}]
- --tier-type=cloud
+ radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
+ --rgw-zone={zone-name} \
+ --endpoints={http://fqdn}[,{http://fqdn}] \
+ --tier-type=cloud
The tier configuration can be then done using the following command
-::
+.. prompt:: bash #
+
+ radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
+ --rgw-zone={zone-name} \
+ --tier-config={key}={val}[,{key}={val}]
- # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
- --rgw-zone={zone-name} \
- --tier-config={key}={val}[,{key}={val}]
The ``key`` in the configuration specifies the config variable that needs to be updated, and
the ``val`` specifies its new value. Nested values can be accessed using period. For example:
-::
+.. prompt:: bash #
- # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
- --rgw-zone={zone-name} \
- --tier-config=connection.access_key={key},connection.secret={secret}
+ radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
+ --rgw-zone={zone-name} \
+ --tier-config=connection.access_key={key},connection.secret={secret}
Configuration array entries can be accessed by specifying the specific entry to be referenced enclosed
again at the same command.
For example, creating a new profile for buckets starting with {prefix}:
-::
+.. prompt:: bash #
- # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
- --rgw-zone={zone-name} \
- --tier-config=profiles[].source_bucket={prefix}'*'
+ radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
+ --rgw-zone={zone-name} \
+ --tier-config=profiles[].source_bucket={prefix}'*'
- # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
- --rgw-zone={zone-name} \
- --tier-config=profiles[-1].connection_id={conn_id},profiles[-1].acls_id={acls_id}
+ radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \
+ --rgw-zone={zone-name} \
+ --tier-config=profiles[-1].connection_id={conn_id},profiles[-1].acls_id={acls_id}
An entry can be removed by using ``--tier-config-rm={key}``.
-==================
- Cloud Transition
-==================
+================
+Cloud Transition
+================
This feature enables transitioning S3 objects to a remote cloud service as part
of `object lifecycle <https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html>`_
the zone/zonegroup changes will take effect once the changes are
committed with ``radosgw-admin period update --commit``.
-::
+.. prompt:: bash #
- $ radosgw-admin zonegroup placement add --rgw-zonegroup={zone-group-name} \
- --placement-id={placement-id} \
- --storage-class={storage-class-name} \
- --tier-type=cloud-s3
+ radosgw-admin zonegroup placement add --rgw-zonegroup={zone-group-name} \
+ --placement-id={placement-id} \
+ --storage-class={storage-class-name} \
+ --tier-type=cloud-s3
-For example::
+For example
+
+.. prompt:: bash #
+
+ radosgw-admin zonegroup placement add --rgw-zonegroup=default \
+ --placement-id=default-placement \
+ --storage-class=CLOUDTIER --tier-type=cloud-s3
+
+::
- $ radosgw-admin zonegroup placement add --rgw-zonegroup=default \
- --placement-id=default-placement \
- --storage-class=CLOUDTIER --tier-type=cloud-s3
[
{
"key": "default-placement",
The tier configuration can be then performed using the following command::
- $ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
- --placement-id={placement-id} \
- --storage-class={storage-class-name} \
- --tier-config={key}={val}[,{key}={val}]
+.. prompt:: bash #
+
+ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
+ --placement-id={placement-id} \
+ --storage-class={storage-class-name} \
+ --tier-config={key}={val}[,{key}={val}]
The ``key`` in the configuration specifies the config variable to be updated, and
the ``val`` specifies its new value.
For example::
- $ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
- --placement-id default-placement \
- --storage-class CLOUDTIER \
- --tier-config=endpoint=http://XX.XX.XX.XX:YY,\
- access_key=<access_key>,secret=<secret>, \
- multipart_sync_threshold=44432, \
- multipart_min_part_size=44432, \
- retain_head_object=true
+.. prompt:: bash #
+
+ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
+ --placement-id default-placement \
+ --storage-class CLOUDTIER \
+ --tier-config=endpoint=http://XX.XX.XX.XX:YY,\
+ access_key=<access_key>,secret=<secret>, \
+ multipart_sync_threshold=44432, \
+ multipart_min_part_size=44432, \
+ retain_head_object=true
Nested tier configuration values can be accessed using periods. This notation
works similarly to how nested fields are accessed in JSON with tools like ``jq``.
and should not be confused with Ceph RGW patterns for realm/zonegroup/zone.
For example::
- $ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
- --placement-id={placement-id} \
- --storage-class={storage-class-name} \
- --tier-config=acls.source_id=${source-id}, \
- acls.dest_id=${dest-id}
+.. prompt:: bash #
+
+ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
+ --placement-id={placement-id} \
+ --storage-class={storage-class-name} \
+ --tier-config=acls.source_id=${source-id}, \
+ acls.dest_id=${dest-id}
Configuration array entries can be accessed by specifying the specific entry to
be referenced enclosed in square brackets, and adding a new array entry can be
performed with an empty array `[]`.
For example, creating a new ``acl`` array entry::
- $ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
- --placement-id={placement-id} \
- --storage-class={storage-class-name} \
- --tier-config=acls[].source_id=${source-id}, \
- acls[${source-id}].dest_id=${dest-id}, \
- acls[${source-id}].type=email
+.. prompt:: bash #
+
+ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
+ --placement-id={placement-id} \
+ --storage-class={storage-class-name} \
+ --tier-config=acls[].source_id=${source-id}, \
+ acls[${source-id}].dest_id=${dest-id}, \
+ acls[${source-id}].type=email
An entry can be removed by supplying ``--tier-config-rm={key}``.
For example::
- $ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
- --placement-id default-placement \
- --storage-class CLOUDTIER \
- --tier-config-rm=acls.source_id=testid
+.. prompt:: bash #
- $ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
- --placement-id default-placement \
- --storage-class CLOUDTIER \
- --tier-config-rm=target_path
+ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
+ --placement-id default-placement \
+ --storage-class CLOUDTIER \
+ --tier-config-rm=acls.source_id=testid
+
+ radosgw-admin zonegroup placement modify --rgw-zonegroup default \
+ --placement-id default-placement \
+ --storage-class CLOUDTIER \
+ --tier-config-rm=target_path
The storage class can be removed using the following command::
- $ radosgw-admin zonegroup placement rm --rgw-zonegroup={zone-group-name} \
- --placement-id={placement-id} \
- --storage-class={storage-class-name}
+.. prompt:: bash #
-For example::
+ radosgw-admin zonegroup placement rm --rgw-zonegroup={zone-group-name} \
+ --placement-id={placement-id} \
+ --storage-class={storage-class-name}
+
+For example:
+
+.. prompt:: bash #
+
+ radosgw-admin zonegroup placement rm --rgw-zonegroup default \
+ --placement-id default-placement \
+ --storage-class CLOUDTIER
+
+::
- $ radosgw-admin zonegroup placement rm --rgw-zonegroup default \
- --placement-id default-placement \
- --storage-class CLOUDTIER
[
{
"key": "default-placement",
]
+ radosgw-admin zonegroup placement rm --rgw-zonegroup default \
+ --placement-id default-placement \
+ --storage-class CLOUDTIER
+
Object Modification and Limitations
-----------------------------------
For example::
- # s3cmd info s3://bucket/lc.txt
+.. prompt:: bash $
+
+ s3cmd info s3://bucket/lc.txt
+
+::
+
s3://bucket/lc.txt (object):
File size: 12
Last mod: Mon, 21 Dec 2020 10:25:56 GMT
ACL: M. Tester: FULL_CONTROL
x-amz-meta-s3cmd-attrs: atime:1608466266/ctime:1597606156/gid:0/gname:root/md5:ed076287532e86365e841e92bfc50d8c/mode:33188/mtime:1597605793/uid:0/uname:root
- # s3cmd get s3://bucket/lc.txt lc_restore.txt
+.. prompt:: bash $
+
+ s3cmd get s3://bucket/lc.txt lc_restore.txt
+
+::
+
download: 's3://bucket/lc.txt' -> 'lc_restore.txt' [1 of 1]
ERROR: S3 error: 403 (InvalidObjectState)
A policy can be defined at the bucket level. A bucket level sync policy inherits the data flow of the zonegroup policy, and can only define a subset of what the zonegroup allows.
-A wildcard zone, and a wildcard bucket parameter in the policy defines all relevant zones, or all relevant buckets. In the context of a bucket policy it means the current bucket instance. A disaster recovery configuration where entire zones are mirrored doesn't require configuring anything on the buckets. However, for a fine grained bucket sync it would be better to configure the pipes to be synced by allowing (status=allowed) them at the zonegroup level (e.g., using wildcards), but only enable the specific sync at the bucket level (status=enabled). If needed, the policy at the bucket level can limit the data movement to specific relevant zones.
+A wildcard zone, and a wildcard bucket parameter in the policy defines all relevant zones, or all relevant buckets. In the context of a bucket policy it means the current bucket instance. A disaster recovery configuration where entire zones are mirrored doesn't require configuring anything on the buckets. However, for a fine grained bucket sync it would be better to configure the pipes to be synced by allowing (``status=allowed``) them at the zonegroup level (e.g., using wildcards), but only enable the specific sync at the bucket level (``status=enabled``). If needed, the policy at the bucket level can limit the data movement to specific relevant zones.
.. important:: Any changes to the zonegroup policy needs to be applied on the
zonegroup master zone, and require period update and commit. Changes
To retrieve the current zonegroup sync policy, or a specific bucket policy:
-::
+.. prompt:: bash #
- # radosgw-admin sync policy get [--bucket=<bucket>]
+ radosgw-admin sync policy get [--bucket=<bucket>]
Create Sync Policy Group
To create a sync policy group:
-::
+.. prompt:: bash #
+
+ radosgw-admin sync group create [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --status=<enabled | allowed | forbidden>
+
- # radosgw-admin sync group create [--bucket=<bucket>] \
- --group-id=<group-id> \
- --status=<enabled | allowed | forbidden> \
-
-
Modify Sync Policy Group
~~~~~~~~~~~~~~~~~~~~~~~~
To modify a sync policy group:
-::
+.. prompt:: bash #
- # radosgw-admin sync group modify [--bucket=<bucket>] \
- --group-id=<group-id> \
- --status=<enabled | allowed | forbidden> \
+ radosgw-admin sync group modify [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --status=<enabled | allowed | forbidden>
Show Sync Policy Group
-~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~
To show a sync policy group:
-::
+.. prompt:: bash #
+
+ radosgw-admin sync group get [--bucket=<bucket>] \
+ --group-id=<group-id>
- # radosgw-admin sync group get [--bucket=<bucket>] \
- --group-id=<group-id>
-
Remove Sync Policy Group
~~~~~~~~~~~~~~~~~~~~~~~~
To remove a sync policy group:
-::
-
- # radosgw-admin sync group remove [--bucket=<bucket>] \
- --group-id=<group-id>
+.. prompt:: bash #
+ radosgw-admin sync group remove [--bucket=<bucket>] \
+ --group-id=<group-id>
Create Sync Flow
- To create or update directional sync flow:
-::
+.. prompt:: bash #
- # radosgw-admin sync group flow create [--bucket=<bucket>] \
- --group-id=<group-id> \
- --flow-id=<flow-id> \
- --flow-type=directional \
- --source-zone=<source_zone> \
- --dest-zone=<dest_zone>
+ radosgw-admin sync group flow create [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --flow-id=<flow-id> \
+ --flow-type=directional \
+ --source-zone=<source_zone> \
+ --dest-zone=<dest_zone>
- To create or update symmetrical sync flow:
-::
+.. prompt:: bash #
- # radosgw-admin sync group flow create [--bucket=<bucket>] \
- --group-id=<group-id> \
- --flow-id=<flow-id> \
- --flow-type=symmetrical \
- --zones=<zones>
+ radosgw-admin sync group flow create [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --flow-id=<flow-id> \
+ --flow-type=symmetrical \
+ --zones=<zones>
Where zones are a comma separated list of all the zones that need to add to the flow.
- To remove directional sync flow:
-::
+.. prompt:: bash #
- # radosgw-admin sync group flow remove [--bucket=<bucket>] \
- --group-id=<group-id> \
- --flow-id=<flow-id> \
- --flow-type=directional \
- --source-zone=<source_zone> \
- --dest-zone=<dest_zone>
+ radosgw-admin sync group flow remove [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --flow-id=<flow-id> \
+ --flow-type=directional \
+ --source-zone=<source_zone> \
+ --dest-zone=<dest_zone>
- To remove specific zones from symmetrical sync flow:
-::
+.. prompt:: bash #
- # radosgw-admin sync group flow remove [--bucket=<bucket>] \
- --group-id=<group-id> \
- --flow-id=<flow-id> \
- --flow-type=symmetrical \
- --zones=<zones>
+ radosgw-admin sync group flow remove [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --flow-id=<flow-id> \
+ --flow-type=symmetrical \
+ --zones=<zones>
Where zones are a comma separated list of all zones to remove from the flow.
- To remove symmetrical sync flow:
-::
+.. prompt:: bash #
- # radosgw-admin sync group flow remove [--bucket=<bucket>] \
- --group-id=<group-id> \
- --flow-id=<flow-id> \
- --flow-type=symmetrical
+ radosgw-admin sync group flow remove [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --flow-id=<flow-id> \
+ --flow-type=symmetrical
Create Sync Pipe
To create sync group pipe, or update its parameters:
-::
+.. prompt:: bash #
- # radosgw-admin sync group pipe create [--bucket=<bucket>] \
- --group-id=<group-id> \
- --pipe-id=<pipe-id> \
- --source-zones=<source_zones> \
- [--source-bucket=<source_buckets>] \
- [--source-bucket-id=<source_bucket_id>] \
- --dest-zones=<dest_zones> \
- [--dest-bucket=<dest_buckets>] \
- [--dest-bucket-id=<dest_bucket_id>] \
- [--prefix=<source_prefix>] \
- [--prefix-rm] \
- [--tags-add=<tags>] \
- [--tags-rm=<tags>] \
- [--dest-owner=<owner>] \
- [--storage-class=<storage_class>] \
- [--mode=<system | user>] \
- [--uid=<user_id>]
+ radosgw-admin sync group pipe create [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --pipe-id=<pipe-id> \
+ --source-zones=<source_zones> \
+ [--source-bucket=<source_buckets>] \
+ [--source-bucket-id=<source_bucket_id>] \
+ --dest-zones=<dest_zones> \
+ [--dest-bucket=<dest_buckets>] \
+ [--dest-bucket-id=<dest_bucket_id>] \
+ [--prefix=<source_prefix>] \
+ [--prefix-rm] \
+ [--tags-add=<tags>] \
+ [--tags-rm=<tags>] \
+ [--dest-owner=<owner>] \
+ [--storage-class=<storage_class>] \
+ [--mode=<system | user>] \
+ [--uid=<user_id>]
Zones are either a list of zones, or '*' (wildcard). Wildcard zones mean any zone that matches the sync flow rules.
To remove specific sync group pipe params, or the entire pipe:
-::
+.. prompt:: bash #
- # radosgw-admin sync group pipe remove [--bucket=<bucket>] \
- --group-id=<group-id> \
- --pipe-id=<pipe-id> \
- [--source-zones=<source_zones>] \
- [--source-bucket=<source_buckets>] \
- [--source-bucket-id=<source_bucket_id>] \
- [--dest-zones=<dest_zones>] \
- [--dest-bucket=<dest_buckets>] \
- [--dest-bucket-id=<dest_bucket_id>]
+ radosgw-admin sync group pipe remove [--bucket=<bucket>] \
+ --group-id=<group-id> \
+ --pipe-id=<pipe-id> \
+ [--source-zones=<source_zones>] \
+ [--source-bucket=<source_buckets>] \
+ [--source-bucket-id=<source_bucket_id>] \
+ [--dest-zones=<dest_zones>] \
+ [--dest-bucket=<dest_buckets>] \
+ [--dest-bucket-id=<dest_bucket_id>]
Sync Info
To get information about the expected sync sources and targets (as defined by the sync policy):
-::
+.. prompt:: bash #
- # radosgw-admin sync info [--bucket=<bucket>] \
- [--effective-zone-name=<zone>]
+ radosgw-admin sync info [--bucket=<bucket>] \
+ [--effective-zone-name=<zone>]
Since a bucket can define a policy that defines data movement from it towards a different bucket at a different zone, when the policy is created we also generate a list of bucket dependencies that are used as hints when a sync of any particular bucket happens. The fact that a bucket references another bucket does not mean it actually syncs to/from it, as the data flow might not permit it.
This is similar to older (pre ``Octopus``) sync capabilities, but being done via the new sync policy engine. Note that changes to the zonegroup sync policy require a period update and commit.
+.. prompt:: bash [us-east]#
+
+ radosgw-admin sync group create --group-id=group1 --status=allowed
+ radosgw-admin sync group flow create --group-id=group1 \
+ --flow-id=flow-mirror --flow-type=symmetrical \
+ --zones=us-east,us-west
+ radosgw-admin sync group pipe create --group-id=group1 \
+ --pipe-id=pipe1 --source-zones='*' \
+ --source-bucket='*' --dest-zones='*' \
+ --dest-bucket='*'
+ radosgw-admin sync group modify --group-id=group1 --status=enabled
+ radosgw-admin period update --commit
+ radosgw-admin sync info --bucket=buck
+
::
- [us-east] $ radosgw-admin sync group create --group-id=group1 --status=allowed
- [us-east] $ radosgw-admin sync group flow create --group-id=group1 \
- --flow-id=flow-mirror --flow-type=symmetrical \
- --zones=us-east,us-west
- [us-east] $ radosgw-admin sync group pipe create --group-id=group1 \
- --pipe-id=pipe1 --source-zones='*' \
- --source-bucket='*' --dest-zones='*' \
- --dest-bucket='*'
- [us-east] $ radosgw-admin sync group modify --group-id=group1 --status=enabled
- [us-east] $ radosgw-admin period update --commit
-
- $ radosgw-admin sync info --bucket=buck
- {
- "sources": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-east",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "params": {
- ...
- }
- }
- ],
- "dests": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-east",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- ...
- }
- ],
- ...
- }
- }
+ {
+ "sources": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-east",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "params": {
+ ...
+ }
+ }
+ ],
+ "dests": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-east",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ ...
+ }
+ ],
+ ...
+ }
+ }
Note that the "id" field in the output above reflects the pipe rule
that generated that entry, a single rule can generate multiple sync
entries as can be seen in the example.
+.. prompt:: bash [us-west]#
+
+ radosgw-admin sync info --bucket=buck
+
::
- [us-west] $ radosgw-admin sync info --bucket=buck
- {
- "sources": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-east",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- ...
- }
- ],
- "dests": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-east",
- "bucket": "buck:115b12b3-....4409.1"
- },
- ...
- }
- ],
- ...
- }
+ {
+ "sources": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-east",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ ...
+ }
+ ],
+ "dests": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-east",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ ...
+ }
+ ],
+ ...
+ }
Also similar to older sync capabilities. In here we add a third zone, ``us-west-2`` that will be a replica of ``us-west``, but data will not be replicated back from it.
-::
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group flow create --group-id=group1 \
- --flow-id=us-west-backup --flow-type=directional \
- --source-zone=us-west --dest-zone=us-west-2
- [us-east] $ radosgw-admin period update --commit
+ radosgw-admin sync group flow create --group-id=group1 \
+ --flow-id=us-west-backup --flow-type=directional \
+ --source-zone=us-west --dest-zone=us-west-2
+ radosgw-admin period update --commit
Note that us-west has two dests:
+.. prompt:: bash [us-west]#
+
+ radosgw-admin sync info --bucket=buck
+
::
- [us-west] $ radosgw-admin sync info --bucket=buck
- {
- "sources": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-east",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- ...
- }
- ],
- "dests": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-east",
- "bucket": "buck:115b12b3-....4409.1"
- },
- ...
- },
- {
- "id": "pipe1",
- "source": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-west-2",
- "bucket": "buck:115b12b3-....4409.1"
- },
- ...
- }
- ],
- ...
- }
+ {
+ "sources": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-east",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ ...
+ }
+ ],
+ "dests": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-east",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ ...
+ },
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-west-2",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ ...
+ }
+ ],
+ ...
+ }
Whereas us-west-2 has only source and no destinations:
+.. prompt:: bash [us-west-2]#
+
+ radosgw-admin sync info --bucket=buck
+
::
- [us-west-2] $ radosgw-admin sync info --bucket=buck
- {
- "sources": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-west",
- "bucket": "buck:115b12b3-....4409.1"
- },
- "dest": {
- "zone": "us-west-2",
- "bucket": "buck:115b12b3-....4409.1"
- },
- ...
- }
- ],
- "dests": [],
- ...
- }
+ {
+ "sources": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-west",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ "dest": {
+ "zone": "us-west-2",
+ "bucket": "buck:115b12b3-....4409.1"
+ },
+ ...
+ }
+ ],
+ "dests": [],
+ ...
+ }
Using the same group configuration, but this time switching it to ``allowed`` state, which means that sync is allowed but not enabled.
-::
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group modify --group-id=group1 --status=allowed
- [us-east] $ radosgw-admin period update --commit
+ radosgw-admin sync group modify --group-id=group1 --status=allowed
+ radosgw-admin period update --commit
And we will create a bucket level policy rule for existing bucket ``buck2``. Note that the bucket needs to exist before being able to set this policy, and admin commands that modify bucket policies need to run on the master zone, however, they do not require period update. There is no need to change the data flow, as it is inherited from the zonegroup policy. A bucket policy flow will only be a subset of the flow defined in the zonegroup policy. Same goes for pipes, although a bucket policy can enable pipes that are not enabled (albeit not forbidden) at the zonegroup policy.
-::
-
- [us-east] $ radosgw-admin sync group create --bucket=buck2 \
- --group-id=buck2-default --status=enabled
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck2 \
- --group-id=buck2-default --pipe-id=pipe1 \
- --source-zones='*' --dest-zones='*'
+ radosgw-admin sync group create --bucket=buck2 \
+ --group-id=buck2-default --status=enabled
+ radosgw-admin sync group pipe create --bucket=buck2 \
+ --group-id=buck2-default --pipe-id=pipe1 \
+ --source-zones='*' --dest-zones='*'
-Example 4: Limit Bucket Sync To Specific Zones
+Example 4: Limit Bucket Sync to Specific Zones
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This will only sync ``buck3`` to ``us-east`` (from any zone that flow allows to sync into ``us-east``).
-::
-
- [us-east] $ radosgw-admin sync group create --bucket=buck3 \
- --group-id=buck3-default --status=enabled
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck3 \
- --group-id=buck3-default --pipe-id=pipe1 \
- --source-zones='*' --dest-zones=us-east
+ radosgw-admin sync group create --bucket=buck3 \
+ --group-id=buck3-default --status=enabled
+ radosgw-admin sync group pipe create --bucket=buck3 \
+ --group-id=buck3-default --pipe-id=pipe1 \
+ --source-zones='*' --dest-zones=us-east
Set ``buck4`` to pull data from ``buck5``:
-::
-
- [us-east] $ radosgw-admin sync group create --bucket=buck4 '
- --group-id=buck4-default --status=enabled
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck4 \
- --group-id=buck4-default --pipe-id=pipe1 \
- --source-zones='*' --source-bucket=buck5 \
- --dest-zones='*'
+ radosgw-admin sync group create --bucket=buck4 \
+ --group-id=buck4-default --status=enabled
+ radosgw-admin sync group pipe create --bucket=buck4 \
+ --group-id=buck4-default --pipe-id=pipe1 \
+ --source-zones='*' --source-bucket=buck5 \
+ --dest-zones='*'
can also limit it to specific zones, for example the following will
only sync data originated in us-west:
-::
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe modify --bucket=buck4 \
- --group-id=buck4-default --pipe-id=pipe1 \
- --source-zones=us-west --source-bucket=buck5 \
- --dest-zones='*'
+ radosgw-admin sync group pipe modify --bucket=buck4 \
+ --group-id=buck4-default --pipe-id=pipe1 \
+ --source-zones=us-west --source-bucket=buck5 \
+ --dest-zones='*'
Checking the sync info for ``buck5`` on ``us-west`` is interesting:
+.. prompt:: bash [us-west]#
+
+ radosgw-admin sync info --bucket=buck5
+
::
- [us-west] $ radosgw-admin sync info --bucket=buck5
- {
- "sources": [],
- "dests": [],
- "hints": {
- "sources": [],
- "dests": [
- "buck4:115b12b3-....14433.2"
- ]
- },
- "resolved-hints-1": {
- "sources": [],
- "dests": [
- {
- "id": "pipe1",
- "source": {
- "zone": "us-west",
- "bucket": "buck5"
- },
- "dest": {
- "zone": "us-east",
- "bucket": "buck4:115b12b3-....14433.2"
- },
- ...
- },
- {
- "id": "pipe1",
- "source": {
- "zone": "us-west",
- "bucket": "buck5"
- },
- "dest": {
- "zone": "us-west-2",
- "bucket": "buck4:115b12b3-....14433.2"
- },
- ...
- }
- ]
- },
- "resolved-hints": {
- "sources": [],
- "dests": []
- }
- }
+ {
+ "sources": [],
+ "dests": [],
+ "hints": {
+ "sources": [],
+ "dests": [
+ "buck4:115b12b3-....14433.2"
+ ]
+ },
+ "resolved-hints-1": {
+ "sources": [],
+ "dests": [
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-west",
+ "bucket": "buck5"
+ },
+ "dest": {
+ "zone": "us-east",
+ "bucket": "buck4:115b12b3-....14433.2"
+ },
+ ...
+ },
+ {
+ "id": "pipe1",
+ "source": {
+ "zone": "us-west",
+ "bucket": "buck5"
+ },
+ "dest": {
+ "zone": "us-west-2",
+ "bucket": "buck4:115b12b3-....14433.2"
+ },
+ ...
+ }
+ ]
+ },
+ "resolved-hints": {
+ "sources": [],
+ "dests": []
+ }
+ }
Note that there are resolved hints, which means that the bucket ``buck5`` found about ``buck4`` syncing from it indirectly, and not from its own policy (the policy for ``buck5`` itself is empty).
-Example 6: Sync To Different Bucket
+Example 6: Sync to Different Bucket
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The same mechanism can work for configuring data to be synced to (vs. synced from as in the previous example). Note that internally data is still pulled from the source at the destination zone:
Set ``buck6`` to "push" data to ``buck5``:
-::
-
- [us-east] $ radosgw-admin sync group create --bucket=buck6 \
- --group-id=buck6-default --status=enabled
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck6 \
- --group-id=buck6-default --pipe-id=pipe1 \
- --source-zones='*' --source-bucket='*' \
- --dest-zones='*' --dest-bucket=buck5
+ radosgw-admin sync group create --bucket=buck6 \
+ --group-id=buck6-default --status=enabled
+ radosgw-admin sync group pipe create --bucket=buck6 \
+ --group-id=buck6-default --pipe-id=pipe1 \
+ --source-zones='*' --source-bucket='*' \
+ --dest-zones='*' --dest-bucket=buck5
A wildcard bucket name means the current bucket in the context of bucket sync policy.
Sync from ``buck8`` to ``buck9``, but only objects that start with ``foo/``:
-::
-
- [us-east] $ radosgw-admin sync group create --bucket=buck8 \
- --group-id=buck8-default --status=enabled
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck8 \
- --group-id=buck8-default --pipe-id=pipe-prefix \
- --prefix=foo/ --source-zones='*' --dest-zones='*' \
- --dest-bucket=buck9
+ radosgw-admin sync group create --bucket=buck8 \
+ --group-id=buck8-default --status=enabled
+ radosgw-admin sync group pipe create --bucket=buck8 \
+ --group-id=buck8-default --pipe-id=pipe-prefix \
+ --prefix=foo/ --source-zones='*' --dest-zones='*' \
+ --dest-bucket=buck9
Also sync from ``buck8`` to ``buck9`` any object that has the tags ``color=blue`` or ``color=red``:
-::
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck8 \
- --group-id=buck8-default --pipe-id=pipe-tags \
- --tags-add=color=blue,color=red --source-zones='*' \
- --dest-zones='*' --dest-bucket=buck9
+ radosgw-admin sync group pipe create --bucket=buck8 \
+ --group-id=buck8-default --pipe-id=pipe-tags \
+ --tags-add=color=blue,color=red --source-zones='*' \
+ --dest-zones='*' --dest-bucket=buck9
And we can check the expected sync in ``us-east`` (for example):
+.. prompt:: bash [us-east]#
+
+ radosgw-admin sync info --bucket=buck8
+
::
- [us-east] $ radosgw-admin sync info --bucket=buck8
- {
- "sources": [],
- "dests": [
- {
- "id": "pipe-prefix",
- "source": {
- "zone": "us-east",
- "bucket": "buck8:115b12b3-....14433.5"
- },
- "dest": {
- "zone": "us-west",
- "bucket": "buck9"
- },
- "params": {
- "source": {
- "filter": {
- "prefix": "foo/",
- "tags": []
- }
- },
- ...
- }
- },
- {
- "id": "pipe-tags",
- "source": {
- "zone": "us-east",
- "bucket": "buck8:115b12b3-....14433.5"
- },
- "dest": {
- "zone": "us-west",
- "bucket": "buck9"
- },
- "params": {
- "source": {
- "filter": {
- "tags": [
- {
- "key": "color",
- "value": "blue"
- },
- {
- "key": "color",
- "value": "red"
- }
- ]
- }
- },
- ...
- }
- }
- ],
- ...
- }
+ {
+ "sources": [],
+ "dests": [
+ {
+ "id": "pipe-prefix",
+ "source": {
+ "zone": "us-east",
+ "bucket": "buck8:115b12b3-....14433.5"
+ },
+ "dest": {
+ "zone": "us-west",
+ "bucket": "buck9"
+ },
+ "params": {
+ "source": {
+ "filter": {
+ "prefix": "foo/",
+ "tags": []
+ }
+ },
+ ...
+ }
+ },
+ {
+ "id": "pipe-tags",
+ "source": {
+ "zone": "us-east",
+ "bucket": "buck8:115b12b3-....14433.5"
+ },
+ "dest": {
+ "zone": "us-west",
+ "bucket": "buck9"
+ },
+ "params": {
+ "source": {
+ "filter": {
+ "tags": [
+ {
+ "key": "color",
+ "value": "blue"
+ },
+ {
+ "key": "color",
+ "value": "red"
+ }
+ ]
+ }
+ },
+ ...
+ }
+ }
+ ],
+ ...
+ }
Note that there aren't any sources, only two different destinations (one for each configuration). When the sync process happens it will select the relevant rule for each object it syncs.
Storage class of the destination objects can be configured:
-::
-
- [us-east] $ radosgw-admin sync group create --bucket=buck10 \
- --group-id=buck10-default --status=enabled
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck10 \
- --group-id=buck10-default \
- --pipe-id=pipe-storage-class \
- --source-zones='*' --dest-zones=us-west-2 \
- --storage-class=CHEAP_AND_SLOW
+ radosgw-admin sync group create --bucket=buck10 \
+ --group-id=buck10-default --status=enabled
+ radosgw-admin sync group pipe create --bucket=buck10 \
+ --group-id=buck10-default \
+ --pipe-id=pipe-storage-class \
+ --source-zones='*' --dest-zones=us-west-2 \
+ --storage-class=CHEAP_AND_SLOW
Example 9: Destination Params: Destination Owner Translation
Set the destination objects owner as the destination bucket owner.
This requires specifying the uid of the destination bucket:
-::
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group create --bucket=buck11 \
- --group-id=buck11-default --status=enabled
+ radosgw-admin sync group create --bucket=buck11 \
+ --group-id=buck11-default --status=enabled
+ radosgw-admin sync group pipe create --bucket=buck11 \
+ --group-id=buck11-default --pipe-id=pipe-dest-owner \
+ --source-zones='*' --dest-zones='*' \
+ --dest-bucket=buck12 --dest-owner=joe
- [us-east] $ radosgw-admin sync group pipe create --bucket=buck11 \
- --group-id=buck11-default --pipe-id=pipe-dest-owner \
- --source-zones='*' --dest-zones='*' \
- --dest-bucket=buck12 --dest-owner=joe
Example 10: Destination Params: User Mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
User mode makes sure that the user has permissions to both read the objects, and write to the destination bucket. This requires that the uid of the user (which in its context the operation executes) is specified.
-::
+.. prompt:: bash [us-east]#
- [us-east] $ radosgw-admin sync group pipe modify --bucket=buck11 \
- --group-id=buck11-default --pipe-id=pipe-dest-owner \
- --mode=user --uid=jenny
+ radosgw-admin sync group pipe modify --bucket=buck11 \
+ --group-id=buck11-default --pipe-id=pipe-dest-owner \
+ --mode=user --uid=jenny
Archive Sync Module <archive-sync-module>
.. note ``rgw`` is the default sync plugin and there is no need to explicitly
- configure this
+ configure this.
Requirements and Assumptions
----------------------------
Create the third zone similar to the :ref:`multisite` docs, for example
-::
-
- # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-es \
- --access-key={system-key} --secret={secret} --endpoints=http://rgw-es:80
+.. prompt:: bash #
+ radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-es \
+ --access-key={system-key} --secret={secret} --endpoints=http://rgw-es:80
A sync module can be configured for this zone via the following
-::
+.. prompt:: bash #
- # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type={tier-type} --tier-config={set of key=value pairs}
+ radosgw-admin zone modify --rgw-zone={zone-name} --tier-type={tier-type} --tier-config={set of key=value pairs}
For example in the ``elasticsearch`` sync module
-::
+.. prompt:: bash #
- # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type=elasticsearch \
+ radosgw-admin zone modify --rgw-zone={zone-name} --tier-type=elasticsearch \
--tier-config=endpoint=http://localhost:9200,num_shards=10,num_replicas=1
Finally update the period
+.. prompt:: bash #
-::
-
- # radosgw-admin period update --commit
+ radosgw-admin period update --commit
Now start the radosgw in the zone
-::
+.. prompt:: bash #
- # systemctl start ceph-radosgw@rgw.`hostname -s`
- # systemctl enable ceph-radosgw@rgw.`hostname -s`
+ systemctl start ceph-radosgw@rgw.`hostname -s`
+ systemctl enable ceph-radosgw@rgw.`hostname -s`
projects / ceph.git / commitdiff