From b02acfef02b9ced1b087b98593f5fbbe24ee2eb8 Mon Sep 17 00:00:00 2001 From: Ville Ojamo <14869000+bluikko@users.noreply.github.com> Date: Thu, 10 Apr 2025 17:34:57 +0700 Subject: [PATCH] doc/radosgw: Promptify CLI, cosmetic fixes Use the more modern prompt block for CLI commands and use right one $ vs #. Fix indentation on JSON example outputs and some CLI command switches. Add some arguably missing comma in JSON example output. Add a full stop at the end of a one-sentence paragraph. Remove extra comma mid-sentence in another. Fix missing backslashes or typo at end of multiline commands. Lines under section headings as long as heading text. Fix hyperlinks. Fix list items prefixed with - insted of *. Format configuration syntax in the middle of text as code. Fix typo "PI" to "API" and remove extra space. Remove colons at the end of section headers in a few places. Use Title Case in section titles consistently with short words lowercase. Possibly controversial: don't add whitespace before and after main title section header text. Possibly controversial: don't indent line continuation backslashes, leave only 1 space before them. Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com> (cherry picked from commit 7fdaa94181c6fca7d4db0b6f4ee6646528121d43) --- doc/radosgw/archive-sync-module.rst | 12 +- doc/radosgw/cloud-restore.rst | 167 +++--- doc/radosgw/cloud-sync-module.rst | 111 ++-- doc/radosgw/cloud-transition.rst | 140 +++-- doc/radosgw/multisite-sync-policy.rst | 814 +++++++++++++------------- doc/radosgw/sync-modules.rst | 28 +- 6 files changed, 662 insertions(+), 610 deletions(-) diff --git a/doc/radosgw/archive-sync-module.rst b/doc/radosgw/archive-sync-module.rst index f9779f23ceba1..be6f571e428f4 100644 --- a/doc/radosgw/archive-sync-module.rst +++ b/doc/radosgw/archive-sync-module.rst @@ -30,11 +30,13 @@ See `Multisite Configuration`_ for multisite configuration instructions. The archive sync module requires the creation of a new zone. The zone tier type needs to be defined as ``archive``: -:: +.. prompt:: bash # + + radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --endpoints={http://fqdn}[,{http://fqdn}] \ + --tier-type=archive + - # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ - --rgw-zone={zone-name} \ - --endpoints={http://fqdn}[,{http://fqdn}] - --tier-type=archive .. _Multisite Configuration: ../multisite diff --git a/doc/radosgw/cloud-restore.rst b/doc/radosgw/cloud-restore.rst index 701439d950fce..3d7114081c0f6 100644 --- a/doc/radosgw/cloud-restore.rst +++ b/doc/radosgw/cloud-restore.rst @@ -1,6 +1,6 @@ -=============== - Cloud Restore -=============== +============= +Cloud Restore +============= The :doc:`cloud-transition>` feature makes it possible to transition objects to a remote cloud service. The ``cloud-restore`` feature described below enables restoration @@ -12,23 +12,22 @@ S3-compatible cloud services. In order to faciliate this, the ``retain_head_object`` option should be set to ``true`` in the ``tier-config`` when configuring the storage class. -Objects can be restored using the `S3 RestoreObject ` -PI . The restored copies will be retained within RGW only for the number +Objects can be restored using the `S3 RestoreObject `_ +API. The restored copies will be retained within RGW only for the number of ``days`` specified. However if ``days`` is not provided, the restored copies are considered permanent and will be treated as regular objects. In addition, by enabling the ``allow_read_through`` option, -the `S3 GetObject ` +the `S3 GetObject `_ API can be used to restore the object temporarily. Cloud Storage Class Tier Configuration -------------------------------------- -The `tier configuration ` +The `tier configuration `_ of the cloud storage class configured for data transition is used to restore -objects as well: +objects as well:: -``` { "access_key": , "secret": ,` @@ -44,7 +43,6 @@ objects as well: "multipart_min_part_size": {part_size}, "retain_head_object": } -``` The below options have been added to the tier configuration to facilitate object restoration. @@ -53,7 +51,7 @@ The below options have been added to the tier configuration to facilitate object The storage class to which object data is to be restored. Default value is ``STANDARD``. -read-through specific Configurables: +Read-through Specific Configurables ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * ``allow_read_through`` (``true`` | ``false``) @@ -67,20 +65,21 @@ Default value is 1 day. For example: -``` - # radosgw-admin zonegroup placement modify --rgw-zonegroup default \ - --placement-id default-placement \ - --storage-class CLOUDTIER \ - --tier-config=endpoint=http://XX.XX.XX.XX:YY,\ - access_key=,secret=, \ - retain_head_object=true, \ - restore_storage_class=COLDTIER, \ - allow_read_through=true, \ - read_through_restore_days=10 -``` +.. prompt:: bash # + radosgw-admin zonegroup placement modify --rgw-zonegroup default \ + --placement-id default-placement \ + --storage-class CLOUDTIER \ + --tier-config=endpoint=http://XX.XX.XX.XX:YY,\ + access_key=,secret=, \ + retain_head_object=true, \ + restore_storage_class=COLDTIER, \ + allow_read_through=true, \ + read_through_restore_days=10 -S3 Glacier specific Configurables: + + +S3 Glacier Specific Configurables ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To restore objects archived in an S3 Glacier or Tape cloud storage class, the @@ -101,26 +100,27 @@ pricing. Supported options are ``Standard`` and ``Expedited``. For example: -``` - # radosgw-admin zonegroup placement add --rgw-zonegroup=default \ - --placement-id=default-placement \ - --storage-class=CLOUDTIER-GLACIER --tier-type=cloud-s3-glacier - - # radosgw-admin zonegroup placement modify --rgw-zonegroup default \ - --placement-id default-placement \ - --storage-class CLOUDTIER \ - --tier-config=endpoint=http://XX.XX.XX.XX:YY,\ - access_key=XXXXX,secret=YYYYY, \ - retain_head_object=true, \ - target_storage_class=Glacier, \ - ............ - ............ - restore_storage_class=COLDTIER, \ - glacier_restore_days=2, \ - glacier_restore_tier_type=Expedited - - - [ +.. prompt:: bash # + + radosgw-admin zonegroup placement add --rgw-zonegroup=default \ + --placement-id=default-placement \ + --storage-class=CLOUDTIER-GLACIER --tier-type=cloud-s3-glacier + radosgw-admin zonegroup placement modify --rgw-zonegroup default \ + --placement-id default-placement \ + --storage-class CLOUDTIER \ + --tier-config=endpoint=http://XX.XX.XX.XX:YY,\ + access_key=XXXXX,secret=YYYYY, \ + retain_head_object=true, \ + target_storage_class=Glacier, \ + ............ \ + ............ \ + restore_storage_class=COLDTIER, \ + glacier_restore_days=2, \ + glacier_restore_tier_type=Expedited + +:: + + [ { "key": "default-placement", "val": { @@ -159,47 +159,47 @@ For example: } } ] -``` -Examples of Restore Objects: ----------------------------- +Examples of Restore Objects +--------------------------- Using the S3 RestoreObject CLI ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Th `S3 restore-object ` +Th `S3 restore-object `_ CLI supports these options: -``` - $ aws s3api restore-object - --bucket - --key - [--version-id ] - --restore-request (structure) { - Days= - } -``` +.. prompt:: bash $ + + aws s3api restore-object --bucket \ + --key \ + [--version-id ] \ + --restore-request (structure) { \ + Days= \ + } + Note: ``Days`` is optional and if not provided, the object is restored permanently. Example 1: -``` - $ aws s3api restore-object --bucket bucket1 --key doc1.rtf - [--version-id 3sL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo] - --restore-request Days=10 +.. prompt:: bash $ + + aws s3api restore-object --bucket bucket1 --key doc1.rtf \ + [--version-id 3sL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo] \ + --restore-request Days=10 \ .... -``` + This will restore the object ``doc1.rtf`` at an optional version, for the duration of 10 days. Example 2: -``` - $ aws s3api restore-object --bucket bucket1 --key doc2.rtf --restore-request {} .... -``` +.. prompt:: bash $ + + aws s3api restore-object --bucket bucket1 --key doc2.rtf --restore-request {} .... This will restore the object ``doc2.rtf`` permanently and it will be treated as regular object. @@ -212,9 +212,10 @@ Ensure that the ``allow_read_through`` tier-config option is enabled. Example 3: -``` - $ aws s3api get-object --bucket bucket1 --key doc3.rtf .... -``` +.. prompt:: bash $ + + aws s3api get-object --bucket bucket1 --key doc3.rtf .... + This will restore the object `doc3.rtf`` for ``read_through_restore_days`` days. @@ -222,49 +223,51 @@ Note: The above CLI command may time out if object restoration takes too long. You can verify the restore status before reissuing the command. -Verifying the restoration status +Verifying the Restoration Status -------------------------------- Verify the status of the restoration by issuing -an `S3 HeadObject ` +an `S3 HeadObject `_ request. The response includes the ``x-amz-restore`` header if object restoration is in progress or a copy of it is already restored. Example: -``` - $ aws s3api head-object --key doc1.rtf --bucket bucket1 .... -``` +.. prompt:: bash $ + + aws s3api head-object --key doc1.rtf --bucket bucket1 .... + The ``radosgw-admin`` CLI can be used to check restoration status and other details. Example: -``` - $ radosgw-admin object stat --bucket bucket1 --object doc1.rtf -``` +.. prompt:: bash $ + + radosgw-admin object stat --bucket bucket1 --object doc1.rtf + Restored Object Properties -------------------------- Storage -~~~~~~ +~~~~~~~ Objects are restored to the storage class configured via ``restore_storage_class`` in the tier-config. However, as -per `` +per ``_ the storage class of restored objects should remain unchanged. Therefore, for -temporary copies, the ```x-amz-storage-class``` will continue to reflect the +temporary copies, the ``x-amz-storage-class`` will continue to reflect the original cloud-tier storage class. mtime -~~~~ +~~~~~ The ``mtime`` of the transitioned and restored objects should remain unchanged. Lifecycle -~~~~~~~~ +~~~~~~~~~ ``Temporary`` copies are not subject to transition to the cloud. However, as is the case with cloud-transitioned objects, they can be deleted via regular LC (Life Cycle) expiration rules or an external S3 ``delete`` request. @@ -274,7 +277,7 @@ policies. Replication -~~~~~~~~~~ +~~~~~~~~~~~ ``Temporary`` copies are not replicated and will be retained only by the zone on which the restore request is initiated. @@ -282,7 +285,7 @@ on which the restore request is initiated. Versioned Objects -~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~ For versioned objects, if an object has been cloud-transitioned, it is in a non-current state. After a restore, the same non-current object will be updated with the downloaded data, and its ``HEAD`` object will be modified accordingly. diff --git a/doc/radosgw/cloud-sync-module.rst b/doc/radosgw/cloud-sync-module.rst index a601bd503fbab..1b68c4dd36aed 100644 --- a/doc/radosgw/cloud-sync-module.rst +++ b/doc/radosgw/cloud-sync-module.rst @@ -1,6 +1,6 @@ -========================= +================= Cloud Sync Module -========================= +================= .. versionadded:: Mimic @@ -19,10 +19,10 @@ stores these as metadata attributes on the destination objects. Cloud Sync Tier Type Configuration -------------------------------------- +---------------------------------- -Trivial Configuration: -~~~~~~~~~~~~~~~~~~~~~~ +Trivial Configuration +~~~~~~~~~~~~~~~~~~~~~ :: @@ -40,25 +40,25 @@ Trivial Configuration: } -Non Trivial Configuration: -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Non Trivial Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~ :: { "default": { "connection": { - "access_key": , - "secret": , - "endpoint": , - "host_style" , + "access_key": , + "secret": , + "endpoint": , + "host_style" , }, "acls": [ - { - "type" : , # optional, default is id - "source_id": , - "dest_id": - } ... ] + { + "type" : , # optional, default is id + "source_id": , + "dest_id": + } ... ], "target_path": # optional }, "connections": [ @@ -76,15 +76,15 @@ Non Trivial Configuration: "type": , "source_id": , "dest_id": - } ... ] + } ... ], } ], "profiles": [ { - "source_bucket": , - "connection_id": , - "acls_id": , - "target_path": , # optional + "source_bucket": , + "connection_id": , + "acls_id": , + "target_path": , # optional } ... ], } @@ -140,13 +140,14 @@ ID of user in the destination. A string that defines how the target path is created. The target path specifies a prefix to which the source object name is appended. The target path configurable can include any of the following variables: -- ``sid``: unique string that represents the sync instance ID -- ``zonegroup``: the zonegroup name -- ``zonegroup_id``: the zonegroup ID -- ``zone``: the zone name -- ``zone_id``: the zone id -- ``bucket``: source bucket name -- ``owner``: source bucket owner ID + +* ``sid``: unique string that represents the sync instance ID +* ``zonegroup``: the zonegroup name +* ``zonegroup_id``: the zonegroup ID +* ``zone``: the zone name +* ``zone_id``: the zone id +* ``bucket``: source bucket name +* ``owner``: source bucket owner ID For example: ``target_path = rgwx-${zone}-${sid}/${owner}/${bucket}`` @@ -163,14 +164,15 @@ holds a list of ``acl_mappings``. * ``profiles`` (array) A list of profiles. Each profile contains the following: -- ``source_bucket``: either a bucket name, or a bucket prefix (if ends with ``*``) that defines the source bucket(s) for this profile -- ``target_path``: as defined above -- ``connection_id``: ID of the connection that will be used for this profile -- ``acls_id``: ID of ACLs profile that will be used for this profile +* ``source_bucket``: either a bucket name, or a bucket prefix (if ends with ``*``) that defines the source bucket(s) for this profile +* ``target_path``: as defined above +* ``connection_id``: ID of the connection that will be used for this profile +* ``acls_id``: ID of ACLs profile that will be used for this profile -S3 Specific Configurables: -~~~~~~~~~~~~~~~~~~~~~~~~~~ + +S3 Specific Configurables +~~~~~~~~~~~~~~~~~~~~~~~~~ Currently cloud sync will only work with backends that are compatible with AWS S3. There are a few configurables that can be used to tweak its behavior when accessing these cloud services: @@ -198,30 +200,31 @@ How to Configure See :ref:`multisite` for how to multisite config instructions. The cloud sync module requires a creation of a new zone. The zone tier type needs to be defined as ``cloud``: -:: +.. prompt:: bash # - # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ - --rgw-zone={zone-name} \ - --endpoints={http://fqdn}[,{http://fqdn}] - --tier-type=cloud + radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --endpoints={http://fqdn}[,{http://fqdn}] \ + --tier-type=cloud The tier configuration can be then done using the following command -:: +.. prompt:: bash # + + radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --tier-config={key}={val}[,{key}={val}] - # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ - --rgw-zone={zone-name} \ - --tier-config={key}={val}[,{key}={val}] The ``key`` in the configuration specifies the config variable that needs to be updated, and the ``val`` specifies its new value. Nested values can be accessed using period. For example: -:: +.. prompt:: bash # - # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ - --rgw-zone={zone-name} \ - --tier-config=connection.access_key={key},connection.secret={secret} + radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --tier-config=connection.access_key={key},connection.secret={secret} Configuration array entries can be accessed by specifying the specific entry to be referenced enclosed @@ -230,15 +233,15 @@ the last entry in the array. At the moment it is not possible to create a new en again at the same command. For example, creating a new profile for buckets starting with {prefix}: -:: +.. prompt:: bash # - # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ - --rgw-zone={zone-name} \ - --tier-config=profiles[].source_bucket={prefix}'*' + radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --tier-config=profiles[].source_bucket={prefix}'*' - # radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ - --rgw-zone={zone-name} \ - --tier-config=profiles[-1].connection_id={conn_id},profiles[-1].acls_id={acls_id} + radosgw-admin zone modify --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --tier-config=profiles[-1].connection_id={conn_id},profiles[-1].acls_id={acls_id} An entry can be removed by using ``--tier-config-rm={key}``. diff --git a/doc/radosgw/cloud-transition.rst b/doc/radosgw/cloud-transition.rst index f0992bd72b173..3c6f5637fb517 100644 --- a/doc/radosgw/cloud-transition.rst +++ b/doc/radosgw/cloud-transition.rst @@ -1,6 +1,6 @@ -================== - Cloud Transition -================== +================ +Cloud Transition +================ This feature enables transitioning S3 objects to a remote cloud service as part of `object lifecycle `_ @@ -154,18 +154,23 @@ See :ref:`adding_a_storage_class` for how to configure storage-class for a zoneg the zone/zonegroup changes will take effect once the changes are committed with ``radosgw-admin period update --commit``. -:: +.. prompt:: bash # - $ radosgw-admin zonegroup placement add --rgw-zonegroup={zone-group-name} \ - --placement-id={placement-id} \ - --storage-class={storage-class-name} \ - --tier-type=cloud-s3 + radosgw-admin zonegroup placement add --rgw-zonegroup={zone-group-name} \ + --placement-id={placement-id} \ + --storage-class={storage-class-name} \ + --tier-type=cloud-s3 -For example:: +For example + +.. prompt:: bash # + + radosgw-admin zonegroup placement add --rgw-zonegroup=default \ + --placement-id=default-placement \ + --storage-class=CLOUDTIER --tier-type=cloud-s3 + +:: - $ radosgw-admin zonegroup placement add --rgw-zonegroup=default \ - --placement-id=default-placement \ - --storage-class=CLOUDTIER --tier-type=cloud-s3 [ { "key": "default-placement", @@ -207,24 +212,28 @@ For example:: The tier configuration can be then performed using the following command:: - $ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \ - --placement-id={placement-id} \ - --storage-class={storage-class-name} \ - --tier-config={key}={val}[,{key}={val}] +.. prompt:: bash # + + radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \ + --placement-id={placement-id} \ + --storage-class={storage-class-name} \ + --tier-config={key}={val}[,{key}={val}] The ``key`` in the configuration specifies the config variable to be updated, and the ``val`` specifies its new value. For example:: - $ radosgw-admin zonegroup placement modify --rgw-zonegroup default \ - --placement-id default-placement \ - --storage-class CLOUDTIER \ - --tier-config=endpoint=http://XX.XX.XX.XX:YY,\ - access_key=,secret=, \ - multipart_sync_threshold=44432, \ - multipart_min_part_size=44432, \ - retain_head_object=true +.. prompt:: bash # + + radosgw-admin zonegroup placement modify --rgw-zonegroup default \ + --placement-id default-placement \ + --storage-class CLOUDTIER \ + --tier-config=endpoint=http://XX.XX.XX.XX:YY,\ + access_key=,secret=, \ + multipart_sync_threshold=44432, \ + multipart_min_part_size=44432, \ + retain_head_object=true Nested tier configuration values can be accessed using periods. This notation works similarly to how nested fields are accessed in JSON with tools like ``jq``. @@ -232,49 +241,62 @@ Note that the use of period separators ``(.)`` is specific to key access within and should not be confused with Ceph RGW patterns for realm/zonegroup/zone. For example:: - $ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \ - --placement-id={placement-id} \ - --storage-class={storage-class-name} \ - --tier-config=acls.source_id=${source-id}, \ - acls.dest_id=${dest-id} +.. prompt:: bash # + + radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \ + --placement-id={placement-id} \ + --storage-class={storage-class-name} \ + --tier-config=acls.source_id=${source-id}, \ + acls.dest_id=${dest-id} Configuration array entries can be accessed by specifying the specific entry to be referenced enclosed in square brackets, and adding a new array entry can be performed with an empty array `[]`. For example, creating a new ``acl`` array entry:: - $ radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \ - --placement-id={placement-id} \ - --storage-class={storage-class-name} \ - --tier-config=acls[].source_id=${source-id}, \ - acls[${source-id}].dest_id=${dest-id}, \ - acls[${source-id}].type=email +.. prompt:: bash # + + radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \ + --placement-id={placement-id} \ + --storage-class={storage-class-name} \ + --tier-config=acls[].source_id=${source-id}, \ + acls[${source-id}].dest_id=${dest-id}, \ + acls[${source-id}].type=email An entry can be removed by supplying ``--tier-config-rm={key}``. For example:: - $ radosgw-admin zonegroup placement modify --rgw-zonegroup default \ - --placement-id default-placement \ - --storage-class CLOUDTIER \ - --tier-config-rm=acls.source_id=testid +.. prompt:: bash # - $ radosgw-admin zonegroup placement modify --rgw-zonegroup default \ - --placement-id default-placement \ - --storage-class CLOUDTIER \ - --tier-config-rm=target_path + radosgw-admin zonegroup placement modify --rgw-zonegroup default \ + --placement-id default-placement \ + --storage-class CLOUDTIER \ + --tier-config-rm=acls.source_id=testid + + radosgw-admin zonegroup placement modify --rgw-zonegroup default \ + --placement-id default-placement \ + --storage-class CLOUDTIER \ + --tier-config-rm=target_path The storage class can be removed using the following command:: - $ radosgw-admin zonegroup placement rm --rgw-zonegroup={zone-group-name} \ - --placement-id={placement-id} \ - --storage-class={storage-class-name} +.. prompt:: bash # -For example:: + radosgw-admin zonegroup placement rm --rgw-zonegroup={zone-group-name} \ + --placement-id={placement-id} \ + --storage-class={storage-class-name} + +For example: + +.. prompt:: bash # + + radosgw-admin zonegroup placement rm --rgw-zonegroup default \ + --placement-id default-placement \ + --storage-class CLOUDTIER + +:: - $ radosgw-admin zonegroup placement rm --rgw-zonegroup default \ - --placement-id default-placement \ - --storage-class CLOUDTIER [ { "key": "default-placement", @@ -289,6 +311,10 @@ For example:: ] + radosgw-admin zonegroup placement rm --rgw-zonegroup default \ + --placement-id default-placement \ + --storage-class CLOUDTIER + Object Modification and Limitations ----------------------------------- @@ -345,7 +371,12 @@ objects intact. For example:: - # s3cmd info s3://bucket/lc.txt +.. prompt:: bash $ + + s3cmd info s3://bucket/lc.txt + +:: + s3://bucket/lc.txt (object): File size: 12 Last mod: Mon, 21 Dec 2020 10:25:56 GMT @@ -358,7 +389,12 @@ For example:: ACL: M. Tester: FULL_CONTROL x-amz-meta-s3cmd-attrs: atime:1608466266/ctime:1597606156/gid:0/gname:root/md5:ed076287532e86365e841e92bfc50d8c/mode:33188/mtime:1597605793/uid:0/uname:root - # s3cmd get s3://bucket/lc.txt lc_restore.txt +.. prompt:: bash $ + + s3cmd get s3://bucket/lc.txt lc_restore.txt + +:: + download: 's3://bucket/lc.txt' -> 'lc_restore.txt' [1 of 1] ERROR: S3 error: 403 (InvalidObjectState) diff --git a/doc/radosgw/multisite-sync-policy.rst b/doc/radosgw/multisite-sync-policy.rst index 7a6e7105ce637..d377853c44f08 100644 --- a/doc/radosgw/multisite-sync-policy.rst +++ b/doc/radosgw/multisite-sync-policy.rst @@ -29,7 +29,7 @@ A sync policy group can be in 3 states: A policy can be defined at the bucket level. A bucket level sync policy inherits the data flow of the zonegroup policy, and can only define a subset of what the zonegroup allows. -A wildcard zone, and a wildcard bucket parameter in the policy defines all relevant zones, or all relevant buckets. In the context of a bucket policy it means the current bucket instance. A disaster recovery configuration where entire zones are mirrored doesn't require configuring anything on the buckets. However, for a fine grained bucket sync it would be better to configure the pipes to be synced by allowing (status=allowed) them at the zonegroup level (e.g., using wildcards), but only enable the specific sync at the bucket level (status=enabled). If needed, the policy at the bucket level can limit the data movement to specific relevant zones. +A wildcard zone, and a wildcard bucket parameter in the policy defines all relevant zones, or all relevant buckets. In the context of a bucket policy it means the current bucket instance. A disaster recovery configuration where entire zones are mirrored doesn't require configuring anything on the buckets. However, for a fine grained bucket sync it would be better to configure the pipes to be synced by allowing (``status=allowed``) them at the zonegroup level (e.g., using wildcards), but only enable the specific sync at the bucket level (``status=enabled``). If needed, the policy at the bucket level can limit the data movement to specific relevant zones. .. important:: Any changes to the zonegroup policy needs to be applied on the zonegroup master zone, and require period update and commit. Changes @@ -52,9 +52,9 @@ Get Sync Policy To retrieve the current zonegroup sync policy, or a specific bucket policy: -:: +.. prompt:: bash # - # radosgw-admin sync policy get [--bucket=] + radosgw-admin sync policy get [--bucket=] Create Sync Policy Group @@ -62,46 +62,45 @@ Create Sync Policy Group To create a sync policy group: -:: +.. prompt:: bash # + + radosgw-admin sync group create [--bucket=] \ + --group-id= \ + --status= + - # radosgw-admin sync group create [--bucket=] \ - --group-id= \ - --status= \ - - Modify Sync Policy Group ~~~~~~~~~~~~~~~~~~~~~~~~ To modify a sync policy group: -:: +.. prompt:: bash # - # radosgw-admin sync group modify [--bucket=] \ - --group-id= \ - --status= \ + radosgw-admin sync group modify [--bucket=] \ + --group-id= \ + --status= Show Sync Policy Group -~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~ To show a sync policy group: -:: +.. prompt:: bash # + + radosgw-admin sync group get [--bucket=] \ + --group-id= - # radosgw-admin sync group get [--bucket=] \ - --group-id= - Remove Sync Policy Group ~~~~~~~~~~~~~~~~~~~~~~~~ To remove a sync policy group: -:: - - # radosgw-admin sync group remove [--bucket=] \ - --group-id= +.. prompt:: bash # + radosgw-admin sync group remove [--bucket=] \ + --group-id= Create Sync Flow @@ -109,25 +108,25 @@ Create Sync Flow - To create or update directional sync flow: -:: +.. prompt:: bash # - # radosgw-admin sync group flow create [--bucket=] \ - --group-id= \ - --flow-id= \ - --flow-type=directional \ - --source-zone= \ - --dest-zone= + radosgw-admin sync group flow create [--bucket=] \ + --group-id= \ + --flow-id= \ + --flow-type=directional \ + --source-zone= \ + --dest-zone= - To create or update symmetrical sync flow: -:: +.. prompt:: bash # - # radosgw-admin sync group flow create [--bucket=] \ - --group-id= \ - --flow-id= \ - --flow-type=symmetrical \ - --zones= + radosgw-admin sync group flow create [--bucket=] \ + --group-id= \ + --flow-id= \ + --flow-type=symmetrical \ + --zones= Where zones are a comma separated list of all the zones that need to add to the flow. @@ -138,25 +137,25 @@ Remove Sync Flow Zones - To remove directional sync flow: -:: +.. prompt:: bash # - # radosgw-admin sync group flow remove [--bucket=] \ - --group-id= \ - --flow-id= \ - --flow-type=directional \ - --source-zone= \ - --dest-zone= + radosgw-admin sync group flow remove [--bucket=] \ + --group-id= \ + --flow-id= \ + --flow-type=directional \ + --source-zone= \ + --dest-zone= - To remove specific zones from symmetrical sync flow: -:: +.. prompt:: bash # - # radosgw-admin sync group flow remove [--bucket=] \ - --group-id= \ - --flow-id= \ - --flow-type=symmetrical \ - --zones= + radosgw-admin sync group flow remove [--bucket=] \ + --group-id= \ + --flow-id= \ + --flow-type=symmetrical \ + --zones= Where zones are a comma separated list of all zones to remove from the flow. @@ -164,12 +163,12 @@ Where zones are a comma separated list of all zones to remove from the flow. - To remove symmetrical sync flow: -:: +.. prompt:: bash # - # radosgw-admin sync group flow remove [--bucket=] \ - --group-id= \ - --flow-id= \ - --flow-type=symmetrical + radosgw-admin sync group flow remove [--bucket=] \ + --group-id= \ + --flow-id= \ + --flow-type=symmetrical Create Sync Pipe @@ -178,25 +177,25 @@ Create Sync Pipe To create sync group pipe, or update its parameters: -:: +.. prompt:: bash # - # radosgw-admin sync group pipe create [--bucket=] \ - --group-id= \ - --pipe-id= \ - --source-zones= \ - [--source-bucket=] \ - [--source-bucket-id=] \ - --dest-zones= \ - [--dest-bucket=] \ - [--dest-bucket-id=] \ - [--prefix=] \ - [--prefix-rm] \ - [--tags-add=] \ - [--tags-rm=] \ - [--dest-owner=] \ - [--storage-class=] \ - [--mode=] \ - [--uid=] + radosgw-admin sync group pipe create [--bucket=] \ + --group-id= \ + --pipe-id= \ + --source-zones= \ + [--source-bucket=] \ + [--source-bucket-id=] \ + --dest-zones= \ + [--dest-bucket=] \ + [--dest-bucket-id=] \ + [--prefix=] \ + [--prefix-rm] \ + [--tags-add=] \ + [--tags-rm=] \ + [--dest-owner=] \ + [--storage-class=] \ + [--mode=] \ + [--uid=] Zones are either a list of zones, or '*' (wildcard). Wildcard zones mean any zone that matches the sync flow rules. @@ -214,17 +213,17 @@ Remove Sync Pipe To remove specific sync group pipe params, or the entire pipe: -:: +.. prompt:: bash # - # radosgw-admin sync group pipe remove [--bucket=] \ - --group-id= \ - --pipe-id= \ - [--source-zones=] \ - [--source-bucket=] \ - [--source-bucket-id=] \ - [--dest-zones=] \ - [--dest-bucket=] \ - [--dest-bucket-id=] + radosgw-admin sync group pipe remove [--bucket=] \ + --group-id= \ + --pipe-id= \ + [--source-zones=] \ + [--source-bucket=] \ + [--source-bucket-id=] \ + [--dest-zones=] \ + [--dest-bucket=] \ + [--dest-bucket-id=] Sync Info @@ -232,10 +231,10 @@ Sync Info To get information about the expected sync sources and targets (as defined by the sync policy): -:: +.. prompt:: bash # - # radosgw-admin sync info [--bucket=] \ - [--effective-zone-name=] + radosgw-admin sync info [--bucket=] \ + [--effective-zone-name=] Since a bucket can define a policy that defines data movement from it towards a different bucket at a different zone, when the policy is created we also generate a list of bucket dependencies that are used as hints when a sync of any particular bucket happens. The fact that a bucket references another bucket does not mean it actually syncs to/from it, as the data flow might not permit it. @@ -252,94 +251,99 @@ Example 1: Two Zones, Complete Mirror This is similar to older (pre ``Octopus``) sync capabilities, but being done via the new sync policy engine. Note that changes to the zonegroup sync policy require a period update and commit. +.. prompt:: bash [us-east]# + + radosgw-admin sync group create --group-id=group1 --status=allowed + radosgw-admin sync group flow create --group-id=group1 \ + --flow-id=flow-mirror --flow-type=symmetrical \ + --zones=us-east,us-west + radosgw-admin sync group pipe create --group-id=group1 \ + --pipe-id=pipe1 --source-zones='*' \ + --source-bucket='*' --dest-zones='*' \ + --dest-bucket='*' + radosgw-admin sync group modify --group-id=group1 --status=enabled + radosgw-admin period update --commit + radosgw-admin sync info --bucket=buck + :: - [us-east] $ radosgw-admin sync group create --group-id=group1 --status=allowed - [us-east] $ radosgw-admin sync group flow create --group-id=group1 \ - --flow-id=flow-mirror --flow-type=symmetrical \ - --zones=us-east,us-west - [us-east] $ radosgw-admin sync group pipe create --group-id=group1 \ - --pipe-id=pipe1 --source-zones='*' \ - --source-bucket='*' --dest-zones='*' \ - --dest-bucket='*' - [us-east] $ radosgw-admin sync group modify --group-id=group1 --status=enabled - [us-east] $ radosgw-admin period update --commit - - $ radosgw-admin sync info --bucket=buck - { - "sources": [ - { - "id": "pipe1", - "source": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-east", - "bucket": "buck:115b12b3-....4409.1" - }, - "params": { - ... - } - } - ], - "dests": [ - { - "id": "pipe1", - "source": { - "zone": "us-east", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - ... - } - ], - ... - } - } + { + "sources": [ + { + "id": "pipe1", + "source": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-east", + "bucket": "buck:115b12b3-....4409.1" + }, + "params": { + ... + } + } + ], + "dests": [ + { + "id": "pipe1", + "source": { + "zone": "us-east", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + ... + } + ], + ... + } + } Note that the "id" field in the output above reflects the pipe rule that generated that entry, a single rule can generate multiple sync entries as can be seen in the example. +.. prompt:: bash [us-west]# + + radosgw-admin sync info --bucket=buck + :: - [us-west] $ radosgw-admin sync info --bucket=buck - { - "sources": [ - { - "id": "pipe1", - "source": { - "zone": "us-east", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - ... - } - ], - "dests": [ - { - "id": "pipe1", - "source": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-east", - "bucket": "buck:115b12b3-....4409.1" - }, - ... - } - ], - ... - } + { + "sources": [ + { + "id": "pipe1", + "source": { + "zone": "us-east", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + ... + } + ], + "dests": [ + { + "id": "pipe1", + "source": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-east", + "bucket": "buck:115b12b3-....4409.1" + }, + ... + } + ], + ... + } @@ -348,87 +352,93 @@ Example 2: Directional, Entire Zone Backup Also similar to older sync capabilities. In here we add a third zone, ``us-west-2`` that will be a replica of ``us-west``, but data will not be replicated back from it. -:: +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group flow create --group-id=group1 \ - --flow-id=us-west-backup --flow-type=directional \ - --source-zone=us-west --dest-zone=us-west-2 - [us-east] $ radosgw-admin period update --commit + radosgw-admin sync group flow create --group-id=group1 \ + --flow-id=us-west-backup --flow-type=directional \ + --source-zone=us-west --dest-zone=us-west-2 + radosgw-admin period update --commit Note that us-west has two dests: +.. prompt:: bash [us-west]# + + radosgw-admin sync info --bucket=buck + :: - [us-west] $ radosgw-admin sync info --bucket=buck - { - "sources": [ - { - "id": "pipe1", - "source": { - "zone": "us-east", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - ... - } - ], - "dests": [ - { - "id": "pipe1", - "source": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-east", - "bucket": "buck:115b12b3-....4409.1" - }, - ... - }, - { - "id": "pipe1", - "source": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-west-2", - "bucket": "buck:115b12b3-....4409.1" - }, - ... - } - ], - ... - } + { + "sources": [ + { + "id": "pipe1", + "source": { + "zone": "us-east", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + ... + } + ], + "dests": [ + { + "id": "pipe1", + "source": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-east", + "bucket": "buck:115b12b3-....4409.1" + }, + ... + }, + { + "id": "pipe1", + "source": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-west-2", + "bucket": "buck:115b12b3-....4409.1" + }, + ... + } + ], + ... + } Whereas us-west-2 has only source and no destinations: +.. prompt:: bash [us-west-2]# + + radosgw-admin sync info --bucket=buck + :: - [us-west-2] $ radosgw-admin sync info --bucket=buck - { - "sources": [ - { - "id": "pipe1", - "source": { - "zone": "us-west", - "bucket": "buck:115b12b3-....4409.1" - }, - "dest": { - "zone": "us-west-2", - "bucket": "buck:115b12b3-....4409.1" - }, - ... - } - ], - "dests": [], - ... - } + { + "sources": [ + { + "id": "pipe1", + "source": { + "zone": "us-west", + "bucket": "buck:115b12b3-....4409.1" + }, + "dest": { + "zone": "us-west-2", + "bucket": "buck:115b12b3-....4409.1" + }, + ... + } + ], + "dests": [], + ... + } @@ -437,38 +447,36 @@ Example 3: Mirror a Specific Bucket Using the same group configuration, but this time switching it to ``allowed`` state, which means that sync is allowed but not enabled. -:: +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group modify --group-id=group1 --status=allowed - [us-east] $ radosgw-admin period update --commit + radosgw-admin sync group modify --group-id=group1 --status=allowed + radosgw-admin period update --commit And we will create a bucket level policy rule for existing bucket ``buck2``. Note that the bucket needs to exist before being able to set this policy, and admin commands that modify bucket policies need to run on the master zone, however, they do not require period update. There is no need to change the data flow, as it is inherited from the zonegroup policy. A bucket policy flow will only be a subset of the flow defined in the zonegroup policy. Same goes for pipes, although a bucket policy can enable pipes that are not enabled (albeit not forbidden) at the zonegroup policy. -:: - - [us-east] $ radosgw-admin sync group create --bucket=buck2 \ - --group-id=buck2-default --status=enabled +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe create --bucket=buck2 \ - --group-id=buck2-default --pipe-id=pipe1 \ - --source-zones='*' --dest-zones='*' + radosgw-admin sync group create --bucket=buck2 \ + --group-id=buck2-default --status=enabled + radosgw-admin sync group pipe create --bucket=buck2 \ + --group-id=buck2-default --pipe-id=pipe1 \ + --source-zones='*' --dest-zones='*' -Example 4: Limit Bucket Sync To Specific Zones +Example 4: Limit Bucket Sync to Specific Zones ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This will only sync ``buck3`` to ``us-east`` (from any zone that flow allows to sync into ``us-east``). -:: - - [us-east] $ radosgw-admin sync group create --bucket=buck3 \ - --group-id=buck3-default --status=enabled +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe create --bucket=buck3 \ - --group-id=buck3-default --pipe-id=pipe1 \ - --source-zones='*' --dest-zones=us-east + radosgw-admin sync group create --bucket=buck3 \ + --group-id=buck3-default --status=enabled + radosgw-admin sync group pipe create --bucket=buck3 \ + --group-id=buck3-default --pipe-id=pipe1 \ + --source-zones='*' --dest-zones=us-east @@ -479,97 +487,98 @@ Note that bucket sync only works (currently) across zones and not within the sam Set ``buck4`` to pull data from ``buck5``: -:: - - [us-east] $ radosgw-admin sync group create --bucket=buck4 ' - --group-id=buck4-default --status=enabled +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe create --bucket=buck4 \ - --group-id=buck4-default --pipe-id=pipe1 \ - --source-zones='*' --source-bucket=buck5 \ - --dest-zones='*' + radosgw-admin sync group create --bucket=buck4 \ + --group-id=buck4-default --status=enabled + radosgw-admin sync group pipe create --bucket=buck4 \ + --group-id=buck4-default --pipe-id=pipe1 \ + --source-zones='*' --source-bucket=buck5 \ + --dest-zones='*' can also limit it to specific zones, for example the following will only sync data originated in us-west: -:: +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe modify --bucket=buck4 \ - --group-id=buck4-default --pipe-id=pipe1 \ - --source-zones=us-west --source-bucket=buck5 \ - --dest-zones='*' + radosgw-admin sync group pipe modify --bucket=buck4 \ + --group-id=buck4-default --pipe-id=pipe1 \ + --source-zones=us-west --source-bucket=buck5 \ + --dest-zones='*' Checking the sync info for ``buck5`` on ``us-west`` is interesting: +.. prompt:: bash [us-west]# + + radosgw-admin sync info --bucket=buck5 + :: - [us-west] $ radosgw-admin sync info --bucket=buck5 - { - "sources": [], - "dests": [], - "hints": { - "sources": [], - "dests": [ - "buck4:115b12b3-....14433.2" - ] - }, - "resolved-hints-1": { - "sources": [], - "dests": [ - { - "id": "pipe1", - "source": { - "zone": "us-west", - "bucket": "buck5" - }, - "dest": { - "zone": "us-east", - "bucket": "buck4:115b12b3-....14433.2" - }, - ... - }, - { - "id": "pipe1", - "source": { - "zone": "us-west", - "bucket": "buck5" - }, - "dest": { - "zone": "us-west-2", - "bucket": "buck4:115b12b3-....14433.2" - }, - ... - } - ] - }, - "resolved-hints": { - "sources": [], - "dests": [] - } - } + { + "sources": [], + "dests": [], + "hints": { + "sources": [], + "dests": [ + "buck4:115b12b3-....14433.2" + ] + }, + "resolved-hints-1": { + "sources": [], + "dests": [ + { + "id": "pipe1", + "source": { + "zone": "us-west", + "bucket": "buck5" + }, + "dest": { + "zone": "us-east", + "bucket": "buck4:115b12b3-....14433.2" + }, + ... + }, + { + "id": "pipe1", + "source": { + "zone": "us-west", + "bucket": "buck5" + }, + "dest": { + "zone": "us-west-2", + "bucket": "buck4:115b12b3-....14433.2" + }, + ... + } + ] + }, + "resolved-hints": { + "sources": [], + "dests": [] + } + } Note that there are resolved hints, which means that the bucket ``buck5`` found about ``buck4`` syncing from it indirectly, and not from its own policy (the policy for ``buck5`` itself is empty). -Example 6: Sync To Different Bucket +Example 6: Sync to Different Bucket ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The same mechanism can work for configuring data to be synced to (vs. synced from as in the previous example). Note that internally data is still pulled from the source at the destination zone: Set ``buck6`` to "push" data to ``buck5``: -:: - - [us-east] $ radosgw-admin sync group create --bucket=buck6 \ - --group-id=buck6-default --status=enabled +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe create --bucket=buck6 \ - --group-id=buck6-default --pipe-id=pipe1 \ - --source-zones='*' --source-bucket='*' \ - --dest-zones='*' --dest-bucket=buck5 + radosgw-admin sync group create --bucket=buck6 \ + --group-id=buck6-default --status=enabled + radosgw-admin sync group pipe create --bucket=buck6 \ + --group-id=buck6-default --pipe-id=pipe1 \ + --source-zones='*' --source-bucket='*' \ + --dest-zones='*' --dest-bucket=buck5 A wildcard bucket name means the current bucket in the context of bucket sync policy. @@ -581,86 +590,88 @@ Example 7: Source Filters Sync from ``buck8`` to ``buck9``, but only objects that start with ``foo/``: -:: - - [us-east] $ radosgw-admin sync group create --bucket=buck8 \ - --group-id=buck8-default --status=enabled +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe create --bucket=buck8 \ - --group-id=buck8-default --pipe-id=pipe-prefix \ - --prefix=foo/ --source-zones='*' --dest-zones='*' \ - --dest-bucket=buck9 + radosgw-admin sync group create --bucket=buck8 \ + --group-id=buck8-default --status=enabled + radosgw-admin sync group pipe create --bucket=buck8 \ + --group-id=buck8-default --pipe-id=pipe-prefix \ + --prefix=foo/ --source-zones='*' --dest-zones='*' \ + --dest-bucket=buck9 Also sync from ``buck8`` to ``buck9`` any object that has the tags ``color=blue`` or ``color=red``: -:: +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe create --bucket=buck8 \ - --group-id=buck8-default --pipe-id=pipe-tags \ - --tags-add=color=blue,color=red --source-zones='*' \ - --dest-zones='*' --dest-bucket=buck9 + radosgw-admin sync group pipe create --bucket=buck8 \ + --group-id=buck8-default --pipe-id=pipe-tags \ + --tags-add=color=blue,color=red --source-zones='*' \ + --dest-zones='*' --dest-bucket=buck9 And we can check the expected sync in ``us-east`` (for example): +.. prompt:: bash [us-east]# + + radosgw-admin sync info --bucket=buck8 + :: - [us-east] $ radosgw-admin sync info --bucket=buck8 - { - "sources": [], - "dests": [ - { - "id": "pipe-prefix", - "source": { - "zone": "us-east", - "bucket": "buck8:115b12b3-....14433.5" - }, - "dest": { - "zone": "us-west", - "bucket": "buck9" - }, - "params": { - "source": { - "filter": { - "prefix": "foo/", - "tags": [] - } - }, - ... - } - }, - { - "id": "pipe-tags", - "source": { - "zone": "us-east", - "bucket": "buck8:115b12b3-....14433.5" - }, - "dest": { - "zone": "us-west", - "bucket": "buck9" - }, - "params": { - "source": { - "filter": { - "tags": [ - { - "key": "color", - "value": "blue" - }, - { - "key": "color", - "value": "red" - } - ] - } - }, - ... - } - } - ], - ... - } + { + "sources": [], + "dests": [ + { + "id": "pipe-prefix", + "source": { + "zone": "us-east", + "bucket": "buck8:115b12b3-....14433.5" + }, + "dest": { + "zone": "us-west", + "bucket": "buck9" + }, + "params": { + "source": { + "filter": { + "prefix": "foo/", + "tags": [] + } + }, + ... + } + }, + { + "id": "pipe-tags", + "source": { + "zone": "us-east", + "bucket": "buck8:115b12b3-....14433.5" + }, + "dest": { + "zone": "us-west", + "bucket": "buck9" + }, + "params": { + "source": { + "filter": { + "tags": [ + { + "key": "color", + "value": "blue" + }, + { + "key": "color", + "value": "red" + } + ] + } + }, + ... + } + } + ], + ... + } Note that there aren't any sources, only two different destinations (one for each configuration). When the sync process happens it will select the relevant rule for each object it syncs. @@ -673,16 +684,15 @@ Example 8: Destination Params: Storage Class Storage class of the destination objects can be configured: -:: - - [us-east] $ radosgw-admin sync group create --bucket=buck10 \ - --group-id=buck10-default --status=enabled +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe create --bucket=buck10 \ - --group-id=buck10-default \ - --pipe-id=pipe-storage-class \ - --source-zones='*' --dest-zones=us-west-2 \ - --storage-class=CHEAP_AND_SLOW + radosgw-admin sync group create --bucket=buck10 \ + --group-id=buck10-default --status=enabled + radosgw-admin sync group pipe create --bucket=buck10 \ + --group-id=buck10-default \ + --pipe-id=pipe-storage-class \ + --source-zones='*' --dest-zones=us-west-2 \ + --storage-class=CHEAP_AND_SLOW Example 9: Destination Params: Destination Owner Translation @@ -691,26 +701,26 @@ Example 9: Destination Params: Destination Owner Translation Set the destination objects owner as the destination bucket owner. This requires specifying the uid of the destination bucket: -:: +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group create --bucket=buck11 \ - --group-id=buck11-default --status=enabled + radosgw-admin sync group create --bucket=buck11 \ + --group-id=buck11-default --status=enabled + radosgw-admin sync group pipe create --bucket=buck11 \ + --group-id=buck11-default --pipe-id=pipe-dest-owner \ + --source-zones='*' --dest-zones='*' \ + --dest-bucket=buck12 --dest-owner=joe - [us-east] $ radosgw-admin sync group pipe create --bucket=buck11 \ - --group-id=buck11-default --pipe-id=pipe-dest-owner \ - --source-zones='*' --dest-zones='*' \ - --dest-bucket=buck12 --dest-owner=joe Example 10: Destination Params: User Mode ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ User mode makes sure that the user has permissions to both read the objects, and write to the destination bucket. This requires that the uid of the user (which in its context the operation executes) is specified. -:: +.. prompt:: bash [us-east]# - [us-east] $ radosgw-admin sync group pipe modify --bucket=buck11 \ - --group-id=buck11-default --pipe-id=pipe-dest-owner \ - --mode=user --uid=jenny + radosgw-admin sync group pipe modify --bucket=buck11 \ + --group-id=buck11-default --pipe-id=pipe-dest-owner \ + --mode=user --uid=jenny diff --git a/doc/radosgw/sync-modules.rst b/doc/radosgw/sync-modules.rst index 61797edc857c8..53f16c70cb3a9 100644 --- a/doc/radosgw/sync-modules.rst +++ b/doc/radosgw/sync-modules.rst @@ -32,7 +32,7 @@ for configuring any sync plugin Archive Sync Module .. note ``rgw`` is the default sync plugin and there is no need to explicitly - configure this + configure this. Requirements and Assumptions ---------------------------- @@ -50,25 +50,24 @@ Configuring Sync Modules Create the third zone similar to the :ref:`multisite` docs, for example -:: - - # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-es \ - --access-key={system-key} --secret={secret} --endpoints=http://rgw-es:80 +.. prompt:: bash # + radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-es \ + --access-key={system-key} --secret={secret} --endpoints=http://rgw-es:80 A sync module can be configured for this zone via the following -:: +.. prompt:: bash # - # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type={tier-type} --tier-config={set of key=value pairs} + radosgw-admin zone modify --rgw-zone={zone-name} --tier-type={tier-type} --tier-config={set of key=value pairs} For example in the ``elasticsearch`` sync module -:: +.. prompt:: bash # - # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type=elasticsearch \ + radosgw-admin zone modify --rgw-zone={zone-name} --tier-type=elasticsearch \ --tier-config=endpoint=http://localhost:9200,num_shards=10,num_replicas=1 @@ -76,18 +75,17 @@ For the various supported tier-config options refer to the `elasticsearch sync m Finally update the period +.. prompt:: bash # -:: - - # radosgw-admin period update --commit + radosgw-admin period update --commit Now start the radosgw in the zone -:: +.. prompt:: bash # - # systemctl start ceph-radosgw@rgw.`hostname -s` - # systemctl enable ceph-radosgw@rgw.`hostname -s` + systemctl start ceph-radosgw@rgw.`hostname -s` + systemctl enable ceph-radosgw@rgw.`hostname -s` -- 2.39.5