From: Abhishek Lekshmanan Date: Thu, 25 Jan 2018 16:16:23 +0000 (+0100) Subject: doc: add sync modules to toc X-Git-Tag: v13.0.2~140^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=640b2ae2baebc05d9986361d4a55df4ac4aa2917;p=ceph.git doc: add sync modules to toc Adding a seperate toc under sync modules for elastic, also renamed sync-plugins to sync-modules Signed-off-by: Abhishek Lekshmanan --- diff --git a/doc/radosgw/index.rst b/doc/radosgw/index.rst index be10335f2597..d4d435e5a832 100644 --- a/doc/radosgw/index.rst +++ b/doc/radosgw/index.rst @@ -55,6 +55,7 @@ you may write data with one API and retrieve it with the other. Server-Side Encryption Bucket Policy Dynamic bucket index resharding + Sync Modules Data Layout in RADOS troubleshooting Manpage radosgw <../../man/8/radosgw> diff --git a/doc/radosgw/sync-modules.rst b/doc/radosgw/sync-modules.rst new file mode 100644 index 000000000000..2cfba98a7714 --- /dev/null +++ b/doc/radosgw/sync-modules.rst @@ -0,0 +1,94 @@ +============ +Sync Modules +============ + +.. versionadded:: Kraken + +The `Multisite`_ functionality of RGW introduced in Jewel allowed the ability to +create multiple zones and mirror data and metadata between them. ``Sync Modules`` +are built atop of the multisite framework that allows for forwarding data and +metadata to a different external tier. A sync module allows for a set of actions +to be performed whenever a change in data occurs (metadata ops like bucket or +user creation etc. are also regarded as changes in data). As the rgw multisite +changes are eventually consistent at remote sites, changes are propagated +asynchronously. This would allow for unlocking use cases such as backing up the +object storage to an external cloud cluster or a custom backup solution using +tape drives, indexing metadata in ElasticSearch etc. + +A sync module configuration is local to a zone. The sync module determines +whether the zone exports data or can only consume data that was modified in +another zone. As of luminous the supported sync plugins are `elasticsearch`_, +``rgw``, which is the default sync plugin that synchronises data between the +zones and ``log`` which is a trivial sync plugin that logs the metadata +operation that happens in the remote zones. The following docs are written with +the example of a zone using `elasticsearch sync module`_, the process would be similar +for configuring any sync plugin + +.. toctree:: + :maxdepth: 1 + + ElasticSearch Sync Module + +.. note ``rgw`` is the default sync plugin and there is no need to explicitly + configure this + +Requirements and Assumptions +---------------------------- + +Let us assume a simple multisite configuration as described in the `Multisite`_ +docs, of 2 zones ``us-east`` and ``us-west``, let's add a third zone +``us-east-es`` which is a zone that only processes metadata from the other +sites. This zone can be in the same or a different ceph cluster as ``us-east``. +This zone would only consume metadata from other zones and RGWs in this zone +will not serve any end user requests directly. + + +Configuring Sync Modules +------------------------ + +Create the third zone similar to the `Multisite`_ docs, for example + +:: + + # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-es \ + --access-key={system-key} --secret={secret} --endpoints=http://rgw-es:80 + + + +A sync module can be configured for this zone via the following + +:: + + # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type={tier-type} --tier-config={set of key=value pairs} + + +For example in the ``elasticsearch`` sync module + +:: + + # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type=elasticsearch \ + --tier-config=endpoint=http://localhost:9200,num_shards=10,num_replicas=1 + + +For the various supported tier-config options refer to the `elasticsearch sync module`_ docs + +Finally update the period + + +:: + + # radosgw-admin period update --commit + + +Now start the radosgw in the zone + +:: + + # systemctl start ceph-radosgw@rgw.`hostname -s` + # systemctl enable ceph-radosgw@rgw.`hostname -s` + + + +.. _`Multisite`: ../multisite +.. _`elasticsearch sync module`: ../elastic-sync-module +.. _`elasticsearch`: ../elastic-sync-module diff --git a/doc/radosgw/sync-plugins.rst b/doc/radosgw/sync-plugins.rst deleted file mode 100644 index 4b8917488581..000000000000 --- a/doc/radosgw/sync-plugins.rst +++ /dev/null @@ -1,89 +0,0 @@ -============ -Sync Modules -============ - -.. versionadded:: Kraken - -The `Multisite`_ functionality of RGW introduced in Jewel allowed the ability to -create multiple zones and mirror data and metadata between them. ``Sync Modules`` -are built atop of the multisite framework that allows for forwarding data and -metadata to a different external tier. A sync module allows for a set of actions -to be performed whenever a change in data occurs (metadata ops like bucket or -user creation etc. are also regarded as changes in data). As the rgw multisite -changes are eventually consistent at remote sites, changes are propagated -asynchronously. This would allow for unlocking use cases such as backing up the -object storage to an external cloud cluster or a custom backup solution using -tape drives, indexing metadata in ElasticSearch etc. - -A sync module configuration is local to a zone. The sync module determines -whether the zone exports data or can only consume data that was modified in -another zone. As of luminous the supported sync plugins are `elasticsearch`_, -``rgw``, which is the default sync plugin that synchronises data between the -zones and ``log`` which is a trivial sync plugin that logs the metadata -operation that happens in the remote zones. The following docs are written with -the example of a zone using `elasticsearch sync module`_, the process would be similar -for configuring any sync plugin - -.. note ``rgw`` is the default sync plugin and there is no need to explicitly - configure this - -Requirements and Assumptions ----------------------------- - -Let us assume a simple multisite configuration as described in the `Multisite`_ -docs, of 2 zones ``us-east`` and ``us-west``, let's add a third zone -``us-east-es`` which is a zone that only processes metadata from the other -sites. This zone can be in the same or a different ceph cluster as ``us-east``. -This zone would only consume metadata from other zones and RGWs in this zone -will not serve any end user requests directly. - - -Configuring Sync Modules ------------------------- - -Create the third zone similar to the `Multisite`_ docs, for example - -:: - - # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-es \ - --access-key={system-key} --secret={secret} --endpoints=http://rgw-es:80 - - - -A sync module can be configured for this zone via the following - -:: - - # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type={tier-type} --tier-config={set of key=value pairs} - - -For example in the ``elasticsearch`` sync module - -:: - - # radosgw-admin zone modify --rgw-zone={zone-name} --tier-type=elasticsearch \ - --tier-config=endpoint=http://localhost:9200,num_shards=10,num_replicas=1 - - -For the various supported tier-config options refer to the `elasticsearch sync module`_ docs - -Finally update the period - - -:: - - # radosgw-admin period update --commit - - -Now start the radosgw in the zone - -:: - - # systemctl start ceph-radosgw@rgw.`hostname -s` - # systemctl enable ceph-radosgw@rgw.`hostname -s` - - - -.. _`Multisite`: ../multisite -.. _`elasticsearch sync module`: ../elastic-sync-module -.. _`elasticsearch`: ../elastic-sync-module