From: John Wilkins Date: Tue, 29 Oct 2013 00:46:06 +0000 (-0700) Subject: doc: Updated with a verified installation procedure and latest usage. X-Git-Tag: v0.73~26^2~8 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=bd507ef3fdc0ba45996e5c4270ede28a9ff2bbde;p=ceph.git doc: Updated with a verified installation procedure and latest usage. Signed-off-by: John Wilkins --- diff --git a/doc/radosgw/federated-config.rst b/doc/radosgw/federated-config.rst index 9a71cb351f3b..78ab3fbb30db 100644 --- a/doc/radosgw/federated-config.rst +++ b/doc/radosgw/federated-config.rst @@ -2,18 +2,22 @@ Configuring Federated Gateways ================================ -.. versionadded:: 0.71 +.. versionadded:: 0.72 Emperor -In Ceph version 0.71 and beyond, you may configure Ceph Object Gateways in a -federated architecture, with multiple regions, and with multiple zones for a -region. +In Ceph version 0.72 Emperor and beyond, you may configure each :term:`Ceph +Object Gateway` to participate in a federated architecture, with multiple +regions, and with multiple zones for a region. -- **Region**: A region represents a logical geographic area and contains one +- **Region**: A region represents a *logical* geographic area and contains one or more zones. A cluster with multiple regions must specify a master region. -- **Zone**: A zone is a logical grouping of one or more Ceph Object Gateway +- **Zone**: A zone is a *logical* grouping of one or more Ceph Object Gateway instance(s). A region has a master zone that processes client requests. + +Background +---------- + When you deploy a :term:`Ceph Object Store` service that spans geographical locales, configuring Ceph Object Gateway regions and metadata synchronization agents enables the service to maintain a global namespace, even though Ceph @@ -25,421 +29,465 @@ data synchronization agents enables the service to maintain one or more copy(ies) of the master zone's data. Extra copies of the data are important for failover, backup and disaster recovery. -You may deploy a single Ceph Storage Cluster with a federated architecture -if you have low latency network connections (this isn't recommended). You may -also deploy one Ceph Storage Cluster per region with a separate set of -pools for each zone (typical). You may also deploy a separate Ceph Storage -Cluster for each zone if your requirements and resources warrant this level -of redundancy. +You may deploy a single Ceph Storage Cluster with a federated architecture if +you have low latency network connections (this isn't recommended). You may also +deploy one Ceph Storage Cluster per region with a separate set of pools for +each zone (typical). You may also deploy a separate Ceph Storage Cluster for +each zone if your requirements and resources warrant this level of redundancy. +About this Guide +---------------- -Exemplary Cluster -================= +In the following sections, we will demonstrate how to configure a federated +cluster in two logical steps: -For the purposes of this configuration guide, we provide an exemplary procedure -for setting up two regions and two zones for each region. So the cluster will -comprise four gateway instances--one per zone. A production cluster at the -petabyte scale and beyond would likely involve deploying more instances per -zone. +#. **Configure a Master Region:** This section of the guide describes how to + set up a region with multiple zones, and how to synchronize data between the + master zone and the secondary zone(s) within the master region. + +#. **Configure a Secondary Region:** This section of the guide describes how + to repeat the section on setting up a master region and multiple zones so + that you have two regions with intra-zone synchronization in each region. + Finally, you will learn how to set up a metadata synchronization agent so + that you can maintain a global namespace for the regions in your cluster. -Let's assume the first region represents New York and the second region -represents London. For naming purposes, we will refer to them by their standard -abbreviations: -- New York: ``ny`` -- London: ``ldn`` +Configure a Master Region +========================= -Zones are logical containers for the gateway instances. The physical location of -the gateway instances is up to you, but disaster recovery is an important -consideration. A disaster can be as simple as a power failure or a network -failure. Yet, it can also involve a natural disaster or a significant political -or economic event. In such cases, it is prudent to maintain a secondary zone -outside of the geographic (not logical) region. +This section provides an exemplary procedure for setting up a region, and two +zones within the region. The cluster will comprise two gateway daemon +instances--one per zone. This region will serve as the master region. -Let's assume the master zone for each region is physically located in that -region, and the secondary zone is physically located in another region. For -continuity, our naming convention will use ``{region name}-{zone name}`` format, -but you can use any naming convention you prefer. -- New York Region, Master Zone: ``ny-ny`` -- New York Region, Secondary Zone: ``ny-ldn`` -- London Region, Master Zone: ``ldn-ldn`` -- London Region, Secondary Zone: ``ldn-ny`` +Naming for the Master Region +---------------------------- -.. image:: ../images/region-zone-sync.png +Before configuring the cluster, defining region, zone and instance names will +help you manage your cluster. Let's assume the region represents the United +States, and we refer to it by its standard abbreviation. -To configure the exemplary cluster, you must configure regions and zones. -Once you configure regions and zones, you must configure each instance of a -:term:`Ceph Object Gateway` to use the Ceph Storage Cluster as the data storage -backend. +- United States: ``us`` +Let's assume the zones represent the Eastern and Western United States. For +continuity, our naming convention will use ``{region name}-{zone name}`` format, +but you can use any naming convention you prefer. -Configuring Regions and Zones -============================= +- United States, East Region: ``us-east`` +- United States, West Region: ``us-west`` -For each :term:`Ceph Node` that runs a :term:`Ceph Object Gateway`, you must -install Apache, FastCGI and the Ceph Object Gateway daemon (``radosgw``). See -`Install Apache, FastCGI and Gateway`_ for details. +Finally, let's assume that zones may have more than one Ceph Object Gateway +instance per zone. For continuity, our naming convention will use +``{region name}-{zone name}-{instance}`` format, but you can use any naming +convention you prefer. -Default Region and Zone ------------------------ +- United States Region, Master Zone, Instance 1: ``us-east-1`` +- United States Region, Secondary Zone, Instance 1: ``us-west-1`` -The Ceph Object Gateway can generate its own default gateway and zone. These -defaults are the master region and master zone for a cluster. When you configure -your cluster for regions and zones, you will be replacing (and likely deleting, -if it exists) the default region and zone. -In the exemplary cluster, the New York region and New York zone (``ny-ny``) -are the master region and zone respectively. All other instances retrieve -their metadata from the master region and zone. +Create Pools +------------ + +You may have a Ceph Storage Cluster for the entire region or a Ceph Storage +Cluster for each zone. + +For continuity, our naming convention will use ``{region name}-{zone name}`` +format prepended to the pool name, but you can use any naming convention you +prefer. For example: + + +- ``.us.rgw.root`` + +- ``.us-east.rgw.root`` +- ``.us-east.rgw.control`` +- ``.us-east.rgw.gc`` +- ``.us-east.log`` +- ``.us-east.intent-log`` +- ``.us-east.usage`` +- ``.us-east.users`` +- ``.us-east.users.email`` +- ``.us-east.users.swift`` +- ``.us-east.users.uid`` + +- ``.us-west.rgw.root`` +- ``.us-west.rgw.control`` +- ``.us-west.rgw.gc`` +- ``.us-west.log`` +- ``.us-west.intent-log`` +- ``.us-west.usage`` +- ``.us-west.users`` +- ``.us-west.users.email`` +- ``.us-west.users.swift`` +- ``.us-west.users.uid`` + +See `Configuration Reference - Pools`_ for details on the default pools for +gateways. See `Pools`_ for details on creating pools. Execute the following +to create a pool:: + ceph osd pool create {poolname} {pg-num} {pgp-num} -Create Regions --------------- -#. Configure a region infile called ``region.json`` for the ``ny`` region. +.. tip:: When adding a large number of pools, it may take some time for your + cluster to return to a ``active + clean`` state. - Copy the contents of the following example to a text editor. Set - ``is_master`` to ``true``. Replace ``{fqdn}`` with the fully-qualified - domain name of the endpoint. It will specify a master zone as ``ny-ny`` and - list it in the ``zones`` list along with the ``ny-ldn`` zone. - See `Configuration Reference - Regions`_ for details.:: +.. topic:: CRUSH Maps - { "name": "ny", - "api_name": "ny", - "is_master": "true", - "endpoints": [ - "http:\/\/{fqdn}:80\/"], - "master_zone": "ny-ny", - "zones": [ - { "name": "ny-ny", - "endpoints": [ - "http:\/\/{fqdn}:80\/"], - "log_meta": "false", - "log_data": "false"}, - { "name": "ny-ldn", - "endpoints": [ - "http:\/\/{fqdn}:80\/"], - "log_meta": "false", - "log_data": "false"}], - "placement_targets": [], - "default_placement": ""} + When deploying a Ceph Storage Cluster for the entire region, consider + using a CRUSH rule for the the zone such that you do NOT have overlapping + failure domains. See `CRUSH Map`_ for details. +When you have completed this step, execute the following to ensure that +you have created all of the foregoing pools:: -#. Create the ``ny`` region using the ``region.json`` infile you just - created. :: + rados lspools - sudo radosgw-admin region set --infile region.json - Repeat the foregoing process to create region ``ldn``, but set - ``is_master`` to ``false``. Update the ``master_zone`` so that it is - set to ``ldn-ldn``. Ensure that you have entries for ``ldn-ldn`` and - ``ldn-ny`` in the ``zones`` values. +Create a Keyring +---------------- -#. Delete the default region. :: +Each instance must have a user name and key to communicate with a Ceph Storage +Cluster. In the following steps, we use an admin node to create a keyring. +Then, we create a client user name and key for each instance. Next, we add the +keys to the Ceph Storage Cluster(s). Finally, we distribute the key ring to +each node containing an instance. - rados -p .rgw.root rm region_info.default +#. Create a keyring. :: + + sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring + sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring -#. Set the ``ny`` region as the default region. :: - sudo radosgw-admin region default --rgw-region=ny +#. Generate a Ceph Object Gateway user name and key for each instance. :: - Only one region can be the default region for a cluster. + sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.us-east-1 --gen-key + sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.us-west-1 --gen-key -#. Update the region map. :: - sudo radosgw-admin regionmap update +#. Add capabilities to each key. See `Configuration Reference - Pools`_ for details + on the effect of write permissions for the monitor and creating pools. :: + sudo ceph-authtool -n client.radosgw.us-east-1 --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.keyring + sudo ceph-authtool -n client.radosgw.us-west-1 --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.keyring -Create Zone Users ------------------ +#. Once you have created a keyring and key to enable the Ceph Object Gateway + with access to the Ceph Storage Cluster, add each key as an entry to your + Ceph Storage Cluster(s). For example:: -Create zone users before configuring the zones. :: + sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.us-east-1 -i /etc/ceph/ceph.client.radosgw.keyring + sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.us-west-1 -i /etc/ceph/ceph.client.radosgw.keyring - sudo radosgw-admin user create --uid="ny-ny" --display-name="Region-NY Zone-NY" - sudo radosgw-admin user create --uid="ny-ldn" --display-name="Region-NY Zone-LDN" - sudo radosgw-admin user create --uid="ldn-ldn" --display-name="Region-LDN Zone-LDN" - sudo radosgw-admin user create --uid="ldn-ny" --display-name="Region-LDN Zone-NY" -Copy the ``access_key`` and ``secret_key`` fields for each user. You will need them -to configure each zone. +Install Apache/FastCGI +---------------------- -Create a Zone -------------- +For each :term:`Ceph Node` that runs a :term:`Ceph Object Gateway` daemon +instance, you must install Apache, FastCGI, the Ceph Object Gateway daemon +(``radosgw``) and the Ceph Object Gateway Sync Agent (``radosgw-agent``). +See `Install Apache, FastCGI and Gateway`_ for details. -#. Configure a zone infile called ``zone.json`` for the ``ny-ny`` zone. - Copy the contents of the following example to a text editor. - Paste the contents of the ``access_key`` and ``secret_key`` fields from the - step of creating a zone user into the ``system_key`` field. This - configuration uses pool names prepended with the region name and zone name. - See `Configuration Reference - Pools`_ for additional details on gateway - pools. See `Configuration Reference - Zones`_ for additional details on - zones. :: - - { "domain_root": ".ny-ny.rgw", - "control_pool": ".ny-ny.rgw.control", - "gc_pool": ".ny-ny.rgw.gc", - "log_pool": ".ny-ny.log", - "intent_log_pool": ".ny-ny.intent-log", - "usage_log_pool": ".ny-ny.usage", - "user_keys_pool": ".ny-ny.users", - "user_email_pool": ".ny-ny.users.email", - "user_swift_pool": ".ny-ny.users.swift", - "user_uid_pool": ".ny-ny.users.uid", - "system_key": { "access_key": "", "secret_key": ""} - } +Create Data Directories +----------------------- +Create data directories for each daemon instance on their respective +hosts. :: -#. Create the ``ny-ny`` zone using the ``zone.json`` infile you just - created. :: + ssh {us-east-1} + sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.us-east-1 + + ssh {us-west-1} + sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.us-west-1 + - sudo radosgw-admin zone set --rgw-zone=ny-ny --infile zone.json +Create a Gateway Configuration +------------------------------ - Repeat the previous two steps to create zones ``ny-ldn``, ``ldn-ldn``, - and ``ldn-ny`` (replacing ``ny-ny`` in the ``zone.json`` file). +For each instance, create an Ceph Object Gateway configuration file under the +``/etc/apache2/sites-available`` directory on the host(s) where you installed +the Ceph Object Gateway daemon(s). See below for an exemplary embodiment of a +gateway configuration as discussed in the following text. +.. literalinclude:: rgw.conf + :language: ini -#. Delete the default zone. :: +#. Replace the ``/{path}/{socket-name}`` entry with path to the socket and + the socket name. For example, + ``/var/run/ceph/client.radosgw.us-east-1.sock``. Ensure that you use the + same path and socket name in your ``ceph.conf`` entry. - rados -p .rgw.root rm zone_info.default +#. Replace the ``{fqdn}`` entry with the fully-qualified domain name of the + server. + +#. Replace the ``{email.address}`` entry with the email address for the + server administrator. + +#. Add a ``ServerAlias`` if you wish to use S3-style subdomains + (of course you do). +#. Save the configuration to a file (e.g., ``rgw-us-east.conf``). -#. Update the region map. :: +Repeat the process for the secondary zone (e.g., ``rgw-us-west.conf``). - radosgw-admin regionmap update +Enable the Configuration +------------------------ -Create Pools -============ - -If the username(s) and key(s) that provide your Ceph Object Gateway with access -to the Ceph Storage Cluster do not have write capability to the :term:`Ceph -Monitor`, you must create the pools manually. See `Configuration Reference - -Pools`_ for details on the default pools for gateways. See `Pools`_ for -details on creating pools. The default pools for a Ceph Object Gateway are: - -- ``.rgw`` -- ``.rgw.control`` -- ``.rgw.gc`` -- ``.log`` -- ``.intent-log`` -- ``.usage`` -- ``.users`` -- ``.users.email`` -- ``.users.swift`` -- ``.users.uid`` - -The `Exemplary Cluster`_ assumes that you will have a Ceph Storage Cluster for -each region, and that you will create pools for each zone that resides -**physically** in that region (e.g., ``ny-ny`` and ``ldn-ny`` in New York and -``ldn-ldn`` and ``ny-ldn`` in London). For each pool, prepend the name of the -zone name (e.g., ``.ny-ny``, ``.ny-ldn``, ``ldn-ldn``, ``ldn-ny``). For example: - -- ``.ny-ny.rgw`` -- ``.ny-ny.rgw.control`` -- ``.ny-ny.rgw.gc`` -- ``.ny-ny.log`` -- ``.ny-ny.intent-log`` -- ``.ny-ny.usage`` -- ``.ny-ny.users`` -- ``.ny-ny.users.email`` -- ``.ny-ny.users.swift`` -- ``.ny-ny.users.uid`` - - -Execute one of the following:: - - rados mkpool {poolname} - ceph osd pool create {poolname} {pg-num} {pgp-num} +For each instance, enable the gateway configuration and disable the +default site. +#. Enable the site for the gateway configuration. :: -.. tip:: When adding a large number of pools, it may take some time for your - cluster to return to a ``active + clean`` state. + sudo a2ensite {rgw-conf-filename} +#. Disable the default site. :: -Configuring Gateway Instances -============================= + sudo a2dissite default -The `Exemplary Cluster`_ assumes that you will configure an instance for -each zone. In larger deployments, you may need to configure multiple instances -per zone to handle higher loads. +.. note:: Failure to disable the default site can lead to problems. -Before you configure a gateway instance, determine an ID for the instance. You -can name a Ceph Object Gateway instance anything you like. In large clusters -with regions and zones, it may help to add region and zone names into your -instance name. For example:: - ny-ny-instance1 +Add a FastCGI Script +-------------------- -When referring to your instance identifier in the Ceph configuration file, it -is prepended with ``client.radosgw.``. For example, an instance named -``ny-ny-instance1`` will look like this:: +FastCGI requires a script for each Ceph Object Gateway instance to +enable the S3-compatible interface. To create the script, execute +the following procedures. - [client.radosgw.ny-ny-instance1] -Similarly, the default data path for an instance named -``ny-ny-instance1`` is prepended with ``{cluster}-radosgw.``. For -example:: +#. Go to the ``/var/www`` directory. :: - /var/lib/ceph/radosgw/ceph-radosgw.ny-ny-instance1 - + cd /var/www -Create a Data Directory ------------------------ +#. Open an editor with the file name ``s3gw.fcgi``. **Note:** The configuration + file specifies this filename. :: -Create a data directory on the node where you installed ``radosgw``. :: - - sudo mkdir -p /var/lib/ceph/radosgw/ceph-radosgw.ny-ny-instance1 - + sudo vim s3gw.fcgi -Create a Storage Cluster User ------------------------------ +#. Add a shell script with ``exec`` and the path to the gateway binary, + the path to the Ceph configuration file, and the user name (``-n``; + the same user name created in step 2 of `Create a Keyring`_. + Copy the following into the editor. :: -When accessing the Ceph Storage Cluster, each instance of a Ceph Object Gateway -must provide the Ceph Storage Cluster with a user name and key. We recommend -setting up at least one user name and key for each zone. See `Cephx -Administration`_ for a discussion on adding keyrings and keys. + #!/bin/sh + exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.{ID} -#. Create a keyring for the Ceph Object Gateway. For example:: + For example:: - sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.ny.keyring - sudo chmod +r /etc/ceph/ceph.client.radosgw.ny.keyring + #!/bin/sh + exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.us-east-1 +#. Save the file. -#. Generate a key so that the Ceph Object Gateway can provide a user name and - key to authenticate itself with the Ceph Storage Cluster. Then, add - capabilities to the key. See `Configuration Reference - Pools`_ for details - on the effect of write permissions for the monitor and creating pools. :: +#. Change the permissions on the file so that it is executable. :: - sudo ceph-authtool /etc/ceph/ceph.client.radosgw.ny.keyring -n client.radosgw.ny-ny --gen-key - sudo ceph-authtool /etc/ceph/ceph.client.radosgw.ny.keyring -n client.radosgw.ldn-ny --gen-key - sudo ceph-authtool -n client.radosgw.ny-ny --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.ny.keyring - sudo ceph-authtool -n client.radosgw.ldn-ny --cap osd 'allow rwx' --cap mon 'allow rw' /etc/ceph/ceph.client.radosgw.ny.keyring + sudo chmod +x s3gw.fcgi - **Note:** You will need to generate a key for each zone that will access - the Ceph Storage Cluster (assuming 1 per region). - -#. Once you have created a keyring and key to enable the Ceph Object Gateway - with access to the Ceph Storage Cluster, add it as an entry to your Ceph - Storage Cluster. For example:: - sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.ny-ny -i /etc/ceph/ceph.client.radosgw.ny.keyring +Repeat the process for the secondary zone. + + +Add Instances to Ceph Config File +--------------------------------- + +On an admin node, add an entry for each instance in the Ceph configuration file +for your Ceph Storage Cluster(s). For example:: + + ... + [client.radosgw.us-east-1] + rgw region = us + rgw region root pool = .us.rgw.root + rgw zone = us-east + rgw zone root pool = .us-east.rgw.root + keyring = /etc/ceph/ceph.client.radosgw.keyring + rgw dns name = {hostname} + rgw socket path = /var/run/ceph/$name.sock + host = {host-name} + + [client.radosgw.us-west-1] + rgw region = us + rgw region root pool = .us.rgw.root + rgw zone = ny-queens + rgw zone root pool = .us-west.rgw.root + keyring = /etc/ceph/ceph.client.radosgw.keyring + rgw dns name = {hostname} + rgw socket path = /var/run/ceph/$name.sock + host = {host-name} -Create a Gateway Configuration ------------------------------- -Create an ``rgw.conf`` file under the ``/etc/apache2/sites-available`` directory -on the host(s) where you installed the Ceph Object Gateway daemon(s). See below -for an exemplary embodiment of a gateway configuration as discussed in the -following text. +Then, update each :term:`Ceph Node` with the updated Ceph configuration +file. For example:: -.. literalinclude:: rgw.conf - :language: ini + ceph-deploy --overwrite-conf config {node1} {node2} {nodex} -#. Replace the ``{fqdn}`` entry with the fully-qualified domain name of the - server. - -#. Replace the ``{email.address}`` entry with the email address for the - server administrator. - -#. Add a ``ServerAlias`` if you wish to use S3-style subdomains - (of course you do). +Create a Region +--------------- -Enable the Configuration ------------------------- +#. Configure a region infile called ``us.json`` for the ``us`` region. + Copy the contents of the following example to a text editor. Set + ``is_master`` to ``true``. Replace ``{fqdn}`` with the fully-qualified + domain name of the endpoint. It will specify a master zone as ``us-east`` + and list it in the ``zones`` list along with the ``us-west`` zone. + See `Configuration Reference - Regions`_ for details.:: -#. Enable the site for the gateway configuration (e.g., ``rgw.conf``). :: + { "name": "us", + "api_name": "us", + "is_master": "true", + "endpoints": [ + "http:\/\/{fqdn}:80\/"], + "master_zone": "us-east", + "zones": [ + { "name": "us-east", + "endpoints": [ + "http:\/\/{fqdn}:80\/"], + "log_meta": "false", + "log_data": "false"}, + { "name": "us-west", + "endpoints": [ + "http:\/\/{fqdn}:80\/"], + "log_meta": "false", + "log_data": "false"}], + "placement_targets": [], + "default_placement": ""} - sudo a2ensite rgw.conf -#. Disable the default site. :: +#. Create the ``us`` region using the ``us.json`` infile you just + created. :: - sudo a2dissite default + sudo radosgw-admin region set --infile us.json --name client.radosgw.us-east-1 -.. note:: Failure to disable the default site can lead to problems. +#. Delete the default region (if it exists). :: + rados -p .us.rgw.root rm region_info.default + +#. Set the ``us`` region as the default region. :: -Add a FastCGI Script --------------------- + sudo radosgw-admin region default --rgw-region=us --name client.radosgw.us-east-1 -FastCGI requires a script for the S3-compatible interface. To create the -script, execute the following procedures on the server node. + Only one region can be the default region for a cluster. -#. Go to the ``/var/www`` directory. :: +#. Update the region map. :: - cd /var/www + sudo radosgw-admin regionmap update --name client.radosgw.us-east-1 -#. Open an editor with the file name ``s3gw.fcgi``. :: - sudo vim s3gw.fcgi -#. Add a shell script with ``exec`` and the path to the gateway binary, - the path to the Ceph configuration file, and the user name (``-n``; - the same user name created in step 2 of `Create a Storage Cluster User`_. - Copy the following into the editor. :: +Create Zones +------------ - #!/bin/sh - exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.ny-ny-instance1 +#. Configure a zone infile called ``us-east.json`` for the + ``us-east`` zone. -#. Save the file. + Copy the contents of the following example to a text editor. + This configuration uses pool names prepended with the region name and + zone name. See `Configuration Reference - Pools`_ for additional details on + gateway pools. See `Configuration Reference - Zones`_ for additional + details on zones. :: + + { "domain_root": ".us-east.rgw.root", + "control_pool": ".us-east.rgw.control", + "gc_pool": ".us-east.rgw.gc", + "log_pool": ".us-east.log", + "intent_log_pool": ".us-east.intent-log", + "usage_log_pool": ".us-east.usage", + "user_keys_pool": ".us-east.users", + "user_email_pool": ".us-east.users.email", + "user_swift_pool": ".us-east.users.swift", + "user_uid_pool": ".us-east.users.uid", + "system_key": { "access_key": "", "secret_key": ""} + } -#. Change the permissions on the file so that it is executable. :: - sudo chmod +x s3gw.fcgi +#. Add the ``us-east`` zone using the ``us-east.json`` infile you + just created in both the east and west pools by specifying their respective + user names (i.e., ``--name``). :: + sudo radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-east-1 + sudo radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-west-1 -Add Configuration to Ceph Config --------------------------------- + Repeat step 1 to create a zone infile for ``us-west``. Then add the zone + using the ``us-west.json`` infile in both the east and west pools by + specifying their respective user names (i.e., ``--name``). :: -Ceph Object Gateway instances read the `Ceph configuration file`_ (located at -``/etc/ceph/{cluster-name}.conf`` by default). The Ceph Object Gateway is a -client of the Ceph Storage Cluster, so you must place each instance under a -``[client]`` section that specifically refers to the ``radosgw`` daemon and the -instance ID. For example:: + sudo radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-east-1 + sudo radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-west-1 - [client.radosgw.ny-ny-instance1] - - #Region Info - rgw region = ny - rgw region root pool = .ny.rgw.root - - #Zone Info - rgw zone = ny-ny - rgw zone root pool = .ny-ny.rgw.root - keyring = /etc/ceph/ceph.client.radosgw.ny.keyring +#. Delete the default zone (if it exists). :: + + rados -p .rgw.root rm zone_info.default - #DNS Info for S3 Subdomains - rgw dns name = {hostname} - - #Ceph Node Info - rgw socket path = /tmp/$name.sock - host = {host-name} - -Add the foregoing setting (replacing values in braces where appropriate) to the -master copy of the Ceph configuration file you keep with your admin node. Then, -use ``ceph-deploy`` to push a fresh copy the configuration file from your admin -node Ceph Nodes. :: +#. Update the region map. :: + + sudo radosgw-admin regionmap update --name client.radosgw.us-east-1 - ceph-deploy --overwrite-conf config push {host-name [host-name]...} +Create Zone Users +----------------- + +Ceph Object Gateway stores zone users in the zone pools. So you must create zone +users after configuring the zones. Copy the ``access_key`` and ``secret_key`` +fields for each user so you can update your zone configuration once you complete +this step. :: + + sudo radosgw-admin user create --uid="us-east" --display-name="Region-US Zone-East" --name client.radosgw.us-east-1 --system + sudo radosgw-admin user create --uid="us-west" --display-name="Region-US Zone-West" --name client.radosgw.us-west-1 --system + + +Update Zone Configurations +-------------------------- + +You must update the zone configuration with zone users so that +the synchronization agents can authenticate with the zones. + +#. Open your ``us-east.json`` zone configuration file and paste the contents of + the ``access_key`` and ``secret_key`` fields from the step of creating + zone users into the ``system_key`` field of your zone configuration + infile. :: + + { "domain_root": ".us-east.rgw", + "control_pool": ".us-east.rgw.control", + "gc_pool": ".us-east.rgw.gc", + "log_pool": ".us-east.log", + "intent_log_pool": ".us-east.intent-log", + "usage_log_pool": ".us-east.usage", + "user_keys_pool": ".us-east.users", + "user_email_pool": ".us-east.users.email", + "user_swift_pool": ".us-east.users.swift", + "user_uid_pool": ".us-east.users.uid", + "system_key": { + "access_key": "{paste-access_key-here}", + "secret_key": "{paste-secret_key-here}" + }, + "placement_pools": [] + } + +#. Save the ``us-east.json`` file. Then, update your zone configuration. :: + + sudo radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-east-1 + sudo radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.us-west-1 + +#. Repeat step 1 to update the zone infile for ``us-west``. Then, update + your zone configuration. :: + + sudo radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-east-1 + sudo radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.us-west-1 + Restart Services -================ +---------------- Once you have redeployed your Ceph configuration files, we recommend restarting -your ``ceph`` service. +your Ceph Storage Cluster(s) and Apache instances. -For Ubuntu, use the following:: +For Ubuntu, use the following on each :term:`Ceph Node`:: sudo restart ceph-all @@ -451,39 +499,56 @@ To ensure that all components have reloaded their configurations, for each gateway instance we recommend restarting the ``apache2`` service. For example:: sudo service apache2 restart - -Start Gateways -============== -Start up the ``radosgw`` service. When starting the service with other than -the default region and zone, you must specify them explicitly. :: +Start Gateway Instances +----------------------- + +Start up the ``radosgw`` service. :: - sudo /etc/init.d/radosgw start --rgw-region={region} --rgw-zone={zone} + sudo /etc/init.d/radosgw start +If you are running multiple instances on the same host, you must specify the +user name. :: -Synchronize Metadata -==================== + sudo /etc/init.d/radosgw start --name client.radosgw.us-east-1 -The metadata agent maintains a global namespace for the cluster. The master -zone of the master region is the source for all other instances in the cluster. +Open a browser and check the endpoints for each zone. A simple HTTP request +to the domain name should return the following: -Configure an Agent ------------------- +.. code-block:: xml -To configure the metadata synchronization agent, retrieve the following from all -zones: + + + anonymous + + + + + +Replicate Data +-------------- + +The data synchronization agent replicates the data of a master zone to a +secondary zone. The master zone of a region is the source for the secondary zone +of the region. + + +To configure the synchronization agent, retrieve the following from each zone: + +- Zone Name - Access Key - Secret Key - Hostname - Port -You only need the hostname and port for a single instance (assuming all gateway -instances in a region/zone access the same Ceph Storage Cluster). Specify these -values in a configuration file (e.g., ``cluster-md-sync.conf``), and include a -``log_file`` name and an identifier for the ``daemon_id``. For example: +You only need the hostname and port for a single instance (assuming all +gateway instances in a region/zone access the same Ceph Storage Cluster). +Specify these values in a configuration file +(e.g., ``cluster-data-sync.conf``), and include a ``log_file`` name and an +identifier for the ``daemon_id``. For example: .. code-block:: ini @@ -500,74 +565,156 @@ values in a configuration file (e.g., ``cluster-md-sync.conf``), and include a log_file: {log.filename} daemon_id: {daemon-id} -The `Exemplary Cluster`_ assumes that ``ny-ny`` is the master region and zone, -so it is the source for ``ny-ldn``, ``ldn-ldn`` and ``ldn-ny``. +A concrete example may look like this: +.. code-block:: ini -Activate an Agent ------------------ + src_access_key: DG8RE354EFPZBICHIAF0 + src_secret_key: i3U0HiRP8CXaBWrcF8bbh6CbsxGYuPPwRkixfFSb + src_host: ceph-gateway-east + src_port: 80 + src_zone: us-east + dest_access_key: U60RFI6B08F32T2PD30G + dest_secret_key: W3HuUor7Gl1Ee93pA2pq2wFk1JMQ7hTrSDecYExl + dest_host: ceph-gateway-west + dest_port: 80 + dest_zone: us-west + log_file: /var/log/radosgw/radosgw-sync-us-east-west.log + daemon_id: rgw-east-2-west -To activate the metadata synchronization agent, execute the following:: - radosgw-agent -c region-md-sync.conf +To activate the data synchronization agent, open a terminal and +execute the following:: -You must have an agent for each source-destination pair. + radosgw-agent -c region-data-sync.conf +When the synchronization agent is running, you should see output +indicating that the agent is synchronizing shards of data. :: -Replicate Data -============== + INFO:radosgw_agent.sync:Starting incremental sync + INFO:radosgw_agent.worker:17910 is processing shard number 0 + INFO:radosgw_agent.worker:shard 0 has 0 entries after '' + INFO:radosgw_agent.worker:finished processing shard 0 + INFO:radosgw_agent.worker:17910 is processing shard number 1 + INFO:radosgw_agent.sync:1/64 shards processed + INFO:radosgw_agent.worker:shard 1 has 0 entries after '' + INFO:radosgw_agent.worker:finished processing shard 1 + INFO:radosgw_agent.sync:2/64 shards processed + ... -The data synchronization agent replicates the data of a master zone to a -secondary zone. The master zone of a region is the source for the secondary zone -of the region. +.. note:: You must have an agent for each source-destination pair. -Configure an Agent ------------------- +Configure a Secondary Region +============================ -To configure the synchronization agent, retrieve the following from all zones: +This section provides an exemplary procedure for setting up a cluster with +multiple regions. Configuring a cluster that spans regions requires maintaining +a global namespace, so that there are no namespace clashes among object names +stored across in different regions. -- Access Key -- Secret Key -- Hostname -- Port +This section extends the procedure in `Configure a Master Region`_, but +changes the region name and modifies a few procedures. See the following +sections for details. -You only need the hostname and port for a single instance (assuming all gateway -instances in a region/zone access the same Ceph Storage Cluster). Specify these -values in a configuration file (e.g., ``cluster-data-sync.conf``), and include a -``log_file`` name and an identifier for the ``daemon_id``. For example: -.. code-block:: ini +Naming for the Secondary Region +------------------------------- - src_access_key: {source-access-key} - src_secret_key: {source-secret-key} - src_host: {source-hostname} - src_port: {source-port} - src_zone: {source-zone} - dest_access_key: {destination-access-key} - dest_secret_key: {destination-secret-key} - dest_host: {destination-hostname} - dest_port: {destination-port} - dest_zone: {destination-zone} - log_file: {log.filename} - daemon_id: {daemon-id} +Before configuring the cluster, defining region, zone and instance names will +help you manage your cluster. Let's assume the region represents the European +Union, and we refer to it by its standard abbreviation. -The `Exemplary Cluster`_ assumes that ``ny-ny`` and ``ldn-ldn`` are the master -zones, so they are the source for ``ny-ldn`` and ``ldn-ldn`` respectively. +- European Union: ``eu`` +Let's assume the zones represent the Eastern and Western European Union. For +continuity, our naming convention will use ``{region name}-{zone name}`` +format, but you can use any naming convention you prefer. -Activate an Agent ------------------ +- European Union, East Region: ``eu-east`` +- European Union, West Region: ``eu-west`` -To activate the data synchronization agent, execute the following:: +Finally, let's assume that zones may have more than one Ceph Object Gateway +instance per zone. For continuity, our naming convention will use +``{region name}-{zone name}-{instance}`` format, but you can use any naming +convention you prefer. - radosgw-agent -c region-data-sync.conf +- European Union Region, Master Zone, Instance 1: ``eu-east-1`` +- European Union Region, Secondary Zone, Instance 1: ``eu-west-1`` + + +Configuring a Secondary Region +------------------------------ + +Repeat the exemplary procedure of `Configure a Master Region`_ +with the following differences: + +#. Use `Naming for the Secondary Region`_ in lieu of `Naming for + the Master Region`_. + +#. `Create Pools`_ using ``eu`` instead of ``us``. + +#. `Create a Keyring`_ and the corresponding keys using ``eu`` instead of + ``us``. You may use the same keyring if you desire, but ensure that you + create the keys on the Ceph Storage Cluster for that region (or region + and zone). + +#. `Install Apache/FastCGI`_. + +#. `Create Data Directories`_ using ``eu`` instead of ``us``. + +#. `Create a Gateway Configuration`_ using ``eu`` instead of ``us`` for + the socket names. + +#. `Enable the Configuration`_. + +#. `Add a FastCGI Script`_ using ``eu`` instead of ``us`` for the user names. + +#. `Add Instances to Ceph Config File`_ using ``eu`` instead of ``us`` for the + pool names. + +#. `Create a Region`_ using ``eu`` instead of ``us``. Set ``is_master`` to + ``false``. For consistency, create the master region in the secondary region + too. :: + + sudo radosgw-admin region set --infile us.json --name client.radosgw.eu-east-1 + +#. `Create Zones`_ using ``eu`` instead of ``us``. Ensure that you update the + user name (i.e., ``--name``) so that you create the zones in the correct + cluster. + +#. `Update Zone Configurations`_ using ``eu`` instead of ``us``. + +#. Create zones from master region in the secondary region. :: + + sudo radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.eu-east-1 + sudo radosgw-admin zone set --rgw-zone=us-east --infile us-east.json --name client.radosgw.eu-west-1 + sudo radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.eu-east-1 + sudo radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name client.radosgw.eu-west-1 + +#. Create zones from secondary region in the master region. :: + + sudo radosgw-admin zone set --rgw-zone=eu-east --infile eu-east.json --name client.radosgw.us-east-1 + sudo radosgw-admin zone set --rgw-zone=eu-east --infile eu-east.json --name client.radosgw.us-west-1 + sudo radosgw-admin zone set --rgw-zone=eu-west --infile eu-west.json --name client.radosgw.us-east-1 + sudo radosgw-admin zone set --rgw-zone=eu-west --infile eu-west.json --name client.radosgw.us-west-1 + +#. `Restart Services`_. + +#. `Start Gateway Instances`_. -You must have an agent for each source-destination pair. +#. `Replicate Data`_ by specifying the master zone of the master region as the + source zone and the master zone of the secondary region as the secondary + zone. When activating the ``radosgw-agent``, specify ``--metadata-only`` so + that it only copies metadata. +Once you have completed the foregoing procedure, you should have a cluster +consisting of a master region (``us``) and a secondary region (``eu``) where +there is a unified namespace between the two regions. +.. _CRUSH Map: ../../rados/operations/crush-map .. _Install Apache, FastCGI and Gateway: ../manual-install .. _Cephx Administration: ../../rados/operations/authentication/#cephx-administration .. _Ceph configuration file: ../../rados/configuration/ceph-conf