-=============
-RGW Multisite
-=============
+==========
+Multi-Site
+==========
.. versionadded:: Jewel
-From Ceph release Jewel and beyond, you may configure each :term:`Ceph Object
-Gateway` to work in an active active zone configuration, allowing for writing to
-non-master zones. Following are the basic terminologies that would be used:
-
-- **Zone**: A zone is *logical* grouping of one or more Ceph Object Gateway
- instances. There will be one zone that should be designated as the master zone
- in a zonegroup, which will handle all bucket and user creation.
-
-- **Zonegroup**: A zonegroup consists of multiple zones, this approximately
- corresponds to what used to be called as a region in pre Jewel releases for
- federated deployments. There should be a master zonegroup that will handle
- changes to the system configuration. Object data is only replicated between
- zones within a zonegroup, whereas metadata (such as buckets and users) are
- replicated across all zonegroups.
-
-- **Zonegroup map**: A zonegroup map is a configuration structure that holds the
- map of the entire system, ie. which zonegroup is the master, relationships
- between different zonegroups and certain configurables like storage policies.
-
-- **Realm**: A realm is a container for zonegroups, this allows for separation
- of zonegroups themselves between clusters. It is possible to create multiple
- realms, making it easier to run completely different configurations in the same
- cluster.
-
-- **Period**: A period holds the configuration structure for the current state
- of the realm. Every period contains a unique id and an epoch. A period's epoch
- is incremented on every commit operation. Every realm has an associated
- current period, holding the current state of configuration of the zonegroups
- and storage policies. Any configuration change for a non-master zone will
- increment the period's epoch. Changing the master zone to a different zone
- will trigger the following changes:
-
- - A new period is generated with a new period id and epoch of 1
- - Realm's current period is updated to point to the newly generated period id
- - Realm's epoch is incremented
-
-- **Metadata Master Zone**: The master zone of the master zonegroup is
- designated as the metadata master zone. This zone is authoritative for all
- metadata in the system. Any S3/Swift requests to modify metadata (such as
- bucket creation/removal) that are sent to other zones are forwarded to and
- handled by the metadata master zone.
-
-About this guide
-================
-
-In this guide we use 2 Ceph clusters, and create a single zonegroup,
-with three zones, which sync data between them actively. For the
-purpose of the guide, we'll create 2 zones in the same cluster, and 1
-zone in a different cluster. There is no sync agent involved for
-mirroring data changes between the Ceph Object Gateway instances and
-this allows for a much simpler configuration scheme and active-active
-configurations. Please note that metadata operations such as creating
-a user would still need to go through the master zone, however data
-operations such as creation of buckets objects can be handled by any
-of the zones.
-
-System Keys
------------
+A single zone configuration typically consists of one zone group containing one
+zone and one or more `ceph-radosgw` instances where you may load-balance gateway
+client requests between the instances. In a single zone configuration, typically
+multiple gateway instances point to a single Ceph storage cluster. However, Kraken
+supports several multi-site configuration options for the Ceph Object Gateway:
+
+- **Multi-zone:** A more advanced configuration consists of one zone group and
+ multiple zones, each zone with one or more `ceph-radosgw` instances. Each zone
+ is backed by its own Ceph Storage Cluster. Multiple zones in a zone group
+ provides disaster recovery for the zone group should one of the zones experience
+ a significant failure. In Kraken, each zone is active and may receive write
+ operations. In addition to disaster recovery, multiple active zones may also
+ serve as a foundation for content delivery networks.
+
+- **Multi-zone-group:** Formerly called 'regions', Ceph Object Gateway can also
+ support multiple zone groups, each zone group with one or more zones. Objects
+ stored to zones in one zone group within the same realm as another zone
+ group will share a global object namespace, ensuring unique object IDs across
+ zone groups and zones.
+
+- **Multiple Realms:** In Kraken, the Ceph Object Gateway supports the notion
+ of realms, which can be a single zone group or multiple zone groups and
+ a globally unique namespace for the realm. Multiple realms provide the ability
+ to support numerous configurations and namespaces.
+
+Replicating object data between zones within a zone group looks something
+like this:
+
+.. image:: ../images/zone-sync2.png
+ :align: center
+
+For additional details on setting up a cluster, see `Ceph Object Gateway for
+Production <https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ceph_object_gateway_for_production/>`__.
+
+Functional Changes from Infernalis
+==================================
+
+In Kraken, you can configure each Ceph Object Gateway to
+work in an active-active zone configuration, allowing for writes to
+non-master zones.
+
+The multi-site configuration is stored within a container called a
+"realm." The realm stores zone groups, zones, and a time "period" with
+multiple epochs for tracking changes to the configuration. In Kraken,
+the ``ceph-radosgw`` daemons handle the synchronization,
+eliminating the need for a separate synchronization agent. Additionally,
+the new approach to synchronization allows the Ceph Object Gateway to
+operate with an "active-active" configuration instead of
+"active-passive".
+
+Requirements and Assumptions
+============================
+
+A multi-site configuration requires at least two Ceph storage clusters,
+preferably given a distinct cluster name. At least two Ceph object
+gateway instances, one for each Ceph storage cluster.
+
+This guide assumes at least two Ceph storage clusters in geographically
+separate locations; however, the configuration can work on the same
+site. This guide also assumes four Ceph object gateway servers named
+``rgw1`` and ``rgw2``.
+
+A multi-site configuration requires a master zone group and a master
+zone. Additionally, each zone group requires a master zone. Zone groups
+may have one or more secondary or non-master zones.
+
+In this guide, the ``rgw1`` host will serve as the master zone of the
+master zone group; and, the ``rgw2`` host will serve as the secondary zone
+of the master zone group.
+
+Pools
+=====
+
+We recommend using the `Ceph Placement Group’s per Pool
+Calculator <http://ceph.com/pgcalc/>`__ to calculate a
+suitable number of placement groups for the pools the ``ceph-radosgw``
+daemon will create. Set the calculated values as defaults in your Ceph
+configuration file. For example:
-While configuring zones, Ceph Object Gateway instance, expects
-creation of zone a system user, with a S3-like access and secret keys.
-This allows another Ceph Object Gateway instance to pull the
-configuration remotely, with the access and secret keys. For making
-scripting and using of configuration management tools easier, it would
-be easier to generate the access key and secret key before hand and
-use these. For the purpose of the guide we assume an access key and
-secret key are set in the environment. For eg.::
-
- $ SYSTEM_ACCESS_KEY=1555b35654ad1656d805
- $ SYSTEM_SECRET_KEY=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q12
-
-Generally access keys are 20 alphanumeric characters, secret keys are
-40 alphanumeric characters (they can contain +/= chars as well). These
-keys can be generated in shell for eg::
-
- $ SYSTEM_ACCESS_KEY=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1)
- $ SYSTEM_SECRET_KEY=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1)
-
-Then we can create a system user by following command::
+::
- $ radosgw-admin user create --uid=zone.user --display-name="Zone User" \
- --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --system
+ osd pool default pg num = 50
+ osd pool default pgp num = 50
+.. note:: Make this change to the Ceph configuration file on your
+ storage cluster; then, either make a runtime change to the
+ configuration so that it will use those defaults when the gateway
+ instance creates the pools.
-Naming conventions
-------------------
-
-This section describes the process of setting up a master zone. For the purpose
-of the guide, we'll assume a zonegroup called ``us`` spanning the United States,
-which will be our master zonegroup. This will contain two zones written in a
-``{zonegroup}-{zone}`` format, this is just a convention only and you are free
-to choose a format you prefer to set these names. So in summary:
-
-- Master Zonegroup: United States ``us``
-- Master zone: United States, East Region 1: ``us-east-1``
-- Second zone: United States, East Region 2: ``us-east-2``
-- Second zone: United States, West Region: ``us-west``
-
-This will be a part of a larger realm, say ``gold``. The zones ``us-east-1`` and
-``us-east-2`` are part of the same Ceph cluster, of which we'll make
-``us-east-1`` as the primary. ``us-west`` will be in a different Ceph cluster.
-
-Create Pools
-------------
-
-Ceph Object Gateway instance will create pools on its own, when it is
-configured with the appropriate permissions, this will be done with
-the value of ``pg_num`` and ``pgp_num`` from the ``ceph.conf``. See
-`Ceph Object Gateway - Create Pools`_ for more detailed explanation.
-Pools particular to a zone by default follows the convention of
-``{zone-name}.pool-name``. In the default configuration, for eg. in
-``us-east-1`` zone following will be the pools:
-
-- ``.rgw.root``
-- ``us-east-1.rgw.control``
-- ``us-east-1.rgw.data.root``
-- ``us-east-1.rgw.gc``
-- ``us-east-1.rgw.log``
-- ``us-east-1.rgw.intent-log``
-- ``us-east-1.rgw.usage``
-- ``us-east-1.rgw.users.keys``
-- ``us-east-1.rgw.users.email``
-- ``us-east-1.rgw.users.swift``
-- ``us-east-1.rgw.users.uid``
-- ``us-east-1.rgw.buckets.index``
-- ``us-east-1.rgw.buckets.data``
-- ``us-east-1.rgw.meta``
-
-These pools can be created in the other zones as well by replacing
-``us-east-1`` with the appropriate zone name.
-
-Configuring the master zone in the primary cluster
-==================================================
-
-First, we'll configure the master zone ``us-east-1``, in the master
-zonegroup ``us``, this will serve as the master zone for all metadata
-operations, so all operations like creation of users need to be done
-on this zone.
-
-Creating a realm
-----------------
-
-Configure a realm called ``gold``, and also make this as default ::
-
- # radosgw-admin realm create --rgw-realm=gold --default
- {
- "id": "4a367026-bd8f-40ee-b486-8212482ddcd7",
- "name": "gold",
- "current_period": "09559832-67a4-4101-8b3f-10dfcd6b2707",
- "epoch": 1
- }
-
-
-Note that every realm has an id, which allows for flexibility like renaming the
-realm later, should the need arise. The ``current_period`` changes whenever we
-change anything in the master zone. The ``epoch`` is incremented when there's a
-change in configuration of the master zone, which results in a change of current
-period.
-
-Deleting the default zonegroup
-------------------------------
-
-A simple installation of Ceph Object Gateway would assume the default
-zonegroup called "default", since we no longer need this we begin by
-removing the default zonegroup::
-
- # radosgw-admin zonegroup delete --rgw-zonegroup=default
-
-
-Creating a master zonegroup
----------------------------
-
-We'll create a zonegroup called ``us`` as master zonegroup. A master
-zonegroup will be in control of the zonegroup map and propagate
-changes to the rest of the system. We will also set this zonegroup as
-the default, which allows later commands to use this zonegroup without
-explicitly mentioning it with the ``--rgw-zonegroup`` option.
-
-::
-
- # radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --master --default
- {
- "id": "d4018b8d-8c0d-4072-8919-608726fa369e",
- "name": "us",
- "api_name": "us",
- "is_master": "true",
- "endpoints": [
- "http:\/\/rgw1:80"
- ],
- "hostnames": [],
- "hostnames_s3website": [],
- "master_zone": "",
- "zones": [],
- "placement_targets": [],
- "default_placement": "",
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7"
- }
+Alternatively, create the pools manually. See
+`Pools <http://docs.ceph.com/docs/master/rados/operations/pools/#pools>`__
+for details on creating pools.
-Alternatively, we can make this zonegroup as the default zonegroup via
-the following command ::
-
- # radosgw-admin zonegroup default --rgw-zonegroup=us
-
-
-Creating a master zone
-----------------------
-
-Next we create a zone, and make it as the default zone. Note that for metadata
-operations like user creation you would want to use this zone. We also add it to
-the zonegroup
-
-::
-
- # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-1 --endpoints=http://rgw1:80 --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --default --master
- {
- "id": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "name": "us-east-1",
- "domain_root": "us-east-1.rgw.data.root",
- "control_pool": "us-east-1.rgw.control",
- "gc_pool": "us-east-1.rgw.gc",
- "log_pool": "us-east-1.rgw.log",
- "intent_log_pool": "us-east-1.rgw.intent-log",
- "usage_log_pool": "us-east-1.rgw.usage",
- "user_keys_pool": "us-east-1.rgw.users.keys",
- "user_email_pool": "us-east-1.rgw.users.email",
- "user_swift_pool": "us-east-1.rgw.users.swift",
- "user_uid_pool": "us-east-1.rgw.users.uid",
- "system_key": {
- "access_key": "1555b35654ad1656d804",
- "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
- },
- "placement_pools": [
- {
- "key": "default-placement",
- "val": {
- "index_pool": "us-east-1.rgw.buckets.index",
- "data_pool": "us-east-1.rgw.buckets.data",
- "data_extra_pool": "us-east-1.rgw.buckets.non-ec",
- "index_type": 0
- }
- }
- ],
- "metadata_heap": "us-east-1.rgw.meta",
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7"
- }
+Pool names particular to a zone follow the naming convention
+``{zone-name}.pool-name``. For example, a zone named ``us-east`` will
+have the following pools:
+- ``.rgw.root``
-Note that the above ``--rgw-zonegroup`` and ``--default`` options add the
-zone to a zonegroup and makes it the default zone as well. This can also be
-accomplished by the following commands::
+- ``us-east.rgw.control``
- # radosgw-admin zone default --rgw-zone=us-east-1
- # radosgw-admin zonegroup add --rgw-zonegroup=us --rgw-zone=us-east-1
+- ``us-east.rgw.data.root``
+- ``us-east.rgw.gc``
-Creating system users
----------------------
+- ``us-east.rgw.log``
-Next we create the system users for accessing the zone pools, note that these
-keys would be used when configuring the second zone::
+- ``us-east.rgw.intent-log``
- # radosgw-admin user create --uid=zone.user --display-name="Zone
- User" --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --system
- {
- "user_id": "zone.user",
- "display_name": "Zone User",
- "email": "",
- "suspended": 0,
- "max_buckets": 1000,
- "auid": 0,
- "subusers": [],
- "keys": [
- {
- "user": "zone.user",
- "access_key": "1555b35654ad1656d804",
- "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
- }
- ],
- "swift_keys": [],
- "caps": [],
- "op_mask": "read, write, delete",
- "system": "true",
- "default_placement": "",
- "placement_tags": [],
- "bucket_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- },
- "user_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- },
- "temp_url_keys": []
- }
-
-
-.. note:: Please note that system users have super user privileges over the
- entire zone, and will not behave like normal users for rest operations
- like creating buckets, objects etc. as the output would contain
- additional json fields for maintaining metadata.
-
-
-Update the period
------------------
+- ``us-east.rgw.usage``
-Since we have made a change in the master zone configuration, we need to
-commit these zone changes to reflect in the realm configuration structure. This
-is what the period would look like initially::
-
- # radosgw-admin period get
- {
- "id": "09559832-67a4-4101-8b3f-10dfcd6b2707",
- "epoch": 1,
- "predecessor_uuid": "",
- "sync_status": [],
- "period_map": {
- "id": "09559832-67a4-4101-8b3f-10dfcd6b2707",
- "zonegroups": [],
- "short_zone_ids": []
- },
- "master_zonegroup": "",
- "master_zone": "",
- "period_config": {
- "bucket_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- },
- "user_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- }
- },
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7",
- "realm_name": "gold",
- "realm_epoch": 1
- }
+- ``us-east.rgw.users.keys``
-Now we update the period and commit the changes::
-
- # radosgw-admin period update --commit
- {
- "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1",
- "epoch": 1,
- "predecessor_uuid": "09559832-67a4-4101-8b3f-10dfcd6b2707",
- "sync_status": [ ""... # truncating the output here
- ],
- "period_map": {
- "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1",
- "zonegroups": [
- {
- "id": "d4018b8d-8c0d-4072-8919-608726fa369e",
- "name": "us",
- "api_name": "us",
- "is_master": "true",
- "endpoints": [
- "http:\/\/rgw1:80"
- ],
- "hostnames": [],
- "hostnames_s3website": [],
- "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "zones": [
- {
- "id": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "name": "us-east-1",
- "endpoints": [
- "http:\/\/rgw1:80"
- ],
- "log_meta": "true",
- "log_data": "false",
- "bucket_index_max_shards": 0,
- "read_only": "false"
- }
- ],
- "placement_targets": [
- {
- "name": "default-placement",
- "tags": []
- }
- ],
- "default_placement": "default-placement",
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7"
- }
- ],
- "short_zone_ids": [
- {
- "key": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "val": 630926044
- }
- ]
- },
- "master_zonegroup": "d4018b8d-8c0d-4072-8919-608726fa369e",
- "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "period_config": {
- "bucket_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- },
- "user_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- }
- },
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7",
- "realm_name": "gold",
- "realm_epoch": 2
- }
+- ``us-east.rgw.users.email``
+- ``us-east.rgw.users.swift``
-Starting the Ceph Object Gateway instance
------------------------------------------
+- ``us-east.rgw.users.uid``
-Before starting the Ceph Object Gateway instance, the ``rgw_zone`` and
-``rgw_frontends`` options need to be mentioned in the configuration
-file. For more details refer to the `Install Ceph Gateway`_ section of
-the guide. The configuration section for Ceph Object Gateway instance
-should resemble::
+- ``us-east.rgw.buckets.index``
- [client.rgw.us-east-1]
- rgw_frontends="civetweb port=80"
- rgw_zone=us-east-1
+- ``us-east.rgw.buckets.data``
-And start the Ceph Object Gateway instance(according to the OS
-installation) ::
- $ sudo systemctl start ceph-radosgw@rgw.us-east-1
+Configuring a Master Zone
+=========================
+All gateways in a multi-site configuration will retrieve their
+configuration from a ``ceph-radosgw`` daemon on a host within the master
+zone group and master zone. To configure your gateways in a multi-site
+configuration, choose a ``ceph-radosgw`` instance to configure the
+master zone group and master zone.
-Configuring the second zone in same cluster
-===========================================
+Create a Realm
+--------------
-Now we configure the second zone, ``us-east-2``, in the same cluster. For the
-same cluster, all the following commands can be executed in the node hosting the
-primary zone itself.
+A realm contains the multi-site configuration of zone groups and zones
+and also serves to enforce a globally unique namespace within the realm.
-Creating the second zone
-------------------------
+Create a new realm for the multi-site configuration by opening a command
+line interface on a host identified to serve in the master zone group
+and zone. Then, execute the following:
-We follow a similar step to creation of master zone, except dropping the
-``--master`` flag this time ::
-
- # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --endpoints=http://rgw2:80
- {
- "id": "950c1a43-6836-41a2-a161-64777e07e8b8",
- "name": "us-east-2",
- "domain_root": "us-east-2.rgw.data.root",
- "control_pool": "us-east-2.rgw.control",
- "gc_pool": "us-east-2.rgw.gc",
- "log_pool": "us-east-2.rgw.log",
- "intent_log_pool": "us-east-2.rgw.intent-log",
- "usage_log_pool": "us-east-2.rgw.usage",
- "user_keys_pool": "us-east-2.rgw.users.keys",
- "user_email_pool": "us-east-2.rgw.users.email",
- "user_swift_pool": "us-east-2.rgw.users.swift",
- "user_uid_pool": "us-east-2.rgw.users.uid",
- "system_key": {
- "access_key": "1555b35654ad1656d804",
- "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
- },
- "placement_pools": [
- {
- "key": "default-placement",
- "val": {
- "index_pool": "us-east-2.rgw.buckets.index",
- "data_pool": "us-east-2.rgw.buckets.data",
- "data_extra_pool": "us-east-2.rgw.buckets.non-ec",
- "index_type": 0
- }
- }
- ],
- "metadata_heap": "us-east-2.rgw.meta",
- "realm_id": "815d74c2-80d6-4e63-8cfc-232037f7ff5c"
- }
+::
-Updating the period
--------------------
-
-Next we inform all the Ceph Object Gateway instances of the new change
-in the system map by doing a period update and committing the
-changes::
-
- # radosgw-admin period update --commit
- {
- "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1",
- "epoch": 2,
- "predecessor_uuid": "09559832-67a4-4101-8b3f-10dfcd6b2707",
- "sync_status": [ ""... # truncating the output here
- ],
- "period_map": {
- "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1",
- "zonegroups": [
- {
- "id": "d4018b8d-8c0d-4072-8919-608726fa369e",
- "name": "us",
- "api_name": "us",
- "is_master": "true",
- "endpoints": [
- "http:\/\/rgw1:80"
- ],
- "hostnames": [],
- "hostnames_s3website": [],
- "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "zones": [
- {
- "id": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "name": "us-east-1",
- "endpoints": [
- "http:\/\/rgw1:80"
- ],
- "log_meta": "true",
- "log_data": "false",
- "bucket_index_max_shards": 0,
- "read_only": "false"
- },
- {
- "id": "950c1a43-6836-41a2-a161-64777e07e8b8",
- "name": "us-east-2",
- "endpoints": [
- "http:\/\/rgw2:80"
- ],
- "log_meta": "false",
- "log_data": "true",
- "bucket_index_max_shards": 0,
- "read_only": "false"
- }
+ # radosgw-admin realm create --rgw-realm={realm-name} [--default]
- ],
- "placement_targets": [
- {
- "name": "default-placement",
- "tags": []
- }
- ],
- "default_placement": "default-placement",
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7"
- }
- ],
- "short_zone_ids": [
- {
- "key": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "val": 630926044
- },
- {
- "key": "950c1a43-6836-41a2-a161-64777e07e8b8",
- "val": 4276257543
- }
+For example:
- ]
- },
- "master_zonegroup": "d4018b8d-8c0d-4072-8919-608726fa369e",
- "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "period_config": {
- "bucket_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- },
- "user_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- }
- },
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7",
- "realm_name": "gold",
- "realm_epoch": 2
- }
+::
+ # radosgw-admin realm create --rgw-realm=movies --default
-Starting the Ceph Object Gateway instance
------------------------------------------
-
-On the node that is hosting the Ceph Object Gateway instance of the
-second zone, you would start it, similar to the Ceph Object Gateway
-instance ``client.rgw.us-east-1`` , changing the ``rgw
-zone=us-east-2`` in the configuration file this time. For eg::
-
- [client.rgw.us-east-2]
- rgw_frontends="civetweb port=80"
- rgw_zone=us-east-2
-
-And start the Ceph Object Gateway instance depending on your Operating
-system's init system, for eg::
-
- $ sudo systemctl start ceph-radosgw@rgw.us-east-2
-
-
-Configuring the Ceph Object Gateway in the second Ceph cluster
-==============================================================
-
-Now we go on to configure Ceph Object Gateway in the second Ceph
-cluster, which may be geographically apart, that is a part of the same
-zonegroup.
-
-Since a realm was already configured from the first cluster, we pull
-and make that realm as the default here, We also get the configuration
-from the master zone by pulling the period::
-
- # radosgw-admin realm pull --url=http://rgw1:80
- --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY
- {
- "id": "4a367026-bd8f-40ee-b486-8212482ddcd7",
- "name": "gold",
- "current_period": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1",
- "epoch": 2
- }
-
- # radosgw-admin period pull --url=http://rgw1:80 --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY
- # radosgw-admin realm default --rgw-realm=gold
-
-We also set the default zonegroup to the created ``us`` zonegroup::
-
- # radosgw-admin zonegroup default --rgw-zonegroup=us
-
-Configuring the second zone
----------------------------
-
-We create the new zone, ``us-west``, with the same system keys::
-
- # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west
- --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --endpoints=http://rgw3:80 --default
- {
- "id": "d9522067-cb7b-4129-8751-591e45815b16",
- "name": "us-west",
- "domain_root": "us-west.rgw.data.root",
- "control_pool": "us-west.rgw.control",
- "gc_pool": "us-west.rgw.gc",
- "log_pool": "us-west.rgw.log",
- "intent_log_pool": "us-west.rgw.intent-log",
- "usage_log_pool": "us-west.rgw.usage",
- "user_keys_pool": "us-west.rgw.users.keys",
- "user_email_pool": "us-west.rgw.users.email",
- "user_swift_pool": "us-west.rgw.users.swift",
- "user_uid_pool": "us-west.rgw.users.uid",
- "system_key": {
- "access_key": "1555b35654ad1656d804",
- "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
- },
- "placement_pools": [
- {
- "key": "default-placement",
- "val": {
- "index_pool": "us-west.rgw.buckets.index",
- "data_pool": "us-west.rgw.buckets.data",
- "data_extra_pool": "us-west.rgw.buckets.non-ec",
- "index_type": 0
- }
- }
- ],
- "metadata_heap": "us-west.rgw.meta",
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7"
+If the cluster will have a single realm, specify the ``--default`` flag.
+If ``--default`` is specified, ``radosgw-admin`` will use this realm by
+default. If ``--default`` is not specified, adding zone-groups and zones
+requires specifying either the ``--rgw-realm`` flag or the
+``--realm-id`` flag to identify the realm when adding zone groups and
+zones.
+
+After creating the realm, ``radosgw-admin`` will echo back the realm
+configuration. For example:
+
+::
+
+ {
+ "id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62",
+ "name": "movies",
+ "current_period": "1950b710-3e63-4c41-a19e-46a715000980",
+ "epoch": 1
}
+.. note:: Ceph generates a unique ID for the realm, which allows the renaming
+ of a realm if the need arises.
-Updating the period
--------------------
+Create a Master Zone Group
+--------------------------
-In order to propagate the zonegroup-map changes, we need to update and
-commit the period::
+A realm must have at least one zone group, which will serve as the
+master zone group for the realm.
- # radosgw-admin period update --commit --rgw-zone=us-west
- {
- "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1",
- "epoch": 3,
- "predecessor_uuid": "09559832-67a4-4101-8b3f-10dfcd6b2707",
- "sync_status": [
- "", # truncated
- ],
- "period_map": {
- "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1",
- "zonegroups": [
- {
- "id": "d4018b8d-8c0d-4072-8919-608726fa369e",
- "name": "us",
- "api_name": "us",
- "is_master": "true",
- "endpoints": [
- "http:\/\/rgw1:80"
- ],
- "hostnames": [],
- "hostnames_s3website": [],
- "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "zones": [
- {
- "id": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "name": "us-east-1",
- "endpoints": [
- "http:\/\/rgw1:80"
- ],
- "log_meta": "true",
- "log_data": "true",
- "bucket_index_max_shards": 0,
- "read_only": "false"
- },
- {
- "id": "950c1a43-6836-41a2-a161-64777e07e8b8",
- "name": "us-east-2",
- "endpoints": [
- "http:\/\/rgw2:80"
- ],
- "log_meta": "false",
- "log_data": "true",
- "bucket_index_max_shards": 0,
- "read_only": "false"
- },
- {
- "id": "d9522067-cb7b-4129-8751-591e45815b16",
- "name": "us-west",
- "endpoints": [
- "http:\/\/rgw3:80"
- ],
- "log_meta": "false",
- "log_data": "true",
- "bucket_index_max_shards": 0,
- "read_only": "false"
- }
- ],
- "placement_targets": [
- {
- "name": "default-placement",
- "tags": []
- }
- ],
- "default_placement": "default-placement",
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7"
- }
+Create a new master zone group for the multi-site configuration by
+opening a command line interface on a host identified to serve in the
+master zone group and zone. Then, execute the following:
+
+::
+
+ # radosgw-admin zonegroup create --rgw-zonegroup={name} --endpoints={url} [--rgw-realm={realm-name}|--realm-id={realm-id}] --master --default
+
+For example:
+
+::
+
+ # radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --rgw-realm=movies --master --default
+
+If the realm will only have a single zone group, specify the
+``--default`` flag. If ``--default`` is specified, ``radosgw-admin``
+will use this zone group by default when adding new zones. If
+``--default`` is not specified, adding zones will require either the
+``--rgw-zonegroup`` flag or the ``--zonegroup-id`` flag to identify the
+zone group when adding or modifying zones.
+
+After creating the master zone group, ``radosgw-admin`` will echo back
+the zone group configuration. For example:
+
+::
+
+ {
+ "id": "f1a233f5-c354-4107-b36c-df66126475a6",
+ "name": "us",
+ "api_name": "us",
+ "is_master": "true",
+ "endpoints": [
+ "http:\/\/rgw1:80"
],
- "short_zone_ids": [
- {
- "key": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "val": 630926044
- },
- {
- "key": "950c1a43-6836-41a2-a161-64777e07e8b8",
- "val": 4276257543
- },
- {
- "key": "d9522067-cb7b-4129-8751-591e45815b16",
- "val": 329470157
- }
- ]
- },
- "master_zonegroup": "d4018b8d-8c0d-4072-8919-608726fa369e",
- "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0",
- "period_config": {
- "bucket_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- },
- "user_quota": {
- "enabled": false,
- "max_size_kb": -1,
- "max_objects": -1
- }
- },
- "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7",
- "realm_name": "gold",
- "realm_epoch": 2
+ "hostnames": [],
+ "hostnames_s3webzone": [],
+ "master_zone": "",
+ "zones": [],
+ "placement_targets": [],
+ "default_placement": "",
+ "realm_id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62"
}
-You can observe that the period epoch number has incremented, indicating a
-change in the configuration.
+Create a Master Zone
+--------------------
+
+.. important:: Zones must be created on a Ceph Object Gateway node that will be
+ within the zone.
+
+Create a new master zone for the multi-site configuration by opening a
+command line interface on a host identified to serve in the master zone
+group and zone. Then, execute the following:
+
+::
+
+ # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \
+ --rgw-zone={zone-name} \
+ --master --default \
+ --endpoints={http://fqdn}[,{http://fqdn}]
+
+
+For example:
+
+::
+
+ # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east \
+ --master --default \
+ --endpoints={http://fqdn}[,{http://fqdn}]
+
+
+.. note:: The ``--access-key`` and ``--secret`` aren’t specified. These
+ settings will be added to the zone once the user is created in the
+ next section.
+
+.. important:: The following steps assume a multi-site configuration using newly
+ installed systems that aren’t storing data yet. DO NOT DELETE the
+ ``default`` zone and its pools if you are already using it to store
+ data, or the data will be deleted and unrecoverable.
+
+Delete Default Zone Group and Zone
+----------------------------------
+
+Delete the ``default`` zone if it exists. Make sure to remove it from
+the default zone group first.
-Starting the Ceph Object Gateway instance
------------------------------------------
+::
+
+ # radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
+ # radosgw-admin period update --commit
+ # radosgw-admin zone delete --rgw-zone=default
+ # radosgw-admin period update --commit
+ # radosgw-admin zonegroup delete --rgw-zonegroup=default
+ # radosgw-admin period update --commit
+
+Finally, delete the ``default`` pools in your Ceph storage cluster if
+they exist.
+
+.. important:: The following step assumes a multi-site configuration using newly
+ installed systems that aren’t currently storing data. DO NOT DELETE
+ the ``default`` zone group if you are already using it to store
+ data.
+
+::
+
+ # rados rmpool default.rgw.control default.rgw.control --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it
+
+Create a System User
+--------------------
+
+The ``ceph-radosgw`` daemons must authenticate before pulling realm and
+period information. In the master zone, create a system user to
+facilitate authentication between daemons.
+
+::
+
+ # radosgw-admin user create --uid="{user-name}" --display-name="{Display Name}" --system
+
+For example:
+
+::
+
+ # radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system
+
+Make a note of the ``access_key`` and ``secret_key``, as the secondary
+zones will require them to authenticate with the master zone.
+
+Finally, add the system user to the master zone.
+
+::
+
+ # radosgw-admin zone modify --rgw-zone=us-east --access-key={access-key} --secret={secret}
+ # radosgw-admin period update --commit
+
+Update the Period
+-----------------
+
+After updating the master zone configuration, update the period.
+
+::
+
+ # radosgw-admin period update --commit
+
+.. note:: Updating the period changes the epoch, and ensures that other zones
+ will receive the updated configuration.
+
+Update the Ceph Configuration File
+----------------------------------
+
+Update the Ceph configuration file on master zone hosts by adding the
+``rgw_zone`` configuration option and the name of the master zone to the
+instance entry.
+
+::
+
+ [client.rgw.{instance-name}]
+ ...
+ rgw_zone={zone-name}
+
+For example:
+
+::
+
+ [client.rgw.rgw1]
+ host = rgw1
+ rgw frontends = "civetweb port=80"
+ rgw_zone=us-east
+
+Start the Gateway
+-----------------
+
+On the object gateway host, start and enable the Ceph Object Gateway
+service:
+
+::
+
+ # systemctl start ceph-radosgw@rgw.`hostname -s`
+ # systemctl enable ceph-radosgw@rgw.`hostname -s`
+
+Configure Secondary Zones
+=========================
+
+Zones within a zone group replicate all data to ensure that each zone
+has the same data. When creating the secondary zone, execute all of the
+following operations on a host identified to serve the secondary zone.
+
+.. note:: To add a third zone, follow the same procedures as for adding the
+ secondary zone. Use different zone name.
+
+.. important:: You must execute metadata operations, such as user creation, on a
+ host within the master zone. The master zone and the secondary zone
+ can receive bucket operations, but the secondary zone redirects
+ bucket operations to the master zone. If the master zone is down,
+ bucket operations will fail.
+
+Pull the Realm
+--------------
+
+Using the URL path, access key and secret of the master zone in the
+master zone group, pull the realm to the host. To pull a non-default
+realm, specify the realm using the ``--rgw-realm`` or ``--realm-id``
+configuration options.
+
+::
+
+ # radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}
+
+If this realm is the default realm or the only realm, make the realm the
+default realm.
+
+::
+
+ # radosgw-admin realm default --rgw-realm={realm-name}
+
+Pull the Period
+---------------
+
+Using the URL path, access key and secret of the master zone in the
+master zone group, pull the period to the host. To pull a period from a
+non-default realm, specify the realm using the ``--rgw-realm`` or
+``--realm-id`` configuration options.
+
+::
+
+ # radosgw-admin period pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}
+
+
+.. note:: Pulling the period retrieves the latest version of the zone group
+ and zone configurations for the realm.
+
+Create a Secondary Zone
+-----------------------
-This is similar to starting the Ceph Object Gateway instance in the
-first zone, the only difference is the ``rgw zone`` option, which
-should reflect ``us-west``::
+.. important:: Zones must be created on a Ceph Object Gateway node that will be
+ within the zone.
- [client.rgw.us-west]
- rgw_frontends="civetweb port=80"
- rgw_zone=us-west
+Create a secondary zone for the multi-site configuration by opening a
+command line interface on a host identified to serve the secondary zone.
+Specify the zone group ID, the new zone name and an endpoint for the
+zone. **DO NOT** use the ``--master`` or ``--default`` flags. In Kraken,
+all zones run in an active-active configuration by
+default; that is, a gateway client may write data to any zone and the
+zone will replicate the data to all other zones within the zone group.
+If the secondary zone should not accept write operations, specify the
+``--read-only`` flag to create an active-passive configuration between
+the master zone and the secondary zone. Additionally, provide the
+``access_key`` and ``secret_key`` of the generated system user stored in
+the master zone of the master zone group. Execute the following:
-And start the Ceph Object Gateway instance depending on your Operating
-system's init system, for eg::
+::
+
+ # radosgw-admin zone create --rgw-zonegroup={zone-group-name}\
+ --rgw-zone={zone-name} --endpoints={url} \
+ --access-key={system-key} --secret={secret}\
+ --endpoints=http://{fqdn}:80 \
+ [--read-only]
+
+For example:
+
+::
+
+ # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west \
+ --access-key={system-key} --secret={secret} \
+ --endpoints=http://rgw2:80
+
+.. important:: The following steps assume a multi-site configuration using newly
+ installed systems that aren’t storing data. **DO NOT DELETE** the
+ ``default`` zone and its pools if you are already using it to store
+ data, or the data will be lost and unrecoverable.
+
+Delete the default zone if needed.
+
+::
+
+ # radosgw-admin zone delete --rgw-zone=default
+
+Finally, delete the default pools in your Ceph storage cluster if
+needed.
+
+::
+
+ # rados rmpool default.rgw.control default.rgw.control --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it
+ # rados rmpool default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it
+
+Update the Ceph Configuration File
+----------------------------------
+
+Update the Ceph configuration file on the secondary zone hosts by adding
+the ``rgw_zone`` configuration option and the name of the secondary zone
+to the instance entry.
+
+::
+
+ [client.rgw.{instance-name}]
+ ...
+ rgw_zone={zone-name}
+
+For example:
+
+::
+
+ [client.rgw.rgw2]
+ host = rgw2
+ rgw frontends = "civetweb port=80"
+ rgw_zone=us-west
+
+Update the Period
+-----------------
+
+After updating the master zone configuration, update the period.
+
+::
+
+ # radosgw-admin period update --commit
+
+.. note:: Updating the period changes the epoch, and ensures that other zones
+ will receive the updated configuration.
+
+Start the Gateway
+-----------------
+
+On the object gateway host, start and enable the Ceph Object Gateway
+service:
+
+::
+
+ # systemctl start ceph-radosgw@rgw.`hostname -s`
+ # systemctl enable ceph-radosgw@rgw.`hostname -s`
+
+Check Synchronization Status
+----------------------------
+
+Once the secondary zone is up and running, check the synchronization
+status. Synchronization copies users and buckets created in the master
+zone to the secondary zone.
+
+::
+
+ # radosgw-admin sync status
+
+The output will provide the status of synchronization operations. For
+example:
+
+::
+
+ realm f3239bc5-e1a8-4206-a81d-e1576480804d (earth)
+ zonegroup c50dbb7e-d9ce-47cc-a8bb-97d9b399d388 (us)
+ zone 4c453b70-4a16-4ce8-8185-1893b05d346e (us-west)
+ metadata sync syncing
+ full sync: 0/64 shards
+ metadata is caught up with master
+ incremental sync: 64/64 shards
+ data sync source: 1ee9da3e-114d-4ae3-a8a4-056e8a17f532 (us-east)
+ syncing
+ full sync: 0/128 shards
+ incremental sync: 128/128 shards
+ data is caught up with source
- $ sudo systemctl start ceph-radosgw@rgw.us-west
+.. note:: Secondary zones accept bucket operations; however, secondary zones
+ redirect bucket operations to the master zone and then synchronize
+ with the master zone to receive the result of the bucket operations.
+ If the master zone is down, bucket operations executed on the
+ secondary zone will fail, but object operations should succeed.
Maintenance
Information about the replication status of a zone can be queried with::
- $ radosgw-admin sync status
-
+ $ radosgw-admin sync status
realm b3bc1c37-9c44-4b89-a03b-04c269bea5da (earth)
zonegroup f54f9b22-b4b6-4a0e-9211-fa6ac1693f49 (us)
zone adce11c9-b8ed-4a90-8bc5-3fc029ff0816 (us-2)
- metadata sync syncing
+ metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is behind on 1 shards
incremental sync: 128/128 shards
data is caught up with source
-
Changing the Metadata Master Zone
---------------------------------
.. important:: Care must be taken when changing which zone is the metadata
- master. If a zone has not finished syncing metadata from the current master
- zone, it will be unable to serve any remaining entries when promoted to
- master and those changes will be lost. For this reason, waiting for a
- zone's ``radosgw-admin sync status`` to catch up on metadata sync before
- promoting it to master is recommended.
-
- Similarly, if changes to metadata are being processed by the current master
- zone while another zone is being promoted to master, those changes are
- likely to be lost. To avoid this, shutting down any ``radosgw`` instances
- on the previous master zone is recommended. After promoting another zone,
- its new period can be fetched with ``radosgw-admin period pull`` and the
- gateway(s) can be restarted.
+ master. If a zone has not finished syncing metadata from the current master
+ zone, it will be unable to serve any remaining entries when promoted to
+ master and those changes will be lost. For this reason, waiting for a
+ zone's ``radosgw-admin sync status`` to catch up on metadata sync before
+ promoting it to master is recommended.
+
+ Similarly, if changes to metadata are being processed by the current master
+ zone while another zone is being promoted to master, those changes are
+ likely to be lost. To avoid this, shutting down any ``radosgw`` instances
+ on the previous master zone is recommended. After promoting another zone,
+ its new period can be fetched with ``radosgw-admin period pull`` and the
+ gateway(s) can be restarted.
To promote a zone (for example, zone ``us-2`` in zonegroup ``us``) to metadata
master, run the following commands on that zone::
This will generate a new period, and the radosgw instance(s) in zone ``us-2``
will send this period to other zones.
-.. _`Ceph Object Gateway - Create Pools`: ../config#create-pools
-.. _`Install Ceph Gateway`: ../../install/install-ceph-gateway
+Failover and Disaster Recovery
+==============================
+
+If the master zone should fail, failover to the secondary zone for
+disaster recovery.
+
+1. Make the secondary zone the master and default zone. For example:
+
+ ::
+
+ # radosgw-admin zone modify --rgw-zone={zone-name} --master --default
+
+ By default, Ceph Object Gateway will run in an active-active
+ configuration. If the cluster was configured to run in an
+ active-passive configuration, the secondary zone is a read-only zone.
+ Remove the ``--read-only`` status to allow the zone to receive write
+ operations. For example:
+
+ ::
+
+ # radosgw-admin zone modify --rgw-zone={zone-name} --master --default \
+ --read-only=False
+
+2. Update the period to make the changes take effect.
+
+ ::
+
+ # radosgw-admin period update --commit
+
+3. Finally, restart the Ceph Object Gateway.
+
+ ::
+
+ # systemctl restart ceph-radosgw@rgw.`hostname -s`
+
+If the former master zone recovers, revert the operation.
+
+1. From the recovered zone, pull the period from the current master
+ zone.
+
+ ::
+
+ # radosgw-admin period pull --url={url-to-master-zone-gateway} \
+ --access-key={access-key} --secret={secret}
+
+2. Make the recovered zone the master and default zone.
+
+ ::
+
+ # radosgw-admin zone modify --rgw-zone={zone-name} --master --default
+
+3. Update the period to make the changes take effect.
+
+ ::
+
+ # radosgw-admin period update --commit
+
+4. Then, restart the Ceph Object Gateway in the recovered zone.
+
+ ::
+
+ # systemctl restart ceph-radosgw@rgw.`hostname -s`
+
+5. If the secondary zone needs to be a read-only configuration, update
+ the secondary zone.
+
+ ::
+
+ # radosgw-admin zone modify --rgw-zone={zone-name} --read-only
+
+6. Update the period to make the changes take effect.
+
+ ::
+
+ # radosgw-admin period update --commit
+
+7. Finally, restart the Ceph Object Gateway in the secondary zone.
+
+ ::
+
+ # systemctl restart ceph-radosgw@rgw.`hostname -s`
+
+Migrating a Single Site System to Multi-Site
+============================================
+
+To migrate from a single site system with a ``default`` zone group and
+zone to a multi site system, use the following steps:
+
+1. Create a realm. Replace ``<name>`` with the realm name.
+
+ ::
+
+ # radosgw-admin realm create --rgw-realm=<name> --default
+
+2. Rename the default zone and zonegroup. Replace ``<name>`` with the
+ zonegroup or zone name.
+
+ ::
+
+ # radosgw-admin zonegroup rename --rgw-zonegroup default --zonegroup-new-name=<name>
+ # radosgw-admin zone rename --rgw-zone default --zone-new-name us-east-1 --rgw-zonegroup=<name>
+
+3. Configure the master zonegroup. Replace ``<name>`` with the realm or
+ zonegroup name. Replace ``<fqdn>`` with the fully qualified domain
+ name(s) in the zonegroup.
+
+ ::
+
+ # radosgw-admin zonegroup modify --rgw-realm=<name> --rgw-zonegroup=<name> --endpoints http://<fqdn>:80 --master --default
+
+4. Configure the master zone. Replace ``<name>`` with the realm,
+ zonegroup or zone name. Replace ``<fqdn>`` with the fully qualified
+ domain name(s) in the zonegroup.
+
+ ::
+
+ # radosgw-admin zone modify --rgw-realm=<name> --rgw-zonegroup=<name> \
+ --rgw-zone=<name> --endpoints http://<fqdn>:80 \
+ --access-key=<access-key> --secret=<secret-key> \
+ --master --default
+
+5. Create a system user. Replace ``<user-id>`` with the username.
+ Replace ``<display-name>`` with a display name. It may contain
+ spaces.
+
+ ::
+
+ # radosgw-admin user create --uid=<user-id> --display-name="<display-name>"\
+ --access-key=<access-key> --secret=<secret-key> --system
+
+6. Commit the updated configuration.
+
+ ::
+
+ # radosgw-admin period update --commit
+
+7. Finally, restart the Ceph Object Gateway.
+
+ ::
+
+ # systemctl restart ceph-radosgw@rgw.`hostname -s`
+
+After completing this procedure, proceed to `Configure a Secondary
+Zone <#configure-secondary-zones>`__ to create a secondary zone
+in the master zone group.
+
+
+Multi-Site Configuration Reference
+==================================
+
+The following sections provide additional details and command-line
+usage for realms, periods, zone groups and zones.
+
+Realms
+------
+
+A realm represents a globally unique namespace consisting of one or more
+zonegroups containing one or more zones, and zones containing buckets,
+which in turn contain objects. A realm enables the Ceph Object Gateway
+to support multiple namespaces and their configuration on the same
+hardware.
+
+A realm contains the notion of periods. Each period represents the state
+of the zone group and zone configuration in time. Each time you make a
+change to a zonegroup or zone, update the period and commit it.
+
+By default, the Ceph Object Gateway does not create a realm
+for backward compatibility with Infernalis and earlier releases.
+However, as a best practice, we recommend creating realms for new
+clusters.
+
+Create a Realm
+~~~~~~~~~~~~~~
+
+To create a realm, execute ``realm create`` and specify the realm name.
+If the realm is the default, specify ``--default``.
+
+::
+
+ # radosgw-admin realm create --rgw-realm={realm-name} [--default]
+
+For example:
+
+::
+
+ # radosgw-admin realm create --rgw-realm=movies --default
+
+By specifying ``--default``, the realm will be called implicitly with
+each ``radosgw-admin`` call unless ``--rgw-realm`` and the realm name
+are explicitly provided.
+
+Make a Realm the Default
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+One realm in the list of realms should be the default realm. There may
+be only one default realm. If there is only one realm and it wasn’t
+specified as the default realm when it was created, make it the default
+realm. Alternatively, to change which realm is the default, execute:
+
+::
+
+ # radosgw-admin realm default --rgw-realm=movies
+
+..note:: When the realm is default, the command line assumes
+ ``--rgw-realm=<realm-name>`` as an argument.
+
+Delete a Realm
+~~~~~~~~~~~~~~
+
+To delete a realm, execute ``realm delete`` and specify the realm name.
+
+::
+
+ # radosgw-admin realm delete --rgw-realm={realm-name}
+
+For example:
+
+::
+
+ # radosgw-admin realm delete --rgw-realm=movies
+
+Get a Realm
+~~~~~~~~~~~
+
+To get a realm, execute ``realm get`` and specify the realm name.
+
+::
+
+ #radosgw-admin realm get --rgw-realm=<name>
+
+For example:
+
+::
+
+ # radosgw-admin realm get --rgw-realm=movies [> filename.json]
+
+The CLI will echo a JSON object with the realm properties.
+
+::
+
+ {
+ "id": "0a68d52e-a19c-4e8e-b012-a8f831cb3ebc",
+ "name": "movies",
+ "current_period": "b0c5bbef-4337-4edd-8184-5aeab2ec413b",
+ "epoch": 1
+ }
+
+Use ``>`` and an output file name to output the JSON object to a file.
+
+Set a Realm
+~~~~~~~~~~~
+
+To set a realm, execute ``realm set``, specify the realm name, and
+``--infile=`` with an input file name.
+
+::
+
+ #radosgw-admin realm set --rgw-realm=<name> --infile=<infilename>
+
+For example:
+
+::
+
+ # radosgw-admin realm set --rgw-realm=movies --infile=filename.json
+
+List Realms
+~~~~~~~~~~~
+
+To list realms, execute ``realm list``.
+
+::
+
+ # radosgw-admin realm list
+
+List Realm Periods
+~~~~~~~~~~~~~~~~~~
+
+To list realm periods, execute ``realm list-periods``.
+
+::
+
+ # radosgw-admin realm list-periods
+
+Pull a Realm
+~~~~~~~~~~~~
+
+To pull a realm from the node containing the master zone group and
+master zone to a node containing a secondary zone group or zone, execute
+``realm pull`` on the node that will receive the realm configuration.
+
+::
+
+ # radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}
+
+Rename a Realm
+~~~~~~~~~~~~~~
+
+A realm is not part of the period. Consequently, renaming the realm is
+only applied locally, and will not get pulled with ``realm pull``. When
+renaming a realm with multiple zones, run the command on each zone. To
+rename a realm, execute the following:
+
+::
+
+ # radosgw-admin realm rename --rgw-realm=<current-name> --realm-new-name=<new-realm-name>
+
+.. note:: DO NOT use ``realm set`` to change the ``name`` parameter. That
+ changes the internal name only. Specifying ``--rgw-realm`` would
+ still use the old realm name.
+
+Zone Groups
+-----------
+
+The Ceph Object Gateway supports multi-site deployments and a global
+namespace by using the notion of zone groups. Formerly called a region
+in Infernalis, a zone group defines the geographic location of one or more Ceph
+Object Gateway instances within one or more zones.
+
+Configuring zone groups differs from typical configuration procedures,
+because not all of the settings end up in a Ceph configuration file. You
+can list zone groups, get a zone group configuration, and set a zone
+group configuration.
+
+Create a Zone Group
+~~~~~~~~~~~~~~~~~~~
+
+Creating a zone group consists of specifying the zone group name.
+Creating a zone assumes it will live in the default realm unless
+``--rgw-realm=<realm-name>`` is specified. If the zonegroup is the
+default zonegroup, specify the ``--default`` flag. If the zonegroup is
+the master zonegroup, specify the ``--master`` flag. For example:
+
+::
+
+ # radosgw-admin zonegroup create --rgw-zonegroup=<name> [--rgw-realm=<name>][--master] [--default]
+
+
+.. note:: Use ``zonegroup modify --rgw-zonegroup=<zonegroup-name>`` to modify
+ an existing zone group’s settings.
+
+Make a Zone Group the Default
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+One zonegroup in the list of zonegroups should be the default zonegroup.
+There may be only one default zonegroup. If there is only one zonegroup
+and it wasn’t specified as the default zonegroup when it was created,
+make it the default zonegroup. Alternatively, to change which zonegroup
+is the default, execute:
+
+::
+
+ # radosgw-admin zonegroup default --rgw-zonegroup=comedy
+
+.. note:: When the zonegroup is default, the command line assumes
+ ``--rgw-zonegroup=<zonegroup-name>`` as an argument.
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Add a Zone to a Zone Group
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To add a zone to a zonegroup, execute the following:
+
+::
+
+ # radosgw-admin zonegroup add --rgw-zonegroup=<name> --rgw-zone=<name>
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Remove a Zone from a Zone Group
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To remove a zone from a zonegroup, execute the following:
+
+::
+
+ # radosgw-admin zonegroup remove --rgw-zonegroup=<name> --rgw-zone=<name>
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Rename a Zone Group
+~~~~~~~~~~~~~~~~~~~
+
+To rename a zonegroup, execute the following:
+
+::
+
+ # radosgw-admin zonegroup rename --rgw-zonegroup=<name> --zonegroup-new-name=<name>
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Delete a Zone Group
+~~~~~~~~~~~~~~~~~~~
+
+To delete a zonegroup, execute the following:
+
+::
+
+ # radosgw-admin zonegroup delete --rgw-zonegroup=<name>
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+List Zone Groups
+~~~~~~~~~~~~~~~~
+
+A Ceph cluster contains a list of zone groups. To list the zone groups,
+execute:
+
+::
+
+ # radosgw-admin zonegroup list
+
+The ``radosgw-admin`` returns a JSON formatted list of zone groups.
+
+::
+
+ {
+ "default_info": "90b28698-e7c3-462c-a42d-4aa780d24eda",
+ "zonegroups": [
+ "us"
+ ]
+ }
+
+Get a Zone Group Map
+~~~~~~~~~~~~~~~~~~~~
+
+To list the details of each zone group, execute:
+
+::
+
+ # radosgw-admin zonegroup-map get
+
+.. note:: If you receive a ``failed to read zonegroup map`` error, run
+ ``radosgw-admin zonegroup-map update`` as ``root`` first.
+
+Get a Zone Group
+~~~~~~~~~~~~~~~~
+
+To view the configuration of a zone group, execute:
+
+::
+
+ radosgw-admin zonegroup get [--rgw-zonegroup=<zonegroup>]
+
+The zone group configuration looks like this:
+
+::
+
+ {
+ "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
+ "name": "us",
+ "api_name": "us",
+ "is_master": "true",
+ "endpoints": [
+ "http:\/\/rgw1:80"
+ ],
+ "hostnames": [],
+ "hostnames_s3website": [],
+ "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
+ "zones": [
+ {
+ "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
+ "name": "us-east",
+ "endpoints": [
+ "http:\/\/rgw1"
+ ],
+ "log_meta": "true",
+ "log_data": "true",
+ "bucket_index_max_shards": 0,
+ "read_only": "false"
+ },
+ {
+ "id": "d1024e59-7d28-49d1-8222-af101965a939",
+ "name": "us-west",
+ "endpoints": [
+ "http:\/\/rgw2:80"
+ ],
+ "log_meta": "false",
+ "log_data": "true",
+ "bucket_index_max_shards": 0,
+ "read_only": "false"
+ }
+ ],
+ "placement_targets": [
+ {
+ "name": "default-placement",
+ "tags": []
+ }
+ ],
+ "default_placement": "default-placement",
+ "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
+ }
+
+Set a Zone Group
+~~~~~~~~~~~~~~~~
+
+Defining a zone group consists of creating a JSON object, specifying at
+least the required settings:
+
+1. ``name``: The name of the zone group. Required.
+
+2. ``api_name``: The API name for the zone group. Optional.
+
+3. ``is_master``: Determines if the zone group is the master zone group.
+ Required. **note:** You can only have one master zone group.
+
+4. ``endpoints``: A list of all the endpoints in the zone group. For
+ example, you may use multiple domain names to refer to the same zone
+ group. Remember to escape the forward slashes (``\/``). You may also
+ specify a port (``fqdn:port``) for each endpoint. Optional.
+
+5. ``hostnames``: A list of all the hostnames in the zone group. For
+ example, you may use multiple domain names to refer to the same zone
+ group. Optional. The ``rgw dns name`` setting will automatically be
+ included in this list. You should restart the gateway daemon(s) after
+ changing this setting.
+
+6. ``master_zone``: The master zone for the zone group. Optional. Uses
+ the default zone if not specified. **note:** You can only have one
+ master zone per zone group.
+
+7. ``zones``: A list of all zones within the zone group. Each zone has a
+ name (required), a list of endpoints (optional), and whether or not
+ the gateway will log metadata and data operations (false by default).
+
+8. ``placement_targets``: A list of placement targets (optional). Each
+ placement target contains a name (required) for the placement target
+ and a list of tags (optional) so that only users with the tag can use
+ the placement target (i.e., the user’s ``placement_tags`` field in
+ the user info).
+
+9. ``default_placement``: The default placement target for the object
+ index and object data. Set to ``default-placement`` by default. You
+ may also set a per-user default placement in the user info for each
+ user.
+
+To set a zone group, create a JSON object consisting of the required
+fields, save the object to a file (e.g., ``zonegroup.json``); then,
+execute the following command:
+
+::
+
+ # radosgw-admin zonegroup set --infile zonegroup.json
+
+Where ``zonegroup.json`` is the JSON file you created.
+
+.. important:: The ``default`` zone group ``is_master`` setting is ``true`` by
+ default. If you create a new zone group and want to make it the
+ master zone group, you must either set the ``default`` zone group
+ ``is_master`` setting to ``false``, or delete the ``default`` zone
+ group.
+
+Finally, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Set a Zone Group Map
+~~~~~~~~~~~~~~~~~~~~
+
+Setting a zone group map consists of creating a JSON object consisting
+of one or more zone groups, and setting the ``master_zonegroup`` for the
+cluster. Each zone group in the zone group map consists of a key/value
+pair, where the ``key`` setting is equivalent to the ``name`` setting
+for an individual zone group configuration, and the ``val`` is a JSON
+object consisting of an individual zone group configuration.
+
+You may only have one zone group with ``is_master`` equal to ``true``,
+and it must be specified as the ``master_zonegroup`` at the end of the
+zone group map. The following JSON object is an example of a default
+zone group map.
+
+::
+
+ {
+ "zonegroups": [
+ {
+ "key": "90b28698-e7c3-462c-a42d-4aa780d24eda",
+ "val": {
+ "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
+ "name": "us",
+ "api_name": "us",
+ "is_master": "true",
+ "endpoints": [
+ "http:\/\/rgw1:80"
+ ],
+ "hostnames": [],
+ "hostnames_s3website": [],
+ "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
+ "zones": [
+ {
+ "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
+ "name": "us-east",
+ "endpoints": [
+ "http:\/\/rgw1"
+ ],
+ "log_meta": "true",
+ "log_data": "true",
+ "bucket_index_max_shards": 0,
+ "read_only": "false"
+ },
+ {
+ "id": "d1024e59-7d28-49d1-8222-af101965a939",
+ "name": "us-west",
+ "endpoints": [
+ "http:\/\/rgw2:80"
+ ],
+ "log_meta": "false",
+ "log_data": "true",
+ "bucket_index_max_shards": 0,
+ "read_only": "false"
+ }
+ ],
+ "placement_targets": [
+ {
+ "name": "default-placement",
+ "tags": []
+ }
+ ],
+ "default_placement": "default-placement",
+ "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
+ }
+ }
+ ],
+ "master_zonegroup": "90b28698-e7c3-462c-a42d-4aa780d24eda",
+ "bucket_quota": {
+ "enabled": false,
+ "max_size_kb": -1,
+ "max_objects": -1
+ },
+ "user_quota": {
+ "enabled": false,
+ "max_size_kb": -1,
+ "max_objects": -1
+ }
+ }
+
+To set a zone group map, execute the following:
+
+::
+
+ # radosgw-admin zonegroup-map set --infile zonegroupmap.json
+
+Where ``zonegroupmap.json`` is the JSON file you created. Ensure that
+you have zones created for the ones specified in the zone group map.
+Finally, update the period.
+
+::
+
+ # radosgw-admin period update --commit
+
+Zones
+-----
+
+Ceph Object Gateway supports the notion of zones. A zone defines a
+logical group consisting of one or more Ceph Object Gateway instances.
+
+Configuring zones differs from typical configuration procedures, because
+not all of the settings end up in a Ceph configuration file. You can
+list zones, get a zone configuration and set a zone configuration.
+
+Create a Zone
+~~~~~~~~~~~~~
+
+To create a zone, specify a zone name. If it is a master zone, specify
+the ``--master`` option. Only one zone in a zone group may be a master
+zone. To add the zone to a zonegroup, specify the ``--rgw-zonegroup``
+option with the zonegroup name.
+
+::
+
+ # radosgw-admin zone create --rgw-zone=<name> \
+ [--zonegroup=<zonegroup-name]\
+ [--endpoints=<endpoint>[,<endpoint>] \
+ [--master] [--default] \
+ --access-key $SYSTEM_ACCESS_KEY --secret $SYSTEM_SECRET_KEY
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Delete a Zone
+~~~~~~~~~~~~~
+
+To delete zone, first remove it from the zonegroup.
+
+::
+
+ # radosgw-admin zonegroup remove --zonegroup=<name>\
+ --zone=<name>
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Next, delete the zone. Execute the following:
+
+::
+
+ # radosgw-admin zone delete --rgw-zone<name>
+
+Finally, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+.. important:: Do not delete a zone without removing it from a zone group first.
+ Otherwise, updating the period will fail.
+
+If the pools for the deleted zone will not be used anywhere else,
+consider deleting the pools. Replace ``<del-zone>`` in the example below
+with the deleted zone’s name.
+
+.. important:: Only delete the pools with prepended zone names. Deleting the root
+ pool, such as, ``.rgw.root`` will remove all of the system’s
+ configuration.
+
+.. important:: Once the pools are deleted, all of the data within them are deleted
+ in an unrecoverable manner. Only delete the pools if the pool
+ contents are no longer needed.
+
+::
+
+ # rados rmpool <del-zone>.rgw.control <del-zone>.rgw.control --yes-i-really-really-mean-it
+ # rados rmpool <del-zone>.rgw.data.root <del-zone>.rgw.data.root --yes-i-really-really-mean-it
+ # rados rmpool <del-zone>.rgw.gc <del-zone>.rgw.gc --yes-i-really-really-mean-it
+ # rados rmpool <del-zone>.rgw.log <del-zone>.rgw.log --yes-i-really-really-mean-it
+ # rados rmpool <del-zone>.rgw.users.uid <del-zone>.rgw.users.uid --yes-i-really-really-mean-it
+
+Modify a Zone
+~~~~~~~~~~~~~
+
+To modify a zone, specify the zone name and the parameters you wish to
+modify.
+
+::
+
+ # radosgw-admin zone modify [options]
+
+Where ``[options]``:
+
+- ``--access-key=<key>``
+- ``--secret/--secret-key=<key>``
+- ``--master``
+- ``--default``
+- ``--endpoints=<list>``
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+List Zones
+~~~~~~~~~~
+
+As ``root``, to list the zones in a cluster, execute:
+
+::
+
+ # radosgw-admin zone list
+
+Get a Zone
+~~~~~~~~~~
+
+As ``root``, to get the configuration of a zone, execute:
+
+::
+
+ # radosgw-admin zone get [--rgw-zone=<zone>]
+
+The ``default`` zone looks like this:
+
+::
+
+ { "domain_root": ".rgw",
+ "control_pool": ".rgw.control",
+ "gc_pool": ".rgw.gc",
+ "log_pool": ".log",
+ "intent_log_pool": ".intent-log",
+ "usage_log_pool": ".usage",
+ "user_keys_pool": ".users",
+ "user_email_pool": ".users.email",
+ "user_swift_pool": ".users.swift",
+ "user_uid_pool": ".users.uid",
+ "system_key": { "access_key": "", "secret_key": ""},
+ "placement_pools": [
+ { "key": "default-placement",
+ "val": { "index_pool": ".rgw.buckets.index",
+ "data_pool": ".rgw.buckets"}
+ }
+ ]
+ }
+
+Set a Zone
+~~~~~~~~~~
+
+Configuring a zone involves specifying a series of Ceph Object Gateway
+pools. For consistency, we recommend using a pool prefix that is the
+same as the zone name. See
+`Pools <http://docs.ceph.com/docs/master/rados/operations/pools/#pools>`__
+for details of configuring pools.
+
+To set a zone, create a JSON object consisting of the pools, save the
+object to a file (e.g., ``zone.json``); then, execute the following
+command, replacing ``{zone-name}`` with the name of the zone:
+
+::
+
+ # radosgw-admin zone set --rgw-zone={zone-name} --infile zone.json
+
+Where ``zone.json`` is the JSON file you created.
+
+Then, as ``root``, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Rename a Zone
+~~~~~~~~~~~~~
+
+To rename a zone, specify the zone name and the new zone name.
+
+::
+
+ # radosgw-admin zone rename --rgw-zone=<name> --zone-new-name=<name>
+
+Then, update the period:
+
+::
+
+ # radosgw-admin period update --commit
+
+Zone Group and Zone Settings
+----------------------------
+
+When configuring a default zone group and zone, the pool name includes
+the zone name. For example:
+
+- ``default.rgw.control``
+
+To change the defaults, include the following settings in your Ceph
+configuration file under each ``[client.radosgw.{instance-name}]``
+instance.
+
++-------------------------------------+-----------------------------------+---------+-----------------------+
+| Name | Description | Type | Default |
++=====================================+===================================+=========+=======================+
+| ``rgw_zone`` | The name of the zone for the | String | None |
+| | gateway instance. | | |
++-------------------------------------+-----------------------------------+---------+-----------------------+
+| ``rgw_zonegroup`` | The name of the zone group for | String | None |
+| | the gateway instance. | | |
++-------------------------------------+-----------------------------------+---------+-----------------------+
+| ``rgw_zonegroup_root_pool`` | The root pool for the zone group. | String | ``.rgw.root`` |
++-------------------------------------+-----------------------------------+---------+-----------------------+
+| ``rgw_zone_root_pool`` | The root pool for the zone. | String | ``.rgw.root`` |
++-------------------------------------+-----------------------------------+---------+-----------------------+
+| ``rgw_default_zone_group_info_oid`` | The OID for storing the default | String | ``default.zonegroup`` |
+| | zone group. We do not recommend | | |
+| | changing this setting. | | |
++-------------------------------------+-----------------------------------+---------+-----------------------+
+| ``rgw_num_zone_opstate_shards`` | The maximum number of shards for | Integer | ``128`` |
+| | keeping inter-zone group | | |
+| | synchronization progress. | | |
++-------------------------------------+-----------------------------------+---------+-----------------------+
projects / ceph.git / commitdiff