From 925f290023a1fd23a1cb029cd9d22cc3c27c2223 Mon Sep 17 00:00:00 2001 From: John Wilkins Date: Thu, 30 Mar 2017 14:54:42 -0700 Subject: [PATCH] Ported the downstream (and tested) document upstream. Signed-off-by: John Wilkins --- doc/radosgw/multisite.rst | 2179 ++++++++++++++++++++++++------------- 1 file changed, 1417 insertions(+), 762 deletions(-) diff --git a/doc/radosgw/multisite.rst b/doc/radosgw/multisite.rst index 3d243cb2652..e81e9014286 100644 --- a/doc/radosgw/multisite.rst +++ b/doc/radosgw/multisite.rst @@ -1,793 +1,559 @@ -============= -RGW Multisite -============= +========== +Multi-Site +========== .. versionadded:: Jewel -From Ceph release Jewel and beyond, you may configure each :term:`Ceph Object -Gateway` to work in an active active zone configuration, allowing for writing to -non-master zones. Following are the basic terminologies that would be used: - -- **Zone**: A zone is *logical* grouping of one or more Ceph Object Gateway - instances. There will be one zone that should be designated as the master zone - in a zonegroup, which will handle all bucket and user creation. - -- **Zonegroup**: A zonegroup consists of multiple zones, this approximately - corresponds to what used to be called as a region in pre Jewel releases for - federated deployments. There should be a master zonegroup that will handle - changes to the system configuration. Object data is only replicated between - zones within a zonegroup, whereas metadata (such as buckets and users) are - replicated across all zonegroups. - -- **Zonegroup map**: A zonegroup map is a configuration structure that holds the - map of the entire system, ie. which zonegroup is the master, relationships - between different zonegroups and certain configurables like storage policies. - -- **Realm**: A realm is a container for zonegroups, this allows for separation - of zonegroups themselves between clusters. It is possible to create multiple - realms, making it easier to run completely different configurations in the same - cluster. - -- **Period**: A period holds the configuration structure for the current state - of the realm. Every period contains a unique id and an epoch. A period's epoch - is incremented on every commit operation. Every realm has an associated - current period, holding the current state of configuration of the zonegroups - and storage policies. Any configuration change for a non-master zone will - increment the period's epoch. Changing the master zone to a different zone - will trigger the following changes: - - - A new period is generated with a new period id and epoch of 1 - - Realm's current period is updated to point to the newly generated period id - - Realm's epoch is incremented - -- **Metadata Master Zone**: The master zone of the master zonegroup is - designated as the metadata master zone. This zone is authoritative for all - metadata in the system. Any S3/Swift requests to modify metadata (such as - bucket creation/removal) that are sent to other zones are forwarded to and - handled by the metadata master zone. - -About this guide -================ - -In this guide we use 2 Ceph clusters, and create a single zonegroup, -with three zones, which sync data between them actively. For the -purpose of the guide, we'll create 2 zones in the same cluster, and 1 -zone in a different cluster. There is no sync agent involved for -mirroring data changes between the Ceph Object Gateway instances and -this allows for a much simpler configuration scheme and active-active -configurations. Please note that metadata operations such as creating -a user would still need to go through the master zone, however data -operations such as creation of buckets objects can be handled by any -of the zones. - -System Keys ------------ +A single zone configuration typically consists of one zone group containing one +zone and one or more `ceph-radosgw` instances where you may load-balance gateway +client requests between the instances. In a single zone configuration, typically +multiple gateway instances point to a single Ceph storage cluster. However, Kraken +supports several multi-site configuration options for the Ceph Object Gateway: + +- **Multi-zone:** A more advanced configuration consists of one zone group and + multiple zones, each zone with one or more `ceph-radosgw` instances. Each zone + is backed by its own Ceph Storage Cluster. Multiple zones in a zone group + provides disaster recovery for the zone group should one of the zones experience + a significant failure. In Kraken, each zone is active and may receive write + operations. In addition to disaster recovery, multiple active zones may also + serve as a foundation for content delivery networks. + +- **Multi-zone-group:** Formerly called 'regions', Ceph Object Gateway can also + support multiple zone groups, each zone group with one or more zones. Objects + stored to zones in one zone group within the same realm as another zone + group will share a global object namespace, ensuring unique object IDs across + zone groups and zones. + +- **Multiple Realms:** In Kraken, the Ceph Object Gateway supports the notion + of realms, which can be a single zone group or multiple zone groups and + a globally unique namespace for the realm. Multiple realms provide the ability + to support numerous configurations and namespaces. + +Replicating object data between zones within a zone group looks something +like this: + +.. image:: ../images/zone-sync2.png + :align: center + +For additional details on setting up a cluster, see `Ceph Object Gateway for +Production `__. + +Functional Changes from Infernalis +================================== + +In Kraken, you can configure each Ceph Object Gateway to +work in an active-active zone configuration, allowing for writes to +non-master zones. + +The multi-site configuration is stored within a container called a +"realm." The realm stores zone groups, zones, and a time "period" with +multiple epochs for tracking changes to the configuration. In Kraken, +the ``ceph-radosgw`` daemons handle the synchronization, +eliminating the need for a separate synchronization agent. Additionally, +the new approach to synchronization allows the Ceph Object Gateway to +operate with an "active-active" configuration instead of +"active-passive". + +Requirements and Assumptions +============================ + +A multi-site configuration requires at least two Ceph storage clusters, +preferably given a distinct cluster name. At least two Ceph object +gateway instances, one for each Ceph storage cluster. + +This guide assumes at least two Ceph storage clusters in geographically +separate locations; however, the configuration can work on the same +site. This guide also assumes four Ceph object gateway servers named +``rgw1`` and ``rgw2``. + +A multi-site configuration requires a master zone group and a master +zone. Additionally, each zone group requires a master zone. Zone groups +may have one or more secondary or non-master zones. + +In this guide, the ``rgw1`` host will serve as the master zone of the +master zone group; and, the ``rgw2`` host will serve as the secondary zone +of the master zone group. + +Pools +===== + +We recommend using the `Ceph Placement Group’s per Pool +Calculator `__ to calculate a +suitable number of placement groups for the pools the ``ceph-radosgw`` +daemon will create. Set the calculated values as defaults in your Ceph +configuration file. For example: -While configuring zones, Ceph Object Gateway instance, expects -creation of zone a system user, with a S3-like access and secret keys. -This allows another Ceph Object Gateway instance to pull the -configuration remotely, with the access and secret keys. For making -scripting and using of configuration management tools easier, it would -be easier to generate the access key and secret key before hand and -use these. For the purpose of the guide we assume an access key and -secret key are set in the environment. For eg.:: - - $ SYSTEM_ACCESS_KEY=1555b35654ad1656d805 - $ SYSTEM_SECRET_KEY=h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q12 - -Generally access keys are 20 alphanumeric characters, secret keys are -40 alphanumeric characters (they can contain +/= chars as well). These -keys can be generated in shell for eg:: - - $ SYSTEM_ACCESS_KEY=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 1) - $ SYSTEM_SECRET_KEY=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1) - -Then we can create a system user by following command:: +:: - $ radosgw-admin user create --uid=zone.user --display-name="Zone User" \ - --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --system + osd pool default pg num = 50 + osd pool default pgp num = 50 +.. note:: Make this change to the Ceph configuration file on your + storage cluster; then, either make a runtime change to the + configuration so that it will use those defaults when the gateway + instance creates the pools. -Naming conventions ------------------- - -This section describes the process of setting up a master zone. For the purpose -of the guide, we'll assume a zonegroup called ``us`` spanning the United States, -which will be our master zonegroup. This will contain two zones written in a -``{zonegroup}-{zone}`` format, this is just a convention only and you are free -to choose a format you prefer to set these names. So in summary: - -- Master Zonegroup: United States ``us`` -- Master zone: United States, East Region 1: ``us-east-1`` -- Second zone: United States, East Region 2: ``us-east-2`` -- Second zone: United States, West Region: ``us-west`` - -This will be a part of a larger realm, say ``gold``. The zones ``us-east-1`` and -``us-east-2`` are part of the same Ceph cluster, of which we'll make -``us-east-1`` as the primary. ``us-west`` will be in a different Ceph cluster. - -Create Pools ------------- - -Ceph Object Gateway instance will create pools on its own, when it is -configured with the appropriate permissions, this will be done with -the value of ``pg_num`` and ``pgp_num`` from the ``ceph.conf``. See -`Ceph Object Gateway - Create Pools`_ for more detailed explanation. -Pools particular to a zone by default follows the convention of -``{zone-name}.pool-name``. In the default configuration, for eg. in -``us-east-1`` zone following will be the pools: - -- ``.rgw.root`` -- ``us-east-1.rgw.control`` -- ``us-east-1.rgw.data.root`` -- ``us-east-1.rgw.gc`` -- ``us-east-1.rgw.log`` -- ``us-east-1.rgw.intent-log`` -- ``us-east-1.rgw.usage`` -- ``us-east-1.rgw.users.keys`` -- ``us-east-1.rgw.users.email`` -- ``us-east-1.rgw.users.swift`` -- ``us-east-1.rgw.users.uid`` -- ``us-east-1.rgw.buckets.index`` -- ``us-east-1.rgw.buckets.data`` -- ``us-east-1.rgw.meta`` - -These pools can be created in the other zones as well by replacing -``us-east-1`` with the appropriate zone name. - -Configuring the master zone in the primary cluster -================================================== - -First, we'll configure the master zone ``us-east-1``, in the master -zonegroup ``us``, this will serve as the master zone for all metadata -operations, so all operations like creation of users need to be done -on this zone. - -Creating a realm ----------------- - -Configure a realm called ``gold``, and also make this as default :: - - # radosgw-admin realm create --rgw-realm=gold --default - { - "id": "4a367026-bd8f-40ee-b486-8212482ddcd7", - "name": "gold", - "current_period": "09559832-67a4-4101-8b3f-10dfcd6b2707", - "epoch": 1 - } - - -Note that every realm has an id, which allows for flexibility like renaming the -realm later, should the need arise. The ``current_period`` changes whenever we -change anything in the master zone. The ``epoch`` is incremented when there's a -change in configuration of the master zone, which results in a change of current -period. - -Deleting the default zonegroup ------------------------------- - -A simple installation of Ceph Object Gateway would assume the default -zonegroup called "default", since we no longer need this we begin by -removing the default zonegroup:: - - # radosgw-admin zonegroup delete --rgw-zonegroup=default - - -Creating a master zonegroup ---------------------------- - -We'll create a zonegroup called ``us`` as master zonegroup. A master -zonegroup will be in control of the zonegroup map and propagate -changes to the rest of the system. We will also set this zonegroup as -the default, which allows later commands to use this zonegroup without -explicitly mentioning it with the ``--rgw-zonegroup`` option. - -:: - - # radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --master --default - { - "id": "d4018b8d-8c0d-4072-8919-608726fa369e", - "name": "us", - "api_name": "us", - "is_master": "true", - "endpoints": [ - "http:\/\/rgw1:80" - ], - "hostnames": [], - "hostnames_s3website": [], - "master_zone": "", - "zones": [], - "placement_targets": [], - "default_placement": "", - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7" - } +Alternatively, create the pools manually. See +`Pools `__ +for details on creating pools. -Alternatively, we can make this zonegroup as the default zonegroup via -the following command :: - - # radosgw-admin zonegroup default --rgw-zonegroup=us - - -Creating a master zone ----------------------- - -Next we create a zone, and make it as the default zone. Note that for metadata -operations like user creation you would want to use this zone. We also add it to -the zonegroup - -:: - - # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-1 --endpoints=http://rgw1:80 --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --default --master - { - "id": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "name": "us-east-1", - "domain_root": "us-east-1.rgw.data.root", - "control_pool": "us-east-1.rgw.control", - "gc_pool": "us-east-1.rgw.gc", - "log_pool": "us-east-1.rgw.log", - "intent_log_pool": "us-east-1.rgw.intent-log", - "usage_log_pool": "us-east-1.rgw.usage", - "user_keys_pool": "us-east-1.rgw.users.keys", - "user_email_pool": "us-east-1.rgw.users.email", - "user_swift_pool": "us-east-1.rgw.users.swift", - "user_uid_pool": "us-east-1.rgw.users.uid", - "system_key": { - "access_key": "1555b35654ad1656d804", - "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==" - }, - "placement_pools": [ - { - "key": "default-placement", - "val": { - "index_pool": "us-east-1.rgw.buckets.index", - "data_pool": "us-east-1.rgw.buckets.data", - "data_extra_pool": "us-east-1.rgw.buckets.non-ec", - "index_type": 0 - } - } - ], - "metadata_heap": "us-east-1.rgw.meta", - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7" - } +Pool names particular to a zone follow the naming convention +``{zone-name}.pool-name``. For example, a zone named ``us-east`` will +have the following pools: +- ``.rgw.root`` -Note that the above ``--rgw-zonegroup`` and ``--default`` options add the -zone to a zonegroup and makes it the default zone as well. This can also be -accomplished by the following commands:: +- ``us-east.rgw.control`` - # radosgw-admin zone default --rgw-zone=us-east-1 - # radosgw-admin zonegroup add --rgw-zonegroup=us --rgw-zone=us-east-1 +- ``us-east.rgw.data.root`` +- ``us-east.rgw.gc`` -Creating system users ---------------------- +- ``us-east.rgw.log`` -Next we create the system users for accessing the zone pools, note that these -keys would be used when configuring the second zone:: +- ``us-east.rgw.intent-log`` - # radosgw-admin user create --uid=zone.user --display-name="Zone - User" --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --system - { - "user_id": "zone.user", - "display_name": "Zone User", - "email": "", - "suspended": 0, - "max_buckets": 1000, - "auid": 0, - "subusers": [], - "keys": [ - { - "user": "zone.user", - "access_key": "1555b35654ad1656d804", - "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==" - } - ], - "swift_keys": [], - "caps": [], - "op_mask": "read, write, delete", - "system": "true", - "default_placement": "", - "placement_tags": [], - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "temp_url_keys": [] - } - - -.. note:: Please note that system users have super user privileges over the - entire zone, and will not behave like normal users for rest operations - like creating buckets, objects etc. as the output would contain - additional json fields for maintaining metadata. - - -Update the period ------------------ +- ``us-east.rgw.usage`` -Since we have made a change in the master zone configuration, we need to -commit these zone changes to reflect in the realm configuration structure. This -is what the period would look like initially:: - - # radosgw-admin period get - { - "id": "09559832-67a4-4101-8b3f-10dfcd6b2707", - "epoch": 1, - "predecessor_uuid": "", - "sync_status": [], - "period_map": { - "id": "09559832-67a4-4101-8b3f-10dfcd6b2707", - "zonegroups": [], - "short_zone_ids": [] - }, - "master_zonegroup": "", - "master_zone": "", - "period_config": { - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - } - }, - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7", - "realm_name": "gold", - "realm_epoch": 1 - } +- ``us-east.rgw.users.keys`` -Now we update the period and commit the changes:: - - # radosgw-admin period update --commit - { - "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1", - "epoch": 1, - "predecessor_uuid": "09559832-67a4-4101-8b3f-10dfcd6b2707", - "sync_status": [ ""... # truncating the output here - ], - "period_map": { - "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1", - "zonegroups": [ - { - "id": "d4018b8d-8c0d-4072-8919-608726fa369e", - "name": "us", - "api_name": "us", - "is_master": "true", - "endpoints": [ - "http:\/\/rgw1:80" - ], - "hostnames": [], - "hostnames_s3website": [], - "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "zones": [ - { - "id": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "name": "us-east-1", - "endpoints": [ - "http:\/\/rgw1:80" - ], - "log_meta": "true", - "log_data": "false", - "bucket_index_max_shards": 0, - "read_only": "false" - } - ], - "placement_targets": [ - { - "name": "default-placement", - "tags": [] - } - ], - "default_placement": "default-placement", - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7" - } - ], - "short_zone_ids": [ - { - "key": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "val": 630926044 - } - ] - }, - "master_zonegroup": "d4018b8d-8c0d-4072-8919-608726fa369e", - "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "period_config": { - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - } - }, - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7", - "realm_name": "gold", - "realm_epoch": 2 - } +- ``us-east.rgw.users.email`` +- ``us-east.rgw.users.swift`` -Starting the Ceph Object Gateway instance ------------------------------------------ +- ``us-east.rgw.users.uid`` -Before starting the Ceph Object Gateway instance, the ``rgw_zone`` and -``rgw_frontends`` options need to be mentioned in the configuration -file. For more details refer to the `Install Ceph Gateway`_ section of -the guide. The configuration section for Ceph Object Gateway instance -should resemble:: +- ``us-east.rgw.buckets.index`` - [client.rgw.us-east-1] - rgw_frontends="civetweb port=80" - rgw_zone=us-east-1 +- ``us-east.rgw.buckets.data`` -And start the Ceph Object Gateway instance(according to the OS -installation) :: - $ sudo systemctl start ceph-radosgw@rgw.us-east-1 +Configuring a Master Zone +========================= +All gateways in a multi-site configuration will retrieve their +configuration from a ``ceph-radosgw`` daemon on a host within the master +zone group and master zone. To configure your gateways in a multi-site +configuration, choose a ``ceph-radosgw`` instance to configure the +master zone group and master zone. -Configuring the second zone in same cluster -=========================================== +Create a Realm +-------------- -Now we configure the second zone, ``us-east-2``, in the same cluster. For the -same cluster, all the following commands can be executed in the node hosting the -primary zone itself. +A realm contains the multi-site configuration of zone groups and zones +and also serves to enforce a globally unique namespace within the realm. -Creating the second zone ------------------------- +Create a new realm for the multi-site configuration by opening a command +line interface on a host identified to serve in the master zone group +and zone. Then, execute the following: -We follow a similar step to creation of master zone, except dropping the -``--master`` flag this time :: - - # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --endpoints=http://rgw2:80 - { - "id": "950c1a43-6836-41a2-a161-64777e07e8b8", - "name": "us-east-2", - "domain_root": "us-east-2.rgw.data.root", - "control_pool": "us-east-2.rgw.control", - "gc_pool": "us-east-2.rgw.gc", - "log_pool": "us-east-2.rgw.log", - "intent_log_pool": "us-east-2.rgw.intent-log", - "usage_log_pool": "us-east-2.rgw.usage", - "user_keys_pool": "us-east-2.rgw.users.keys", - "user_email_pool": "us-east-2.rgw.users.email", - "user_swift_pool": "us-east-2.rgw.users.swift", - "user_uid_pool": "us-east-2.rgw.users.uid", - "system_key": { - "access_key": "1555b35654ad1656d804", - "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==" - }, - "placement_pools": [ - { - "key": "default-placement", - "val": { - "index_pool": "us-east-2.rgw.buckets.index", - "data_pool": "us-east-2.rgw.buckets.data", - "data_extra_pool": "us-east-2.rgw.buckets.non-ec", - "index_type": 0 - } - } - ], - "metadata_heap": "us-east-2.rgw.meta", - "realm_id": "815d74c2-80d6-4e63-8cfc-232037f7ff5c" - } +:: -Updating the period -------------------- - -Next we inform all the Ceph Object Gateway instances of the new change -in the system map by doing a period update and committing the -changes:: - - # radosgw-admin period update --commit - { - "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1", - "epoch": 2, - "predecessor_uuid": "09559832-67a4-4101-8b3f-10dfcd6b2707", - "sync_status": [ ""... # truncating the output here - ], - "period_map": { - "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1", - "zonegroups": [ - { - "id": "d4018b8d-8c0d-4072-8919-608726fa369e", - "name": "us", - "api_name": "us", - "is_master": "true", - "endpoints": [ - "http:\/\/rgw1:80" - ], - "hostnames": [], - "hostnames_s3website": [], - "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "zones": [ - { - "id": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "name": "us-east-1", - "endpoints": [ - "http:\/\/rgw1:80" - ], - "log_meta": "true", - "log_data": "false", - "bucket_index_max_shards": 0, - "read_only": "false" - }, - { - "id": "950c1a43-6836-41a2-a161-64777e07e8b8", - "name": "us-east-2", - "endpoints": [ - "http:\/\/rgw2:80" - ], - "log_meta": "false", - "log_data": "true", - "bucket_index_max_shards": 0, - "read_only": "false" - } + # radosgw-admin realm create --rgw-realm={realm-name} [--default] - ], - "placement_targets": [ - { - "name": "default-placement", - "tags": [] - } - ], - "default_placement": "default-placement", - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7" - } - ], - "short_zone_ids": [ - { - "key": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "val": 630926044 - }, - { - "key": "950c1a43-6836-41a2-a161-64777e07e8b8", - "val": 4276257543 - } +For example: - ] - }, - "master_zonegroup": "d4018b8d-8c0d-4072-8919-608726fa369e", - "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "period_config": { - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - } - }, - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7", - "realm_name": "gold", - "realm_epoch": 2 - } +:: + # radosgw-admin realm create --rgw-realm=movies --default -Starting the Ceph Object Gateway instance ------------------------------------------ - -On the node that is hosting the Ceph Object Gateway instance of the -second zone, you would start it, similar to the Ceph Object Gateway -instance ``client.rgw.us-east-1`` , changing the ``rgw -zone=us-east-2`` in the configuration file this time. For eg:: - - [client.rgw.us-east-2] - rgw_frontends="civetweb port=80" - rgw_zone=us-east-2 - -And start the Ceph Object Gateway instance depending on your Operating -system's init system, for eg:: - - $ sudo systemctl start ceph-radosgw@rgw.us-east-2 - - -Configuring the Ceph Object Gateway in the second Ceph cluster -============================================================== - -Now we go on to configure Ceph Object Gateway in the second Ceph -cluster, which may be geographically apart, that is a part of the same -zonegroup. - -Since a realm was already configured from the first cluster, we pull -and make that realm as the default here, We also get the configuration -from the master zone by pulling the period:: - - # radosgw-admin realm pull --url=http://rgw1:80 - --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY - { - "id": "4a367026-bd8f-40ee-b486-8212482ddcd7", - "name": "gold", - "current_period": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1", - "epoch": 2 - } - - # radosgw-admin period pull --url=http://rgw1:80 --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY - # radosgw-admin realm default --rgw-realm=gold - -We also set the default zonegroup to the created ``us`` zonegroup:: - - # radosgw-admin zonegroup default --rgw-zonegroup=us - -Configuring the second zone ---------------------------- - -We create the new zone, ``us-west``, with the same system keys:: - - # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west - --access-key=$SYSTEM_ACCESS_KEY --secret=$SYSTEM_SECRET_KEY --endpoints=http://rgw3:80 --default - { - "id": "d9522067-cb7b-4129-8751-591e45815b16", - "name": "us-west", - "domain_root": "us-west.rgw.data.root", - "control_pool": "us-west.rgw.control", - "gc_pool": "us-west.rgw.gc", - "log_pool": "us-west.rgw.log", - "intent_log_pool": "us-west.rgw.intent-log", - "usage_log_pool": "us-west.rgw.usage", - "user_keys_pool": "us-west.rgw.users.keys", - "user_email_pool": "us-west.rgw.users.email", - "user_swift_pool": "us-west.rgw.users.swift", - "user_uid_pool": "us-west.rgw.users.uid", - "system_key": { - "access_key": "1555b35654ad1656d804", - "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r\/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==" - }, - "placement_pools": [ - { - "key": "default-placement", - "val": { - "index_pool": "us-west.rgw.buckets.index", - "data_pool": "us-west.rgw.buckets.data", - "data_extra_pool": "us-west.rgw.buckets.non-ec", - "index_type": 0 - } - } - ], - "metadata_heap": "us-west.rgw.meta", - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7" +If the cluster will have a single realm, specify the ``--default`` flag. +If ``--default`` is specified, ``radosgw-admin`` will use this realm by +default. If ``--default`` is not specified, adding zone-groups and zones +requires specifying either the ``--rgw-realm`` flag or the +``--realm-id`` flag to identify the realm when adding zone groups and +zones. + +After creating the realm, ``radosgw-admin`` will echo back the realm +configuration. For example: + +:: + + { + "id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62", + "name": "movies", + "current_period": "1950b710-3e63-4c41-a19e-46a715000980", + "epoch": 1 } +.. note:: Ceph generates a unique ID for the realm, which allows the renaming + of a realm if the need arises. -Updating the period -------------------- +Create a Master Zone Group +-------------------------- -In order to propagate the zonegroup-map changes, we need to update and -commit the period:: +A realm must have at least one zone group, which will serve as the +master zone group for the realm. - # radosgw-admin period update --commit --rgw-zone=us-west - { - "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1", - "epoch": 3, - "predecessor_uuid": "09559832-67a4-4101-8b3f-10dfcd6b2707", - "sync_status": [ - "", # truncated - ], - "period_map": { - "id": "b5e4d3ec-2a62-4746-b479-4b2bc14b27d1", - "zonegroups": [ - { - "id": "d4018b8d-8c0d-4072-8919-608726fa369e", - "name": "us", - "api_name": "us", - "is_master": "true", - "endpoints": [ - "http:\/\/rgw1:80" - ], - "hostnames": [], - "hostnames_s3website": [], - "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "zones": [ - { - "id": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "name": "us-east-1", - "endpoints": [ - "http:\/\/rgw1:80" - ], - "log_meta": "true", - "log_data": "true", - "bucket_index_max_shards": 0, - "read_only": "false" - }, - { - "id": "950c1a43-6836-41a2-a161-64777e07e8b8", - "name": "us-east-2", - "endpoints": [ - "http:\/\/rgw2:80" - ], - "log_meta": "false", - "log_data": "true", - "bucket_index_max_shards": 0, - "read_only": "false" - }, - { - "id": "d9522067-cb7b-4129-8751-591e45815b16", - "name": "us-west", - "endpoints": [ - "http:\/\/rgw3:80" - ], - "log_meta": "false", - "log_data": "true", - "bucket_index_max_shards": 0, - "read_only": "false" - } - ], - "placement_targets": [ - { - "name": "default-placement", - "tags": [] - } - ], - "default_placement": "default-placement", - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7" - } +Create a new master zone group for the multi-site configuration by +opening a command line interface on a host identified to serve in the +master zone group and zone. Then, execute the following: + +:: + + # radosgw-admin zonegroup create --rgw-zonegroup={name} --endpoints={url} [--rgw-realm={realm-name}|--realm-id={realm-id}] --master --default + +For example: + +:: + + # radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --rgw-realm=movies --master --default + +If the realm will only have a single zone group, specify the +``--default`` flag. If ``--default`` is specified, ``radosgw-admin`` +will use this zone group by default when adding new zones. If +``--default`` is not specified, adding zones will require either the +``--rgw-zonegroup`` flag or the ``--zonegroup-id`` flag to identify the +zone group when adding or modifying zones. + +After creating the master zone group, ``radosgw-admin`` will echo back +the zone group configuration. For example: + +:: + + { + "id": "f1a233f5-c354-4107-b36c-df66126475a6", + "name": "us", + "api_name": "us", + "is_master": "true", + "endpoints": [ + "http:\/\/rgw1:80" ], - "short_zone_ids": [ - { - "key": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "val": 630926044 - }, - { - "key": "950c1a43-6836-41a2-a161-64777e07e8b8", - "val": 4276257543 - }, - { - "key": "d9522067-cb7b-4129-8751-591e45815b16", - "val": 329470157 - } - ] - }, - "master_zonegroup": "d4018b8d-8c0d-4072-8919-608726fa369e", - "master_zone": "83859a9a-9901-4f00-aa6d-285c777e10f0", - "period_config": { - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - } - }, - "realm_id": "4a367026-bd8f-40ee-b486-8212482ddcd7", - "realm_name": "gold", - "realm_epoch": 2 + "hostnames": [], + "hostnames_s3webzone": [], + "master_zone": "", + "zones": [], + "placement_targets": [], + "default_placement": "", + "realm_id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62" } -You can observe that the period epoch number has incremented, indicating a -change in the configuration. +Create a Master Zone +-------------------- + +.. important:: Zones must be created on a Ceph Object Gateway node that will be + within the zone. + +Create a new master zone for the multi-site configuration by opening a +command line interface on a host identified to serve in the master zone +group and zone. Then, execute the following: + +:: + + # radosgw-admin zone create --rgw-zonegroup={zone-group-name} \ + --rgw-zone={zone-name} \ + --master --default \ + --endpoints={http://fqdn}[,{http://fqdn}] + + +For example: + +:: + + # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east \ + --master --default \ + --endpoints={http://fqdn}[,{http://fqdn}] + + +.. note:: The ``--access-key`` and ``--secret`` aren’t specified. These + settings will be added to the zone once the user is created in the + next section. + +.. important:: The following steps assume a multi-site configuration using newly + installed systems that aren’t storing data yet. DO NOT DELETE the + ``default`` zone and its pools if you are already using it to store + data, or the data will be deleted and unrecoverable. + +Delete Default Zone Group and Zone +---------------------------------- + +Delete the ``default`` zone if it exists. Make sure to remove it from +the default zone group first. -Starting the Ceph Object Gateway instance ------------------------------------------ +:: + + # radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default + # radosgw-admin period update --commit + # radosgw-admin zone delete --rgw-zone=default + # radosgw-admin period update --commit + # radosgw-admin zonegroup delete --rgw-zonegroup=default + # radosgw-admin period update --commit + +Finally, delete the ``default`` pools in your Ceph storage cluster if +they exist. + +.. important:: The following step assumes a multi-site configuration using newly + installed systems that aren’t currently storing data. DO NOT DELETE + the ``default`` zone group if you are already using it to store + data. + +:: + + # rados rmpool default.rgw.control default.rgw.control --yes-i-really-really-mean-it + # rados rmpool default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it + # rados rmpool default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it + # rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it + # rados rmpool default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it + +Create a System User +-------------------- + +The ``ceph-radosgw`` daemons must authenticate before pulling realm and +period information. In the master zone, create a system user to +facilitate authentication between daemons. + +:: + + # radosgw-admin user create --uid="{user-name}" --display-name="{Display Name}" --system + +For example: + +:: + + # radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system + +Make a note of the ``access_key`` and ``secret_key``, as the secondary +zones will require them to authenticate with the master zone. + +Finally, add the system user to the master zone. + +:: + + # radosgw-admin zone modify --rgw-zone=us-east --access-key={access-key} --secret={secret} + # radosgw-admin period update --commit + +Update the Period +----------------- + +After updating the master zone configuration, update the period. + +:: + + # radosgw-admin period update --commit + +.. note:: Updating the period changes the epoch, and ensures that other zones + will receive the updated configuration. + +Update the Ceph Configuration File +---------------------------------- + +Update the Ceph configuration file on master zone hosts by adding the +``rgw_zone`` configuration option and the name of the master zone to the +instance entry. + +:: + + [client.rgw.{instance-name}] + ... + rgw_zone={zone-name} + +For example: + +:: + + [client.rgw.rgw1] + host = rgw1 + rgw frontends = "civetweb port=80" + rgw_zone=us-east + +Start the Gateway +----------------- + +On the object gateway host, start and enable the Ceph Object Gateway +service: + +:: + + # systemctl start ceph-radosgw@rgw.`hostname -s` + # systemctl enable ceph-radosgw@rgw.`hostname -s` + +Configure Secondary Zones +========================= + +Zones within a zone group replicate all data to ensure that each zone +has the same data. When creating the secondary zone, execute all of the +following operations on a host identified to serve the secondary zone. + +.. note:: To add a third zone, follow the same procedures as for adding the + secondary zone. Use different zone name. + +.. important:: You must execute metadata operations, such as user creation, on a + host within the master zone. The master zone and the secondary zone + can receive bucket operations, but the secondary zone redirects + bucket operations to the master zone. If the master zone is down, + bucket operations will fail. + +Pull the Realm +-------------- + +Using the URL path, access key and secret of the master zone in the +master zone group, pull the realm to the host. To pull a non-default +realm, specify the realm using the ``--rgw-realm`` or ``--realm-id`` +configuration options. + +:: + + # radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret} + +If this realm is the default realm or the only realm, make the realm the +default realm. + +:: + + # radosgw-admin realm default --rgw-realm={realm-name} + +Pull the Period +--------------- + +Using the URL path, access key and secret of the master zone in the +master zone group, pull the period to the host. To pull a period from a +non-default realm, specify the realm using the ``--rgw-realm`` or +``--realm-id`` configuration options. + +:: + + # radosgw-admin period pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret} + + +.. note:: Pulling the period retrieves the latest version of the zone group + and zone configurations for the realm. + +Create a Secondary Zone +----------------------- -This is similar to starting the Ceph Object Gateway instance in the -first zone, the only difference is the ``rgw zone`` option, which -should reflect ``us-west``:: +.. important:: Zones must be created on a Ceph Object Gateway node that will be + within the zone. - [client.rgw.us-west] - rgw_frontends="civetweb port=80" - rgw_zone=us-west +Create a secondary zone for the multi-site configuration by opening a +command line interface on a host identified to serve the secondary zone. +Specify the zone group ID, the new zone name and an endpoint for the +zone. **DO NOT** use the ``--master`` or ``--default`` flags. In Kraken, +all zones run in an active-active configuration by +default; that is, a gateway client may write data to any zone and the +zone will replicate the data to all other zones within the zone group. +If the secondary zone should not accept write operations, specify the +``--read-only`` flag to create an active-passive configuration between +the master zone and the secondary zone. Additionally, provide the +``access_key`` and ``secret_key`` of the generated system user stored in +the master zone of the master zone group. Execute the following: -And start the Ceph Object Gateway instance depending on your Operating -system's init system, for eg:: +:: + + # radosgw-admin zone create --rgw-zonegroup={zone-group-name}\ + --rgw-zone={zone-name} --endpoints={url} \ + --access-key={system-key} --secret={secret}\ + --endpoints=http://{fqdn}:80 \ + [--read-only] + +For example: + +:: + + # radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west \ + --access-key={system-key} --secret={secret} \ + --endpoints=http://rgw2:80 + +.. important:: The following steps assume a multi-site configuration using newly + installed systems that aren’t storing data. **DO NOT DELETE** the + ``default`` zone and its pools if you are already using it to store + data, or the data will be lost and unrecoverable. + +Delete the default zone if needed. + +:: + + # radosgw-admin zone delete --rgw-zone=default + +Finally, delete the default pools in your Ceph storage cluster if +needed. + +:: + + # rados rmpool default.rgw.control default.rgw.control --yes-i-really-really-mean-it + # rados rmpool default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it + # rados rmpool default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it + # rados rmpool default.rgw.log default.rgw.log --yes-i-really-really-mean-it + # rados rmpool default.rgw.users.uid default.rgw.users.uid --yes-i-really-really-mean-it + +Update the Ceph Configuration File +---------------------------------- + +Update the Ceph configuration file on the secondary zone hosts by adding +the ``rgw_zone`` configuration option and the name of the secondary zone +to the instance entry. + +:: + + [client.rgw.{instance-name}] + ... + rgw_zone={zone-name} + +For example: + +:: + + [client.rgw.rgw2] + host = rgw2 + rgw frontends = "civetweb port=80" + rgw_zone=us-west + +Update the Period +----------------- + +After updating the master zone configuration, update the period. + +:: + + # radosgw-admin period update --commit + +.. note:: Updating the period changes the epoch, and ensures that other zones + will receive the updated configuration. + +Start the Gateway +----------------- + +On the object gateway host, start and enable the Ceph Object Gateway +service: + +:: + + # systemctl start ceph-radosgw@rgw.`hostname -s` + # systemctl enable ceph-radosgw@rgw.`hostname -s` + +Check Synchronization Status +---------------------------- + +Once the secondary zone is up and running, check the synchronization +status. Synchronization copies users and buckets created in the master +zone to the secondary zone. + +:: + + # radosgw-admin sync status + +The output will provide the status of synchronization operations. For +example: + +:: + + realm f3239bc5-e1a8-4206-a81d-e1576480804d (earth) + zonegroup c50dbb7e-d9ce-47cc-a8bb-97d9b399d388 (us) + zone 4c453b70-4a16-4ce8-8185-1893b05d346e (us-west) + metadata sync syncing + full sync: 0/64 shards + metadata is caught up with master + incremental sync: 64/64 shards + data sync source: 1ee9da3e-114d-4ae3-a8a4-056e8a17f532 (us-east) + syncing + full sync: 0/128 shards + incremental sync: 128/128 shards + data is caught up with source - $ sudo systemctl start ceph-radosgw@rgw.us-west +.. note:: Secondary zones accept bucket operations; however, secondary zones + redirect bucket operations to the master zone and then synchronize + with the master zone to receive the result of the bucket operations. + If the master zone is down, bucket operations executed on the + secondary zone will fail, but object operations should succeed. Maintenance @@ -798,12 +564,11 @@ Checking the Sync Status Information about the replication status of a zone can be queried with:: - $ radosgw-admin sync status - + $ radosgw-admin sync status realm b3bc1c37-9c44-4b89-a03b-04c269bea5da (earth) zonegroup f54f9b22-b4b6-4a0e-9211-fa6ac1693f49 (us) zone adce11c9-b8ed-4a90-8bc5-3fc029ff0816 (us-2) - metadata sync syncing + metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is behind on 1 shards @@ -819,23 +584,22 @@ Information about the replication status of a zone can be queried with:: incremental sync: 128/128 shards data is caught up with source - Changing the Metadata Master Zone --------------------------------- .. important:: Care must be taken when changing which zone is the metadata - master. If a zone has not finished syncing metadata from the current master - zone, it will be unable to serve any remaining entries when promoted to - master and those changes will be lost. For this reason, waiting for a - zone's ``radosgw-admin sync status`` to catch up on metadata sync before - promoting it to master is recommended. - - Similarly, if changes to metadata are being processed by the current master - zone while another zone is being promoted to master, those changes are - likely to be lost. To avoid this, shutting down any ``radosgw`` instances - on the previous master zone is recommended. After promoting another zone, - its new period can be fetched with ``radosgw-admin period pull`` and the - gateway(s) can be restarted. + master. If a zone has not finished syncing metadata from the current master + zone, it will be unable to serve any remaining entries when promoted to + master and those changes will be lost. For this reason, waiting for a + zone's ``radosgw-admin sync status`` to catch up on metadata sync before + promoting it to master is recommended. + + Similarly, if changes to metadata are being processed by the current master + zone while another zone is being promoted to master, those changes are + likely to be lost. To avoid this, shutting down any ``radosgw`` instances + on the previous master zone is recommended. After promoting another zone, + its new period can be fetched with ``radosgw-admin period pull`` and the + gateway(s) can be restarted. To promote a zone (for example, zone ``us-2`` in zonegroup ``us``) to metadata master, run the following commands on that zone:: @@ -847,5 +611,896 @@ master, run the following commands on that zone:: This will generate a new period, and the radosgw instance(s) in zone ``us-2`` will send this period to other zones. -.. _`Ceph Object Gateway - Create Pools`: ../config#create-pools -.. _`Install Ceph Gateway`: ../../install/install-ceph-gateway +Failover and Disaster Recovery +============================== + +If the master zone should fail, failover to the secondary zone for +disaster recovery. + +1. Make the secondary zone the master and default zone. For example: + + :: + + # radosgw-admin zone modify --rgw-zone={zone-name} --master --default + + By default, Ceph Object Gateway will run in an active-active + configuration. If the cluster was configured to run in an + active-passive configuration, the secondary zone is a read-only zone. + Remove the ``--read-only`` status to allow the zone to receive write + operations. For example: + + :: + + # radosgw-admin zone modify --rgw-zone={zone-name} --master --default \ + --read-only=False + +2. Update the period to make the changes take effect. + + :: + + # radosgw-admin period update --commit + +3. Finally, restart the Ceph Object Gateway. + + :: + + # systemctl restart ceph-radosgw@rgw.`hostname -s` + +If the former master zone recovers, revert the operation. + +1. From the recovered zone, pull the period from the current master + zone. + + :: + + # radosgw-admin period pull --url={url-to-master-zone-gateway} \ + --access-key={access-key} --secret={secret} + +2. Make the recovered zone the master and default zone. + + :: + + # radosgw-admin zone modify --rgw-zone={zone-name} --master --default + +3. Update the period to make the changes take effect. + + :: + + # radosgw-admin period update --commit + +4. Then, restart the Ceph Object Gateway in the recovered zone. + + :: + + # systemctl restart ceph-radosgw@rgw.`hostname -s` + +5. If the secondary zone needs to be a read-only configuration, update + the secondary zone. + + :: + + # radosgw-admin zone modify --rgw-zone={zone-name} --read-only + +6. Update the period to make the changes take effect. + + :: + + # radosgw-admin period update --commit + +7. Finally, restart the Ceph Object Gateway in the secondary zone. + + :: + + # systemctl restart ceph-radosgw@rgw.`hostname -s` + +Migrating a Single Site System to Multi-Site +============================================ + +To migrate from a single site system with a ``default`` zone group and +zone to a multi site system, use the following steps: + +1. Create a realm. Replace ```` with the realm name. + + :: + + # radosgw-admin realm create --rgw-realm= --default + +2. Rename the default zone and zonegroup. Replace ```` with the + zonegroup or zone name. + + :: + + # radosgw-admin zonegroup rename --rgw-zonegroup default --zonegroup-new-name= + # radosgw-admin zone rename --rgw-zone default --zone-new-name us-east-1 --rgw-zonegroup= + +3. Configure the master zonegroup. Replace ```` with the realm or + zonegroup name. Replace ```` with the fully qualified domain + name(s) in the zonegroup. + + :: + + # radosgw-admin zonegroup modify --rgw-realm= --rgw-zonegroup= --endpoints http://:80 --master --default + +4. Configure the master zone. Replace ```` with the realm, + zonegroup or zone name. Replace ```` with the fully qualified + domain name(s) in the zonegroup. + + :: + + # radosgw-admin zone modify --rgw-realm= --rgw-zonegroup= \ + --rgw-zone= --endpoints http://:80 \ + --access-key= --secret= \ + --master --default + +5. Create a system user. Replace ```` with the username. + Replace ```` with a display name. It may contain + spaces. + + :: + + # radosgw-admin user create --uid= --display-name=""\ + --access-key= --secret= --system + +6. Commit the updated configuration. + + :: + + # radosgw-admin period update --commit + +7. Finally, restart the Ceph Object Gateway. + + :: + + # systemctl restart ceph-radosgw@rgw.`hostname -s` + +After completing this procedure, proceed to `Configure a Secondary +Zone <#configure-secondary-zones>`__ to create a secondary zone +in the master zone group. + + +Multi-Site Configuration Reference +================================== + +The following sections provide additional details and command-line +usage for realms, periods, zone groups and zones. + +Realms +------ + +A realm represents a globally unique namespace consisting of one or more +zonegroups containing one or more zones, and zones containing buckets, +which in turn contain objects. A realm enables the Ceph Object Gateway +to support multiple namespaces and their configuration on the same +hardware. + +A realm contains the notion of periods. Each period represents the state +of the zone group and zone configuration in time. Each time you make a +change to a zonegroup or zone, update the period and commit it. + +By default, the Ceph Object Gateway does not create a realm +for backward compatibility with Infernalis and earlier releases. +However, as a best practice, we recommend creating realms for new +clusters. + +Create a Realm +~~~~~~~~~~~~~~ + +To create a realm, execute ``realm create`` and specify the realm name. +If the realm is the default, specify ``--default``. + +:: + + # radosgw-admin realm create --rgw-realm={realm-name} [--default] + +For example: + +:: + + # radosgw-admin realm create --rgw-realm=movies --default + +By specifying ``--default``, the realm will be called implicitly with +each ``radosgw-admin`` call unless ``--rgw-realm`` and the realm name +are explicitly provided. + +Make a Realm the Default +~~~~~~~~~~~~~~~~~~~~~~~~ + +One realm in the list of realms should be the default realm. There may +be only one default realm. If there is only one realm and it wasn’t +specified as the default realm when it was created, make it the default +realm. Alternatively, to change which realm is the default, execute: + +:: + + # radosgw-admin realm default --rgw-realm=movies + +..note:: When the realm is default, the command line assumes + ``--rgw-realm=`` as an argument. + +Delete a Realm +~~~~~~~~~~~~~~ + +To delete a realm, execute ``realm delete`` and specify the realm name. + +:: + + # radosgw-admin realm delete --rgw-realm={realm-name} + +For example: + +:: + + # radosgw-admin realm delete --rgw-realm=movies + +Get a Realm +~~~~~~~~~~~ + +To get a realm, execute ``realm get`` and specify the realm name. + +:: + + #radosgw-admin realm get --rgw-realm= + +For example: + +:: + + # radosgw-admin realm get --rgw-realm=movies [> filename.json] + +The CLI will echo a JSON object with the realm properties. + +:: + + { + "id": "0a68d52e-a19c-4e8e-b012-a8f831cb3ebc", + "name": "movies", + "current_period": "b0c5bbef-4337-4edd-8184-5aeab2ec413b", + "epoch": 1 + } + +Use ``>`` and an output file name to output the JSON object to a file. + +Set a Realm +~~~~~~~~~~~ + +To set a realm, execute ``realm set``, specify the realm name, and +``--infile=`` with an input file name. + +:: + + #radosgw-admin realm set --rgw-realm= --infile= + +For example: + +:: + + # radosgw-admin realm set --rgw-realm=movies --infile=filename.json + +List Realms +~~~~~~~~~~~ + +To list realms, execute ``realm list``. + +:: + + # radosgw-admin realm list + +List Realm Periods +~~~~~~~~~~~~~~~~~~ + +To list realm periods, execute ``realm list-periods``. + +:: + + # radosgw-admin realm list-periods + +Pull a Realm +~~~~~~~~~~~~ + +To pull a realm from the node containing the master zone group and +master zone to a node containing a secondary zone group or zone, execute +``realm pull`` on the node that will receive the realm configuration. + +:: + + # radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret} + +Rename a Realm +~~~~~~~~~~~~~~ + +A realm is not part of the period. Consequently, renaming the realm is +only applied locally, and will not get pulled with ``realm pull``. When +renaming a realm with multiple zones, run the command on each zone. To +rename a realm, execute the following: + +:: + + # radosgw-admin realm rename --rgw-realm= --realm-new-name= + +.. note:: DO NOT use ``realm set`` to change the ``name`` parameter. That + changes the internal name only. Specifying ``--rgw-realm`` would + still use the old realm name. + +Zone Groups +----------- + +The Ceph Object Gateway supports multi-site deployments and a global +namespace by using the notion of zone groups. Formerly called a region +in Infernalis, a zone group defines the geographic location of one or more Ceph +Object Gateway instances within one or more zones. + +Configuring zone groups differs from typical configuration procedures, +because not all of the settings end up in a Ceph configuration file. You +can list zone groups, get a zone group configuration, and set a zone +group configuration. + +Create a Zone Group +~~~~~~~~~~~~~~~~~~~ + +Creating a zone group consists of specifying the zone group name. +Creating a zone assumes it will live in the default realm unless +``--rgw-realm=`` is specified. If the zonegroup is the +default zonegroup, specify the ``--default`` flag. If the zonegroup is +the master zonegroup, specify the ``--master`` flag. For example: + +:: + + # radosgw-admin zonegroup create --rgw-zonegroup= [--rgw-realm=][--master] [--default] + + +.. note:: Use ``zonegroup modify --rgw-zonegroup=`` to modify + an existing zone group’s settings. + +Make a Zone Group the Default +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One zonegroup in the list of zonegroups should be the default zonegroup. +There may be only one default zonegroup. If there is only one zonegroup +and it wasn’t specified as the default zonegroup when it was created, +make it the default zonegroup. Alternatively, to change which zonegroup +is the default, execute: + +:: + + # radosgw-admin zonegroup default --rgw-zonegroup=comedy + +.. note:: When the zonegroup is default, the command line assumes + ``--rgw-zonegroup=`` as an argument. + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +Add a Zone to a Zone Group +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To add a zone to a zonegroup, execute the following: + +:: + + # radosgw-admin zonegroup add --rgw-zonegroup= --rgw-zone= + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +Remove a Zone from a Zone Group +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To remove a zone from a zonegroup, execute the following: + +:: + + # radosgw-admin zonegroup remove --rgw-zonegroup= --rgw-zone= + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +Rename a Zone Group +~~~~~~~~~~~~~~~~~~~ + +To rename a zonegroup, execute the following: + +:: + + # radosgw-admin zonegroup rename --rgw-zonegroup= --zonegroup-new-name= + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +Delete a Zone Group +~~~~~~~~~~~~~~~~~~~ + +To delete a zonegroup, execute the following: + +:: + + # radosgw-admin zonegroup delete --rgw-zonegroup= + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +List Zone Groups +~~~~~~~~~~~~~~~~ + +A Ceph cluster contains a list of zone groups. To list the zone groups, +execute: + +:: + + # radosgw-admin zonegroup list + +The ``radosgw-admin`` returns a JSON formatted list of zone groups. + +:: + + { + "default_info": "90b28698-e7c3-462c-a42d-4aa780d24eda", + "zonegroups": [ + "us" + ] + } + +Get a Zone Group Map +~~~~~~~~~~~~~~~~~~~~ + +To list the details of each zone group, execute: + +:: + + # radosgw-admin zonegroup-map get + +.. note:: If you receive a ``failed to read zonegroup map`` error, run + ``radosgw-admin zonegroup-map update`` as ``root`` first. + +Get a Zone Group +~~~~~~~~~~~~~~~~ + +To view the configuration of a zone group, execute: + +:: + + radosgw-admin zonegroup get [--rgw-zonegroup=] + +The zone group configuration looks like this: + +:: + + { + "id": "90b28698-e7c3-462c-a42d-4aa780d24eda", + "name": "us", + "api_name": "us", + "is_master": "true", + "endpoints": [ + "http:\/\/rgw1:80" + ], + "hostnames": [], + "hostnames_s3website": [], + "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e", + "zones": [ + { + "id": "9248cab2-afe7-43d8-a661-a40bf316665e", + "name": "us-east", + "endpoints": [ + "http:\/\/rgw1" + ], + "log_meta": "true", + "log_data": "true", + "bucket_index_max_shards": 0, + "read_only": "false" + }, + { + "id": "d1024e59-7d28-49d1-8222-af101965a939", + "name": "us-west", + "endpoints": [ + "http:\/\/rgw2:80" + ], + "log_meta": "false", + "log_data": "true", + "bucket_index_max_shards": 0, + "read_only": "false" + } + ], + "placement_targets": [ + { + "name": "default-placement", + "tags": [] + } + ], + "default_placement": "default-placement", + "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe" + } + +Set a Zone Group +~~~~~~~~~~~~~~~~ + +Defining a zone group consists of creating a JSON object, specifying at +least the required settings: + +1. ``name``: The name of the zone group. Required. + +2. ``api_name``: The API name for the zone group. Optional. + +3. ``is_master``: Determines if the zone group is the master zone group. + Required. **note:** You can only have one master zone group. + +4. ``endpoints``: A list of all the endpoints in the zone group. For + example, you may use multiple domain names to refer to the same zone + group. Remember to escape the forward slashes (``\/``). You may also + specify a port (``fqdn:port``) for each endpoint. Optional. + +5. ``hostnames``: A list of all the hostnames in the zone group. For + example, you may use multiple domain names to refer to the same zone + group. Optional. The ``rgw dns name`` setting will automatically be + included in this list. You should restart the gateway daemon(s) after + changing this setting. + +6. ``master_zone``: The master zone for the zone group. Optional. Uses + the default zone if not specified. **note:** You can only have one + master zone per zone group. + +7. ``zones``: A list of all zones within the zone group. Each zone has a + name (required), a list of endpoints (optional), and whether or not + the gateway will log metadata and data operations (false by default). + +8. ``placement_targets``: A list of placement targets (optional). Each + placement target contains a name (required) for the placement target + and a list of tags (optional) so that only users with the tag can use + the placement target (i.e., the user’s ``placement_tags`` field in + the user info). + +9. ``default_placement``: The default placement target for the object + index and object data. Set to ``default-placement`` by default. You + may also set a per-user default placement in the user info for each + user. + +To set a zone group, create a JSON object consisting of the required +fields, save the object to a file (e.g., ``zonegroup.json``); then, +execute the following command: + +:: + + # radosgw-admin zonegroup set --infile zonegroup.json + +Where ``zonegroup.json`` is the JSON file you created. + +.. important:: The ``default`` zone group ``is_master`` setting is ``true`` by + default. If you create a new zone group and want to make it the + master zone group, you must either set the ``default`` zone group + ``is_master`` setting to ``false``, or delete the ``default`` zone + group. + +Finally, update the period: + +:: + + # radosgw-admin period update --commit + +Set a Zone Group Map +~~~~~~~~~~~~~~~~~~~~ + +Setting a zone group map consists of creating a JSON object consisting +of one or more zone groups, and setting the ``master_zonegroup`` for the +cluster. Each zone group in the zone group map consists of a key/value +pair, where the ``key`` setting is equivalent to the ``name`` setting +for an individual zone group configuration, and the ``val`` is a JSON +object consisting of an individual zone group configuration. + +You may only have one zone group with ``is_master`` equal to ``true``, +and it must be specified as the ``master_zonegroup`` at the end of the +zone group map. The following JSON object is an example of a default +zone group map. + +:: + + { + "zonegroups": [ + { + "key": "90b28698-e7c3-462c-a42d-4aa780d24eda", + "val": { + "id": "90b28698-e7c3-462c-a42d-4aa780d24eda", + "name": "us", + "api_name": "us", + "is_master": "true", + "endpoints": [ + "http:\/\/rgw1:80" + ], + "hostnames": [], + "hostnames_s3website": [], + "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e", + "zones": [ + { + "id": "9248cab2-afe7-43d8-a661-a40bf316665e", + "name": "us-east", + "endpoints": [ + "http:\/\/rgw1" + ], + "log_meta": "true", + "log_data": "true", + "bucket_index_max_shards": 0, + "read_only": "false" + }, + { + "id": "d1024e59-7d28-49d1-8222-af101965a939", + "name": "us-west", + "endpoints": [ + "http:\/\/rgw2:80" + ], + "log_meta": "false", + "log_data": "true", + "bucket_index_max_shards": 0, + "read_only": "false" + } + ], + "placement_targets": [ + { + "name": "default-placement", + "tags": [] + } + ], + "default_placement": "default-placement", + "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe" + } + } + ], + "master_zonegroup": "90b28698-e7c3-462c-a42d-4aa780d24eda", + "bucket_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + }, + "user_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + } + } + +To set a zone group map, execute the following: + +:: + + # radosgw-admin zonegroup-map set --infile zonegroupmap.json + +Where ``zonegroupmap.json`` is the JSON file you created. Ensure that +you have zones created for the ones specified in the zone group map. +Finally, update the period. + +:: + + # radosgw-admin period update --commit + +Zones +----- + +Ceph Object Gateway supports the notion of zones. A zone defines a +logical group consisting of one or more Ceph Object Gateway instances. + +Configuring zones differs from typical configuration procedures, because +not all of the settings end up in a Ceph configuration file. You can +list zones, get a zone configuration and set a zone configuration. + +Create a Zone +~~~~~~~~~~~~~ + +To create a zone, specify a zone name. If it is a master zone, specify +the ``--master`` option. Only one zone in a zone group may be a master +zone. To add the zone to a zonegroup, specify the ``--rgw-zonegroup`` +option with the zonegroup name. + +:: + + # radosgw-admin zone create --rgw-zone= \ + [--zonegroup=[,] \ + [--master] [--default] \ + --access-key $SYSTEM_ACCESS_KEY --secret $SYSTEM_SECRET_KEY + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +Delete a Zone +~~~~~~~~~~~~~ + +To delete zone, first remove it from the zonegroup. + +:: + + # radosgw-admin zonegroup remove --zonegroup=\ + --zone= + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +Next, delete the zone. Execute the following: + +:: + + # radosgw-admin zone delete --rgw-zone + +Finally, update the period: + +:: + + # radosgw-admin period update --commit + +.. important:: Do not delete a zone without removing it from a zone group first. + Otherwise, updating the period will fail. + +If the pools for the deleted zone will not be used anywhere else, +consider deleting the pools. Replace ```` in the example below +with the deleted zone’s name. + +.. important:: Only delete the pools with prepended zone names. Deleting the root + pool, such as, ``.rgw.root`` will remove all of the system’s + configuration. + +.. important:: Once the pools are deleted, all of the data within them are deleted + in an unrecoverable manner. Only delete the pools if the pool + contents are no longer needed. + +:: + + # rados rmpool .rgw.control .rgw.control --yes-i-really-really-mean-it + # rados rmpool .rgw.data.root .rgw.data.root --yes-i-really-really-mean-it + # rados rmpool .rgw.gc .rgw.gc --yes-i-really-really-mean-it + # rados rmpool .rgw.log .rgw.log --yes-i-really-really-mean-it + # rados rmpool .rgw.users.uid .rgw.users.uid --yes-i-really-really-mean-it + +Modify a Zone +~~~~~~~~~~~~~ + +To modify a zone, specify the zone name and the parameters you wish to +modify. + +:: + + # radosgw-admin zone modify [options] + +Where ``[options]``: + +- ``--access-key=`` +- ``--secret/--secret-key=`` +- ``--master`` +- ``--default`` +- ``--endpoints=`` + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +List Zones +~~~~~~~~~~ + +As ``root``, to list the zones in a cluster, execute: + +:: + + # radosgw-admin zone list + +Get a Zone +~~~~~~~~~~ + +As ``root``, to get the configuration of a zone, execute: + +:: + + # radosgw-admin zone get [--rgw-zone=] + +The ``default`` zone looks like this: + +:: + + { "domain_root": ".rgw", + "control_pool": ".rgw.control", + "gc_pool": ".rgw.gc", + "log_pool": ".log", + "intent_log_pool": ".intent-log", + "usage_log_pool": ".usage", + "user_keys_pool": ".users", + "user_email_pool": ".users.email", + "user_swift_pool": ".users.swift", + "user_uid_pool": ".users.uid", + "system_key": { "access_key": "", "secret_key": ""}, + "placement_pools": [ + { "key": "default-placement", + "val": { "index_pool": ".rgw.buckets.index", + "data_pool": ".rgw.buckets"} + } + ] + } + +Set a Zone +~~~~~~~~~~ + +Configuring a zone involves specifying a series of Ceph Object Gateway +pools. For consistency, we recommend using a pool prefix that is the +same as the zone name. See +`Pools `__ +for details of configuring pools. + +To set a zone, create a JSON object consisting of the pools, save the +object to a file (e.g., ``zone.json``); then, execute the following +command, replacing ``{zone-name}`` with the name of the zone: + +:: + + # radosgw-admin zone set --rgw-zone={zone-name} --infile zone.json + +Where ``zone.json`` is the JSON file you created. + +Then, as ``root``, update the period: + +:: + + # radosgw-admin period update --commit + +Rename a Zone +~~~~~~~~~~~~~ + +To rename a zone, specify the zone name and the new zone name. + +:: + + # radosgw-admin zone rename --rgw-zone= --zone-new-name= + +Then, update the period: + +:: + + # radosgw-admin period update --commit + +Zone Group and Zone Settings +---------------------------- + +When configuring a default zone group and zone, the pool name includes +the zone name. For example: + +- ``default.rgw.control`` + +To change the defaults, include the following settings in your Ceph +configuration file under each ``[client.radosgw.{instance-name}]`` +instance. + ++-------------------------------------+-----------------------------------+---------+-----------------------+ +| Name | Description | Type | Default | ++=====================================+===================================+=========+=======================+ +| ``rgw_zone`` | The name of the zone for the | String | None | +| | gateway instance. | | | ++-------------------------------------+-----------------------------------+---------+-----------------------+ +| ``rgw_zonegroup`` | The name of the zone group for | String | None | +| | the gateway instance. | | | ++-------------------------------------+-----------------------------------+---------+-----------------------+ +| ``rgw_zonegroup_root_pool`` | The root pool for the zone group. | String | ``.rgw.root`` | ++-------------------------------------+-----------------------------------+---------+-----------------------+ +| ``rgw_zone_root_pool`` | The root pool for the zone. | String | ``.rgw.root`` | ++-------------------------------------+-----------------------------------+---------+-----------------------+ +| ``rgw_default_zone_group_info_oid`` | The OID for storing the default | String | ``default.zonegroup`` | +| | zone group. We do not recommend | | | +| | changing this setting. | | | ++-------------------------------------+-----------------------------------+---------+-----------------------+ +| ``rgw_num_zone_opstate_shards`` | The maximum number of shards for | Integer | ``128`` | +| | keeping inter-zone group | | | +| | synchronization progress. | | | ++-------------------------------------+-----------------------------------+---------+-----------------------+ -- 2.39.5