Install Ceph Object Gateway
===========================
-As of `firefly` (v0.80), Ceph Object Gateway is runnig on Civetweb (embedded into the ``ceph-radosgw`` daemon) instead of Apache and FastCGI. Using Civetweb simplifies the Ceph Object Gateway installation and configuration.
+As of `firefly` (v0.80), Ceph Object Gateway is runnig on Civetweb (embedded
+into the ``ceph-radosgw`` daemon) instead of Apache and FastCGI. Using Civetweb
+simplifies the Ceph Object Gateway installation and configuration.
-.. note:: To run the Ceph Object Gateway service, you should have a running Ceph storage cluster, and the gateway host should have access to the public network.
+.. note:: To run the Ceph Object Gateway service, you should have a running
+ Ceph storage cluster, and the gateway host should have access to the
+ public network.
-.. note:: In version 0.80, the Ceph Object Gateway does not support SSL. You may setup a reverse proxy server with SSL to dispatch HTTPS requests as HTTP requests to CivetWeb.
+.. note:: In version 0.80, the Ceph Object Gateway does not support SSL. You
+ may setup a reverse proxy server with SSL to dispatch HTTPS requests
+ as HTTP requests to CivetWeb.
Execute the Pre-Installation Procedure
--------------------------------------
-Refer to the Red Hat Ceph Storage Installation Guide, and execute the pre-installation procedures on your Ceph Object Gateway node. Specifically, you should disable ``requiretty`` on your Ceph Deploy user, set SELinux to ``Permissive`` and set up a Ceph Deploy user with password-less ``sudo``. For Ceph Object Gateways, you will need to open the port that Civetweb will use in production.
+See Preflight_ and execute the pre-installation procedures on your Ceph Object
+Gateway node. Specifically, you should disable ``requiretty`` on your Ceph
+Deploy user, set SELinux to ``Permissive`` and set up a Ceph Deploy user with
+password-less ``sudo``. For Ceph Object Gateways, you will need to open the
+port that Civetweb will use in production.
.. note:: Civetweb runs on port ``7480`` by default.
Install Ceph Object Gateway
---------------------------
-From the working directory of your administration server, install the Ceph Object Gateway package on the Ceph Object Gateway node. For example::
+From the working directory of your administration server, install the Ceph
+Object Gateway package on the Ceph Object Gateway node. For example::
ceph-deploy install --rgw <gateway-node1> [<gateway-node2> ...]
-The ``ceph-common`` package is a dependency, so ``ceph-deploy`` will install this too. The ``ceph`` CLI tools are intended for administrators. To make your Ceph Object Gateway node an administrator node, execute the following from the working directory of your administration server::
+The ``ceph-common`` package is a dependency, so ``ceph-deploy`` will install
+this too. The ``ceph`` CLI tools are intended for administrators. To make your
+Ceph Object Gateway node an administrator node, execute the following from the
+working directory of your administration server::
ceph-deploy admin <node-name>
Create a Gateway Instance
-------------------------
-From the working directory of your administration server, create an instance of the Ceph Object Gateway on the Ceph Object Gateway. For example::
+From the working directory of your administration server, create an instance of
+the Ceph Object Gateway on the Ceph Object Gateway. For example::
ceph-deploy rgw create <gateway-node1>
-Once the gateway is running, you should be able to access it on port ``7480`` with an unauthenticated request like this::
+Once the gateway is running, you should be able to access it on port ``7480``
+with an unauthenticated request like this::
http://client-node:7480
-If the gateway instance is working properly, you should receive a response like this::
+If the gateway instance is working properly, you should receive a response like
+this::
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
- <Owner>
+ <Owner>
<ID>anonymous</ID>
<DisplayName></DisplayName>
</Owner>
</Buckets>
</ListAllMyBucketsResult>
-If at any point you run into trouble and you want to start over, execute the following to purge the configuration::
+If at any point you run into trouble and you want to start over, execute the
+following to purge the configuration::
ceph-deploy purge <gateway-node1> [<gateway-node2>]
ceph-deploy purgedata <gateway-node1> [<gateway-node2>]
Change the Default Port
-----------------------
-Civetweb runs on port ``7480`` by default. To change the default port (e.g,. to port ``80``), modify your Ceph configuration file in the working directory of your administration server. Add a section entitled ``[client.rgw.<gateway-node>]``, replacing ``<gateway-node>`` with the short node name of your Ceph Object Gateway node (i.e., ``hostname -s``).
+Civetweb runs on port ``7480`` by default. To change the default port (e.g,. to
+port ``80``), modify your Ceph configuration file in the working directory of
+your administration server. Add a section entitled
+``[client.rgw.<gateway-node>]``, replacing ``<gateway-node>`` with the short
+node name of your Ceph Object Gateway node (i.e., ``hostname -s``).
-.. note:: In version 0.94, the Ceph Object Gateway does not support SSL. You may setup a reverse proxy web server with SSL to dispatch HTTPS requests as HTTP requests to CivetWeb.
+.. note:: In version 0.94, the Ceph Object Gateway does not support SSL. You
+ may setup a reverse proxy web server with SSL to dispatch HTTPS
+ requests as HTTP requests to CivetWeb.
-For example, if your node name is ``gateway-node1``, add a section like this after the ``[global]`` section::
+For example, if your node name is ``gateway-node1``, add a section like this
+after the ``[global]`` section::
[client.rgw.gateway-node1]
rgw_frontends = "civetweb port=80"
-.. note:: Ensure that you leave no whitespace between ``port=<port-number>`` in the ``rgw_frontends`` key/value pair. The ``[client.rgw.gateway-node1]`` heading identifies this portion of the Ceph configuration file as configuring a Ceph Storage Cluster client where the client type is a Ceph Object Gateway (i.e., ``rgw``), and the name of the instance is ``gateway-node1``.
+.. note:: Ensure that you leave no whitespace between ``port=<port-number>`` in
+ the ``rgw_frontends`` key/value pair. The ``[client.rgw.gateway-node1]``
+ heading identifies this portion of the Ceph configuration file as
+ configuring a Ceph Storage Cluster client where the client type is a Ceph
+ Object Gateway (i.e., ``rgw``), and the name of the instance is )
+ ``gateway-node1``.
Push the updated configuration file to your Ceph Object Gateway node
(and other Ceph nodes)::
sudo systemctl restart ceph-radosgw.service
-Finally, check to ensure that the port you selected is open on the node's firewall (e.g., port ``80``). If it is not open, add the port and reload the firewall configuration. For example::
+Finally, check to ensure that the port you selected is open on the node's
+firewall (e.g., port ``80``). If it is not open, add the port and reload the
+firewall configuration. For Red Hat Enterprise Linux execute::
sudo firewall-cmd --list-all
sudo firewall-cmd --zone=public --add-port 80/tcp --permanent
sudo firewall-cmd --reload
-Migrating from Apache to Civetweb
----------------------------------
+For Ubuntu, execute::
+
+ sudo iptables --list
+ sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT
+
+Replace ``<iface>`` and ``<ip-address>/<netmask>`` with the relevant values for
+your Ceph Object Gateway node.
+
+Once you have finished configuring ``iptables``, ensure that you make the
+change persistent so that it will be in effect when your Ceph Object Gateway
+node reboots. Execute::
+
+ sudo apt-get install iptables-persistent
+
+A terminal UI will open up. Select ``yes`` for the prompts to save current
+``IPv4`` iptables rules to ``/etc/iptables/rules.v4`` and current ``IPv6``
+iptables rules to ``/etc/iptables/rules.v6``.
-If you're running the Ceph Object Gateway on Apache and FastCGI with Ceph Storage v0.80 or above, you're already running Civetweb--it starts with the ``ceph-radosgw`` daemon and it's running on port 7480 by default so that it doesn't conflict with your Apache and FastCGI installation and other commonly used web service ports. Migrating to use Civetweb basically involves removing your Apache installation. Then, you must remove Apache and FastCGI settings from your Ceph configuration file and reset ``rgw_frontends`` to Civetweb.
+The ``IPv4`` iptables rule that you set in the earlier step will be loaded in
+``/etc/iptables/rules.v4`` and will be persistent across reboots.
-Referring back to the description for installing a Ceph Object Gateway with ``ceph-deploy``, notice that the configuration file only has one setting ``rgw_frontends`` (and that's assuming you elected to change the default port). The ``ceph-deploy`` utility generates the data directory and the keyring for you--placing the keyring in ``/var/lib/ceph/radosgw/{rgw-intance}``. The daemon looks in default locations, whereas you may have specified different settings in your Ceph configuration file. Since you already have keys and a data directory, you will want to maintain those paths in your Ceph configuration file if you used something other than default paths.
+If you add a new ``IPv4`` iptables rule after installing
+``iptables-persistent`` you will have to add it to the rule file. In such case,
+execute the following as the ``root`` user::
-A typical Ceph Object Gateway configuration file for an Apache-based deployment looks something like this::
+ iptables-save > /etc/iptables/rules.v4
+
+Migrating from Apache to Civetweb
+---------------------------------
+
+If you're running the Ceph Object Gateway on Apache and FastCGI with Ceph
+Storage v0.80 or above, you're already running Civetweb--it starts with the
+``ceph-radosgw`` daemon and it's running on port 7480 by default so that it
+doesn't conflict with your Apache and FastCGI installation and other commonly
+used web service ports. Migrating to use Civetweb basically involves removing
+your Apache installation. Then, you must remove Apache and FastCGI settings
+from your Ceph configuration file and reset ``rgw_frontends`` to Civetweb.
+
+Referring back to the description for installing a Ceph Object Gateway with
+``ceph-deploy``, notice that the configuration file only has one setting
+``rgw_frontends`` (and that's assuming you elected to change the default port).
+The ``ceph-deploy`` utility generates the data directory and the keyring for
+you--placing the keyring in ``/var/lib/ceph/radosgw/{rgw-intance}``. The daemon
+looks in default locations, whereas you may have specified different settings
+in your Ceph configuration file. Since you already have keys and a data
+directory, you will want to maintain those paths in your Ceph configuration
+file if you used something other than default paths.
+
+A typical Ceph Object Gateway configuration file for an Apache-based deployment
+looks something similar as the following::
+
+On Red Hat Enterprise Linux::
[client.radosgw.gateway-node1]
host = {hostname}
rgw frontends = fastcgi socket\_port=9000 socket\_host=0.0.0.0
rgw print continue = false
-To modify it for use with Civetweb, simply remove the Apache-specific settings such as ``rgw_socket_path`` and ``rgw_print_continue``. Then, change the ``rgw_frontends`` setting to reflect Civetweb rather than the Apache FastCGI front end and specify the port number you intend to use. For example::
+On Ubuntu::
+
+ [client.radosgw.gateway-node]
+ host = {hostname}
+ keyring = /etc/ceph/ceph.client.radosgw.keyring
+ rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
+ log file = /var/log/radosgw/client.radosgw.gateway-node1.log
+
+To modify it for use with Civetweb, simply remove the Apache-specific settings
+such as ``rgw_socket_path`` and ``rgw_print_continue``. Then, change the
+``rgw_frontends`` setting to reflect Civetweb rather than the Apache FastCGI
+front end and specify the port number you intend to use. For example::
[client.radosgw.gateway-node1]
host = {hostname}
log file = /var/log/radosgw/client.radosgw.gateway-node1.log
rgw_frontends = civetweb port=80
-Finally, restart the Ceph Object Gateway::
+Finally, restart the Ceph Object Gateway. On Red Hat Enterprise Linux execute::
sudo systemctl restart ceph-radosgw.service
-If you used a port number that is not open, you will also need to open that port on your firewall.
+On Ubuntu execute::
+
+ sudo service radosgw restart id=rgw.<short-hostname>
+
+If you used a port number that is not open, you will also need to open that
+port on your firewall.
Configure Bucket Sharding
-------------------------
-A Ceph Object Gateway stores bucket index data in the ``index_pool``, which defaults to ``.rgw.buckets.index``. Sometimes users like to put many objects (hundreds of thousands to millions of objects) in a single bucket. If you do not use the gateway administration interface to set quotas for the maximum number of objects per bucket, the bucket index can suffer significant performance degradation when users place large numbers of objects into a bucket.
+A Ceph Object Gateway stores bucket index data in the ``index_pool``, which
+defaults to ``.rgw.buckets.index``. Sometimes users like to put many objects
+(hundreds of thousands to millions of objects) in a single bucket. If you do
+not use the gateway administration interface to set quotas for the maximum
+number of objects per bucket, the bucket index can suffer significant
+performance degradation when users place large numbers of objects into a
+bucket.
-In Ceph 0.94, you may shard bucket indices to help prevent performance bottlenecks when you allow a high number of objects per bucket. The ``rgw_override_bucket_index_max_shards`` setting allows you to set a maximum number of shards per bucket. The default value is ``0``, which means bucket index sharding is off by default.
+In Ceph 0.94, you may shard bucket indices to help prevent performance
+bottlenecks when you allow a high number of objects per bucket. The
+``rgw_override_bucket_index_max_shards`` setting allows you to set a maximum
+number of shards per bucket. The default value is ``0``, which means bucket
+index sharding is off by default.
-To turn bucket index sharding on, set ``rgw_override_bucket_index_max_shards`` to a value greater than ``0``.
+To turn bucket index sharding on, set ``rgw_override_bucket_index_max_shards``
+to a value greater than ``0``.
-For simple configurations, you may add ``rgw_override_bucket_index_max_shards`` to your Ceph configuration file. Add it under ``[global]`` to create a system-wide value. You can also set it for each instance in your Ceph configuration file.
+For simple configurations, you may add ``rgw_override_bucket_index_max_shards``
+to your Ceph configuration file. Add it under ``[global]`` to create a
+system-wide value. You can also set it for each instance in your Ceph
+configuration file.
-Once you have changed your bucket sharding configuration in your Ceph configuration file, restart your gateway::
+Once you have changed your bucket sharding configuration in your Ceph
+configuration file, restart your gateway. On Red Hat Enteprise Linux execute::
sudo systemctl restart ceph-radosgw.service
-For federated configurations, each zone may have a different ``index_pool`` setting for failover. To make the value consistent for a region's zones, you may set ``rgw_override_bucket_index_max_shards`` in a gateway's region configuration. For example::
+On Ubuntu execute::
+
+ sudo service radosgw restart id=rgw.<short-hostname>sudo service radosgw restart id=rgw.<short-hostname>
+
+For federated configurations, each zone may have a different ``index_pool``
+setting for failover. To make the value consistent for a region's zones, you
+may set ``rgw_override_bucket_index_max_shards`` in a gateway's region
+configuration. For example::
radosgw-admin region get > region.json
-Open the ``region.json`` file and edit the ``bucket_index_max_shards`` setting for each named zone. Save the ``region.json`` file and reset the region. For example::
+Open the ``region.json`` file and edit the ``bucket_index_max_shards`` setting
+for each named zone. Save the ``region.json`` file and reset the region. For
+example::
radosgw-admin region set < region.json
Where ``client.rgw.ceph-client`` is the name of the gateway user.
-.. note:: Mapping the index pool (for each zone, if applicable) to a CRUSH ruleset of SSD-based OSDs may also help with bucket index performance.
+.. note:: Mapping the index pool (for each zone, if applicable) to a CRUSH
+ ruleset of SSD-based OSDs may also help with bucket index performance.
Add Wildcard to DNS
-------------------
-To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you need to add a wildcard to the DNS record of the DNS server you use with the ``ceph-radosgw`` daemon.
+To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you
+need to add a wildcard to the DNS record of the DNS server you use with the
+``ceph-radosgw`` daemon.
-The address of the DNS must also be specified in the Ceph configuration file with the ``rgw dns name = {hostname}`` setting.
+The address of the DNS must also be specified in the Ceph configuration file
+with the ``rgw dns name = {hostname}`` setting.
-For ``dnsmasq``, add the following address setting with a dot (.) prepended to the host name::
+For ``dnsmasq``, add the following address setting with a dot (.) prepended to
+the host name::
address=/.{hostname-or-fqdn}/{host-ip-address}
For example::
- address=/.gateway-node1/192.168.122.75
+ address=/.gateway-node1/192.168.122.75
For ``bind``, add a wildcard to the DNS record. For example::
@ IN A 192.168.122.113
* IN CNAME @
-Restart your DNS server and ping your server with a subdomain to ensure that your ``ceph-radosgw`` daemon can process the subdomain requests::
+Restart your DNS server and ping your server with a subdomain to ensure that
+your ``ceph-radosgw`` daemon can process the subdomain requests::
ping mybucket.{hostname}
Add Debugging (if needed)
-------------------------
-Once you finish the setup procedure, if you encounter issues with your configuration, you can add debugging to the ``[global]`` section of your Ceph configuration file and restart the gateway(s) to help troubleshoot any configuration issues. For example::
+Once you finish the setup procedure, if you encounter issues with your
+configuration, you can add debugging to the ``[global]`` section of your Ceph
+configuration file and restart the gateway(s) to help troubleshoot any
+configuration issues. For example::
[global]
#append the following in the global section.
debug ms = 1
debug rgw = 20
-Using The Gateway
+Using the Gateway
-----------------
-To use the REST interfaces, first create an initial Ceph Object Gateway user for the S3 interface. Then, create a subuser for the Swift interface. You then need to verify if the created users are able to access the gateway.
+To use the REST interfaces, first create an initial Ceph Object Gateway user
+for the S3 interface. Then, create a subuser for the Swift interface. You then
+need to verify if the created users are able to access the gateway.
-Create a radosgw user for S3 access
+Create a RADOSGW User for S3 Access
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-A ``radosgw`` user needs to be created and granted access. The command ``man radosgw-admin`` will provide information on additional command options.
+A ``radosgw`` user needs to be created and granted access. The command ``man
+radosgw-admin`` will provide information on additional command options.
To create the user, execute the following on the ``gateway host``::
"temp_url_keys": []
}
-.. note:: The values of ``keys->access_key`` and ``keys->secret_key`` are needed for access validation.
-
-.. important:: Check the key output. Sometimes ``radosgw-admin`` generates a JSON escape character ``\`` in ``access_key`` or ``secret_key`` and some clients do not know how to handle JSON escape characters. Remedies include removing the JSON escape character ``\``, encapsulating the string in quotes, regenerating the key and ensuring that it does not have a JSON escape character or specify the key and secret manually. Also, if ``radosgw-admin`` generates a JSON escape character ``\`` and a forward slash ``/`` together in a key, like ``\/``, only remove the JSON escape character ``\``. Do not remove the forward slash ``/`` as it is a valid character in the key.
-
-Create a Swift user
+.. note:: The values of ``keys->access_key`` and ``keys->secret_key`` are
+ needed for access validation.
+
+.. important:: Check the key output. Sometimes ``radosgw-admin`` generates a
+ JSON escape character ``\`` in ``access_key`` or ``secret_key``
+ and some clients do not know how to handle JSON escape
+ characters. Remedies include removing the JSON escape character
+ ``\``, encapsulating the string in quotes, regenerating the key
+ and ensuring that it does not have a JSON escape character or
+ specify the key and secret manually. Also, if ``radosgw-admin``
+ generates a JSON escape character ``\`` and a forward slash ``/``
+ together in a key, like ``\/``, only remove the JSON escape
+ character ``\``. Do not remove the forward slash ``/`` as it is
+ a valid character in the key.
+
+Create a Swift User
^^^^^^^^^^^^^^^^^^^
-A Swift subuser needs to be created if this kind of access is needed. Creating a Swift user is a two step process. The first step is to create the user. The second is to create the secret key.
+A Swift subuser needs to be created if this kind of access is needed. Creating
+a Swift user is a two step process. The first step is to create the user. The
+second is to create the secret key.
Execute the following steps on the ``gateway host``:
Access Verification
^^^^^^^^^^^^^^^^^^^
-Test S3 access
+Test S3 Access
""""""""""""""
-You need to write and run a Python test script for verifying S3 access. The S3 access test script will connect to the ``radosgw``, create a new bucket and list all buckets. The values for ``aws_access_key_id`` and ``aws_secret_access_key`` are taken from the values of ``access_key`` and ``secret_key`` returned by the ``radosgw_admin`` command.
+You need to write and run a Python test script for verifying S3 access. The S3
+access test script will connect to the ``radosgw``, create a new bucket and
+list all buckets. The values for ``aws_access_key_id`` and
+``aws_secret_access_key`` are taken from the values of ``access_key`` and
+``secret_key`` returned by the ``radosgw_admin`` command.
Execute the following steps:
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = '{hostname}', port = {port},
- is_secure=False, calling_format = boto.s3.connection.OrdinaryCallingFormat(), )
+ is_secure=False, calling_format = boto.s3.connection.OrdinaryCallingFormat(),
+ )
bucket = conn.create_bucket('my-new-bucket')
- for bucket in conn.get_all_buckets():
- print "{name}".format(
- name = bucket.name,
- created = bucket.creation_date,
- )
+ for bucket in conn.get_all_buckets():
+ print "{name}".format(
+ name = bucket.name,
+ created = bucket.creation_date,
+ )
- Replace ``{hostname}`` with the hostname of the host where you have configured the gateway service i.e, the ``gateway host``. Replace {port} with the port number you are using with Civetweb.
+ Replace ``{hostname}`` with the hostname of the host where you have
+ configured the gateway service i.e, the ``gateway host``. Replace {port}
+ with the port number you are using with Civetweb.
#. Run the script::
Test swift access
"""""""""""""""""
-Swift access can be verified via the ``swift`` command line client. The command ``man swift`` will provide more information on available command line options.
+Swift access can be verified via the ``swift`` command line client. The command
+``man swift`` will provide more information on available command line options.
-To install ``swift`` client, execute the following::
+To install ``swift`` client, execute the following commands. On Red Hat
+Enterprise Linux::
sudo yum install python-setuptools
- sudo easy_install pip
- sudo pip install --upgrade setuptools
- sudo pip install --upgrade python-swiftclient
+ sudo easy_install pip
+ sudo pip install --upgrade setuptools
+ sudo pip install --upgrade python-swiftclient
+
+On Ubuntu::
+
+ sudo apt-get install python-setuptools
+ sudo easy_install pip
+ sudo pip install --upgrade setuptools
+ sudo pip install --upgrade python-swiftclient
To test swift access, execute the following::
swift -A http://{IP ADDRESS}:{port}/auth/1.0 -U testuser:swift -K '{swift_secret_key}' list
-Replace ``{IP ADDRESS}`` with the public IP address of the gateway server and ``{swift_secret_key}`` with its value from the output of ``radosgw-admin key create`` command executed for the ``swift`` user. Replace {port} with the port number you are using with Civetweb (e.g., ``7480`` is the default). If you don't replace the port, it will default to port ``80``.
+Replace ``{IP ADDRESS}`` with the public IP address of the gateway server and
+``{swift_secret_key}`` with its value from the output of ``radosgw-admin key
+create`` command executed for the ``swift`` user. Replace {port} with the port
+number you are using with Civetweb (e.g., ``7480`` is the default). If you
+don't replace the port, it will default to port ``80``.
For example::
The output should be::
my-new-bucket
+
+.. _Preflight: ../../start/quick-start-preflight
projects / ceph.git / commitdiff