From: Sebastian Wagner Date: Fri, 13 Mar 2020 14:15:29 +0000 (+0100) Subject: doc: Move ceph-deploy docs to doc/install/ceph-deploy X-Git-Tag: v15.1.1~1^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=refs%2Fpull%2F33953%2Fhead;p=ceph.git doc: Move ceph-deploy docs to doc/install/ceph-deploy Signed-off-by: Sebastian Wagner --- diff --git a/doc/index.rst b/doc/index.rst index 0306df6eaf1d..af3b49c6c33c 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -93,7 +93,6 @@ about Ceph, see our `Architecture`_ section. start/intro cephadm/index - start/index install/index rados/index cephfs/index diff --git a/doc/install/ceph-deploy/index.rst b/doc/install/ceph-deploy/index.rst new file mode 100644 index 000000000000..9579d3b3be8e --- /dev/null +++ b/doc/install/ceph-deploy/index.rst @@ -0,0 +1,56 @@ +.. _ceph-deploy-index: + +============================ + Installation (ceph-deploy) +============================ + +.. raw:: html + + +

Step 1: Preflight

+ +A :term:`Ceph Client` and a :term:`Ceph Node` may require some basic +configuration work prior to deploying a Ceph Storage Cluster. You can also +avail yourself of help by getting involved in the Ceph community. + +.. toctree:: + + Preflight + +.. raw:: html + +

Step 2: Storage Cluster

+ +Once you have completed your preflight checklist, you should be able to begin +deploying a Ceph Storage Cluster. + +.. toctree:: + + Storage Cluster Quick Start + + +.. raw:: html + +

Step 3: Ceph Client(s)

+ +Most Ceph users don't store objects directly in the Ceph Storage Cluster. They typically use at least one of +Ceph Block Devices, the Ceph File System, and Ceph Object Storage. + +.. toctree:: + + Block Device Quick Start <../../start/quick-rbd> + Filesystem Quick Start + Object Storage Quick Start + +.. raw:: html + +
+ + +.. toctree:: + :hidden: + + Upgrading Ceph + Install Ceph Object Gateway + + diff --git a/doc/install/ceph-deploy/install-ceph-gateway.rst b/doc/install/ceph-deploy/install-ceph-gateway.rst new file mode 100644 index 000000000000..fe5b6b574cd4 --- /dev/null +++ b/doc/install/ceph-deploy/install-ceph-gateway.rst @@ -0,0 +1,615 @@ +=========================== +Install Ceph Object Gateway +=========================== + +As of `firefly` (v0.80), Ceph Object Gateway is running on Civetweb (embedded +into the ``ceph-radosgw`` daemon) instead of Apache and FastCGI. Using Civetweb +simplifies the Ceph Object Gateway installation and configuration. + +.. note:: To run the Ceph Object Gateway service, you should have a running + Ceph storage cluster, and the gateway host should have access to the + public network. + +.. note:: In version 0.80, the Ceph Object Gateway does not support SSL. You + may setup a reverse proxy server with SSL to dispatch HTTPS requests + as HTTP requests to CivetWeb. + +Execute the Pre-Installation Procedure +-------------------------------------- + +See Preflight_ and execute the pre-installation procedures on your Ceph Object +Gateway node. Specifically, you should disable ``requiretty`` on your Ceph +Deploy user, set SELinux to ``Permissive`` and set up a Ceph Deploy user with +password-less ``sudo``. For Ceph Object Gateways, you will need to open the +port that Civetweb will use in production. + +.. note:: Civetweb runs on port ``7480`` by default. + +Install Ceph Object Gateway +--------------------------- + +From the working directory of your administration server, install the Ceph +Object Gateway package on the Ceph Object Gateway node. For example:: + + ceph-deploy install --rgw [ ...] + +The ``ceph-common`` package is a dependency, so ``ceph-deploy`` will install +this too. The ``ceph`` CLI tools are intended for administrators. To make your +Ceph Object Gateway node an administrator node, execute the following from the +working directory of your administration server:: + + ceph-deploy admin + +Create a Gateway Instance +------------------------- + +From the working directory of your administration server, create an instance of +the Ceph Object Gateway on the Ceph Object Gateway. For example:: + + ceph-deploy rgw create + +Once the gateway is running, you should be able to access it on port ``7480`` +with an unauthenticated request like this:: + + http://client-node:7480 + +If the gateway instance is working properly, you should receive a response like +this:: + + + + + anonymous + + + + + + +If at any point you run into trouble and you want to start over, execute the +following to purge the configuration:: + + ceph-deploy purge [] + ceph-deploy purgedata [] + +If you execute ``purge``, you must re-install Ceph. + +Change the Default Port +----------------------- + +Civetweb runs on port ``7480`` by default. To change the default port (e.g., to +port ``80``), modify your Ceph configuration file in the working directory of +your administration server. Add a section entitled +``[client.rgw.]``, replacing ```` with the short +node name of your Ceph Object Gateway node (i.e., ``hostname -s``). + +.. note:: As of version 11.0.1, the Ceph Object Gateway **does** support SSL. + See `Using SSL with Civetweb`_ for information on how to set that up. + +For example, if your node name is ``gateway-node1``, add a section like this +after the ``[global]`` section:: + + [client.rgw.gateway-node1] + rgw_frontends = "civetweb port=80" + +.. note:: Ensure that you leave no whitespace between ``port=`` in + the ``rgw_frontends`` key/value pair. The ``[client.rgw.gateway-node1]`` + heading identifies this portion of the Ceph configuration file as + configuring a Ceph Storage Cluster client where the client type is a Ceph + Object Gateway (i.e., ``rgw``), and the name of the instance is + ``gateway-node1``. + +Push the updated configuration file to your Ceph Object Gateway node +(and other Ceph nodes):: + + ceph-deploy --overwrite-conf config push [] + +To make the new port setting take effect, restart the Ceph Object +Gateway:: + + sudo systemctl restart ceph-radosgw.service + +Finally, check to ensure that the port you selected is open on the node's +firewall (e.g., port ``80``). If it is not open, add the port and reload the +firewall configuration. If you use the ``firewalld`` daemon, execute:: + + sudo firewall-cmd --list-all + sudo firewall-cmd --zone=public --add-port 80/tcp --permanent + sudo firewall-cmd --reload + +If you use ``iptables``, execute:: + + sudo iptables --list + sudo iptables -I INPUT 1 -i -p tcp -s / --dport 80 -j ACCEPT + +Replace ```` and ``/`` with the relevant values for +your Ceph Object Gateway node. + +Once you have finished configuring ``iptables``, ensure that you make the +change persistent so that it will be in effect when your Ceph Object Gateway +node reboots. Execute:: + + sudo apt-get install iptables-persistent + +A terminal UI will open up. Select ``yes`` for the prompts to save current +``IPv4`` iptables rules to ``/etc/iptables/rules.v4`` and current ``IPv6`` +iptables rules to ``/etc/iptables/rules.v6``. + +The ``IPv4`` iptables rule that you set in the earlier step will be loaded in +``/etc/iptables/rules.v4`` and will be persistent across reboots. + +If you add a new ``IPv4`` iptables rule after installing +``iptables-persistent`` you will have to add it to the rule file. In such case, +execute the following as the ``root`` user:: + + iptables-save > /etc/iptables/rules.v4 + +Using SSL with Civetweb +----------------------- +.. _Using SSL with Civetweb: + +Before using SSL with civetweb, you will need a certificate that will match +the host name that that will be used to access the Ceph Object Gateway. +You may wish to obtain one that has `subject alternate name` fields for +more flexibility. If you intend to use S3-style subdomains +(`Add Wildcard to DNS`_), you will need a `wildcard` certificate. + +Civetweb requires that the server key, server certificate, and any other +CA or intermediate certificates be supplied in one file. Each of these +items must be in `pem` form. Because the combined file contains the +secret key, it should be protected from unauthorized access. + +To configure ssl operation, append ``s`` to the port number. For eg:: + + [client.rgw.gateway-node1] + rgw_frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/keyandcert.pem + +.. versionadded :: Luminous + +Furthermore, civetweb can be made to bind to multiple ports, by separating them +with ``+`` in the configuration. This allows for use cases where both ssl and +non-ssl connections are hosted by a single rgw instance. For eg:: + + [client.rgw.gateway-node1] + rgw_frontends = civetweb port=80+443s ssl_certificate=/etc/ceph/private/keyandcert.pem + +Additional Civetweb Configuration Options +----------------------------------------- +Some additional configuration options can be adjusted for the embedded Civetweb web server +in the **Ceph Object Gateway** section of the ``ceph.conf`` file. +A list of supported options, including an example, can be found in the `HTTP Frontends`_. + +Migrating from Apache to Civetweb +--------------------------------- + +If you are running the Ceph Object Gateway on Apache and FastCGI with Ceph +Storage v0.80 or above, you are already running Civetweb--it starts with the +``ceph-radosgw`` daemon and it's running on port 7480 by default so that it +doesn't conflict with your Apache and FastCGI installation and other commonly +used web service ports. Migrating to use Civetweb basically involves removing +your Apache installation. Then, you must remove Apache and FastCGI settings +from your Ceph configuration file and reset ``rgw_frontends`` to Civetweb. + +Referring back to the description for installing a Ceph Object Gateway with +``ceph-deploy``, notice that the configuration file only has one setting +``rgw_frontends`` (and that's assuming you elected to change the default port). +The ``ceph-deploy`` utility generates the data directory and the keyring for +you--placing the keyring in ``/var/lib/ceph/radosgw/{rgw-instance}``. The daemon +looks in default locations, whereas you may have specified different settings +in your Ceph configuration file. Since you already have keys and a data +directory, you will want to maintain those paths in your Ceph configuration +file if you used something other than default paths. + +A typical Ceph Object Gateway configuration file for an Apache-based deployment +looks something similar as the following: + +On Red Hat Enterprise Linux:: + + [client.radosgw.gateway-node1] + host = {hostname} + keyring = /etc/ceph/ceph.client.radosgw.keyring + rgw socket path = "" + log file = /var/log/radosgw/client.radosgw.gateway-node1.log + rgw frontends = fastcgi socket\_port=9000 socket\_host=0.0.0.0 + rgw print continue = false + +On Ubuntu:: + + [client.radosgw.gateway-node] + host = {hostname} + keyring = /etc/ceph/ceph.client.radosgw.keyring + rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock + log file = /var/log/radosgw/client.radosgw.gateway-node1.log + +To modify it for use with Civetweb, simply remove the Apache-specific settings +such as ``rgw_socket_path`` and ``rgw_print_continue``. Then, change the +``rgw_frontends`` setting to reflect Civetweb rather than the Apache FastCGI +front end and specify the port number you intend to use. For example:: + + [client.radosgw.gateway-node1] + host = {hostname} + keyring = /etc/ceph/ceph.client.radosgw.keyring + log file = /var/log/radosgw/client.radosgw.gateway-node1.log + rgw_frontends = civetweb port=80 + +Finally, restart the Ceph Object Gateway. On Red Hat Enterprise Linux execute:: + + sudo systemctl restart ceph-radosgw.service + +On Ubuntu execute:: + + sudo service radosgw restart id=rgw. + +If you used a port number that is not open, you will also need to open that +port on your firewall. + +Configure Bucket Sharding +------------------------- + +A Ceph Object Gateway stores bucket index data in the ``index_pool``, which +defaults to ``.rgw.buckets.index``. Sometimes users like to put many objects +(hundreds of thousands to millions of objects) in a single bucket. If you do +not use the gateway administration interface to set quotas for the maximum +number of objects per bucket, the bucket index can suffer significant +performance degradation when users place large numbers of objects into a +bucket. + +In Ceph 0.94, you may shard bucket indices to help prevent performance +bottlenecks when you allow a high number of objects per bucket. The +``rgw_override_bucket_index_max_shards`` setting allows you to set a maximum +number of shards per bucket. The default value is ``0``, which means bucket +index sharding is off by default. + +To turn bucket index sharding on, set ``rgw_override_bucket_index_max_shards`` +to a value greater than ``0``. + +For simple configurations, you may add ``rgw_override_bucket_index_max_shards`` +to your Ceph configuration file. Add it under ``[global]`` to create a +system-wide value. You can also set it for each instance in your Ceph +configuration file. + +Once you have changed your bucket sharding configuration in your Ceph +configuration file, restart your gateway. On Red Hat Enterprise Linux execute:: + + sudo systemctl restart ceph-radosgw.service + +On Ubuntu execute:: + + sudo service radosgw restart id=rgw. + +For federated configurations, each zone may have a different ``index_pool`` +setting for failover. To make the value consistent for a zonegroup's zones, you +may set ``rgw_override_bucket_index_max_shards`` in a gateway's zonegroup +configuration. For example:: + + radosgw-admin zonegroup get > zonegroup.json + +Open the ``zonegroup.json`` file and edit the ``bucket_index_max_shards`` setting +for each named zone. Save the ``zonegroup.json`` file and reset the zonegroup. +For example:: + + radosgw-admin zonegroup set < zonegroup.json + +Once you have updated your zonegroup, update and commit the period. +For example:: + + radosgw-admin period update --commit + +.. note:: Mapping the index pool (for each zone, if applicable) to a CRUSH + rule of SSD-based OSDs may also help with bucket index performance. + +Add Wildcard to DNS +------------------- +.. _Add Wildcard to DNS: + +To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you +need to add a wildcard to the DNS record of the DNS server you use with the +``ceph-radosgw`` daemon. + +The address of the DNS must also be specified in the Ceph configuration file +with the ``rgw dns name = {hostname}`` setting. + +For ``dnsmasq``, add the following address setting with a dot (.) prepended to +the host name:: + + address=/.{hostname-or-fqdn}/{host-ip-address} + +For example:: + + address=/.gateway-node1/192.168.122.75 + + +For ``bind``, add a wildcard to the DNS record. For example:: + + $TTL 604800 + @ IN SOA gateway-node1. root.gateway-node1. ( + 2 ; Serial + 604800 ; Refresh + 86400 ; Retry + 2419200 ; Expire + 604800 ) ; Negative Cache TTL + ; + @ IN NS gateway-node1. + @ IN A 192.168.122.113 + * IN CNAME @ + +Restart your DNS server and ping your server with a subdomain to ensure that +your DNS configuration works as expected:: + + ping mybucket.{hostname} + +For example:: + + ping mybucket.gateway-node1 + +Add Debugging (if needed) +------------------------- + +Once you finish the setup procedure, if you encounter issues with your +configuration, you can add debugging to the ``[global]`` section of your Ceph +configuration file and restart the gateway(s) to help troubleshoot any +configuration issues. For example:: + + [global] + #append the following in the global section. + debug ms = 1 + debug rgw = 20 + +Using the Gateway +----------------- + +To use the REST interfaces, first create an initial Ceph Object Gateway user +for the S3 interface. Then, create a subuser for the Swift interface. You then +need to verify if the created users are able to access the gateway. + +Create a RADOSGW User for S3 Access +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A ``radosgw`` user needs to be created and granted access. The command ``man +radosgw-admin`` will provide information on additional command options. + +To create the user, execute the following on the ``gateway host``:: + + sudo radosgw-admin user create --uid="testuser" --display-name="First User" + +The output of the command will be something like the following:: + + { + "user_id": "testuser", + "display_name": "First User", + "email": "", + "suspended": 0, + "max_buckets": 1000, + "subusers": [], + "keys": [{ + "user": "testuser", + "access_key": "I0PJDPCIYZ665MW88W9R", + "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" + }], + "swift_keys": [], + "caps": [], + "op_mask": "read, write, delete", + "default_placement": "", + "placement_tags": [], + "bucket_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + }, + "user_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + }, + "temp_url_keys": [] + } + +.. note:: The values of ``keys->access_key`` and ``keys->secret_key`` are + needed for access validation. + +.. important:: Check the key output. Sometimes ``radosgw-admin`` generates a + JSON escape character ``\`` in ``access_key`` or ``secret_key`` + and some clients do not know how to handle JSON escape + characters. Remedies include removing the JSON escape character + ``\``, encapsulating the string in quotes, regenerating the key + and ensuring that it does not have a JSON escape character or + specify the key and secret manually. Also, if ``radosgw-admin`` + generates a JSON escape character ``\`` and a forward slash ``/`` + together in a key, like ``\/``, only remove the JSON escape + character ``\``. Do not remove the forward slash ``/`` as it is + a valid character in the key. + +Create a Swift User +^^^^^^^^^^^^^^^^^^^ + +A Swift subuser needs to be created if this kind of access is needed. Creating +a Swift user is a two step process. The first step is to create the user. The +second is to create the secret key. + +Execute the following steps on the ``gateway host``: + +Create the Swift user:: + + sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full + +The output will be something like the following:: + + { + "user_id": "testuser", + "display_name": "First User", + "email": "", + "suspended": 0, + "max_buckets": 1000, + "subusers": [{ + "id": "testuser:swift", + "permissions": "full-control" + }], + "keys": [{ + "user": "testuser:swift", + "access_key": "3Y1LNW4Q6X0Y53A52DET", + "secret_key": "" + }, { + "user": "testuser", + "access_key": "I0PJDPCIYZ665MW88W9R", + "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" + }], + "swift_keys": [], + "caps": [], + "op_mask": "read, write, delete", + "default_placement": "", + "placement_tags": [], + "bucket_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + }, + "user_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + }, + "temp_url_keys": [] + } + +Create the secret key:: + + sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret + +The output will be something like the following:: + + { + "user_id": "testuser", + "display_name": "First User", + "email": "", + "suspended": 0, + "max_buckets": 1000, + "subusers": [{ + "id": "testuser:swift", + "permissions": "full-control" + }], + "keys": [{ + "user": "testuser:swift", + "access_key": "3Y1LNW4Q6X0Y53A52DET", + "secret_key": "" + }, { + "user": "testuser", + "access_key": "I0PJDPCIYZ665MW88W9R", + "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" + }], + "swift_keys": [{ + "user": "testuser:swift", + "secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA" + }], + "caps": [], + "op_mask": "read, write, delete", + "default_placement": "", + "placement_tags": [], + "bucket_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + }, + "user_quota": { + "enabled": false, + "max_size_kb": -1, + "max_objects": -1 + }, + "temp_url_keys": [] + } + +Access Verification +^^^^^^^^^^^^^^^^^^^ + +Test S3 Access +"""""""""""""" + +You need to write and run a Python test script for verifying S3 access. The S3 +access test script will connect to the ``radosgw``, create a new bucket and +list all buckets. The values for ``aws_access_key_id`` and +``aws_secret_access_key`` are taken from the values of ``access_key`` and +``secret_key`` returned by the ``radosgw-admin`` command. + +Execute the following steps: + +#. You will need to install the ``python-boto`` package:: + + sudo yum install python-boto + +#. Create the Python script:: + + vi s3test.py + +#. Add the following contents to the file:: + + import boto.s3.connection + + access_key = 'I0PJDPCIYZ665MW88W9R' + secret_key = 'dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA' + conn = boto.connect_s3( + aws_access_key_id=access_key, + aws_secret_access_key=secret_key, + host='{hostname}', port={port}, + is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(), + ) + + bucket = conn.create_bucket('my-new-bucket') + for bucket in conn.get_all_buckets(): + print "{name} {created}".format( + name=bucket.name, + created=bucket.creation_date, + ) + + + Replace ``{hostname}`` with the hostname of the host where you have + configured the gateway service i.e., the ``gateway host``. Replace ``{port}`` + with the port number you are using with Civetweb. + +#. Run the script:: + + python s3test.py + + The output will be something like the following:: + + my-new-bucket 2015-02-16T17:09:10.000Z + +Test swift access +""""""""""""""""" + +Swift access can be verified via the ``swift`` command line client. The command +``man swift`` will provide more information on available command line options. + +To install ``swift`` client, execute the following commands. On Red Hat +Enterprise Linux:: + + sudo yum install python-setuptools + sudo easy_install pip + sudo pip install --upgrade setuptools + sudo pip install --upgrade python-swiftclient + +On Debian-based distributions:: + + sudo apt-get install python-setuptools + sudo easy_install pip + sudo pip install --upgrade setuptools + sudo pip install --upgrade python-swiftclient + +To test swift access, execute the following:: + + swift -V 1 -A http://{IP ADDRESS}:{port}/auth -U testuser:swift -K '{swift_secret_key}' list + +Replace ``{IP ADDRESS}`` with the public IP address of the gateway server and +``{swift_secret_key}`` with its value from the output of ``radosgw-admin key +create`` command executed for the ``swift`` user. Replace {port} with the port +number you are using with Civetweb (e.g., ``7480`` is the default). If you +don't replace the port, it will default to port ``80``. + +For example:: + + swift -V 1 -A http://10.19.143.116:7480/auth -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list + +The output should be:: + + my-new-bucket + +.. _Preflight: ../../start/quick-start-preflight +.. _HTTP Frontends: ../../radosgw/frontends diff --git a/doc/install/ceph-deploy/quick-ceph-deploy.rst b/doc/install/ceph-deploy/quick-ceph-deploy.rst new file mode 100644 index 000000000000..c4589c7b3d3c --- /dev/null +++ b/doc/install/ceph-deploy/quick-ceph-deploy.rst @@ -0,0 +1,346 @@ +============================= + Storage Cluster Quick Start +============================= + +If you haven't completed your `Preflight Checklist`_, do that first. This +**Quick Start** sets up a :term:`Ceph Storage Cluster` using ``ceph-deploy`` +on your admin node. Create a three Ceph Node cluster so you can +explore Ceph functionality. + +.. include:: quick-common.rst + +As a first exercise, create a Ceph Storage Cluster with one Ceph Monitor and three +Ceph OSD Daemons. Once the cluster reaches a ``active + clean`` state, expand it +by adding a fourth Ceph OSD Daemon, and two more Ceph Monitors. +For best results, create a directory on your admin node for maintaining the +configuration files and keys that ``ceph-deploy`` generates for your cluster. :: + + mkdir my-cluster + cd my-cluster + +The ``ceph-deploy`` utility will output files to the current directory. Ensure you +are in this directory when executing ``ceph-deploy``. + +.. important:: Do not call ``ceph-deploy`` with ``sudo`` or run it as ``root`` + if you are logged in as a different user, because it will not issue ``sudo`` + commands needed on the remote host. + + +Starting over +============= + +If at any point you run into trouble and you want to start over, execute +the following to purge the Ceph packages, and erase all its data and configuration:: + + ceph-deploy purge {ceph-node} [{ceph-node}] + ceph-deploy purgedata {ceph-node} [{ceph-node}] + ceph-deploy forgetkeys + rm ceph.* + +If you execute ``purge``, you must re-install Ceph. The last ``rm`` +command removes any files that were written out by ceph-deploy locally +during a previous installation. + + +Create a Cluster +================ + +On your admin node from the directory you created for holding your +configuration details, perform the following steps using ``ceph-deploy``. + +#. Create the cluster. :: + + ceph-deploy new {initial-monitor-node(s)} + + Specify node(s) as hostname, fqdn or hostname:fqdn. For example:: + + ceph-deploy new node1 + + Check the output of ``ceph-deploy`` with ``ls`` and ``cat`` in the + current directory. You should see a Ceph configuration file + (``ceph.conf``), a monitor secret keyring (``ceph.mon.keyring``), + and a log file for the new cluster. See `ceph-deploy new -h`_ for + additional details. + + Note for users of Ubuntu 18.04: Python 2 is a prerequisite of Ceph. + Install the ``python-minimal`` package on Ubuntu 18.04 to provide + Python 2:: + + [Ubuntu 18.04] $ sudo apt install python-minimal + +#. If you have more than one network interface, add the ``public network`` + setting under the ``[global]`` section of your Ceph configuration file. + See the `Network Configuration Reference`_ for details. :: + + public network = {ip-address}/{bits} + + For example,:: + + public network = 10.1.2.0/24 + + to use IPs in the 10.1.2.0/24 (or 10.1.2.0/255.255.255.0) network. + +#. If you are deploying in an IPv6 environment, add the following to + ``ceph.conf`` in the local directory:: + + echo ms bind ipv6 = true >> ceph.conf + +#. Install Ceph packages.:: + + ceph-deploy install {ceph-node} [...] + + For example:: + + ceph-deploy install node1 node2 node3 + + The ``ceph-deploy`` utility will install Ceph on each node. + +#. Deploy the initial monitor(s) and gather the keys:: + + ceph-deploy mon create-initial + + Once you complete the process, your local directory should have the following + keyrings: + + - ``ceph.client.admin.keyring`` + - ``ceph.bootstrap-mgr.keyring`` + - ``ceph.bootstrap-osd.keyring`` + - ``ceph.bootstrap-mds.keyring`` + - ``ceph.bootstrap-rgw.keyring`` + - ``ceph.bootstrap-rbd.keyring`` + - ``ceph.bootstrap-rbd-mirror.keyring`` + + .. note:: If this process fails with a message similar to "Unable to + find /etc/ceph/ceph.client.admin.keyring", please ensure that the + IP listed for the monitor node in ceph.conf is the Public IP, not + the Private IP. + +#. Use ``ceph-deploy`` to copy the configuration file and admin key to + your admin node and your Ceph Nodes so that you can use the ``ceph`` + CLI without having to specify the monitor address and + ``ceph.client.admin.keyring`` each time you execute a command. :: + + ceph-deploy admin {ceph-node(s)} + + For example:: + + ceph-deploy admin node1 node2 node3 + +#. Deploy a manager daemon. (Required only for luminous+ builds):: + + ceph-deploy mgr create node1 *Required only for luminous+ builds, i.e >= 12.x builds* + +#. Add three OSDs. For the purposes of these instructions, we assume you have an + unused disk in each node called ``/dev/vdb``. *Be sure that the device is not currently in use and does not contain any important data.* :: + + ceph-deploy osd create --data {device} {ceph-node} + + For example:: + + ceph-deploy osd create --data /dev/vdb node1 + ceph-deploy osd create --data /dev/vdb node2 + ceph-deploy osd create --data /dev/vdb node3 + + .. note:: If you are creating an OSD on an LVM volume, the argument to + ``--data`` *must* be ``volume_group/lv_name``, rather than the path to + the volume's block device. + +#. Check your cluster's health. :: + + ssh node1 sudo ceph health + + Your cluster should report ``HEALTH_OK``. You can view a more complete + cluster status with:: + + ssh node1 sudo ceph -s + + +Expanding Your Cluster +====================== + +Once you have a basic cluster up and running, the next step is to expand +cluster. Then add a Ceph Monitor and Ceph Manager to ``node2`` and ``node3`` +to improve reliability and availability. + +.. ditaa:: + /------------------\ /----------------\ + | ceph-deploy | | node1 | + | Admin Node | | cCCC | + | +-------->+ | + | | | mon.node1 | + | | | osd.0 | + | | | mgr.node1 | + \---------+--------/ \----------------/ + | + | /----------------\ + | | node2 | + | | cCCC | + +----------------->+ | + | | osd.1 | + | | mon.node2 | + | \----------------/ + | + | /----------------\ + | | node3 | + | | cCCC | + +----------------->+ | + | osd.2 | + | mon.node3 | + \----------------/ + +Adding Monitors +--------------- + +A Ceph Storage Cluster requires at least one Ceph Monitor and Ceph +Manager to run. For high availability, Ceph Storage Clusters typically +run multiple Ceph Monitors so that the failure of a single Ceph +Monitor will not bring down the Ceph Storage Cluster. Ceph uses the +Paxos algorithm, which requires a majority of monitors (i.e., greater +than *N/2* where *N* is the number of monitors) to form a quorum. +Odd numbers of monitors tend to be better, although this is not required. + +.. tip: If you did not define the ``public network`` option above then + the new monitor will not know which IP address to bind to on the + new hosts. You can add this line to your ``ceph.conf`` by editing + it now and then push it out to each node with + ``ceph-deploy --overwrite-conf config push {ceph-nodes}``. + +Add two Ceph Monitors to your cluster:: + + ceph-deploy mon add {ceph-nodes} + +For example:: + + ceph-deploy mon add node2 node3 + +Once you have added your new Ceph Monitors, Ceph will begin synchronizing +the monitors and form a quorum. You can check the quorum status by executing +the following:: + + ceph quorum_status --format json-pretty + + +.. tip:: When you run Ceph with multiple monitors, you SHOULD install and + configure NTP on each monitor host. Ensure that the + monitors are NTP peers. + +Adding Managers +--------------- + +The Ceph Manager daemons operate in an active/standby pattern. Deploying +additional manager daemons ensures that if one daemon or host fails, another +one can take over without interrupting service. + +To deploy additional manager daemons:: + + ceph-deploy mgr create node2 node3 + +You should see the standby managers in the output from:: + + ssh node1 sudo ceph -s + + +Add an RGW Instance +------------------- + +To use the :term:`Ceph Object Gateway` component of Ceph, you must deploy an +instance of :term:`RGW`. Execute the following to create an new instance of +RGW:: + + ceph-deploy rgw create {gateway-node} + +For example:: + + ceph-deploy rgw create node1 + +By default, the :term:`RGW` instance will listen on port 7480. This can be +changed by editing ceph.conf on the node running the :term:`RGW` as follows: + +.. code-block:: ini + + [client] + rgw frontends = civetweb port=80 + +To use an IPv6 address, use: + +.. code-block:: ini + + [client] + rgw frontends = civetweb port=[::]:80 + + + +Storing/Retrieving Object Data +============================== + +To store object data in the Ceph Storage Cluster, a Ceph client must: + +#. Set an object name +#. Specify a `pool`_ + +The Ceph Client retrieves the latest cluster map and the CRUSH algorithm +calculates how to map the object to a `placement group`_, and then calculates +how to assign the placement group to a Ceph OSD Daemon dynamically. To find the +object location, all you need is the object name and the pool name. For +example:: + + ceph osd map {poolname} {object-name} + +.. topic:: Exercise: Locate an Object + + As an exercise, lets create an object. Specify an object name, a path to + a test file containing some object data and a pool name using the + ``rados put`` command on the command line. For example:: + + echo {Test-data} > testfile.txt + ceph osd pool create mytest + rados put {object-name} {file-path} --pool=mytest + rados put test-object-1 testfile.txt --pool=mytest + + To verify that the Ceph Storage Cluster stored the object, execute + the following:: + + rados -p mytest ls + + Now, identify the object location:: + + ceph osd map {pool-name} {object-name} + ceph osd map mytest test-object-1 + + Ceph should output the object's location. For example:: + + osdmap e537 pool 'mytest' (1) object 'test-object-1' -> pg 1.d1743484 (1.4) -> up [1,0] acting [1,0] + + To remove the test object, simply delete it using the ``rados rm`` + command. + + For example:: + + rados rm test-object-1 --pool=mytest + + To delete the ``mytest`` pool:: + + ceph osd pool rm mytest + + (For safety reasons you will need to supply additional arguments as + prompted; deleting pools destroys data.) + +As the cluster evolves, the object location may change dynamically. One benefit +of Ceph's dynamic rebalancing is that Ceph relieves you from having to perform +data migration or balancing manually. + + +.. _Preflight Checklist: ../quick-start-preflight +.. _Ceph Deploy: ../../rados/deployment +.. _ceph-deploy install -h: ../../rados/deployment/ceph-deploy-install +.. _ceph-deploy new -h: ../../rados/deployment/ceph-deploy-new +.. _ceph-deploy osd: ../../rados/deployment/ceph-deploy-osd +.. _Running Ceph with Upstart: ../../rados/operations/operating#running-ceph-with-upstart +.. _Running Ceph with sysvinit: ../../rados/operations/operating#running-ceph-with-sysvinit +.. _CRUSH Map: ../../rados/operations/crush-map +.. _pool: ../../rados/operations/pools +.. _placement group: ../../rados/operations/placement-groups +.. _Monitoring a Cluster: ../../rados/operations/monitoring +.. _Monitoring OSDs and PGs: ../../rados/operations/monitoring-osd-pg +.. _Network Configuration Reference: ../../rados/configuration/network-config-ref +.. _User Management: ../../rados/operations/user-management diff --git a/doc/install/ceph-deploy/quick-cephfs.rst b/doc/install/ceph-deploy/quick-cephfs.rst new file mode 100644 index 000000000000..e8ca28f86ee1 --- /dev/null +++ b/doc/install/ceph-deploy/quick-cephfs.rst @@ -0,0 +1,212 @@ +=================== + CephFS Quick Start +=================== + +To use the :term:`CephFS` Quick Start guide, you must have executed the +procedures in the `Storage Cluster Quick Start`_ guide first. Execute this +quick start on the admin host. + +Prerequisites +============= + +#. Verify that you have an appropriate version of the Linux kernel. + See `OS Recommendations`_ for details. :: + + lsb_release -a + uname -r + +#. On the admin node, use ``ceph-deploy`` to install Ceph on your + ``ceph-client`` node. :: + + ceph-deploy install ceph-client + +#. Optionally, if you want a FUSE-mounted file system, you would need to + install ``ceph-fuse`` package as well. + +#. Ensure that the :term:`Ceph Storage Cluster` is running and in an ``active + + clean`` state. :: + + ceph -s [-m {monitor-ip-address}] [-k {path/to/ceph.client.admin.keyring}] + + +Deploy Metadata Server +====================== + +All metadata operations in CephFS happen via a metadata server, so you need at +least one metadata server. Execute the following to create a metadata server:: + + ceph-deploy mds create {ceph-node} + +For example:: + + ceph-deploy mds create node1 + +Now, your Ceph cluster would look like this: + +.. ditaa:: + /------------------\ /----------------\ + | ceph-deploy | | node1 | + | Admin Node | | cCCC | + | +-------->+ mon.node1 | + | | | osd.0 | + | | | mgr.node1 | + | | | mds.node1 | + \---------+--------/ \----------------/ + | + | /----------------\ + | | node2 | + | | cCCC | + +----------------->+ | + | | osd.1 | + | | mon.node2 | + | \----------------/ + | + | /----------------\ + | | node3 | + | | cCCC | + +----------------->+ | + | osd.2 | + | mon.node3 | + \----------------/ + +Create a File System +==================== + +You have already created an MDS (`Storage Cluster Quick Start`_) but it will not +become active until you create some pools and a file system. See +:doc:`/cephfs/createfs`. :: + + ceph osd pool create cephfs_data 32 + ceph osd pool create cephfs_meta 32 + ceph fs new mycephfs cephfs_meta cephfs_data + +.. note:: In case you have multiple Ceph applications and/or have multiple + CephFSs on the same cluster, it would be easier to name your pools as + ... In that case, the above pools would + be named as cephfs.mycehfs.data and cephfs.mycehfs.meta. + +Quick word about Pools and PGs +------------------------------ + +Replication Number/Pool Size +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Since the default replication number/size is 3, you'd need 3 OSDs to get +``active+clean`` for all PGs. Alternatively, you may change the replication +number for the pool to match the number of OSDs:: + + ceph osd pool set cephfs_data size {number-of-osds} + ceph osd pool set cephfs_meta size {number-of-osds} + +Usually, setting ``pg_num`` to 32 gives a perfectly healthy cluster. To pick +appropriate value for ``pg_num``, refer `Placement Group`_. You can also use +pg_autoscaler plugin instead. Introduced by Nautilus release, it can +automatically increase/decrease value of ``pg_num``; refer the +`Placement Group`_ to find out more about it. + +When all OSDs are on the same node... +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +And, in case you have deployed all of the OSDs on the same node, you would need +to create a new CRUSH rule to replicate data across OSDs and set the rule on the +CephFS pools, since the default CRUSH rule is to replicate data across +different nodes:: + + ceph osd crush rule create-replicated rule_foo default osd + ceph osd pool set cephfs_data crush_rule rule_foo + ceph osd pool set cephfs_meta crush_rule rule_foo + +Using Erasure Coded pools +^^^^^^^^^^^^^^^^^^^^^^^^^ +You may also use Erasure Coded pools which can be more effecient and +cost-saving since they allow stripping object data across OSDs and +replicating these stripes with encoded redundancy information. The number +of OSDs across which the data is stripped is `k` and number of replica is `m`. +You'll need to pick up these values before creating CephFS pools. The +following commands create a erasure code profile, creates a pool that'll +use it and then enables it on the pool:: + + ceph osd erasure-code-profile set ec-42-profile k=4 m=2 crush-failure-domain=host crush-device-class=ssd + ceph osd pool create cephfs_data_ec42 64 erasure ec-42-profile + ceph osd pool set cephfs_data_ec42 allow_ec_overwrites true + ceph fs add_data_pool mycephfs cephfs_data_ec42 + +You can also mark directories so that they are only stored on certain pools:: + + setfattr -n ceph.dir.layout -v pool=cephfs_data_ec42 /mnt/mycephfs/logs + +This way you can choose the replication strategy for each directory on your +Ceph file system. + +.. note:: Erasure Coded pools can not be used for CephFS metadata pools. + +Erasure coded pool were introduced in Firefly and could be used directly by +CephFS Luminous onwards. Refer `this article `_ +by Sage Weil to understand EC, it's background, limitations and other details +in Ceph's context. Read more about `Erasure Code`_ here. + +Mounting the File System +======================== + +Using Kernel Driver +------------------- + +The command to mount CephFS using kernel driver looks like this:: + + sudo mount -t ceph :{path-to-mounted} {mount-point} -o name={user-name} + sudo mount -t ceph :/ /mnt/mycephfs -o name=admin # usable version + +``{path-to-be-mounted}`` is the path within CephFS that will be mounted, +``{mount-point}`` is the point in your file system upon which CephFS will be +mounted and ``{user-name}`` is the name of CephX user that has the +authorization to mount CephFS on the machine. Following command is the +extended form, however these extra details are automatically figured out by +by the mount.ceph helper program:: + + sudo mount -t ceph {ip-address-of-MON}:{port-number-of-MON}:{path-to-be-mounted} -o name={user-name},secret={secret-key} {mount-point} + +If you have multiple file systems on your cluster you would need to pass +``fs={fs-name}`` option to ``-o`` option to the ``mount`` command:: + + sudo mount -t ceph :/ /mnt/kcephfs2 -o name=admin,fs=mycephfs2 + +Refer `mount.ceph man page`_ and `Mount CephFS using Kernel Driver`_ to read +more about this. + + +Using FUSE +---------- + +To mount CephFS using FUSE (Filesystem in User Space) run:: + + sudo ceph-fuse /mnt/mycephfs + +To mount a particular directory within CephFS you can use ``-r``:: + + sudo ceph-fuse -r {path-to-be-mounted} /mnt/mycephfs + +If you have multiple file systems on your cluster you would need to pass +``--client_fs {fs-name}`` to the ``ceph-fuse`` command:: + + sudo ceph-fuse /mnt/mycephfs2 --client_fs mycephfs2 + +Refer `ceph-fuse man page`_ and `Mount CephFS using FUSE`_ to read more about +this. + +.. note:: Mount the CephFS file system on the admin node, not the server node. + + +Additional Information +====================== + +See `CephFS`_ for additional information. See `Troubleshooting`_ if you +encounter trouble. + +.. _Storage Cluster Quick Start: ../quick-ceph-deploy +.. _CephFS: ../../cephfs/ +.. _Troubleshooting: ../../cephfs/troubleshooting +.. _OS Recommendations: ../os-recommendations +.. _Placement Group: ../../rados/operations/placement-groups +.. _mount.ceph man page: ../../man/8/mount.ceph +.. _Mount CephFS using Kernel Driver: ../cephfs/kernel +.. _ceph-fuse man page: ../../man/8/ceph-fuse +.. _Mount CephFS using FUSE: ../../cephfs/fuse +.. _Erasure Code: ../../rados/operations/erasure-code diff --git a/doc/install/ceph-deploy/quick-common.rst b/doc/install/ceph-deploy/quick-common.rst new file mode 100644 index 000000000000..915a7b886422 --- /dev/null +++ b/doc/install/ceph-deploy/quick-common.rst @@ -0,0 +1,20 @@ +.. ditaa:: + /------------------\ /-----------------\ + | admin-node | | node1 | + | +-------->+ cCCC | + | ceph-deploy | | mon.node1 | + | | | osd.0 | + \---------+--------/ \-----------------/ + | + | /----------------\ + | | node2 | + +----------------->+ cCCC | + | | osd.1 | + | \----------------/ + | + | /----------------\ + | | node3 | + +----------------->| cCCC | + | osd.2 | + \----------------/ + diff --git a/doc/install/ceph-deploy/quick-rgw-old.rst b/doc/install/ceph-deploy/quick-rgw-old.rst new file mode 100644 index 000000000000..db6474de5145 --- /dev/null +++ b/doc/install/ceph-deploy/quick-rgw-old.rst @@ -0,0 +1,30 @@ +:orphan: + +=========================== + Quick Ceph Object Storage +=========================== + +To use the :term:`Ceph Object Storage` Quick Start guide, you must have executed the +procedures in the `Storage Cluster Quick Start`_ guide first. Make sure that you +have at least one :term:`RGW` instance running. + +Configure new RGW instance +========================== + +The :term:`RGW` instance created by the `Storage Cluster Quick Start`_ will run using +the embedded CivetWeb webserver. ``ceph-deploy`` will create the instance and start +it automatically with default parameters. + +To administer the :term:`RGW` instance, see details in the the +`RGW Admin Guide`_. + +Additional details may be found in the `Configuring Ceph Object Gateway`_ guide, but +the steps specific to Apache are no longer needed. + +.. note:: Deploying RGW using ``ceph-deploy`` and using the CivetWeb webserver instead + of Apache is new functionality as of **Hammer** release. + + +.. _Storage Cluster Quick Start: ../quick-ceph-deploy +.. _RGW Admin Guide: ../../radosgw/admin +.. _Configuring Ceph Object Gateway: ../../radosgw/config-fcgi diff --git a/doc/install/ceph-deploy/quick-rgw.rst b/doc/install/ceph-deploy/quick-rgw.rst new file mode 100644 index 000000000000..5efda04f9ba1 --- /dev/null +++ b/doc/install/ceph-deploy/quick-rgw.rst @@ -0,0 +1,101 @@ +=============================== +Ceph Object Gateway Quick Start +=============================== + +As of `firefly` (v0.80), Ceph Storage dramatically simplifies installing and +configuring a Ceph Object Gateway. The Gateway daemon embeds Civetweb, so you +do not have to install a web server or configure FastCGI. Additionally, +``ceph-deploy`` can install the gateway package, generate a key, configure a +data directory and create a gateway instance for you. + +.. tip:: Civetweb uses port ``7480`` by default. You must either open port + ``7480``, or set the port to a preferred port (e.g., port ``80``) in your Ceph + configuration file. + +To start a Ceph Object Gateway, follow the steps below: + +Installing Ceph Object Gateway +============================== + +#. Execute the pre-installation steps on your ``client-node``. If you intend to + use Civetweb's default port ``7480``, you must open it using either + ``firewall-cmd`` or ``iptables``. See `Preflight Checklist`_ for more + information. + +#. From the working directory of your administration server, install the Ceph + Object Gateway package on the ``client-node`` node. For example:: + + ceph-deploy install --rgw [ ...] + +Creating the Ceph Object Gateway Instance +========================================= + +From the working directory of your administration server, create an instance of +the Ceph Object Gateway on the ``client-node``. For example:: + + ceph-deploy rgw create + +Once the gateway is running, you should be able to access it on port ``7480``. +(e.g., ``http://client-node:7480``). + +Configuring the Ceph Object Gateway Instance +============================================ + +#. To change the default port (e.g., to port ``80``), modify your Ceph + configuration file. Add a section entitled ``[client.rgw.]``, + replacing ```` with the short node name of your Ceph client + node (i.e., ``hostname -s``). For example, if your node name is + ``client-node``, add a section like this after the ``[global]`` section:: + + [client.rgw.client-node] + rgw_frontends = "civetweb port=80" + + .. note:: Ensure that you leave no whitespace between ``port=`` + in the ``rgw_frontends`` key/value pair. + + .. important:: If you intend to use port 80, make sure that the Apache + server is not running otherwise it will conflict with Civetweb. We recommend + to remove Apache in this case. + +#. To make the new port setting take effect, restart the Ceph Object Gateway. + On Red Hat Enterprise Linux 7 and Fedora, run the following command:: + + sudo systemctl restart ceph-radosgw.service + + On Red Hat Enterprise Linux 6 and Ubuntu, run the following command:: + + sudo service radosgw restart id=rgw. + +#. Finally, check to ensure that the port you selected is open on the node's + firewall (e.g., port ``80``). If it is not open, add the port and reload the + firewall configuration. For example:: + + sudo firewall-cmd --list-all + sudo firewall-cmd --zone=public --add-port 80/tcp --permanent + sudo firewall-cmd --reload + + See `Preflight Checklist`_ for more information on configuring firewall with + ``firewall-cmd`` or ``iptables``. + + You should be able to make an unauthenticated request, and receive a + response. For example, a request with no parameters like this:: + + http://:80 + + Should result in a response like this:: + + + + + anonymous + + + + + + +See the `Configuring Ceph Object Gateway`_ guide for additional administration +and API details. + +.. _Configuring Ceph Object Gateway: ../../radosgw/config-ref +.. _Preflight Checklist: ../quick-start-preflight diff --git a/doc/install/ceph-deploy/quick-start-preflight.rst b/doc/install/ceph-deploy/quick-start-preflight.rst new file mode 100644 index 000000000000..b1fdc92d2286 --- /dev/null +++ b/doc/install/ceph-deploy/quick-start-preflight.rst @@ -0,0 +1,364 @@ +===================== + Preflight Checklist +===================== + +The ``ceph-deploy`` tool operates out of a directory on an admin +:term:`node`. Any host with network connectivity and a modern python +environment and ssh (such as Linux) should work. + +In the descriptions below, :term:`Node` refers to a single machine. + +.. include:: quick-common.rst + + +Ceph-deploy Setup +================= + +Add Ceph repositories to the ``ceph-deploy`` admin node. Then, install +``ceph-deploy``. + +Debian/Ubuntu +------------- + +For Debian and Ubuntu distributions, perform the following steps: + +#. Add the release key:: + + wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - + +#. Add the Ceph packages to your repository. Use the command below and + replace ``{ceph-stable-release}`` with a stable Ceph release (e.g., + ``luminous``.) For example:: + + echo deb https://download.ceph.com/debian-{ceph-stable-release}/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list + +#. Update your repository and install ``ceph-deploy``:: + + sudo apt update + sudo apt install ceph-deploy + +.. note:: You can also use the EU mirror eu.ceph.com for downloading your packages by replacing ``https://ceph.com/`` by ``http://eu.ceph.com/`` + + +RHEL/CentOS +----------- + +For CentOS 7, perform the following steps: + +#. On Red Hat Enterprise Linux 7, register the target machine with + ``subscription-manager``, verify your subscriptions, and enable the + "Extras" repository for package dependencies. For example:: + + sudo subscription-manager repos --enable=rhel-7-server-extras-rpms + +#. Install and enable the Extra Packages for Enterprise Linux (EPEL) + repository:: + + sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + + Please see the `EPEL wiki`_ page for more information. + +#. Add the Ceph repository to your yum configuration file at ``/etc/yum.repos.d/ceph.repo`` with the following command. Replace ``{ceph-stable-release}`` with a stable Ceph release (e.g., + ``luminous``.) For example:: + + cat << EOM > /etc/yum.repos.d/ceph.repo + [ceph-noarch] + name=Ceph noarch packages + baseurl=https://download.ceph.com/rpm-{ceph-stable-release}/el7/noarch + enabled=1 + gpgcheck=1 + type=rpm-md + gpgkey=https://download.ceph.com/keys/release.asc + EOM + +#. You may need to install python setuptools required by ceph-deploy: + + sudo yum install python-setuptools + +#. Update your repository and install ``ceph-deploy``:: + + sudo yum update + sudo yum install ceph-deploy + +.. note:: You can also use the EU mirror eu.ceph.com for downloading your packages by replacing ``https://ceph.com/`` by ``http://eu.ceph.com/`` + + +openSUSE +-------- + +The Ceph project does not currently publish release RPMs for openSUSE, but +a stable version of Ceph is included in the default update repository, so +installing it is just a matter of:: + + sudo zypper install ceph + sudo zypper install ceph-deploy + +If the distro version is out-of-date, open a bug at +https://bugzilla.opensuse.org/index.cgi and possibly try your luck with one of +the following repositories: + +#. Hammer:: + + https://software.opensuse.org/download.html?project=filesystems%3Aceph%3Ahammer&package=ceph + +#. Jewel:: + + https://software.opensuse.org/download.html?project=filesystems%3Aceph%3Ajewel&package=ceph + + +Ceph Node Setup +=============== + +The admin node must have password-less SSH access to Ceph nodes. +When ceph-deploy logs in to a Ceph node as a user, that particular +user must have passwordless ``sudo`` privileges. + + +Install NTP +----------- + +We recommend installing NTP on Ceph nodes (especially on Ceph Monitor nodes) to +prevent issues arising from clock drift. See `Clock`_ for details. + +On CentOS / RHEL, execute:: + + sudo yum install ntp ntpdate ntp-doc + +On Debian / Ubuntu, execute:: + + sudo apt install ntpsec + +or:: + + sudo apt install chrony + +Ensure that you enable the NTP service. Ensure that each Ceph Node uses the +same NTP time server. See `NTP`_ for details. + + +Install SSH Server +------------------ + +For **ALL** Ceph Nodes perform the following steps: + +#. Install an SSH server (if necessary) on each Ceph Node:: + + sudo apt install openssh-server + + or:: + + sudo yum install openssh-server + + +#. Ensure the SSH server is running on **ALL** Ceph Nodes. + + +Create a Ceph Deploy User +------------------------- + +The ``ceph-deploy`` utility must login to a Ceph node as a user +that has passwordless ``sudo`` privileges, because it needs to install +software and configuration files without prompting for passwords. + +Recent versions of ``ceph-deploy`` support a ``--username`` option so you can +specify any user that has password-less ``sudo`` (including ``root``, although +this is **NOT** recommended). To use ``ceph-deploy --username {username}``, the +user you specify must have password-less SSH access to the Ceph node, as +``ceph-deploy`` will not prompt you for a password. + +We recommend creating a specific user for ``ceph-deploy`` on **ALL** Ceph nodes +in the cluster. Please do **NOT** use "ceph" as the user name. A uniform user +name across the cluster may improve ease of use (not required), but you should +avoid obvious user names, because hackers typically use them with brute force +hacks (e.g., ``root``, ``admin``, ``{productname}``). The following procedure, +substituting ``{username}`` for the user name you define, describes how to +create a user with passwordless ``sudo``. + +.. note:: Starting with the :ref:`Infernalis release `, the "ceph" user name is reserved + for the Ceph daemons. If the "ceph" user already exists on the Ceph nodes, + removing the user must be done before attempting an upgrade. + +#. Create a new user on each Ceph Node. :: + + ssh user@ceph-server + sudo useradd -d /home/{username} -m {username} + sudo passwd {username} + +#. For the new user you added to each Ceph node, ensure that the user has + ``sudo`` privileges. :: + + echo "{username} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/{username} + sudo chmod 0440 /etc/sudoers.d/{username} + + +Enable Password-less SSH +------------------------ + +Since ``ceph-deploy`` will not prompt for a password, you must generate +SSH keys on the admin node and distribute the public key to each Ceph +node. ``ceph-deploy`` will attempt to generate the SSH keys for initial +monitors. + +#. Generate the SSH keys, but do not use ``sudo`` or the + ``root`` user. Leave the passphrase empty:: + + ssh-keygen + + Generating public/private key pair. + Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /ceph-admin/.ssh/id_rsa. + Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub. + +#. Copy the key to each Ceph Node, replacing ``{username}`` with the user name + you created with `Create a Ceph Deploy User`_. :: + + ssh-copy-id {username}@node1 + ssh-copy-id {username}@node2 + ssh-copy-id {username}@node3 + +#. (Recommended) Modify the ``~/.ssh/config`` file of your ``ceph-deploy`` + admin node so that ``ceph-deploy`` can log in to Ceph nodes as the user you + created without requiring you to specify ``--username {username}`` each + time you execute ``ceph-deploy``. This has the added benefit of streamlining + ``ssh`` and ``scp`` usage. Replace ``{username}`` with the user name you + created:: + + Host node1 + Hostname node1 + User {username} + Host node2 + Hostname node2 + User {username} + Host node3 + Hostname node3 + User {username} + + +Enable Networking On Bootup +--------------------------- + +Ceph OSDs peer with each other and report to Ceph Monitors over the network. +If networking is ``off`` by default, the Ceph cluster cannot come online +during bootup until you enable networking. + +The default configuration on some distributions (e.g., CentOS) has the +networking interface(s) off by default. Ensure that, during boot up, your +network interface(s) turn(s) on so that your Ceph daemons can communicate over +the network. For example, on Red Hat and CentOS, navigate to +``/etc/sysconfig/network-scripts`` and ensure that the ``ifcfg-{iface}`` file +has ``ONBOOT`` set to ``yes``. + + +Ensure Connectivity +------------------- + +Ensure connectivity using ``ping`` with short hostnames (``hostname -s``). +Address hostname resolution issues as necessary. + +.. note:: Hostnames should resolve to a network IP address, not to the + loopback IP address (e.g., hostnames should resolve to an IP address other + than ``127.0.0.1``). If you use your admin node as a Ceph node, you + should also ensure that it resolves to its hostname and IP address + (i.e., not its loopback IP address). + + +Open Required Ports +------------------- + +Ceph Monitors communicate using port ``6789`` by default. Ceph OSDs communicate +in a port range of ``6800:7300`` by default. See the `Network Configuration +Reference`_ for details. Ceph OSDs can use multiple network connections to +communicate with clients, monitors, other OSDs for replication, and other OSDs +for heartbeats. + +On some distributions (e.g., RHEL), the default firewall configuration is fairly +strict. You may need to adjust your firewall settings allow inbound requests so +that clients in your network can communicate with daemons on your Ceph nodes. + +For ``firewalld`` on RHEL 7, add the ``ceph-mon`` service for Ceph Monitor +nodes and the ``ceph`` service for Ceph OSDs and MDSs to the public zone and +ensure that you make the settings permanent so that they are enabled on reboot. + +For example, on monitors:: + + sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent + +and on OSDs and MDSs:: + + sudo firewall-cmd --zone=public --add-service=ceph --permanent + +Once you have finished configuring firewalld with the ``--permanent`` flag, you can make the changes live immediately without rebooting:: + + sudo firewall-cmd --reload + +For ``iptables``, add port ``6789`` for Ceph Monitors and ports ``6800:7300`` +for Ceph OSDs. For example:: + + sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT + +Once you have finished configuring ``iptables``, ensure that you make the +changes persistent on each node so that they will be in effect when your nodes +reboot. For example:: + + /sbin/service iptables save + +TTY +--- + +On CentOS and RHEL, you may receive an error while trying to execute +``ceph-deploy`` commands. If ``requiretty`` is set by default on your Ceph +nodes, disable it by executing ``sudo visudo`` and locate the ``Defaults +requiretty`` setting. Change it to ``Defaults:ceph !requiretty`` or comment it +out to ensure that ``ceph-deploy`` can connect using the user you created with +`Create a Ceph Deploy User`_. + +.. note:: If editing, ``/etc/sudoers``, ensure that you use + ``sudo visudo`` rather than a text editor. + + +SELinux +------- + +On CentOS and RHEL, SELinux is set to ``Enforcing`` by default. To streamline your +installation, we recommend setting SELinux to ``Permissive`` or disabling it +entirely and ensuring that your installation and cluster are working properly +before hardening your configuration. To set SELinux to ``Permissive``, execute the +following:: + + sudo setenforce 0 + +To configure SELinux persistently (recommended if SELinux is an issue), modify +the configuration file at ``/etc/selinux/config``. + + +Priorities/Preferences +---------------------- + +Ensure that your package manager has priority/preferences packages installed and +enabled. On CentOS, you may need to install EPEL. On RHEL, you may need to +enable optional repositories. :: + + sudo yum install yum-plugin-priorities + +For example, on RHEL 7 server, execute the following to install +``yum-plugin-priorities`` and enable the ``rhel-7-server-optional-rpms`` +repository:: + + sudo yum install yum-plugin-priorities --enablerepo=rhel-7-server-optional-rpms + + +Summary +======= + +This completes the Quick Start Preflight. Proceed to the `Storage Cluster +Quick Start`_. + +.. _Storage Cluster Quick Start: ../quick-ceph-deploy +.. _OS Recommendations: ../os-recommendations +.. _Network Configuration Reference: ../../rados/configuration/network-config-ref +.. _Clock: ../../rados/configuration/mon-config-ref#clock +.. _NTP: http://www.ntp.org/ +.. _Infernalis release: ../../release-notes/#v9-1-0-infernalis-release-candidate +.. _EPEL wiki: https://fedoraproject.org/wiki/EPEL diff --git a/doc/install/ceph-deploy/upgrading-ceph.rst b/doc/install/ceph-deploy/upgrading-ceph.rst new file mode 100644 index 000000000000..6fbf43a236dc --- /dev/null +++ b/doc/install/ceph-deploy/upgrading-ceph.rst @@ -0,0 +1,235 @@ +================ + Upgrading Ceph +================ + +Each release of Ceph may have additional steps. Refer to the `release notes +document of your release`_ to identify release-specific procedures for your +cluster before using the upgrade procedures. + + +Summary +======= + +You can upgrade daemons in your Ceph cluster while the cluster is online and in +service! Certain types of daemons depend upon others. For example, Ceph Metadata +Servers and Ceph Object Gateways depend upon Ceph Monitors and Ceph OSD Daemons. +We recommend upgrading in this order: + +#. `Ceph Deploy`_ +#. Ceph Monitors +#. Ceph OSD Daemons +#. Ceph Metadata Servers +#. Ceph Object Gateways + +As a general rule, we recommend upgrading all the daemons of a specific type +(e.g., all ``ceph-mon`` daemons, all ``ceph-osd`` daemons, etc.) to ensure that +they are all on the same release. We also recommend that you upgrade all the +daemons in your cluster before you try to exercise new functionality in a +release. + +The `Upgrade Procedures`_ are relatively simple, but do look at the `release +notes document of your release`_ before upgrading. The basic process involves +three steps: + +#. Use ``ceph-deploy`` on your admin node to upgrade the packages for + multiple hosts (using the ``ceph-deploy install`` command), or login to each + host and upgrade the Ceph package `using your distro's package manager`_. + For example, when `Upgrading Monitors`_, the ``ceph-deploy`` syntax might + look like this:: + + ceph-deploy install --release {release-name} ceph-node1[ ceph-node2] + ceph-deploy install --release firefly mon1 mon2 mon3 + + **Note:** The ``ceph-deploy install`` command will upgrade the packages + in the specified node(s) from the old release to the release you specify. + There is no ``ceph-deploy upgrade`` command. + +#. Login in to each Ceph node and restart each Ceph daemon. + See `Operating a Cluster`_ for details. + +#. Ensure your cluster is healthy. See `Monitoring a Cluster`_ for details. + +.. important:: Once you upgrade a daemon, you cannot downgrade it. + + +Ceph Deploy +=========== + +Before upgrading Ceph daemons, upgrade the ``ceph-deploy`` tool. :: + + sudo pip install -U ceph-deploy + +Or:: + + sudo apt-get install ceph-deploy + +Or:: + + sudo yum install ceph-deploy python-pushy + + +Upgrade Procedures +================== + +The following sections describe the upgrade process. + +.. important:: Each release of Ceph may have some additional steps. Refer to + the `release notes document of your release`_ for details **BEFORE** you + begin upgrading daemons. + + +Upgrading Monitors +------------------ + +To upgrade monitors, perform the following steps: + +#. Upgrade the Ceph package for each daemon instance. + + You may use ``ceph-deploy`` to address all monitor nodes at once. + For example:: + + ceph-deploy install --release {release-name} ceph-node1[ ceph-node2] + ceph-deploy install --release hammer mon1 mon2 mon3 + + You may also use the package manager for your Linux distribution on + each individual node. To upgrade packages manually on each Debian/Ubuntu + host, perform the following steps:: + + ssh {mon-host} + sudo apt-get update && sudo apt-get install ceph + + On CentOS/Red Hat hosts, perform the following steps:: + + ssh {mon-host} + sudo yum update && sudo yum install ceph + + +#. Restart each monitor. For Ubuntu distributions, use:: + + sudo systemctl restart ceph-mon@{hostname}.service + + For CentOS/Red Hat/Debian distributions, use:: + + sudo /etc/init.d/ceph restart {mon-id} + + For CentOS/Red Hat distributions deployed with ``ceph-deploy``, + the monitor ID is usually ``mon.{hostname}``. + +#. Ensure each monitor has rejoined the quorum:: + + ceph mon stat + +Ensure that you have completed the upgrade cycle for all of your Ceph Monitors. + + +Upgrading an OSD +---------------- + +To upgrade a Ceph OSD Daemon, perform the following steps: + +#. Upgrade the Ceph OSD Daemon package. + + You may use ``ceph-deploy`` to address all Ceph OSD Daemon nodes at + once. For example:: + + ceph-deploy install --release {release-name} ceph-node1[ ceph-node2] + ceph-deploy install --release hammer osd1 osd2 osd3 + + You may also use the package manager on each node to upgrade packages + `using your distro's package manager`_. For Debian/Ubuntu hosts, perform the + following steps on each host:: + + ssh {osd-host} + sudo apt-get update && sudo apt-get install ceph + + For CentOS/Red Hat hosts, perform the following steps:: + + ssh {osd-host} + sudo yum update && sudo yum install ceph + + +#. Restart the OSD, where ``N`` is the OSD number. For Ubuntu, use:: + + sudo systemctl restart ceph-osd@{N}.service + + For multiple OSDs on a host, you may restart all of them with systemd. :: + + sudo systemctl restart ceph-osd + + For CentOS/Red Hat/Debian distributions, use:: + + sudo /etc/init.d/ceph restart N + + +#. Ensure each upgraded Ceph OSD Daemon has rejoined the cluster:: + + ceph osd stat + +Ensure that you have completed the upgrade cycle for all of your +Ceph OSD Daemons. + + +Upgrading a Metadata Server +--------------------------- + +To upgrade a Ceph Metadata Server, perform the following steps: + +#. Upgrade the Ceph Metadata Server package. You may use ``ceph-deploy`` to + address all Ceph Metadata Server nodes at once, or use the package manager + on each node. For example:: + + ceph-deploy install --release {release-name} ceph-node1 + ceph-deploy install --release hammer mds1 + + To upgrade packages manually, perform the following steps on each + Debian/Ubuntu host:: + + ssh {mon-host} + sudo apt-get update && sudo apt-get install ceph-mds + + Or the following steps on CentOS/Red Hat hosts:: + + ssh {mon-host} + sudo yum update && sudo yum install ceph-mds + + +#. Restart the metadata server. For Ubuntu, use:: + + sudo systemctl restart ceph-mds@{hostname}.service + + For CentOS/Red Hat/Debian distributions, use:: + + sudo /etc/init.d/ceph restart mds.{hostname} + + For clusters deployed with ``ceph-deploy``, the name is usually either + the name you specified on creation or the hostname. + +#. Ensure the metadata server is up and running:: + + ceph mds stat + + +Upgrading a Client +------------------ + +Once you have upgraded the packages and restarted daemons on your Ceph +cluster, we recommend upgrading ``ceph-common`` and client libraries +(``librbd1`` and ``librados2``) on your client nodes too. + +#. Upgrade the package:: + + ssh {client-host} + apt-get update && sudo apt-get install ceph-common librados2 librbd1 python-rados python-rbd + +#. Ensure that you have the latest version:: + + ceph --version + +If you do not have the latest version, you may need to uninstall, auto remove +dependencies and reinstall. + + +.. _using your distro's package manager: ../install-storage-cluster/ +.. _Operating a Cluster: ../../rados/operations/operating +.. _Monitoring a Cluster: ../../rados/operations/monitoring +.. _release notes document of your release: ../../releases diff --git a/doc/install/index.rst b/doc/install/index.rst index 298b3b39a75e..69a48a6d003c 100644 --- a/doc/install/index.rst +++ b/doc/install/index.rst @@ -69,6 +69,11 @@ Install ceph using ceph-deploy ceph-deploy is no longer actively maintained. It is not tested on versions of Ceph newer than Nautilus. It does not support RHEL8, CentOS 8, or newer operating systems. +.. toctree:: + :hidden: + + ceph-deploy/index + DeepSea ------- @@ -92,6 +97,11 @@ Manually install Ceph using packages. * :ref:`install-manual` +.. toctree:: + :hidden: + + index_manual + Puppet ------ @@ -100,9 +110,3 @@ Installs Ceph using Puppet * `github.com/openstack/puppet-ceph `_ - - -.. toctree:: - :hidden: - - index_manual diff --git a/doc/install/index_manual.rst b/doc/install/index_manual.rst index 68520735ac67..817247c9532b 100644 --- a/doc/install/index_manual.rst +++ b/doc/install/index_manual.rst @@ -42,7 +42,6 @@ QEMU. Install ceph-deploy Install Ceph Storage Cluster - Install Ceph Object Gateway Install Virtualization for Block @@ -69,6 +68,4 @@ sequence. .. toctree:: :maxdepth: 2 - Upgrading Ceph - .. _get packages: ../install/get-packages diff --git a/doc/install/install-ceph-gateway.rst b/doc/install/install-ceph-gateway.rst deleted file mode 100644 index fe5b6b574cd4..000000000000 --- a/doc/install/install-ceph-gateway.rst +++ /dev/null @@ -1,615 +0,0 @@ -=========================== -Install Ceph Object Gateway -=========================== - -As of `firefly` (v0.80), Ceph Object Gateway is running on Civetweb (embedded -into the ``ceph-radosgw`` daemon) instead of Apache and FastCGI. Using Civetweb -simplifies the Ceph Object Gateway installation and configuration. - -.. note:: To run the Ceph Object Gateway service, you should have a running - Ceph storage cluster, and the gateway host should have access to the - public network. - -.. note:: In version 0.80, the Ceph Object Gateway does not support SSL. You - may setup a reverse proxy server with SSL to dispatch HTTPS requests - as HTTP requests to CivetWeb. - -Execute the Pre-Installation Procedure --------------------------------------- - -See Preflight_ and execute the pre-installation procedures on your Ceph Object -Gateway node. Specifically, you should disable ``requiretty`` on your Ceph -Deploy user, set SELinux to ``Permissive`` and set up a Ceph Deploy user with -password-less ``sudo``. For Ceph Object Gateways, you will need to open the -port that Civetweb will use in production. - -.. note:: Civetweb runs on port ``7480`` by default. - -Install Ceph Object Gateway ---------------------------- - -From the working directory of your administration server, install the Ceph -Object Gateway package on the Ceph Object Gateway node. For example:: - - ceph-deploy install --rgw [ ...] - -The ``ceph-common`` package is a dependency, so ``ceph-deploy`` will install -this too. The ``ceph`` CLI tools are intended for administrators. To make your -Ceph Object Gateway node an administrator node, execute the following from the -working directory of your administration server:: - - ceph-deploy admin - -Create a Gateway Instance -------------------------- - -From the working directory of your administration server, create an instance of -the Ceph Object Gateway on the Ceph Object Gateway. For example:: - - ceph-deploy rgw create - -Once the gateway is running, you should be able to access it on port ``7480`` -with an unauthenticated request like this:: - - http://client-node:7480 - -If the gateway instance is working properly, you should receive a response like -this:: - - - - - anonymous - - - - - - -If at any point you run into trouble and you want to start over, execute the -following to purge the configuration:: - - ceph-deploy purge [] - ceph-deploy purgedata [] - -If you execute ``purge``, you must re-install Ceph. - -Change the Default Port ------------------------ - -Civetweb runs on port ``7480`` by default. To change the default port (e.g., to -port ``80``), modify your Ceph configuration file in the working directory of -your administration server. Add a section entitled -``[client.rgw.]``, replacing ```` with the short -node name of your Ceph Object Gateway node (i.e., ``hostname -s``). - -.. note:: As of version 11.0.1, the Ceph Object Gateway **does** support SSL. - See `Using SSL with Civetweb`_ for information on how to set that up. - -For example, if your node name is ``gateway-node1``, add a section like this -after the ``[global]`` section:: - - [client.rgw.gateway-node1] - rgw_frontends = "civetweb port=80" - -.. note:: Ensure that you leave no whitespace between ``port=`` in - the ``rgw_frontends`` key/value pair. The ``[client.rgw.gateway-node1]`` - heading identifies this portion of the Ceph configuration file as - configuring a Ceph Storage Cluster client where the client type is a Ceph - Object Gateway (i.e., ``rgw``), and the name of the instance is - ``gateway-node1``. - -Push the updated configuration file to your Ceph Object Gateway node -(and other Ceph nodes):: - - ceph-deploy --overwrite-conf config push [] - -To make the new port setting take effect, restart the Ceph Object -Gateway:: - - sudo systemctl restart ceph-radosgw.service - -Finally, check to ensure that the port you selected is open on the node's -firewall (e.g., port ``80``). If it is not open, add the port and reload the -firewall configuration. If you use the ``firewalld`` daemon, execute:: - - sudo firewall-cmd --list-all - sudo firewall-cmd --zone=public --add-port 80/tcp --permanent - sudo firewall-cmd --reload - -If you use ``iptables``, execute:: - - sudo iptables --list - sudo iptables -I INPUT 1 -i -p tcp -s / --dport 80 -j ACCEPT - -Replace ```` and ``/`` with the relevant values for -your Ceph Object Gateway node. - -Once you have finished configuring ``iptables``, ensure that you make the -change persistent so that it will be in effect when your Ceph Object Gateway -node reboots. Execute:: - - sudo apt-get install iptables-persistent - -A terminal UI will open up. Select ``yes`` for the prompts to save current -``IPv4`` iptables rules to ``/etc/iptables/rules.v4`` and current ``IPv6`` -iptables rules to ``/etc/iptables/rules.v6``. - -The ``IPv4`` iptables rule that you set in the earlier step will be loaded in -``/etc/iptables/rules.v4`` and will be persistent across reboots. - -If you add a new ``IPv4`` iptables rule after installing -``iptables-persistent`` you will have to add it to the rule file. In such case, -execute the following as the ``root`` user:: - - iptables-save > /etc/iptables/rules.v4 - -Using SSL with Civetweb ------------------------ -.. _Using SSL with Civetweb: - -Before using SSL with civetweb, you will need a certificate that will match -the host name that that will be used to access the Ceph Object Gateway. -You may wish to obtain one that has `subject alternate name` fields for -more flexibility. If you intend to use S3-style subdomains -(`Add Wildcard to DNS`_), you will need a `wildcard` certificate. - -Civetweb requires that the server key, server certificate, and any other -CA or intermediate certificates be supplied in one file. Each of these -items must be in `pem` form. Because the combined file contains the -secret key, it should be protected from unauthorized access. - -To configure ssl operation, append ``s`` to the port number. For eg:: - - [client.rgw.gateway-node1] - rgw_frontends = civetweb port=443s ssl_certificate=/etc/ceph/private/keyandcert.pem - -.. versionadded :: Luminous - -Furthermore, civetweb can be made to bind to multiple ports, by separating them -with ``+`` in the configuration. This allows for use cases where both ssl and -non-ssl connections are hosted by a single rgw instance. For eg:: - - [client.rgw.gateway-node1] - rgw_frontends = civetweb port=80+443s ssl_certificate=/etc/ceph/private/keyandcert.pem - -Additional Civetweb Configuration Options ------------------------------------------ -Some additional configuration options can be adjusted for the embedded Civetweb web server -in the **Ceph Object Gateway** section of the ``ceph.conf`` file. -A list of supported options, including an example, can be found in the `HTTP Frontends`_. - -Migrating from Apache to Civetweb ---------------------------------- - -If you are running the Ceph Object Gateway on Apache and FastCGI with Ceph -Storage v0.80 or above, you are already running Civetweb--it starts with the -``ceph-radosgw`` daemon and it's running on port 7480 by default so that it -doesn't conflict with your Apache and FastCGI installation and other commonly -used web service ports. Migrating to use Civetweb basically involves removing -your Apache installation. Then, you must remove Apache and FastCGI settings -from your Ceph configuration file and reset ``rgw_frontends`` to Civetweb. - -Referring back to the description for installing a Ceph Object Gateway with -``ceph-deploy``, notice that the configuration file only has one setting -``rgw_frontends`` (and that's assuming you elected to change the default port). -The ``ceph-deploy`` utility generates the data directory and the keyring for -you--placing the keyring in ``/var/lib/ceph/radosgw/{rgw-instance}``. The daemon -looks in default locations, whereas you may have specified different settings -in your Ceph configuration file. Since you already have keys and a data -directory, you will want to maintain those paths in your Ceph configuration -file if you used something other than default paths. - -A typical Ceph Object Gateway configuration file for an Apache-based deployment -looks something similar as the following: - -On Red Hat Enterprise Linux:: - - [client.radosgw.gateway-node1] - host = {hostname} - keyring = /etc/ceph/ceph.client.radosgw.keyring - rgw socket path = "" - log file = /var/log/radosgw/client.radosgw.gateway-node1.log - rgw frontends = fastcgi socket\_port=9000 socket\_host=0.0.0.0 - rgw print continue = false - -On Ubuntu:: - - [client.radosgw.gateway-node] - host = {hostname} - keyring = /etc/ceph/ceph.client.radosgw.keyring - rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock - log file = /var/log/radosgw/client.radosgw.gateway-node1.log - -To modify it for use with Civetweb, simply remove the Apache-specific settings -such as ``rgw_socket_path`` and ``rgw_print_continue``. Then, change the -``rgw_frontends`` setting to reflect Civetweb rather than the Apache FastCGI -front end and specify the port number you intend to use. For example:: - - [client.radosgw.gateway-node1] - host = {hostname} - keyring = /etc/ceph/ceph.client.radosgw.keyring - log file = /var/log/radosgw/client.radosgw.gateway-node1.log - rgw_frontends = civetweb port=80 - -Finally, restart the Ceph Object Gateway. On Red Hat Enterprise Linux execute:: - - sudo systemctl restart ceph-radosgw.service - -On Ubuntu execute:: - - sudo service radosgw restart id=rgw. - -If you used a port number that is not open, you will also need to open that -port on your firewall. - -Configure Bucket Sharding -------------------------- - -A Ceph Object Gateway stores bucket index data in the ``index_pool``, which -defaults to ``.rgw.buckets.index``. Sometimes users like to put many objects -(hundreds of thousands to millions of objects) in a single bucket. If you do -not use the gateway administration interface to set quotas for the maximum -number of objects per bucket, the bucket index can suffer significant -performance degradation when users place large numbers of objects into a -bucket. - -In Ceph 0.94, you may shard bucket indices to help prevent performance -bottlenecks when you allow a high number of objects per bucket. The -``rgw_override_bucket_index_max_shards`` setting allows you to set a maximum -number of shards per bucket. The default value is ``0``, which means bucket -index sharding is off by default. - -To turn bucket index sharding on, set ``rgw_override_bucket_index_max_shards`` -to a value greater than ``0``. - -For simple configurations, you may add ``rgw_override_bucket_index_max_shards`` -to your Ceph configuration file. Add it under ``[global]`` to create a -system-wide value. You can also set it for each instance in your Ceph -configuration file. - -Once you have changed your bucket sharding configuration in your Ceph -configuration file, restart your gateway. On Red Hat Enterprise Linux execute:: - - sudo systemctl restart ceph-radosgw.service - -On Ubuntu execute:: - - sudo service radosgw restart id=rgw. - -For federated configurations, each zone may have a different ``index_pool`` -setting for failover. To make the value consistent for a zonegroup's zones, you -may set ``rgw_override_bucket_index_max_shards`` in a gateway's zonegroup -configuration. For example:: - - radosgw-admin zonegroup get > zonegroup.json - -Open the ``zonegroup.json`` file and edit the ``bucket_index_max_shards`` setting -for each named zone. Save the ``zonegroup.json`` file and reset the zonegroup. -For example:: - - radosgw-admin zonegroup set < zonegroup.json - -Once you have updated your zonegroup, update and commit the period. -For example:: - - radosgw-admin period update --commit - -.. note:: Mapping the index pool (for each zone, if applicable) to a CRUSH - rule of SSD-based OSDs may also help with bucket index performance. - -Add Wildcard to DNS -------------------- -.. _Add Wildcard to DNS: - -To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you -need to add a wildcard to the DNS record of the DNS server you use with the -``ceph-radosgw`` daemon. - -The address of the DNS must also be specified in the Ceph configuration file -with the ``rgw dns name = {hostname}`` setting. - -For ``dnsmasq``, add the following address setting with a dot (.) prepended to -the host name:: - - address=/.{hostname-or-fqdn}/{host-ip-address} - -For example:: - - address=/.gateway-node1/192.168.122.75 - - -For ``bind``, add a wildcard to the DNS record. For example:: - - $TTL 604800 - @ IN SOA gateway-node1. root.gateway-node1. ( - 2 ; Serial - 604800 ; Refresh - 86400 ; Retry - 2419200 ; Expire - 604800 ) ; Negative Cache TTL - ; - @ IN NS gateway-node1. - @ IN A 192.168.122.113 - * IN CNAME @ - -Restart your DNS server and ping your server with a subdomain to ensure that -your DNS configuration works as expected:: - - ping mybucket.{hostname} - -For example:: - - ping mybucket.gateway-node1 - -Add Debugging (if needed) -------------------------- - -Once you finish the setup procedure, if you encounter issues with your -configuration, you can add debugging to the ``[global]`` section of your Ceph -configuration file and restart the gateway(s) to help troubleshoot any -configuration issues. For example:: - - [global] - #append the following in the global section. - debug ms = 1 - debug rgw = 20 - -Using the Gateway ------------------ - -To use the REST interfaces, first create an initial Ceph Object Gateway user -for the S3 interface. Then, create a subuser for the Swift interface. You then -need to verify if the created users are able to access the gateway. - -Create a RADOSGW User for S3 Access -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -A ``radosgw`` user needs to be created and granted access. The command ``man -radosgw-admin`` will provide information on additional command options. - -To create the user, execute the following on the ``gateway host``:: - - sudo radosgw-admin user create --uid="testuser" --display-name="First User" - -The output of the command will be something like the following:: - - { - "user_id": "testuser", - "display_name": "First User", - "email": "", - "suspended": 0, - "max_buckets": 1000, - "subusers": [], - "keys": [{ - "user": "testuser", - "access_key": "I0PJDPCIYZ665MW88W9R", - "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" - }], - "swift_keys": [], - "caps": [], - "op_mask": "read, write, delete", - "default_placement": "", - "placement_tags": [], - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "temp_url_keys": [] - } - -.. note:: The values of ``keys->access_key`` and ``keys->secret_key`` are - needed for access validation. - -.. important:: Check the key output. Sometimes ``radosgw-admin`` generates a - JSON escape character ``\`` in ``access_key`` or ``secret_key`` - and some clients do not know how to handle JSON escape - characters. Remedies include removing the JSON escape character - ``\``, encapsulating the string in quotes, regenerating the key - and ensuring that it does not have a JSON escape character or - specify the key and secret manually. Also, if ``radosgw-admin`` - generates a JSON escape character ``\`` and a forward slash ``/`` - together in a key, like ``\/``, only remove the JSON escape - character ``\``. Do not remove the forward slash ``/`` as it is - a valid character in the key. - -Create a Swift User -^^^^^^^^^^^^^^^^^^^ - -A Swift subuser needs to be created if this kind of access is needed. Creating -a Swift user is a two step process. The first step is to create the user. The -second is to create the secret key. - -Execute the following steps on the ``gateway host``: - -Create the Swift user:: - - sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full - -The output will be something like the following:: - - { - "user_id": "testuser", - "display_name": "First User", - "email": "", - "suspended": 0, - "max_buckets": 1000, - "subusers": [{ - "id": "testuser:swift", - "permissions": "full-control" - }], - "keys": [{ - "user": "testuser:swift", - "access_key": "3Y1LNW4Q6X0Y53A52DET", - "secret_key": "" - }, { - "user": "testuser", - "access_key": "I0PJDPCIYZ665MW88W9R", - "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" - }], - "swift_keys": [], - "caps": [], - "op_mask": "read, write, delete", - "default_placement": "", - "placement_tags": [], - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "temp_url_keys": [] - } - -Create the secret key:: - - sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret - -The output will be something like the following:: - - { - "user_id": "testuser", - "display_name": "First User", - "email": "", - "suspended": 0, - "max_buckets": 1000, - "subusers": [{ - "id": "testuser:swift", - "permissions": "full-control" - }], - "keys": [{ - "user": "testuser:swift", - "access_key": "3Y1LNW4Q6X0Y53A52DET", - "secret_key": "" - }, { - "user": "testuser", - "access_key": "I0PJDPCIYZ665MW88W9R", - "secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA" - }], - "swift_keys": [{ - "user": "testuser:swift", - "secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA" - }], - "caps": [], - "op_mask": "read, write, delete", - "default_placement": "", - "placement_tags": [], - "bucket_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "user_quota": { - "enabled": false, - "max_size_kb": -1, - "max_objects": -1 - }, - "temp_url_keys": [] - } - -Access Verification -^^^^^^^^^^^^^^^^^^^ - -Test S3 Access -"""""""""""""" - -You need to write and run a Python test script for verifying S3 access. The S3 -access test script will connect to the ``radosgw``, create a new bucket and -list all buckets. The values for ``aws_access_key_id`` and -``aws_secret_access_key`` are taken from the values of ``access_key`` and -``secret_key`` returned by the ``radosgw-admin`` command. - -Execute the following steps: - -#. You will need to install the ``python-boto`` package:: - - sudo yum install python-boto - -#. Create the Python script:: - - vi s3test.py - -#. Add the following contents to the file:: - - import boto.s3.connection - - access_key = 'I0PJDPCIYZ665MW88W9R' - secret_key = 'dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA' - conn = boto.connect_s3( - aws_access_key_id=access_key, - aws_secret_access_key=secret_key, - host='{hostname}', port={port}, - is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(), - ) - - bucket = conn.create_bucket('my-new-bucket') - for bucket in conn.get_all_buckets(): - print "{name} {created}".format( - name=bucket.name, - created=bucket.creation_date, - ) - - - Replace ``{hostname}`` with the hostname of the host where you have - configured the gateway service i.e., the ``gateway host``. Replace ``{port}`` - with the port number you are using with Civetweb. - -#. Run the script:: - - python s3test.py - - The output will be something like the following:: - - my-new-bucket 2015-02-16T17:09:10.000Z - -Test swift access -""""""""""""""""" - -Swift access can be verified via the ``swift`` command line client. The command -``man swift`` will provide more information on available command line options. - -To install ``swift`` client, execute the following commands. On Red Hat -Enterprise Linux:: - - sudo yum install python-setuptools - sudo easy_install pip - sudo pip install --upgrade setuptools - sudo pip install --upgrade python-swiftclient - -On Debian-based distributions:: - - sudo apt-get install python-setuptools - sudo easy_install pip - sudo pip install --upgrade setuptools - sudo pip install --upgrade python-swiftclient - -To test swift access, execute the following:: - - swift -V 1 -A http://{IP ADDRESS}:{port}/auth -U testuser:swift -K '{swift_secret_key}' list - -Replace ``{IP ADDRESS}`` with the public IP address of the gateway server and -``{swift_secret_key}`` with its value from the output of ``radosgw-admin key -create`` command executed for the ``swift`` user. Replace {port} with the port -number you are using with Civetweb (e.g., ``7480`` is the default). If you -don't replace the port, it will default to port ``80``. - -For example:: - - swift -V 1 -A http://10.19.143.116:7480/auth -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list - -The output should be:: - - my-new-bucket - -.. _Preflight: ../../start/quick-start-preflight -.. _HTTP Frontends: ../../radosgw/frontends diff --git a/doc/install/upgrading-ceph.rst b/doc/install/upgrading-ceph.rst deleted file mode 100644 index 6fbf43a236dc..000000000000 --- a/doc/install/upgrading-ceph.rst +++ /dev/null @@ -1,235 +0,0 @@ -================ - Upgrading Ceph -================ - -Each release of Ceph may have additional steps. Refer to the `release notes -document of your release`_ to identify release-specific procedures for your -cluster before using the upgrade procedures. - - -Summary -======= - -You can upgrade daemons in your Ceph cluster while the cluster is online and in -service! Certain types of daemons depend upon others. For example, Ceph Metadata -Servers and Ceph Object Gateways depend upon Ceph Monitors and Ceph OSD Daemons. -We recommend upgrading in this order: - -#. `Ceph Deploy`_ -#. Ceph Monitors -#. Ceph OSD Daemons -#. Ceph Metadata Servers -#. Ceph Object Gateways - -As a general rule, we recommend upgrading all the daemons of a specific type -(e.g., all ``ceph-mon`` daemons, all ``ceph-osd`` daemons, etc.) to ensure that -they are all on the same release. We also recommend that you upgrade all the -daemons in your cluster before you try to exercise new functionality in a -release. - -The `Upgrade Procedures`_ are relatively simple, but do look at the `release -notes document of your release`_ before upgrading. The basic process involves -three steps: - -#. Use ``ceph-deploy`` on your admin node to upgrade the packages for - multiple hosts (using the ``ceph-deploy install`` command), or login to each - host and upgrade the Ceph package `using your distro's package manager`_. - For example, when `Upgrading Monitors`_, the ``ceph-deploy`` syntax might - look like this:: - - ceph-deploy install --release {release-name} ceph-node1[ ceph-node2] - ceph-deploy install --release firefly mon1 mon2 mon3 - - **Note:** The ``ceph-deploy install`` command will upgrade the packages - in the specified node(s) from the old release to the release you specify. - There is no ``ceph-deploy upgrade`` command. - -#. Login in to each Ceph node and restart each Ceph daemon. - See `Operating a Cluster`_ for details. - -#. Ensure your cluster is healthy. See `Monitoring a Cluster`_ for details. - -.. important:: Once you upgrade a daemon, you cannot downgrade it. - - -Ceph Deploy -=========== - -Before upgrading Ceph daemons, upgrade the ``ceph-deploy`` tool. :: - - sudo pip install -U ceph-deploy - -Or:: - - sudo apt-get install ceph-deploy - -Or:: - - sudo yum install ceph-deploy python-pushy - - -Upgrade Procedures -================== - -The following sections describe the upgrade process. - -.. important:: Each release of Ceph may have some additional steps. Refer to - the `release notes document of your release`_ for details **BEFORE** you - begin upgrading daemons. - - -Upgrading Monitors ------------------- - -To upgrade monitors, perform the following steps: - -#. Upgrade the Ceph package for each daemon instance. - - You may use ``ceph-deploy`` to address all monitor nodes at once. - For example:: - - ceph-deploy install --release {release-name} ceph-node1[ ceph-node2] - ceph-deploy install --release hammer mon1 mon2 mon3 - - You may also use the package manager for your Linux distribution on - each individual node. To upgrade packages manually on each Debian/Ubuntu - host, perform the following steps:: - - ssh {mon-host} - sudo apt-get update && sudo apt-get install ceph - - On CentOS/Red Hat hosts, perform the following steps:: - - ssh {mon-host} - sudo yum update && sudo yum install ceph - - -#. Restart each monitor. For Ubuntu distributions, use:: - - sudo systemctl restart ceph-mon@{hostname}.service - - For CentOS/Red Hat/Debian distributions, use:: - - sudo /etc/init.d/ceph restart {mon-id} - - For CentOS/Red Hat distributions deployed with ``ceph-deploy``, - the monitor ID is usually ``mon.{hostname}``. - -#. Ensure each monitor has rejoined the quorum:: - - ceph mon stat - -Ensure that you have completed the upgrade cycle for all of your Ceph Monitors. - - -Upgrading an OSD ----------------- - -To upgrade a Ceph OSD Daemon, perform the following steps: - -#. Upgrade the Ceph OSD Daemon package. - - You may use ``ceph-deploy`` to address all Ceph OSD Daemon nodes at - once. For example:: - - ceph-deploy install --release {release-name} ceph-node1[ ceph-node2] - ceph-deploy install --release hammer osd1 osd2 osd3 - - You may also use the package manager on each node to upgrade packages - `using your distro's package manager`_. For Debian/Ubuntu hosts, perform the - following steps on each host:: - - ssh {osd-host} - sudo apt-get update && sudo apt-get install ceph - - For CentOS/Red Hat hosts, perform the following steps:: - - ssh {osd-host} - sudo yum update && sudo yum install ceph - - -#. Restart the OSD, where ``N`` is the OSD number. For Ubuntu, use:: - - sudo systemctl restart ceph-osd@{N}.service - - For multiple OSDs on a host, you may restart all of them with systemd. :: - - sudo systemctl restart ceph-osd - - For CentOS/Red Hat/Debian distributions, use:: - - sudo /etc/init.d/ceph restart N - - -#. Ensure each upgraded Ceph OSD Daemon has rejoined the cluster:: - - ceph osd stat - -Ensure that you have completed the upgrade cycle for all of your -Ceph OSD Daemons. - - -Upgrading a Metadata Server ---------------------------- - -To upgrade a Ceph Metadata Server, perform the following steps: - -#. Upgrade the Ceph Metadata Server package. You may use ``ceph-deploy`` to - address all Ceph Metadata Server nodes at once, or use the package manager - on each node. For example:: - - ceph-deploy install --release {release-name} ceph-node1 - ceph-deploy install --release hammer mds1 - - To upgrade packages manually, perform the following steps on each - Debian/Ubuntu host:: - - ssh {mon-host} - sudo apt-get update && sudo apt-get install ceph-mds - - Or the following steps on CentOS/Red Hat hosts:: - - ssh {mon-host} - sudo yum update && sudo yum install ceph-mds - - -#. Restart the metadata server. For Ubuntu, use:: - - sudo systemctl restart ceph-mds@{hostname}.service - - For CentOS/Red Hat/Debian distributions, use:: - - sudo /etc/init.d/ceph restart mds.{hostname} - - For clusters deployed with ``ceph-deploy``, the name is usually either - the name you specified on creation or the hostname. - -#. Ensure the metadata server is up and running:: - - ceph mds stat - - -Upgrading a Client ------------------- - -Once you have upgraded the packages and restarted daemons on your Ceph -cluster, we recommend upgrading ``ceph-common`` and client libraries -(``librbd1`` and ``librados2``) on your client nodes too. - -#. Upgrade the package:: - - ssh {client-host} - apt-get update && sudo apt-get install ceph-common librados2 librbd1 python-rados python-rbd - -#. Ensure that you have the latest version:: - - ceph --version - -If you do not have the latest version, you may need to uninstall, auto remove -dependencies and reinstall. - - -.. _using your distro's package manager: ../install-storage-cluster/ -.. _Operating a Cluster: ../../rados/operations/operating -.. _Monitoring a Cluster: ../../rados/operations/monitoring -.. _release notes document of your release: ../../releases diff --git a/doc/radosgw/index.rst b/doc/radosgw/index.rst index bb7280f7009a..957c3641d639 100644 --- a/doc/radosgw/index.rst +++ b/doc/radosgw/index.rst @@ -38,7 +38,7 @@ you may write data with one API and retrieve it with the other. .. toctree:: :maxdepth: 1 - Manual Install w/Civetweb <../../install/install-ceph-gateway> + Manual Install w/Civetweb <../../install/ceph-deploy/install-ceph-gateway> HTTP Frontends Pool Placement and Storage Classes Multisite Configuration diff --git a/doc/start/index.rst b/doc/start/index.rst deleted file mode 100644 index 40143214dfb0..000000000000 --- a/doc/start/index.rst +++ /dev/null @@ -1,49 +0,0 @@ -.. _ceph-deploy-index: - -============================ - Installation (ceph-deploy) -============================ - -.. raw:: html - - -

Step 1: Preflight

- -A :term:`Ceph Client` and a :term:`Ceph Node` may require some basic -configuration work prior to deploying a Ceph Storage Cluster. You can also -avail yourself of help by getting involved in the Ceph community. - -.. toctree:: - - Preflight - -.. raw:: html - -

Step 2: Storage Cluster

- -Once you have completed your preflight checklist, you should be able to begin -deploying a Ceph Storage Cluster. - -.. toctree:: - - Storage Cluster Quick Start - - -.. raw:: html - -

Step 3: Ceph Client(s)

- -Most Ceph users don't store objects directly in the Ceph Storage Cluster. They typically use at least one of -Ceph Block Devices, the Ceph File System, and Ceph Object Storage. - -.. toctree:: - - Block Device Quick Start - Filesystem Quick Start - Object Storage Quick Start - -.. raw:: html - -
- - diff --git a/doc/start/quick-ceph-deploy.rst b/doc/start/quick-ceph-deploy.rst deleted file mode 100644 index c4589c7b3d3c..000000000000 --- a/doc/start/quick-ceph-deploy.rst +++ /dev/null @@ -1,346 +0,0 @@ -============================= - Storage Cluster Quick Start -============================= - -If you haven't completed your `Preflight Checklist`_, do that first. This -**Quick Start** sets up a :term:`Ceph Storage Cluster` using ``ceph-deploy`` -on your admin node. Create a three Ceph Node cluster so you can -explore Ceph functionality. - -.. include:: quick-common.rst - -As a first exercise, create a Ceph Storage Cluster with one Ceph Monitor and three -Ceph OSD Daemons. Once the cluster reaches a ``active + clean`` state, expand it -by adding a fourth Ceph OSD Daemon, and two more Ceph Monitors. -For best results, create a directory on your admin node for maintaining the -configuration files and keys that ``ceph-deploy`` generates for your cluster. :: - - mkdir my-cluster - cd my-cluster - -The ``ceph-deploy`` utility will output files to the current directory. Ensure you -are in this directory when executing ``ceph-deploy``. - -.. important:: Do not call ``ceph-deploy`` with ``sudo`` or run it as ``root`` - if you are logged in as a different user, because it will not issue ``sudo`` - commands needed on the remote host. - - -Starting over -============= - -If at any point you run into trouble and you want to start over, execute -the following to purge the Ceph packages, and erase all its data and configuration:: - - ceph-deploy purge {ceph-node} [{ceph-node}] - ceph-deploy purgedata {ceph-node} [{ceph-node}] - ceph-deploy forgetkeys - rm ceph.* - -If you execute ``purge``, you must re-install Ceph. The last ``rm`` -command removes any files that were written out by ceph-deploy locally -during a previous installation. - - -Create a Cluster -================ - -On your admin node from the directory you created for holding your -configuration details, perform the following steps using ``ceph-deploy``. - -#. Create the cluster. :: - - ceph-deploy new {initial-monitor-node(s)} - - Specify node(s) as hostname, fqdn or hostname:fqdn. For example:: - - ceph-deploy new node1 - - Check the output of ``ceph-deploy`` with ``ls`` and ``cat`` in the - current directory. You should see a Ceph configuration file - (``ceph.conf``), a monitor secret keyring (``ceph.mon.keyring``), - and a log file for the new cluster. See `ceph-deploy new -h`_ for - additional details. - - Note for users of Ubuntu 18.04: Python 2 is a prerequisite of Ceph. - Install the ``python-minimal`` package on Ubuntu 18.04 to provide - Python 2:: - - [Ubuntu 18.04] $ sudo apt install python-minimal - -#. If you have more than one network interface, add the ``public network`` - setting under the ``[global]`` section of your Ceph configuration file. - See the `Network Configuration Reference`_ for details. :: - - public network = {ip-address}/{bits} - - For example,:: - - public network = 10.1.2.0/24 - - to use IPs in the 10.1.2.0/24 (or 10.1.2.0/255.255.255.0) network. - -#. If you are deploying in an IPv6 environment, add the following to - ``ceph.conf`` in the local directory:: - - echo ms bind ipv6 = true >> ceph.conf - -#. Install Ceph packages.:: - - ceph-deploy install {ceph-node} [...] - - For example:: - - ceph-deploy install node1 node2 node3 - - The ``ceph-deploy`` utility will install Ceph on each node. - -#. Deploy the initial monitor(s) and gather the keys:: - - ceph-deploy mon create-initial - - Once you complete the process, your local directory should have the following - keyrings: - - - ``ceph.client.admin.keyring`` - - ``ceph.bootstrap-mgr.keyring`` - - ``ceph.bootstrap-osd.keyring`` - - ``ceph.bootstrap-mds.keyring`` - - ``ceph.bootstrap-rgw.keyring`` - - ``ceph.bootstrap-rbd.keyring`` - - ``ceph.bootstrap-rbd-mirror.keyring`` - - .. note:: If this process fails with a message similar to "Unable to - find /etc/ceph/ceph.client.admin.keyring", please ensure that the - IP listed for the monitor node in ceph.conf is the Public IP, not - the Private IP. - -#. Use ``ceph-deploy`` to copy the configuration file and admin key to - your admin node and your Ceph Nodes so that you can use the ``ceph`` - CLI without having to specify the monitor address and - ``ceph.client.admin.keyring`` each time you execute a command. :: - - ceph-deploy admin {ceph-node(s)} - - For example:: - - ceph-deploy admin node1 node2 node3 - -#. Deploy a manager daemon. (Required only for luminous+ builds):: - - ceph-deploy mgr create node1 *Required only for luminous+ builds, i.e >= 12.x builds* - -#. Add three OSDs. For the purposes of these instructions, we assume you have an - unused disk in each node called ``/dev/vdb``. *Be sure that the device is not currently in use and does not contain any important data.* :: - - ceph-deploy osd create --data {device} {ceph-node} - - For example:: - - ceph-deploy osd create --data /dev/vdb node1 - ceph-deploy osd create --data /dev/vdb node2 - ceph-deploy osd create --data /dev/vdb node3 - - .. note:: If you are creating an OSD on an LVM volume, the argument to - ``--data`` *must* be ``volume_group/lv_name``, rather than the path to - the volume's block device. - -#. Check your cluster's health. :: - - ssh node1 sudo ceph health - - Your cluster should report ``HEALTH_OK``. You can view a more complete - cluster status with:: - - ssh node1 sudo ceph -s - - -Expanding Your Cluster -====================== - -Once you have a basic cluster up and running, the next step is to expand -cluster. Then add a Ceph Monitor and Ceph Manager to ``node2`` and ``node3`` -to improve reliability and availability. - -.. ditaa:: - /------------------\ /----------------\ - | ceph-deploy | | node1 | - | Admin Node | | cCCC | - | +-------->+ | - | | | mon.node1 | - | | | osd.0 | - | | | mgr.node1 | - \---------+--------/ \----------------/ - | - | /----------------\ - | | node2 | - | | cCCC | - +----------------->+ | - | | osd.1 | - | | mon.node2 | - | \----------------/ - | - | /----------------\ - | | node3 | - | | cCCC | - +----------------->+ | - | osd.2 | - | mon.node3 | - \----------------/ - -Adding Monitors ---------------- - -A Ceph Storage Cluster requires at least one Ceph Monitor and Ceph -Manager to run. For high availability, Ceph Storage Clusters typically -run multiple Ceph Monitors so that the failure of a single Ceph -Monitor will not bring down the Ceph Storage Cluster. Ceph uses the -Paxos algorithm, which requires a majority of monitors (i.e., greater -than *N/2* where *N* is the number of monitors) to form a quorum. -Odd numbers of monitors tend to be better, although this is not required. - -.. tip: If you did not define the ``public network`` option above then - the new monitor will not know which IP address to bind to on the - new hosts. You can add this line to your ``ceph.conf`` by editing - it now and then push it out to each node with - ``ceph-deploy --overwrite-conf config push {ceph-nodes}``. - -Add two Ceph Monitors to your cluster:: - - ceph-deploy mon add {ceph-nodes} - -For example:: - - ceph-deploy mon add node2 node3 - -Once you have added your new Ceph Monitors, Ceph will begin synchronizing -the monitors and form a quorum. You can check the quorum status by executing -the following:: - - ceph quorum_status --format json-pretty - - -.. tip:: When you run Ceph with multiple monitors, you SHOULD install and - configure NTP on each monitor host. Ensure that the - monitors are NTP peers. - -Adding Managers ---------------- - -The Ceph Manager daemons operate in an active/standby pattern. Deploying -additional manager daemons ensures that if one daemon or host fails, another -one can take over without interrupting service. - -To deploy additional manager daemons:: - - ceph-deploy mgr create node2 node3 - -You should see the standby managers in the output from:: - - ssh node1 sudo ceph -s - - -Add an RGW Instance -------------------- - -To use the :term:`Ceph Object Gateway` component of Ceph, you must deploy an -instance of :term:`RGW`. Execute the following to create an new instance of -RGW:: - - ceph-deploy rgw create {gateway-node} - -For example:: - - ceph-deploy rgw create node1 - -By default, the :term:`RGW` instance will listen on port 7480. This can be -changed by editing ceph.conf on the node running the :term:`RGW` as follows: - -.. code-block:: ini - - [client] - rgw frontends = civetweb port=80 - -To use an IPv6 address, use: - -.. code-block:: ini - - [client] - rgw frontends = civetweb port=[::]:80 - - - -Storing/Retrieving Object Data -============================== - -To store object data in the Ceph Storage Cluster, a Ceph client must: - -#. Set an object name -#. Specify a `pool`_ - -The Ceph Client retrieves the latest cluster map and the CRUSH algorithm -calculates how to map the object to a `placement group`_, and then calculates -how to assign the placement group to a Ceph OSD Daemon dynamically. To find the -object location, all you need is the object name and the pool name. For -example:: - - ceph osd map {poolname} {object-name} - -.. topic:: Exercise: Locate an Object - - As an exercise, lets create an object. Specify an object name, a path to - a test file containing some object data and a pool name using the - ``rados put`` command on the command line. For example:: - - echo {Test-data} > testfile.txt - ceph osd pool create mytest - rados put {object-name} {file-path} --pool=mytest - rados put test-object-1 testfile.txt --pool=mytest - - To verify that the Ceph Storage Cluster stored the object, execute - the following:: - - rados -p mytest ls - - Now, identify the object location:: - - ceph osd map {pool-name} {object-name} - ceph osd map mytest test-object-1 - - Ceph should output the object's location. For example:: - - osdmap e537 pool 'mytest' (1) object 'test-object-1' -> pg 1.d1743484 (1.4) -> up [1,0] acting [1,0] - - To remove the test object, simply delete it using the ``rados rm`` - command. - - For example:: - - rados rm test-object-1 --pool=mytest - - To delete the ``mytest`` pool:: - - ceph osd pool rm mytest - - (For safety reasons you will need to supply additional arguments as - prompted; deleting pools destroys data.) - -As the cluster evolves, the object location may change dynamically. One benefit -of Ceph's dynamic rebalancing is that Ceph relieves you from having to perform -data migration or balancing manually. - - -.. _Preflight Checklist: ../quick-start-preflight -.. _Ceph Deploy: ../../rados/deployment -.. _ceph-deploy install -h: ../../rados/deployment/ceph-deploy-install -.. _ceph-deploy new -h: ../../rados/deployment/ceph-deploy-new -.. _ceph-deploy osd: ../../rados/deployment/ceph-deploy-osd -.. _Running Ceph with Upstart: ../../rados/operations/operating#running-ceph-with-upstart -.. _Running Ceph with sysvinit: ../../rados/operations/operating#running-ceph-with-sysvinit -.. _CRUSH Map: ../../rados/operations/crush-map -.. _pool: ../../rados/operations/pools -.. _placement group: ../../rados/operations/placement-groups -.. _Monitoring a Cluster: ../../rados/operations/monitoring -.. _Monitoring OSDs and PGs: ../../rados/operations/monitoring-osd-pg -.. _Network Configuration Reference: ../../rados/configuration/network-config-ref -.. _User Management: ../../rados/operations/user-management diff --git a/doc/start/quick-cephfs.rst b/doc/start/quick-cephfs.rst deleted file mode 100644 index e8ca28f86ee1..000000000000 --- a/doc/start/quick-cephfs.rst +++ /dev/null @@ -1,212 +0,0 @@ -=================== - CephFS Quick Start -=================== - -To use the :term:`CephFS` Quick Start guide, you must have executed the -procedures in the `Storage Cluster Quick Start`_ guide first. Execute this -quick start on the admin host. - -Prerequisites -============= - -#. Verify that you have an appropriate version of the Linux kernel. - See `OS Recommendations`_ for details. :: - - lsb_release -a - uname -r - -#. On the admin node, use ``ceph-deploy`` to install Ceph on your - ``ceph-client`` node. :: - - ceph-deploy install ceph-client - -#. Optionally, if you want a FUSE-mounted file system, you would need to - install ``ceph-fuse`` package as well. - -#. Ensure that the :term:`Ceph Storage Cluster` is running and in an ``active + - clean`` state. :: - - ceph -s [-m {monitor-ip-address}] [-k {path/to/ceph.client.admin.keyring}] - - -Deploy Metadata Server -====================== - -All metadata operations in CephFS happen via a metadata server, so you need at -least one metadata server. Execute the following to create a metadata server:: - - ceph-deploy mds create {ceph-node} - -For example:: - - ceph-deploy mds create node1 - -Now, your Ceph cluster would look like this: - -.. ditaa:: - /------------------\ /----------------\ - | ceph-deploy | | node1 | - | Admin Node | | cCCC | - | +-------->+ mon.node1 | - | | | osd.0 | - | | | mgr.node1 | - | | | mds.node1 | - \---------+--------/ \----------------/ - | - | /----------------\ - | | node2 | - | | cCCC | - +----------------->+ | - | | osd.1 | - | | mon.node2 | - | \----------------/ - | - | /----------------\ - | | node3 | - | | cCCC | - +----------------->+ | - | osd.2 | - | mon.node3 | - \----------------/ - -Create a File System -==================== - -You have already created an MDS (`Storage Cluster Quick Start`_) but it will not -become active until you create some pools and a file system. See -:doc:`/cephfs/createfs`. :: - - ceph osd pool create cephfs_data 32 - ceph osd pool create cephfs_meta 32 - ceph fs new mycephfs cephfs_meta cephfs_data - -.. note:: In case you have multiple Ceph applications and/or have multiple - CephFSs on the same cluster, it would be easier to name your pools as - ... In that case, the above pools would - be named as cephfs.mycehfs.data and cephfs.mycehfs.meta. - -Quick word about Pools and PGs ------------------------------- - -Replication Number/Pool Size -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Since the default replication number/size is 3, you'd need 3 OSDs to get -``active+clean`` for all PGs. Alternatively, you may change the replication -number for the pool to match the number of OSDs:: - - ceph osd pool set cephfs_data size {number-of-osds} - ceph osd pool set cephfs_meta size {number-of-osds} - -Usually, setting ``pg_num`` to 32 gives a perfectly healthy cluster. To pick -appropriate value for ``pg_num``, refer `Placement Group`_. You can also use -pg_autoscaler plugin instead. Introduced by Nautilus release, it can -automatically increase/decrease value of ``pg_num``; refer the -`Placement Group`_ to find out more about it. - -When all OSDs are on the same node... -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -And, in case you have deployed all of the OSDs on the same node, you would need -to create a new CRUSH rule to replicate data across OSDs and set the rule on the -CephFS pools, since the default CRUSH rule is to replicate data across -different nodes:: - - ceph osd crush rule create-replicated rule_foo default osd - ceph osd pool set cephfs_data crush_rule rule_foo - ceph osd pool set cephfs_meta crush_rule rule_foo - -Using Erasure Coded pools -^^^^^^^^^^^^^^^^^^^^^^^^^ -You may also use Erasure Coded pools which can be more effecient and -cost-saving since they allow stripping object data across OSDs and -replicating these stripes with encoded redundancy information. The number -of OSDs across which the data is stripped is `k` and number of replica is `m`. -You'll need to pick up these values before creating CephFS pools. The -following commands create a erasure code profile, creates a pool that'll -use it and then enables it on the pool:: - - ceph osd erasure-code-profile set ec-42-profile k=4 m=2 crush-failure-domain=host crush-device-class=ssd - ceph osd pool create cephfs_data_ec42 64 erasure ec-42-profile - ceph osd pool set cephfs_data_ec42 allow_ec_overwrites true - ceph fs add_data_pool mycephfs cephfs_data_ec42 - -You can also mark directories so that they are only stored on certain pools:: - - setfattr -n ceph.dir.layout -v pool=cephfs_data_ec42 /mnt/mycephfs/logs - -This way you can choose the replication strategy for each directory on your -Ceph file system. - -.. note:: Erasure Coded pools can not be used for CephFS metadata pools. - -Erasure coded pool were introduced in Firefly and could be used directly by -CephFS Luminous onwards. Refer `this article `_ -by Sage Weil to understand EC, it's background, limitations and other details -in Ceph's context. Read more about `Erasure Code`_ here. - -Mounting the File System -======================== - -Using Kernel Driver -------------------- - -The command to mount CephFS using kernel driver looks like this:: - - sudo mount -t ceph :{path-to-mounted} {mount-point} -o name={user-name} - sudo mount -t ceph :/ /mnt/mycephfs -o name=admin # usable version - -``{path-to-be-mounted}`` is the path within CephFS that will be mounted, -``{mount-point}`` is the point in your file system upon which CephFS will be -mounted and ``{user-name}`` is the name of CephX user that has the -authorization to mount CephFS on the machine. Following command is the -extended form, however these extra details are automatically figured out by -by the mount.ceph helper program:: - - sudo mount -t ceph {ip-address-of-MON}:{port-number-of-MON}:{path-to-be-mounted} -o name={user-name},secret={secret-key} {mount-point} - -If you have multiple file systems on your cluster you would need to pass -``fs={fs-name}`` option to ``-o`` option to the ``mount`` command:: - - sudo mount -t ceph :/ /mnt/kcephfs2 -o name=admin,fs=mycephfs2 - -Refer `mount.ceph man page`_ and `Mount CephFS using Kernel Driver`_ to read -more about this. - - -Using FUSE ----------- - -To mount CephFS using FUSE (Filesystem in User Space) run:: - - sudo ceph-fuse /mnt/mycephfs - -To mount a particular directory within CephFS you can use ``-r``:: - - sudo ceph-fuse -r {path-to-be-mounted} /mnt/mycephfs - -If you have multiple file systems on your cluster you would need to pass -``--client_fs {fs-name}`` to the ``ceph-fuse`` command:: - - sudo ceph-fuse /mnt/mycephfs2 --client_fs mycephfs2 - -Refer `ceph-fuse man page`_ and `Mount CephFS using FUSE`_ to read more about -this. - -.. note:: Mount the CephFS file system on the admin node, not the server node. - - -Additional Information -====================== - -See `CephFS`_ for additional information. See `Troubleshooting`_ if you -encounter trouble. - -.. _Storage Cluster Quick Start: ../quick-ceph-deploy -.. _CephFS: ../../cephfs/ -.. _Troubleshooting: ../../cephfs/troubleshooting -.. _OS Recommendations: ../os-recommendations -.. _Placement Group: ../../rados/operations/placement-groups -.. _mount.ceph man page: ../../man/8/mount.ceph -.. _Mount CephFS using Kernel Driver: ../cephfs/kernel -.. _ceph-fuse man page: ../../man/8/ceph-fuse -.. _Mount CephFS using FUSE: ../../cephfs/fuse -.. _Erasure Code: ../../rados/operations/erasure-code diff --git a/doc/start/quick-common.rst b/doc/start/quick-common.rst deleted file mode 100644 index 915a7b886422..000000000000 --- a/doc/start/quick-common.rst +++ /dev/null @@ -1,20 +0,0 @@ -.. ditaa:: - /------------------\ /-----------------\ - | admin-node | | node1 | - | +-------->+ cCCC | - | ceph-deploy | | mon.node1 | - | | | osd.0 | - \---------+--------/ \-----------------/ - | - | /----------------\ - | | node2 | - +----------------->+ cCCC | - | | osd.1 | - | \----------------/ - | - | /----------------\ - | | node3 | - +----------------->| cCCC | - | osd.2 | - \----------------/ - diff --git a/doc/start/quick-rgw-old.rst b/doc/start/quick-rgw-old.rst deleted file mode 100644 index db6474de5145..000000000000 --- a/doc/start/quick-rgw-old.rst +++ /dev/null @@ -1,30 +0,0 @@ -:orphan: - -=========================== - Quick Ceph Object Storage -=========================== - -To use the :term:`Ceph Object Storage` Quick Start guide, you must have executed the -procedures in the `Storage Cluster Quick Start`_ guide first. Make sure that you -have at least one :term:`RGW` instance running. - -Configure new RGW instance -========================== - -The :term:`RGW` instance created by the `Storage Cluster Quick Start`_ will run using -the embedded CivetWeb webserver. ``ceph-deploy`` will create the instance and start -it automatically with default parameters. - -To administer the :term:`RGW` instance, see details in the the -`RGW Admin Guide`_. - -Additional details may be found in the `Configuring Ceph Object Gateway`_ guide, but -the steps specific to Apache are no longer needed. - -.. note:: Deploying RGW using ``ceph-deploy`` and using the CivetWeb webserver instead - of Apache is new functionality as of **Hammer** release. - - -.. _Storage Cluster Quick Start: ../quick-ceph-deploy -.. _RGW Admin Guide: ../../radosgw/admin -.. _Configuring Ceph Object Gateway: ../../radosgw/config-fcgi diff --git a/doc/start/quick-rgw.rst b/doc/start/quick-rgw.rst deleted file mode 100644 index 5efda04f9ba1..000000000000 --- a/doc/start/quick-rgw.rst +++ /dev/null @@ -1,101 +0,0 @@ -=============================== -Ceph Object Gateway Quick Start -=============================== - -As of `firefly` (v0.80), Ceph Storage dramatically simplifies installing and -configuring a Ceph Object Gateway. The Gateway daemon embeds Civetweb, so you -do not have to install a web server or configure FastCGI. Additionally, -``ceph-deploy`` can install the gateway package, generate a key, configure a -data directory and create a gateway instance for you. - -.. tip:: Civetweb uses port ``7480`` by default. You must either open port - ``7480``, or set the port to a preferred port (e.g., port ``80``) in your Ceph - configuration file. - -To start a Ceph Object Gateway, follow the steps below: - -Installing Ceph Object Gateway -============================== - -#. Execute the pre-installation steps on your ``client-node``. If you intend to - use Civetweb's default port ``7480``, you must open it using either - ``firewall-cmd`` or ``iptables``. See `Preflight Checklist`_ for more - information. - -#. From the working directory of your administration server, install the Ceph - Object Gateway package on the ``client-node`` node. For example:: - - ceph-deploy install --rgw [ ...] - -Creating the Ceph Object Gateway Instance -========================================= - -From the working directory of your administration server, create an instance of -the Ceph Object Gateway on the ``client-node``. For example:: - - ceph-deploy rgw create - -Once the gateway is running, you should be able to access it on port ``7480``. -(e.g., ``http://client-node:7480``). - -Configuring the Ceph Object Gateway Instance -============================================ - -#. To change the default port (e.g., to port ``80``), modify your Ceph - configuration file. Add a section entitled ``[client.rgw.]``, - replacing ```` with the short node name of your Ceph client - node (i.e., ``hostname -s``). For example, if your node name is - ``client-node``, add a section like this after the ``[global]`` section:: - - [client.rgw.client-node] - rgw_frontends = "civetweb port=80" - - .. note:: Ensure that you leave no whitespace between ``port=`` - in the ``rgw_frontends`` key/value pair. - - .. important:: If you intend to use port 80, make sure that the Apache - server is not running otherwise it will conflict with Civetweb. We recommend - to remove Apache in this case. - -#. To make the new port setting take effect, restart the Ceph Object Gateway. - On Red Hat Enterprise Linux 7 and Fedora, run the following command:: - - sudo systemctl restart ceph-radosgw.service - - On Red Hat Enterprise Linux 6 and Ubuntu, run the following command:: - - sudo service radosgw restart id=rgw. - -#. Finally, check to ensure that the port you selected is open on the node's - firewall (e.g., port ``80``). If it is not open, add the port and reload the - firewall configuration. For example:: - - sudo firewall-cmd --list-all - sudo firewall-cmd --zone=public --add-port 80/tcp --permanent - sudo firewall-cmd --reload - - See `Preflight Checklist`_ for more information on configuring firewall with - ``firewall-cmd`` or ``iptables``. - - You should be able to make an unauthenticated request, and receive a - response. For example, a request with no parameters like this:: - - http://:80 - - Should result in a response like this:: - - - - - anonymous - - - - - - -See the `Configuring Ceph Object Gateway`_ guide for additional administration -and API details. - -.. _Configuring Ceph Object Gateway: ../../radosgw/config-ref -.. _Preflight Checklist: ../quick-start-preflight diff --git a/doc/start/quick-start-preflight.rst b/doc/start/quick-start-preflight.rst deleted file mode 100644 index b1fdc92d2286..000000000000 --- a/doc/start/quick-start-preflight.rst +++ /dev/null @@ -1,364 +0,0 @@ -===================== - Preflight Checklist -===================== - -The ``ceph-deploy`` tool operates out of a directory on an admin -:term:`node`. Any host with network connectivity and a modern python -environment and ssh (such as Linux) should work. - -In the descriptions below, :term:`Node` refers to a single machine. - -.. include:: quick-common.rst - - -Ceph-deploy Setup -================= - -Add Ceph repositories to the ``ceph-deploy`` admin node. Then, install -``ceph-deploy``. - -Debian/Ubuntu -------------- - -For Debian and Ubuntu distributions, perform the following steps: - -#. Add the release key:: - - wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add - - -#. Add the Ceph packages to your repository. Use the command below and - replace ``{ceph-stable-release}`` with a stable Ceph release (e.g., - ``luminous``.) For example:: - - echo deb https://download.ceph.com/debian-{ceph-stable-release}/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list - -#. Update your repository and install ``ceph-deploy``:: - - sudo apt update - sudo apt install ceph-deploy - -.. note:: You can also use the EU mirror eu.ceph.com for downloading your packages by replacing ``https://ceph.com/`` by ``http://eu.ceph.com/`` - - -RHEL/CentOS ------------ - -For CentOS 7, perform the following steps: - -#. On Red Hat Enterprise Linux 7, register the target machine with - ``subscription-manager``, verify your subscriptions, and enable the - "Extras" repository for package dependencies. For example:: - - sudo subscription-manager repos --enable=rhel-7-server-extras-rpms - -#. Install and enable the Extra Packages for Enterprise Linux (EPEL) - repository:: - - sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm - - Please see the `EPEL wiki`_ page for more information. - -#. Add the Ceph repository to your yum configuration file at ``/etc/yum.repos.d/ceph.repo`` with the following command. Replace ``{ceph-stable-release}`` with a stable Ceph release (e.g., - ``luminous``.) For example:: - - cat << EOM > /etc/yum.repos.d/ceph.repo - [ceph-noarch] - name=Ceph noarch packages - baseurl=https://download.ceph.com/rpm-{ceph-stable-release}/el7/noarch - enabled=1 - gpgcheck=1 - type=rpm-md - gpgkey=https://download.ceph.com/keys/release.asc - EOM - -#. You may need to install python setuptools required by ceph-deploy: - - sudo yum install python-setuptools - -#. Update your repository and install ``ceph-deploy``:: - - sudo yum update - sudo yum install ceph-deploy - -.. note:: You can also use the EU mirror eu.ceph.com for downloading your packages by replacing ``https://ceph.com/`` by ``http://eu.ceph.com/`` - - -openSUSE --------- - -The Ceph project does not currently publish release RPMs for openSUSE, but -a stable version of Ceph is included in the default update repository, so -installing it is just a matter of:: - - sudo zypper install ceph - sudo zypper install ceph-deploy - -If the distro version is out-of-date, open a bug at -https://bugzilla.opensuse.org/index.cgi and possibly try your luck with one of -the following repositories: - -#. Hammer:: - - https://software.opensuse.org/download.html?project=filesystems%3Aceph%3Ahammer&package=ceph - -#. Jewel:: - - https://software.opensuse.org/download.html?project=filesystems%3Aceph%3Ajewel&package=ceph - - -Ceph Node Setup -=============== - -The admin node must have password-less SSH access to Ceph nodes. -When ceph-deploy logs in to a Ceph node as a user, that particular -user must have passwordless ``sudo`` privileges. - - -Install NTP ------------ - -We recommend installing NTP on Ceph nodes (especially on Ceph Monitor nodes) to -prevent issues arising from clock drift. See `Clock`_ for details. - -On CentOS / RHEL, execute:: - - sudo yum install ntp ntpdate ntp-doc - -On Debian / Ubuntu, execute:: - - sudo apt install ntpsec - -or:: - - sudo apt install chrony - -Ensure that you enable the NTP service. Ensure that each Ceph Node uses the -same NTP time server. See `NTP`_ for details. - - -Install SSH Server ------------------- - -For **ALL** Ceph Nodes perform the following steps: - -#. Install an SSH server (if necessary) on each Ceph Node:: - - sudo apt install openssh-server - - or:: - - sudo yum install openssh-server - - -#. Ensure the SSH server is running on **ALL** Ceph Nodes. - - -Create a Ceph Deploy User -------------------------- - -The ``ceph-deploy`` utility must login to a Ceph node as a user -that has passwordless ``sudo`` privileges, because it needs to install -software and configuration files without prompting for passwords. - -Recent versions of ``ceph-deploy`` support a ``--username`` option so you can -specify any user that has password-less ``sudo`` (including ``root``, although -this is **NOT** recommended). To use ``ceph-deploy --username {username}``, the -user you specify must have password-less SSH access to the Ceph node, as -``ceph-deploy`` will not prompt you for a password. - -We recommend creating a specific user for ``ceph-deploy`` on **ALL** Ceph nodes -in the cluster. Please do **NOT** use "ceph" as the user name. A uniform user -name across the cluster may improve ease of use (not required), but you should -avoid obvious user names, because hackers typically use them with brute force -hacks (e.g., ``root``, ``admin``, ``{productname}``). The following procedure, -substituting ``{username}`` for the user name you define, describes how to -create a user with passwordless ``sudo``. - -.. note:: Starting with the :ref:`Infernalis release `, the "ceph" user name is reserved - for the Ceph daemons. If the "ceph" user already exists on the Ceph nodes, - removing the user must be done before attempting an upgrade. - -#. Create a new user on each Ceph Node. :: - - ssh user@ceph-server - sudo useradd -d /home/{username} -m {username} - sudo passwd {username} - -#. For the new user you added to each Ceph node, ensure that the user has - ``sudo`` privileges. :: - - echo "{username} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/{username} - sudo chmod 0440 /etc/sudoers.d/{username} - - -Enable Password-less SSH ------------------------- - -Since ``ceph-deploy`` will not prompt for a password, you must generate -SSH keys on the admin node and distribute the public key to each Ceph -node. ``ceph-deploy`` will attempt to generate the SSH keys for initial -monitors. - -#. Generate the SSH keys, but do not use ``sudo`` or the - ``root`` user. Leave the passphrase empty:: - - ssh-keygen - - Generating public/private key pair. - Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /ceph-admin/.ssh/id_rsa. - Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub. - -#. Copy the key to each Ceph Node, replacing ``{username}`` with the user name - you created with `Create a Ceph Deploy User`_. :: - - ssh-copy-id {username}@node1 - ssh-copy-id {username}@node2 - ssh-copy-id {username}@node3 - -#. (Recommended) Modify the ``~/.ssh/config`` file of your ``ceph-deploy`` - admin node so that ``ceph-deploy`` can log in to Ceph nodes as the user you - created without requiring you to specify ``--username {username}`` each - time you execute ``ceph-deploy``. This has the added benefit of streamlining - ``ssh`` and ``scp`` usage. Replace ``{username}`` with the user name you - created:: - - Host node1 - Hostname node1 - User {username} - Host node2 - Hostname node2 - User {username} - Host node3 - Hostname node3 - User {username} - - -Enable Networking On Bootup ---------------------------- - -Ceph OSDs peer with each other and report to Ceph Monitors over the network. -If networking is ``off`` by default, the Ceph cluster cannot come online -during bootup until you enable networking. - -The default configuration on some distributions (e.g., CentOS) has the -networking interface(s) off by default. Ensure that, during boot up, your -network interface(s) turn(s) on so that your Ceph daemons can communicate over -the network. For example, on Red Hat and CentOS, navigate to -``/etc/sysconfig/network-scripts`` and ensure that the ``ifcfg-{iface}`` file -has ``ONBOOT`` set to ``yes``. - - -Ensure Connectivity -------------------- - -Ensure connectivity using ``ping`` with short hostnames (``hostname -s``). -Address hostname resolution issues as necessary. - -.. note:: Hostnames should resolve to a network IP address, not to the - loopback IP address (e.g., hostnames should resolve to an IP address other - than ``127.0.0.1``). If you use your admin node as a Ceph node, you - should also ensure that it resolves to its hostname and IP address - (i.e., not its loopback IP address). - - -Open Required Ports -------------------- - -Ceph Monitors communicate using port ``6789`` by default. Ceph OSDs communicate -in a port range of ``6800:7300`` by default. See the `Network Configuration -Reference`_ for details. Ceph OSDs can use multiple network connections to -communicate with clients, monitors, other OSDs for replication, and other OSDs -for heartbeats. - -On some distributions (e.g., RHEL), the default firewall configuration is fairly -strict. You may need to adjust your firewall settings allow inbound requests so -that clients in your network can communicate with daemons on your Ceph nodes. - -For ``firewalld`` on RHEL 7, add the ``ceph-mon`` service for Ceph Monitor -nodes and the ``ceph`` service for Ceph OSDs and MDSs to the public zone and -ensure that you make the settings permanent so that they are enabled on reboot. - -For example, on monitors:: - - sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent - -and on OSDs and MDSs:: - - sudo firewall-cmd --zone=public --add-service=ceph --permanent - -Once you have finished configuring firewalld with the ``--permanent`` flag, you can make the changes live immediately without rebooting:: - - sudo firewall-cmd --reload - -For ``iptables``, add port ``6789`` for Ceph Monitors and ports ``6800:7300`` -for Ceph OSDs. For example:: - - sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT - -Once you have finished configuring ``iptables``, ensure that you make the -changes persistent on each node so that they will be in effect when your nodes -reboot. For example:: - - /sbin/service iptables save - -TTY ---- - -On CentOS and RHEL, you may receive an error while trying to execute -``ceph-deploy`` commands. If ``requiretty`` is set by default on your Ceph -nodes, disable it by executing ``sudo visudo`` and locate the ``Defaults -requiretty`` setting. Change it to ``Defaults:ceph !requiretty`` or comment it -out to ensure that ``ceph-deploy`` can connect using the user you created with -`Create a Ceph Deploy User`_. - -.. note:: If editing, ``/etc/sudoers``, ensure that you use - ``sudo visudo`` rather than a text editor. - - -SELinux -------- - -On CentOS and RHEL, SELinux is set to ``Enforcing`` by default. To streamline your -installation, we recommend setting SELinux to ``Permissive`` or disabling it -entirely and ensuring that your installation and cluster are working properly -before hardening your configuration. To set SELinux to ``Permissive``, execute the -following:: - - sudo setenforce 0 - -To configure SELinux persistently (recommended if SELinux is an issue), modify -the configuration file at ``/etc/selinux/config``. - - -Priorities/Preferences ----------------------- - -Ensure that your package manager has priority/preferences packages installed and -enabled. On CentOS, you may need to install EPEL. On RHEL, you may need to -enable optional repositories. :: - - sudo yum install yum-plugin-priorities - -For example, on RHEL 7 server, execute the following to install -``yum-plugin-priorities`` and enable the ``rhel-7-server-optional-rpms`` -repository:: - - sudo yum install yum-plugin-priorities --enablerepo=rhel-7-server-optional-rpms - - -Summary -======= - -This completes the Quick Start Preflight. Proceed to the `Storage Cluster -Quick Start`_. - -.. _Storage Cluster Quick Start: ../quick-ceph-deploy -.. _OS Recommendations: ../os-recommendations -.. _Network Configuration Reference: ../../rados/configuration/network-config-ref -.. _Clock: ../../rados/configuration/mon-config-ref#clock -.. _NTP: http://www.ntp.org/ -.. _Infernalis release: ../../release-notes/#v9-1-0-infernalis-release-candidate -.. _EPEL wiki: https://fedoraproject.org/wiki/EPEL