From: John Wilkins Date: Thu, 21 Nov 2013 03:27:47 +0000 (-0800) Subject: doc: Added a manual deployment doc. X-Git-Tag: v0.74~11^2~5 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=de2bcd5da92767abd99a72de1fb778de7689eccd;p=ceph.git doc: Added a manual deployment doc. Signed-off-by: John Wilkins --- diff --git a/doc/install/manual-deployment.rst b/doc/install/manual-deployment.rst new file mode 100644 index 000000000000..edafa28e82c7 --- /dev/null +++ b/doc/install/manual-deployment.rst @@ -0,0 +1,364 @@ +=================== + Manual Deployment +=================== + +All Ceph clusters require at least one monitor, and at least as many OSDs as +copies of an object stored on the cluster. Bootstrapping the initial monitor(s) +is the first step in deploying a Ceph Storage Cluster. Monitor deployment also +sets important criteria for the entire cluster, such as the number of replicas +for pools, the number of placement groups per OSD, the heart beat intervals, +whether authentication is required, etc. Most of these values are set by +default, so it's useful to know about them when setting up your cluster for +production. + +Following the same configuration as `Installation (Quick)`_, we will set up a +cluster with ``node1`` as the monitor node, and ``node2`` and ``node3`` for +OSD nodes. + + + +.. ditaa:: + /------------------\ /----------------\ + | Admin Node | | node1 | + | +-------->+ | + | | | cCCC | + \---------+--------/ \----------------/ + | + | /----------------\ + | | node2 | + +----------------->+ | + | | cCCC | + | \----------------/ + | + | /----------------\ + | | node3 | + +----------------->| | + | cCCC | + \----------------/ + + +Monitor Bootstrapping +===================== + +Bootstrapping a monitor (a Ceph Storage Cluster, in theory) requires +a number of things: + +- **Unique Identifier:** The ``fsid`` is a unique identifier for the cluster, + and stands for File System ID from the days when the Ceph Storage Cluster was + principally for the Ceph Filesystem. Ceph now supports native interfaces, + block device, and object storage gateway interfaces too, so ``fsid`` is a bit + of a misnomer. + +- **Cluster Name:** All Ceph clusters have a name, but the default name is + ``ceph``. Setting the cluster name to something other than ``ceph`` is + especially useful when you are working with multiple clusters and you need to + clearly understand which cluster your are working with. For example, when you + run multiple clusters on the same hardware, or if you are running Ceph in + federated architectures, the cluster name defines which cluster you are + using (e.g., ``us-west``, ``us-east``). + +- **Monitor Name:** Each monitor instance within a cluster has a unique name. + In common practice, the Ceph Monitor name is the host name (we recommend one + Ceph Monitor per host, and no commingling of Ceph OSD Daemons with + Ceph Monitors). + +- **Cluster Map:** Bootstrapping the initial monitor(s) requires you to + generate a cluster map. The cluster map requires the ``fsid``, the cluster + name (or uses the default), and at least one host name and its IP address. + +- **Monitor Keyring**: Monitors communicate with each other via a + secret key. You must generate a keyring with a monitor secret and provide + it when bootstrapping the initial monitor(s). + +- **Administrator Keyring**: To use the ``ceph`` CLI tools, you must have + a ``client.admin`` user. So you must generate the admin user and keyring, + and you must also add the ``client.admin`` user to the monitor keyring. + +The foregoing requirements do not imply the creation of a Ceph Configuration +file. However, as a best practice, we recommend creating a Ceph configuration +file and populating it with the ``fsid``, the ``mon initial members`` and the +``mon host`` settings. + +You can get and set all of the monitor settings at runtime as well. However, +a Ceph Configuration file may contain only those settings that override the +default values. When you add settings to a Ceph configuration file, these +settings override the default settings. Maintaining those settings in a +Ceph configuration file makes it easier to maintain your cluster. + +The procedure is as follows: + + +#. Log in to the initial monitor node(s):: + + ssh {hostname} + ssh node1 + + +#. Ensure you have a directory for the Ceph configuration file. By default, + Ceph uses ``/etc/ceph``. When you install ``ceph``, the directory will + get created automatically. :: + + ls /etc/ceph + + +#. Create a Ceph configuration file (optional). By default, Ceph uses + ``ceph.conf``, where ``ceph`` reflects the cluster name. :: + + sudo vim /etc/ceph/ceph.conf + + +#. Generate a unique ID (i.e., ``fsid``) for your cluster. :: + + uuidgen + + +#. Add the unique ID to your Ceph configuration file (optional). :: + + fsid = {UUID} + fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993 + + +#. Add the initial monitor(s) to your Ceph configuration file (optional). :: + + mon initial members = {hostname}[,{hostname}] + mon initial members = node1 + + +#. Add the IP address(es) of the initial monitor(s) to your Ceph configuration + file and save the file (optional). :: + + mon host = {ip-address}[,{ip-address}] + mon host = 192.168.0.1 + + +#. Create a keyring for your cluster and generate a monitor secret key. :: + + ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' + + +#. Generate an administrator keyring, generate a ``client.admin`` user and add + the user to the keyring. :: + + ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow' + + +#. Add the ``client.admin`` key to the ``ceph.mon.keyring``. :: + + ceph-authtool ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring + + +#. Generate a monitor map using the hostname(s), host IP address(es) and the FSID. + Save it as ``/tmp/monmap``:: + + monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap + monmaptool --create --add node1 192.168.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap + + +#. Create a default data directory (or directories) on the monitor host(s). :: + + sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname} + sudo mkdir /var/lib/ceph/mon/ceph-node1 + + +#. Populate the monitor daemon(s) with the monitor map and keyring. :: + + ceph-mon --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring + ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring + + +#. Consider settings for a Ceph configuration file. Common settings include + the following:: + + [global] + fsid = {cluster-id} + mon initial members = {hostname}[, {hostname}] + mon host = {ip-address}[, {ip-address}] + public network = {network}[, {network}] + cluster network = {network}[, {network}] + auth cluster required = cephx + auth service required = cephx + auth client required = cephx + osd journal size = {n} + filestore xattr use omap = true + osd pool default size = {n} # Write an object n times. + osd pool default min size = {n} # Allow writing n copy in a degraded state. + osd pool default pg num = {n} + osd pool default pgp num = {n} + osd crush chooseleaf type = {n} + + In the foregoing example, the ``[global]`` section of the configuration might + look like this:: + + [global] + fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993 + mon initial members = node1 + mon host = 192.168.0.1 + public network = 192.168.0.0/24 + auth cluster required = cephx + auth service required = cephx + auth client required = cephx + osd journal size = 1024 + filestore xattr use omap = true + osd pool default size = 2 + osd pool default min size = 1 + osd pool default pg num = 333 + osd pool default pgp num = 333 + osd crush chooseleaf type = 1 + + +#. Start the monitor(s). + + For Debian/Ubuntu, use Upstart:: + + sudo start ceph-mon id=node1 + + For CentOS/RHEL, use sysvinit:: + + sudo /etc/init.d/ceph start mon.node1 + + +#. Verify that Ceph created the default pools. :: + + ceph osd lspools + + You should see output like this:: + + 0 data,1 metadata,2 rbd, + + +#. Verify that the monitor is running. :: + + ceph -s + + You should see output that looks something like this:: + + cluster a7f64266-0894-4f1e-a635-d0aeaca0e993 + health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds + monmap e1: 1 mons at {node1=192.168.0.1:6789/0}, election epoch 1, quorum 0 node1 + osdmap e1: 0 osds: 0 up, 0 in + pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects + 0 kB used, 0 kB / 0 kB avail + 192 creating + + + +Adding OSDs +=========== + +Once you have your initial monitor(s) running, you should add OSDs. Your cluster +cannot reach an ``active + clean`` state until you have enough OSDs to handle the +number of copies of an object (e.g., ``osd pool default size = 2`` requires at +least two OSDs). After bootstrapping your monitor, your cluster has a default +CRUSH map; however, the CRUSH map doesn't have any Ceph OSD Daemons mapped to +a Ceph Node. + +To create the first two OSDs, perform this procedure on ``node2`` and ``node3``: + + +#. Generate a UUID for the OSD. :: + + uuidgen + + +#. Create the OSD. If no UUID is given, it will be set automatically when the + OSD starts up. The following command will output the OSD number, which you + will need for subsequent steps. :: + + ceph osd create [{uuid}] + + +#. Create the default directory on your new OSD. :: + + ssh {new-osd-host} + sudo mkdir /var/lib/ceph/osd/ceph-{osd-number} + + +#. If the OSD is for a drive other than the OS drive, prepare it + for use with Ceph, and mount it to the directory you just created:: + + ssh {new-osd-host} + sudo mkfs -t {fstype} /dev/{drive} + sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number} + + +#. Initialize the OSD data directory. :: + + ssh {new-osd-host} + sudo ceph-osd -i {osd-num} --mkfs --mkkey + + The directory must be empty before you can run ``ceph-osd``. + + +#. Register the OSD authentication key. The value of ``ceph`` for + ``ceph-{osd-num}`` in the path is the ``$cluster-$id``. If your + cluster name differs from ``ceph``, use your cluster name instead.:: + + sudo ceph auth add osd.{osd-num} osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-{osd-num}/keyring + + +#. Add your Ceph Node to the CRUSH map. :: + + ceph osd crush add-bucket {hostname} host + ceph osd crush add-bucket node1 host + + +#. Place the Ceph Node under the root ``default``. :: + + ceph osd crush move node1 root=default + + +#. Add the OSD to the CRUSH map so that it can begin receiving data. You may + also decompile the CRUSH map, add the OSD to the device list, add the host as a + bucket (if it's not already in the CRUSH map), add the device as an item in the + host, assign it a weight, recompile it and set it. :: + + ceph osd crush add {id-or-name} {weight} [{bucket-type}={bucket-name} ...] + ceph osd crush add osd.0 1.0 host=node1 + + +#. After you add an OSD to Ceph, the OSD is in your configuration. However, + it is not yet running. The OSD is ``down`` and ``in``. You must start + your new OSD before it can begin receiving data. + + For Debian/Ubuntu, use Upstart:: + + sudo start ceph-osd id={osd-num} + sudo start ceph-osd id=0 + sudo start ceph-osd id=1 + + For CentOS/RHEL, use sysvinit:: + + sudo /etc/init.d/ceph start osd.{osd-num} + sudo /etc/init.d/ceph start osd.0 + sudo /etc/init.d/ceph start osd.1 + + Once you start your OSD, it is ``up`` and ``in``. + + +Summary +======= + +Once you have your monitor and two OSDs up and running, you can watch the +placement groups peer by executing the following:: + + ceph -w + +To view the tree, execute the following:: + + ceph osd tree + +You should see output that looks something like this:: + + # id weight type name up/down reweight + -1 2 root default + -2 2 host node1 + 0 1 osd.0 up 1 + -3 1 host node2 + 1 1 osd.1 up 1 + +To add (or remove) additional monitors, see `Add/Remove Monitors`_. +To add (or remove) additional Ceph OSD Daemons, see `Add/Remove OSDs`_. + +.. _Installation (Quick): ../../start +.. _Add/Remove Monitors: ../../rados/operations/add-or-rm-mons +.. _Add/Remove OSDs: ../../rados/operations/add-or-rm-osds \ No newline at end of file