==================
When you start the Ceph service, the initialization process activates a series
-of daemons that run in the background. The hosts in a typical Ceph cluster run
-at least one of four daemons:
+of daemons that run in the background. All Ceph clusters run two types of
+daemons:
-- Object Storage Device (``ceph-osd``)
- Monitor (``ceph-mon``)
-- Metadata Server (``ceph-mds``)
-- Ceph Gateway (``radosgw``)
+- Object Storage Device (``ceph-osd``)
+
+Clusters that support CephFS run Metadata Servers (``ceph-mds``). Clusters that
+support a RESTful object store run Ceph Gateways (``radosgw``). For your
+convenience, each daemon has a series of default values (*i.e.*, many are set by
+``ceph/src/common/config_opts.h``). You may override these settings with a Ceph
+configuration file.
-For your convenience, each daemon has a series of default values (*i.e.*, many
-are set by ``ceph/src/common/config_opts.h``). You may override these settings
-with a Ceph configuration file.
.. _ceph-conf-file:
When you start a Ceph cluster, each daemon looks for a Ceph configuration file
(i.e., ``ceph.conf`` by default) that provides the cluster's configuration
settings. For manual deployments, you need to create a Ceph configuration file.
-For third party tools that create configuration files for you (*e.g.*, Chef),
-you may use the information contained herein as a reference. The Ceph
-Configuration file defines:
+For tools that create configuration files for you (*e.g.*, ``ceph-deploy``,
+Chef, etc.), you may use the information contained herein as a reference. The
+Ceph Configuration file defines:
+- Cluster Identity
- Authentication settings
- Cluster membership
- Host names
The default ``ceph.conf`` locations in sequential order include:
-#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` environment variable)
+#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF``
+ environment variable)
#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument)
#. ``/etc/ceph/ceph.conf``
#. ``~/.ceph/config``
The Ceph configuration file uses an *ini* style syntax. You can add comments
-by preceding comments with a semi-colon (;) or a pound sign (#). For example:
+by preceding comments with a pound sign (#) or a semi-colon (;). For example:
.. code-block:: ini
# The end of the line terminates a comment.
# We recommend that you provide comments in your configuration file(s).
+
.. _ceph-conf-settings:
Config Sections
``[osd]``
:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in
- the cluster.
+ the cluster, and override the same setting in ``[global]``.
:Example: ``osd journal size = 1000``
``[mon]``
:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in
- the cluster.
+ the cluster, and override the same setting in ``[global]``.
:Example: ``mon addr = 10.0.0.101:6789``
``[mds]``
:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in
- the cluster.
+ the cluster, and override the same setting in ``[global]``.
:Example: ``host = myserver01``
``[client]``
:Description: Settings under ``[client]`` affect all clients (e.g., mounted
- CephFS filesystems, mounted block devices, etc.)
+ CephFS filesystems, mounted block devices, etc.).
:Example: ``log file = /var/log/ceph/radosgw.log``
-Global settings affect all instances of all daemon in the cluster. Use the ``[global]``
-setting for values that are common for all daemons in the cluster. You can override each
-``[global]`` setting by:
+
+Global settings affect all instances of all daemon in the cluster. Use the
+``[global]`` setting for values that are common for all daemons in the cluster.
+You can override each ``[global]`` setting by:
#. Changing the setting in a particular process type
(*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ).
-#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` )
+#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` ).
Overriding a global setting affects all child processes, except those that
-you specifically override.
+you specifically override in a particular daemon.
A typical global setting involves activating authentication. For example:
particular instance, the setting will apply to all OSDs, monitors or metadata
daemons respectively.
+A typical daemon-wide setting involves setting journal sizes, filestore
+settings, etc. For example:
+
+.. code-block:: ini
+
+ [osd]
+ osd journal size = 1000
+ filestore xattr use omap = true
+
+
You may specify settings for particular instances of a daemon. You may specify
an instance by entering its type, delimited by a period (.) and by the
instance ID. The instance ID for an OSD is always numeric, but it may be
The `Hardware Recommendations`_ section provides some hardware guidelines for
configuring the cluster. It is possible for a single host to run multiple
daemons. For example, a single host with multiple disks or RAIDs may run one
-``ceph-osd`` for each disk or RAID. Additionally, a host may run both a
-``ceph-mon`` and an ``ceph-osd`` daemon on the same host. Ideally, you will have
-a host for a particular type of process. For example, one host may run
-``ceph-osd`` daemons, another host may run a ``ceph-mds`` daemon, and other
-hosts may run ``ceph-mon`` daemons.
+``ceph-osd`` for each disk or RAID. Ideally, you will have a host for a
+particular type of process. For example, one host may run ``ceph-osd`` daemons,
+another host may run a ``ceph-mds`` daemon, and other hosts may run ``ceph-mon``
+daemons.
-Each host has a name identified by the ``host`` setting. Monitors also specify
+Each host has a name identified by the ``host`` setting. Monitors also specify
a network address and port (i.e., domain name or IP address) identified by the
``addr`` setting. A basic configuration file will typically specify only
minimal settings for each instance of a daemon. For example:
an fqdn). It is **NOT** an IP address either. Enter ``hostname -s`` on
the command line to retrieve the name of the host. Also, this setting is
**ONLY** for ``mkcephfs`` and manual deployment. It **MUST NOT**
- be used with ``chef`` or ``ceph-deploy``.
+ be used with ``chef`` or ``ceph-deploy``, as those tools will enter the
+ appropriate values for you.
.. _ceph-network-config:
configuring a network for use with Ceph.
-Authentication
-==============
-
-.. versionadded:: Bobtail 0.56
-
-For Bobtail (v 0.56) and beyond, you should expressly enable or disable authentication
-in the ``[global]`` section of your Ceph configuration file. ::
-
- auth cluster required = cephx
- auth service required = cephx
- auth client required = cephx
-
-Additionally, you should enable message signing. See `Cephx Config Reference`_
-and `Cephx Authentication`_ for details.
-
-.. important:: When upgrading, we recommend expressly disabling authentication first,
- then perform the upgrade. Once the upgrade is complete, re-enable authentication.
-
-.. _Cephx Authentication: ../../operations/authentication
-.. _Cephx Config Reference: ../auth-config-ref
-
-
-.. _ceph-monitor-config:
-
Monitors
========
/var/lib/ceph/mon/$cluster-$id
-You must create the corresponding directory yourself. With metavariables fully
-expressed and a cluster named "ceph", the foregoing directory would evaluate to::
+You or a deployment tool (e.g., ``ceph-deploy``) must create the corresponding
+directory. With metavariables fully expressed and a cluster named "ceph", the
+foregoing directory would evaluate to::
/var/lib/ceph/mon/ceph-a
-You may override this path using the ``mon data`` setting. We don't recommend
-changing the default location. Create the default directory on your new monitor host. ::
+For additional details, see the `Monitor Config Reference`_.
- ssh {new-mon-host}
- sudo mkdir /var/lib/ceph/mon/ceph-{mon-letter}
+.. _Monitor Config Reference: ../mon-config-ref
.. _ceph-osd-config:
+
+Authentication
+==============
+
+.. versionadded:: Bobtail 0.56
+
+For Bobtail (v 0.56) and beyond, you should expressly enable or disable authentication
+in the ``[global]`` section of your Ceph configuration file. ::
+
+ auth cluster required = cephx
+ auth service required = cephx
+ auth client required = cephx
+
+Additionally, you should enable message signing. See `Cephx Config Reference`_
+and `Cephx Authentication`_ for details.
+
+.. important:: When upgrading, we recommend expressly disabling authentication first,
+ then perform the upgrade. Once the upgrade is complete, re-enable authentication.
+
+.. _Cephx Authentication: ../../operations/authentication
+.. _Cephx Config Reference: ../auth-config-ref
+
+
+.. _ceph-monitor-config:
+
+
OSDs
====
/var/lib/ceph/osd/$cluster-$id
-You must create the corresponding directory yourself. With metavariables fully
-expressed and a cluster named "ceph", the foregoing directory would evaluate to::
+You or a deployment tool (e.g., ``ceph-deploy``) must create the corresponding
+directory. With metavariables fully expressed and a cluster named "ceph", the
+foregoing directory would evaluate to::
/var/lib/ceph/osd/ceph-0
You may override this path using the ``osd data`` setting. We don't recommend
-changing the default location. Create the default directory on your new OSD host. ::
+changing the default location. Create the default directory on your OSD host. ::
- ssh {new-osd-host}
+ ssh {osd-host}
sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
The ``osd data`` path ideally leads to a mount point with a hard disk that is
We recommend using the ``xfs`` file system or the ``btrfs`` file system when
running :command:mkfs.
-By default, Ceph expects that you will store an OSDs journal with the
-following path::
+See the `OSD Config Reference`_ for additional configuration details.
- /var/lib/ceph/osd/$cluster-$id/journal
-Without performance optimization, Ceph stores the journal on the same disk as
-the OSDs data. An OSD optimized for performance may use a separate disk to store
-journal data (e.g., a solid state drive delivers high performance journaling).
+Heartbeats
+==========
-Ceph's default ``osd journal size`` is 0, so you will need to set this in your
-``ceph.conf`` file. A journal size should find the product of the ``filestore
-max sync interval`` and the expected throughput, and multiply the product by
-two (2)::
-
- osd journal size = {2 * (expected throughput * filestore max sync interval)}
+During runtime operations, OSDs check up on other OSDs and report their
+findings to the monitor. You do not have to provide any settings. However,
+if you have network latency issues, you may wish to modify the settings.
-The expected throughput number should include the expected disk throughput
-(i.e., sustained data transfer rate), and network throughput. For example,
-a 7200 RPM disk will likely have approximately 100 MB/s. Taking the ``min()``
-of the disk and network throughput should provide a reasonable expected
-throughput. Some users just start off with a 10GB journal size. For
-example::
+See `Configuring Monitor/OSD Interaction`_ for additional details.
- osd journal size = 10000
.. _ceph-logging-and-debugging:
================
Ceph is still on the leading edge, so you may encounter situations that require
-modifying logging output and using Ceph's debugging. To activate Ceph's
-debugging output (*i.e.*, ``dout()``), you may add ``debug`` settings to your
-configuration. Ceph's logging levels operate on a scale of 1 to 20, where 1 is
-terse and 20 is verbose.
-
-.. note:: See `Debugging and Logging`_ for details on log rotation.
+modifying logging output and using Ceph's debugging. See `Debugging and
+Logging`_ for details on log rotation.
-.. _Debugging and Logging: ../../operations/debug
-
-Subsystems common to each daemon may be set under ``[global]`` in your
-configuration file. Subsystems for particular daemons are set under the daemon
-section in your configuration file (*e.g.*, ``[mon]``, ``[osd]``, ``[mds]``).
-For example::
-
- [global]
- debug ms = 1
-
- [mon]
- debug mon = 20
- debug paxos = 20
- debug auth = 20
-
- [osd]
- debug osd = 20
- debug filestore = 20
- debug journal = 20
- debug monc = 20
-
- [mds]
- debug mds = 20
- debug mds balancer = 20
- debug mds log = 20
- debug mds migrator = 20
-
-When your system is running well, choose appropriate logging levels and remove
-unnecessary debugging settings to ensure your cluster runs optimally. Logging
-debug output messages is relatively slow, and a waste of resources when
-operating your cluster.
-
-.. tip:: When debug output slows down your system, the latency can hide
- race conditions.
-
-Each subsystem has a logging level for its output logs, and for its logs
-in-memory. You may set different values for each of these subsystems by setting
-a log file level and a memory level for debug logging. For example::
-
- debug {subsystem} {log-level}/{memory-level}
- #for example
- debug mds log 1/20
-
-+--------------------+-----------+--------------+
-| Subsystem | Log Level | Memory Level |
-+====================+===========+==============+
-| ``default`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``lockdep`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``context`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``crush`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds balancer`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds locker`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds log`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds log expire`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds migrator`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``buffer`` | 0 | 0 |
-+--------------------+-----------+--------------+
-| ``timer`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``filer`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``objecter`` | 0 | 0 |
-+--------------------+-----------+--------------+
-| ``rados`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``rbd`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``journaler`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``objectcacher`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``client`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``osd`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``optracker`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``objclass`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``filestore`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``journal`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``ms`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``mon`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``monc`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``paxos`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``tp`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``auth`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``finisher`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``heartbeatmap`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``perfcounter`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``rgw`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``hadoop`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``asok`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``throttle`` | 1 | 5 |
-+--------------------+-----------+--------------+
+.. _Debugging and Logging: ../../operations/troubleshooting/debug
Example ceph.conf
.. _Hardware Recommendations: ../../../install/hardware-recommendations
.. _hardware recommendations: ../../../install/hardware-recommendations
-.. _Network Configuration Reference: ../network-config-ref
\ No newline at end of file
+.. _Network Configuration Reference: ../network-config-ref
+.. _OSD Config Reference: ../osd-config-ref
+.. _Configuring Monitor/OSD Interaction: ../mon-osd-interaction
\ No newline at end of file
projects / ceph.git / commitdiff