From: Sage Weil Date: Thu, 18 Jan 2018 17:31:39 +0000 (-0600) Subject: doc/rados/configuration: big update X-Git-Tag: v13.0.2~78^2~44 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=5ddc55151615a16d9bf31be5216d3d86280ac41e;p=ceph.git doc/rados/configuration: big update - remove a bunch of outdated info - update to reflect new monitor-based configuration - move a bunch of common settings to another document/page Signed-off-by: Sage Weil --- diff --git a/doc/rados/configuration/ceph-conf.rst b/doc/rados/configuration/ceph-conf.rst index 988f525d2f8d..dbfa5f58f2a8 100644 --- a/doc/rados/configuration/ceph-conf.rst +++ b/doc/rados/configuration/ceph-conf.rst @@ -4,180 +4,169 @@ When you start the Ceph service, the initialization process activates a series of daemons that run in the background. A :term:`Ceph Storage Cluster` runs -two types of daemons: +three types of daemons: - :term:`Ceph Monitor` (``ceph-mon``) +- :term:`Ceph Manager` (``ceph-mgr``) - :term:`Ceph OSD Daemon` (``ceph-osd``) -Ceph Storage Clusters that support the :term:`Ceph Filesystem` run at least one -:term:`Ceph Metadata Server` (``ceph-mds``). Clusters that support :term:`Ceph -Object Storage` run Ceph Gateway daemons (``radosgw``). For your convenience, -each daemon has a series of default values (*i.e.*, many are set by -``ceph/src/common/config_opts.h``). You may override these settings with a Ceph -configuration file. +Ceph Storage Clusters that support the :term:`Ceph Filesystem` run at +least one :term:`Ceph Metadata Server` (``ceph-mds``). Clusters that +support :term:`Ceph Object Storage` run Ceph Gateway daemons +(``radosgw``). +Each daemon has a series of configuration options, each of which has a +default values. You may adjust the behavior of the system by changing these +configuration options. -.. _ceph-conf-file: +Option names +============ -The Configuration File -====================== +All Ceph configuration options have a unique name consisting of words +formed with lower-case characters and connected with underscore +(``_``) characters. -When you start a Ceph Storage Cluster, each daemon looks for a Ceph -configuration file (i.e., ``ceph.conf`` by default) that provides the cluster's -configuration settings. For manual deployments, you need to create a Ceph -configuration file. For tools that create configuration files for you (*e.g.*, -``ceph-deploy``, Chef, etc.), you may use the information contained herein as a -reference. The Ceph configuration file defines: - -- Cluster Identity -- Authentication settings -- Cluster membership -- Host names -- Host addresses -- Paths to keyrings -- Paths to journals -- Paths to data -- Other runtime options - -The default Ceph configuration file locations in sequential order include: - -#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` - environment variable) -#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument) -#. ``/etc/ceph/$cluster.conf`` -#. ``~/.ceph/$cluster.conf`` -#. ``./$cluster.conf`` (*i.e.,* in the current working directory) -#. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf`` +When option names are specified on the command line, either underscore +(``_``) or dash (``-``) characters can be used interchangeable (e.g., +``--mon-host`` is equivalent to ``--mon_host``). -where ``$cluster`` is the cluster's name (default ``ceph``). +When option names appear in configuration files, spaces can also be +used in place of underscore or dash. -The Ceph configuration file uses an *ini* style syntax. You can add comments -by preceding comments with a pound sign (#) or a semi-colon (;). For example: +Config sources +============== -.. code-block:: ini +Each Ceph daemon, process, and library will pull its configuration +from several sources, listed below. Sources later in the list will +override those earlier in the list when both are present. + +- the compiled-in default value +- the monitor cluster's centralized configuration database +- a configuration file stored on the local host +- environment variables +- command line arguments +- runtime overrides set by an administrator + +One of the first things a Ceph process does on startup is parse the +configuration options provided via the command line, environment, and +local configuration file. The process will then contact the monitor +cluster to retrieve configuration stored centrally for the entire +cluster. Once a complete view of the configuration is available, the +daemon or process startup will proceed. + +Bootstrap options +----------------- + +Because some configuration options affect the process's ability to +contact the monitors, authenticate, and retrieve the cluster-stored +configuration, they may need to be stored locally on the node and set +in a local configuration file. These options include: + + - ``mon_host``, the list of monitors for the cluster + - ``mon_dns_serv_name`` (default: `ceph-mon`), the name of the DNS + SRV record to check to identify the cluster monitors via DNS + - ``mon_data``, ``osd_data``, ``mds_data``, ``mgr_data``, and + similar options that define which local directory the daemon + stores its data in. + - ``keyring``, ``keyfile``, and/or ``key``, which can be used to + specify the authentication credential to use to authenticate with + the monitor. Note that in most cases the default keyring location + is in the data directory specified above. + +In the vast majority of cases the default values of these are +appropriate, with the exception of the ``mon_host`` option that +identifies the addresses of the cluster's monitors. When DNS is used +to identify monitors a local ceph configuration file can be avoided +entirely. + +Skipping monitor config +----------------------- + +Any process may be passed the option ``--no-mon-config`` to skip the +step that retrieves configuration from the cluster monitors. This is +useful in cases where configuration is managed entirely via +configuration files or where the monitor cluster is currently down but +some maintenance activity needs to be done. - # <--A number (#) sign precedes a comment. - ; A comment may be anything. - # Comments always follow a semi-colon (;) or a pound (#) on each line. - # The end of the line terminates a comment. - # We recommend that you provide comments in your configuration file(s). +.. _ceph-conf-file: -.. _ceph-conf-settings: - -Config Sections -=============== -The configuration file can configure all Ceph daemons in a Ceph Storage Cluster, -or all Ceph daemons of a particular type. To configure a series of daemons, the -settings must be included under the processes that will receive the -configuration as follows: +Configuration sections +====================== -``[global]`` +Any given process or daemon has a single value for each configuration +option. However, values for an option may vary across different +daemon types even daemons of the same type. Ceph options that are +stored in the monitor configuration database or in local configuration +files are grouped into sections to indicate which daemons or clients +they apply to. -:Description: Settings under ``[global]`` affect all daemons in a Ceph Storage - Cluster. - -:Example: ``auth supported = cephx`` +These sections include: -``[osd]`` +``global`` -:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in - the Ceph Storage Cluster, and override the same setting in - ``[global]``. +:Description: Settings under ``global`` affect all daemons and clients + in a Ceph Storage Cluster. -:Example: ``osd journal size = 1000`` +:Example: ``log_file = /var/log/ceph/$cluster-$type.$id.log`` -``[mon]`` +``mon`` -:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in +:Description: Settings under ``mon`` affect all ``ceph-mon`` daemons in the Ceph Storage Cluster, and override the same setting in - ``[global]``. + ``global``. -:Example: ``mon addr = 10.0.0.101:6789`` +:Example: ``mon_cluster_log_to_syslog = true`` -``[mds]`` +``mgr`` -:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in +:Description: Settings in the ``mgr`` section affect all ``ceph-mgr`` daemons in the Ceph Storage Cluster, and override the same setting in - ``[global]``. - -:Example: ``host = myserver01`` - -``[client]`` + ``global``. -:Description: Settings under ``[client]`` affect all Ceph Clients - (e.g., mounted Ceph Filesystems, mounted Ceph Block Devices, - etc.). +:Example: ``mgr_stats_period = 10`` -:Example: ``log file = /var/log/ceph/radosgw.log`` +``osd`` +:Description: Settings under ``osd`` affect all ``ceph-osd`` daemons in + the Ceph Storage Cluster, and override the same setting in + ``global``. -Global settings affect all instances of all daemon in the Ceph Storage Cluster. -Use the ``[global]`` setting for values that are common for all daemons in the -Ceph Storage Cluster. You can override each ``[global]`` setting by: +:Example: ``osd_op_queue = wpq`` -#. Changing the setting in a particular process type - (*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ). +``mds`` -#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` ). +:Description: Settings in the ``mds`` section affect all ``ceph-mds`` daemons in + the Ceph Storage Cluster, and override the same setting in + ``global``. -Overriding a global setting affects all child processes, except those that -you specifically override in a particular daemon. +:Example: ``mds_cache_size = 10G`` -A typical global setting involves activating authentication. For example: +``client`` -.. code-block:: ini - - [global] - #Enable authentication between hosts within the cluster. - #v 0.54 and earlier - auth supported = cephx - - #v 0.55 and after - auth cluster required = cephx - auth service required = cephx - auth client required = cephx - - -You can specify settings that apply to a particular type of daemon. When you -specify settings under ``[osd]``, ``[mon]`` or ``[mds]`` without specifying a -particular instance, the setting will apply to all OSDs, monitors or metadata -daemons respectively. - -A typical daemon-wide setting involves setting journal sizes, filestore -settings, etc. For example: - -.. code-block:: ini - - [osd] - osd journal size = 1000 - - -You may specify settings for particular instances of a daemon. You may specify -an instance by entering its type, delimited by a period (.) and by the instance -ID. The instance ID for a Ceph OSD Daemon is always numeric, but it may be -alphanumeric for Ceph Monitors and Ceph Metadata Servers. +:Description: Settings under ``client`` affect all Ceph Clients + (e.g., mounted Ceph Filesystems, mounted Ceph Block Devices, + etc.). -.. code-block:: ini +:Example: ``objecter_inflight_ops = 512`` - [osd.1] - # settings affect osd.1 only. - - [mon.a] - # settings affect mon.a only. - - [mds.b] - # settings affect mds.b only. +Sections may also specify an individual daemon or client name. For example, +``mon.foo``, ``osd.123``, and ``client.smith`` are all valid section names. -If the daemon you specify is a Ceph Gateway client, specify the daemon and the -instance, delimited by a period (.). For example:: - [client.radosgw.instance-name] - # settings affect client.radosgw.instance-name only. +Any given daemon will draw it's settings from the global section, the +daemon or client type section, and the section sharing its name. +Settings in the most-specific section take precedence, so for example +if the same option is specified in both ``global``, ``mon``, and +``mon.foo`` on the same source (i.e., in the same configurationfile), +the ``mon.foo`` value will be used. +Note that values from the local configuration file always take +precedence over values from the monitor configuration database, +regardless of which section they appear in. .. _ceph-metavariables: @@ -185,15 +174,13 @@ instance, delimited by a period (.). For example:: Metavariables ============= -Metavariables simplify Ceph Storage Cluster configuration dramatically. When a -metavariable is set in a configuration value, Ceph expands the metavariable into -a concrete value. Metavariables are very powerful when used within the -``[global]``, ``[osd]``, ``[mon]``, ``[mds]`` or ``[client]`` sections of your -configuration file. Ceph metavariables are similar to Bash shell expansion. +Metavariables simplify Ceph Storage Cluster configuration +dramatically. When a metavariable is set in a configuration value, +Ceph expands the metavariable into a concrete value at the time the +configuration value is used. Ceph metavariables are similar to variable expansion in the Bash shell. Ceph supports the following metavariables: - ``$cluster`` :Description: Expands to the Ceph Storage Cluster name. Useful when running @@ -205,23 +192,23 @@ Ceph supports the following metavariables: ``$type`` -:Description: Expands to one of ``mds``, ``osd``, or ``mon``, depending on the - type of the instant daemon. +:Description: Expands to a daemon or process type (e.g., ``mds``, ``osd``, or ``mon``) :Example: ``/var/lib/ceph/$type`` ``$id`` -:Description: Expands to the daemon identifier. For ``osd.0``, this would be - ``0``; for ``mds.a``, it would be ``a``. +:Description: Expands to the daemon or client identifier. For + ``osd.0``, this would be ``0``; for ``mds.a``, it would + be ``a``. :Example: ``/var/lib/ceph/$type/$cluster-$id`` ``$host`` -:Description: Expands to the host name of the instant daemon. +:Description: Expands to the host name where the process is running. ``$name`` @@ -235,397 +222,281 @@ Ceph supports the following metavariables: :Example: ``/var/run/ceph/$cluster-$name-$pid.asok`` -.. _ceph-conf-common-settings: - -Common Settings -=============== - -The `Hardware Recommendations`_ section provides some hardware guidelines for -configuring a Ceph Storage Cluster. It is possible for a single :term:`Ceph -Node` to run multiple daemons. For example, a single node with multiple drives -may run one ``ceph-osd`` for each drive. Ideally, you will have a node for a -particular type of process. For example, some nodes may run ``ceph-osd`` -daemons, other nodes may run ``ceph-mds`` daemons, and still other nodes may -run ``ceph-mon`` daemons. - -Each node has a name identified by the ``host`` setting. Monitors also specify -a network address and port (i.e., domain name or IP address) identified by the -``addr`` setting. A basic configuration file will typically specify only -minimal settings for each instance of monitor daemons. For example: - -.. code-block:: ini - [global] - mon_initial_members = ceph1 - mon_host = 10.0.0.1 +The Configuration File +====================== +On startup, Ceph processes search for a configuration file in the +following locations: -.. important:: The ``host`` setting is the short name of the node (i.e., not - an fqdn). It is **NOT** an IP address either. Enter ``hostname -s`` on - the command line to retrieve the name of the node. Do not use ``host`` - settings for anything other than initial monitors unless you are deploying - Ceph manually. You **MUST NOT** specify ``host`` under individual daemons - when using deployment tools like ``chef`` or ``ceph-deploy``, as those tools - will enter the appropriate values for you in the cluster map. +#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` + environment variable) +#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument) +#. ``/etc/ceph/$cluster.conf`` +#. ``~/.ceph/$cluster.conf`` +#. ``./$cluster.conf`` (*i.e.,* in the current working directory) +#. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf`` +where ``$cluster`` is the cluster's name (default ``ceph``). -.. _ceph-network-config: +The Ceph configuration file uses an *ini* style syntax. You can add comments +by preceding comments with a pound sign (#) or a semi-colon (;). For example: -Networks -======== +.. code-block:: ini -See the `Network Configuration Reference`_ for a detailed discussion about -configuring a network for use with Ceph. + # <--A number (#) sign precedes a comment. + ; A comment may be anything. + # Comments always follow a semi-colon (;) or a pound (#) on each line. + # The end of the line terminates a comment. + # We recommend that you provide comments in your configuration file(s). -Monitors -======== +.. _ceph-conf-settings: -Ceph production clusters typically deploy with a minimum 3 :term:`Ceph Monitor` -daemons to ensure high availability should a monitor instance crash. At least -three (3) monitors ensures that the Paxos algorithm can determine which version -of the :term:`Ceph Cluster Map` is the most recent from a majority of Ceph -Monitors in the quorum. +Config file section names +------------------------- -.. note:: You may deploy Ceph with a single monitor, but if the instance fails, - the lack of other monitors may interrupt data service availability. +The configuration file is divided into sections like ``[global]`` or +``[mon.foo]``, where the section is a valid Ceph section name +(`global`, a daemon type, or a daemon name) surrounded by square +brackets. -Ceph Monitors typically listen on port ``6789``. For example: +Global settings affect all instances of all daemon in the Ceph Storage Cluster. +Use the ``[global]`` setting for values that are common for all daemons in the +Ceph Storage Cluster. You can override each ``[global]`` setting by: -.. code-block:: ini +#. Changing the setting in a particular process type + (*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ). - [mon.a] - host = hostName - mon addr = 150.140.130.120:6789 +#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` ). -By default, Ceph expects that you will store a monitor's data under the -following path:: +Overriding a global setting affects all child processes, except those that +you specifically override in a particular daemon. For example, - /var/lib/ceph/mon/$cluster-$id - -You or a deployment tool (e.g., ``ceph-deploy``) must create the corresponding -directory. With metavariables fully expressed and a cluster named "ceph", the -foregoing directory would evaluate to:: +.. code-block:: ini - /var/lib/ceph/mon/ceph-a + [global] + debug ms = 0 -For additional details, see the `Monitor Config Reference`_. - -.. _Monitor Config Reference: ../mon-config-ref - - -.. _ceph-osd-config: - - -Authentication -============== - -.. versionadded:: Bobtail 0.56 - -For Bobtail (v 0.56) and beyond, you should expressly enable or disable -authentication in the ``[global]`` section of your Ceph configuration file. :: - - auth cluster required = cephx - auth service required = cephx - auth client required = cephx - -Additionally, you should enable message signing. See `Cephx Config Reference`_ for details. - -.. important:: When upgrading, we recommend expressly disabling authentication - first, then perform the upgrade. Once the upgrade is complete, re-enable - authentication. - -.. _Cephx Config Reference: ../auth-config-ref - - -.. _ceph-monitor-config: - - -OSDs -==== - -Ceph production clusters typically deploy :term:`Ceph OSD Daemons` where one node -has one OSD daemon running a filestore on one storage drive. A typical -deployment specifies a journal size. For example: + [osd] + debug ms = 1 -.. code-block:: ini + [osd.1] + debug ms = 10 - [osd] - osd journal size = 10000 - - [osd.0] - host = {hostname} #manual deployments only. + [osd.2] + debug ms = 10 -By default, Ceph expects that you will store a Ceph OSD Daemon's data with the -following path:: - /var/lib/ceph/osd/$cluster-$id - -You or a deployment tool (e.g., ``ceph-deploy``) must create the corresponding -directory. With metavariables fully expressed and a cluster named "ceph", the -foregoing directory would evaluate to:: +Monitor configuration database +============================== - /var/lib/ceph/osd/ceph-0 - -You may override this path using the ``osd data`` setting. We don't recommend -changing the default location. Create the default directory on your OSD host. +The monitor cluster manages a database of configuration options that +can be consumed by the entire cluster, enabling streamlined central +configuration management for the entire system. The vast majority of +configuration options can and should be stored here for ease of +administration and transparency. -:: +A handful of settings may still need to be stored in local +configuration files because they affect the ability to connect to the +monitors, authenticate, and fetch configuration information. In most +cases this is limited to the ``mon_host`` option, although this can +also be avoided through the use of DNS SRV records. - ssh {osd-host} - sudo mkdir /var/lib/ceph/osd/ceph-{osd-number} +Sections and masks +------------------ -The ``osd data`` path ideally leads to a mount point with a hard disk that is -separate from the hard disk storing and running the operating system and -daemons. If the OSD is for a disk other than the OS disk, prepare it for -use with Ceph, and mount it to the directory you just created:: +Configuration options stored by the monitor can live in a global +section, daemon type section, or specific daemon section, just like +options in a configuration file can. - ssh {new-osd-host} - sudo mkfs -t {fstype} /dev/{disk} - sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number} +In addition, options may also have a *mask* associated with them to +further restrict which daemons or clients the option applies to. +Masks take two forms: -We recommend using the ``xfs`` file system when running -:command:`mkfs`. (``btrfs`` and ``ext4`` are not recommended and no -longer tested.) +#. ``type:location`` where *type* is a CRUSH property like `rack` or + `host`, and *location* is a value for that property. For example, + ``host:foo`` would limit the option only to daemons or clients + running on a particular host. +#. ``class:device-class`` where *device-class* is the name of a CRUSH + device class (e.g., ``hdd`` or ``ssd``). For example, + ``class:ssd`` would limit the option only to OSDs backed by SSDs. + (This mask has no effect for non-OSD daemons or clients.) -See the `OSD Config Reference`_ for additional configuration details. +When setting a configuration option, the `who` may be a section name, +a mask, or a combination of both separated by a slash (``/``) +character. For example, ``osd/rack:foo`` would mean all OSD daemons +in the ``foo`` rack. +When viewing configuration options, the section name and mask are +generally separated out into separate fields or columns to ease readability. -Heartbeats -========== -During runtime operations, Ceph OSD Daemons check up on other Ceph OSD Daemons -and report their findings to the Ceph Monitor. You do not have to provide any -settings. However, if you have network latency issues, you may wish to modify -the settings. +Commands +-------- -See `Configuring Monitor/OSD Interaction`_ for additional details. +The following CLI commands are used to configure the cluster: +* ``ceph config dump`` will dump the entire configuration database for + the cluster. -.. _ceph-logging-and-debugging: +* ``ceph config get `` will dump the configuration for a specific + daemon or client (e.g., ``mds.a``), as stored in the monitors' + configuration database. -Logs / Debugging -================ +* ``ceph config set