one metadata server in your ``ceph.conf`` configuration file.
.. toctree::
+ :maxdepth: 1
Mount Ceph FS<kernel>
Mount Ceph FS as FUSE <fuse>
Mount Ceph FS in ``fstab`` <fstab>
Using Ceph with Hadoop <hadoop>
+ MDS Configuration <mds-config-ref>
+ Manpage cephfs <../../man/8/cephfs>
+ Manpage ceph-fuse <../../man/8/ceph-fuse>
+ Manpage mount.ceph <../../man/8/mount.ceph>
--- /dev/null
+======================
+ MDS Config Reference
+======================
+
+``mds max file size``
+
+:Description:
+:Type: 64-bit Integer Unsigned
+:Default: 1ULL << 40
+
+``mds cache size``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 100000
+
+``mds cache mid``
+
+:Description:
+:Type: Float
+:Default: 0.7
+
+``mds mem max``
+
+:Description: // KB
+:Type: 32-bit Integer
+:Default: 1048576
+
+``mds dir commit ratio``
+
+:Description:
+:Type: Float
+:Default: 0.5
+
+``mds dir max commit size``
+
+:Description: // MB
+:Type: 32-bit Integer
+:Default: 90
+
+``mds decay halflife``
+
+:Description:
+:Type: Float
+:Default: 5
+
+``mds beacon interval``
+
+:Description:
+:Type: Float
+:Default: 4
+
+``mds beacon grace``
+
+:Description:
+:Type: Float
+:Default: 15
+
+``mds blacklist interval``
+
+:Description: // how long to blacklist failed nodes
+:Type: Float
+:Default: 24.0*60.0
+
+``mds session timeout``
+
+:Description: // cap bits and leases time out if client idle
+:Type: Float
+:Default: 60
+
+``mds session autoclose``
+
+:Description: // autoclose idle session
+:Type: Float
+:Default: 300
+
+``mds reconnect timeout``
+
+:Description: // secs to wait for clients during mds restart
+:Type: Float
+:Default: 45
+
+``mds tick interval``
+
+:Description:
+:Type: Float
+:Default: 5
+
+``mds dirstat min interval``
+
+:Description: //try to avoid propagating more often than x
+:Type: Float
+:Default: 1
+
+``mds scatter nudge interval``
+
+:Description: // how quickly dirstat changes propagate up
+:Type: Float
+:Default: 5
+
+``mds client prealloc inos``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 1000
+
+``mds early reply``
+
+:Description:
+:Type: Boolean
+:Default: true
+
+``mds use tmap``
+
+:Description: // use trivialmap for dir updates
+:Type: Boolean
+:Default: true
+
+``mds default dir hash``
+
+:Description: CEPH STR HASH RJENKINS
+:Type: 32-bit Integer
+:Default:
+
+``mds log``
+
+:Description:
+:Type: Boolean
+:Default: true
+
+``mds log skip corrupt events``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds log max events``
+
+:Description:
+:Type: 32-bit Integer
+:Default: -1
+
+``mds log max segments``
+
+:Description: // segment size defined by FileLayout above
+:Type: 32-bit Integer
+:Default: 30
+
+``mds log max expiring``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 20
+
+``mds log eopen size``
+
+:Description: // # open inodes per log entry
+:Type: 32-bit Integer
+:Default: 100
+
+``mds bal sample interval``
+
+:Description: // every 5 seconds
+:Type: Float
+:Default: 3
+
+``mds bal replicate threshold``
+
+:Description:
+:Type: Float
+:Default: 8000
+
+``mds bal unreplicate threshold``
+
+:Description:
+:Type: Float
+:Default: 0
+
+``mds bal frag``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds bal split size``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 10000
+
+``mds bal split rd``
+
+:Description:
+:Type: Float
+:Default: 25000
+
+``mds bal split wr``
+
+:Description:
+:Type: Float
+:Default: 10000
+
+``mds bal split bits``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 3
+
+``mds bal merge size``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 50
+
+``mds bal merge rd``
+
+:Description:
+:Type: Float
+:Default: 1000
+
+``mds bal merge wr``
+
+:Description:
+:Type: Float
+:Default: 1000
+
+``mds bal interval``
+
+:Description: // seconds
+:Type: 32-bit Integer
+:Default: 10
+
+``mds bal fragment interval``
+
+:Description: // seconds
+:Type: 32-bit Integer
+:Default: 5
+
+``mds bal idle threshold``
+
+:Description:
+:Type: Float
+:Default: 0
+
+``mds bal max``
+
+:Description:
+:Type: 32-bit Integer
+:Default: -1
+
+``mds bal max until``
+
+:Description:
+:Type: 32-bit Integer
+:Default: -1
+
+``mds bal mode``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds bal min rebalance``
+
+:Description: // must be x above avg before we export
+:Type: Float
+:Default: 0.1
+
+``mds bal min start``
+
+:Description: // if we need less x. we don't do anything
+:Type: Float
+:Default: 0.2
+
+``mds bal need min``
+
+:Description: // take within this range of what we need
+:Type: Float
+:Default: 0.8
+
+``mds bal need max``
+
+:Description:
+:Type: Float
+:Default: 1.2
+
+``mds bal midchunk``
+
+:Description: // any sub bigger than this taken in full
+:Type: Float
+:Default: 0.3
+
+``mds bal minchunk``
+
+:Description: // never take anything smaller than this
+:Type: Float
+:Default: 0.001
+
+``mds bal target removal min``
+
+:Description: // min bal iters before old target is removed
+:Type: 32-bit Integer
+:Default: 5
+
+``mds bal target removal max``
+
+:Description: // max bal iters before old target is removed
+:Type: 32-bit Integer
+:Default: 10
+
+``mds replay interval``
+
+:Description: // time to wait before starting replay again
+:Type: Float
+:Default: 1
+
+``mds shutdown check``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds thrash exports``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds thrash fragments``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds dump cache on map``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds dump cache after rejoin``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds verify scatter``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds debug scatterstat``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds debug frag``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds debug auth pins``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds debug subtrees``
+
+:Description:
+:Type: Boolean
+:Default: false
+
+``mds kill mdstable at``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds kill export at``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds kill import at``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds kill link at``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds kill rename at``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``mds wipe sessions``
+
+:Description:
+:Type: Boolean
+:Default: 0
+
+``mds wipe ino prealloc``
+
+:Description:
+:Type: Boolean
+:Default: 0
+
+``mds skip ino``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 0
+
+``max mds``
+
+:Description:
+:Type: 32-bit Integer
+:Default: 1
+
+``mds standby for name``
+
+:Description:
+:Type: String
+:Default:
+
+``mds standby for rank``
+
+:Description:
+:Type: 32-bit Integer
+:Default: -1
+
+``mds standby replay``
+
+:Description:
+:Type: Boolean
+:Default: false
\ No newline at end of file
+++ /dev/null
-==================
- Configuring Ceph
-==================
-
-When you start the Ceph service, the initialization process activates a series
-of daemons that run in the background. The hosts in a typical Ceph cluster run
-at least one of four daemons:
-
-- Object Storage Device (``ceph-osd``)
-- Monitor (``ceph-mon``)
-- Metadata Server (``ceph-mds``)
-- Ceph Gateway (``radosgw``)
-
-For your convenience, each daemon has a series of default values (*i.e.*, many
-are set by ``ceph/src/common/config_opts.h``). You may override these settings
-with a Ceph configuration file.
-
-.. _ceph-conf-file:
-
-The ceph.conf File
-==================
-
-When you start a Ceph cluster, each daemon looks for a ``ceph.conf`` file that
-provides its configuration settings. For manual deployments, you need to create
-a ``ceph.conf`` file to configure your cluster. For third party tools that
-create configuration files for you (*e.g.*, Chef), you may use the information
-contained herein as a reference. The ``ceph.conf`` file defines:
-
-- Cluster membership
-- Host names
-- Host addresses
-- Paths to keyrings
-- Paths to journals
-- Paths to data
-- Other runtime options
-
-The default ``ceph.conf`` locations in sequential order include:
-
-#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` environment variable)
-#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument)
-#. ``/etc/ceph/ceph.conf``
-#. ``~/.ceph/config``
-#. ``./ceph.conf`` (*i.e.,* in the current working directory)
-
-
-The ``ceph.conf`` file uses an *ini* style syntax. You can add comments to the
-``ceph.conf`` file by preceding comments with a semi-colon (;) or a pound sign
-(#). For example:
-
-.. code-block:: ini
-
- # <--A number (#) sign precedes a comment.
- ; A comment may be anything.
- # Comments always follow a semi-colon (;) or a pound (#) on each line.
- # The end of the line terminates a comment.
- # We recommend that you provide comments in your configuration file(s).
-
-.. _ceph-conf-settings:
-
-ceph.conf Settings
-==================
-
-The ``ceph.conf`` file can configure all daemons in a cluster, or all daemons of
-a particular type. To configure a series of daemons, the settings must be
-included under the processes that will receive the configuration as follows:
-
-``[global]``
-
-:Description: Settings under ``[global]`` affect all daemons in a Ceph cluster.
-:Example: ``auth supported = cephx``
-
-``[osd]``
-
-:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in the cluster.
-:Example: ``osd journal size = 1000``
-
-``[mon]``
-
-:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in the cluster.
-:Example: ``mon addr = 10.0.0.101:6789``
-
-
-``[mds]``
-
-:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in the cluster.
-:Example: ``host = myserver01``
-
-
-Global settings affect all instances of all daemon in the cluster. Use the ``[global]``
-setting for values that are common for all daemons in the cluster. You can override each
-``[global]`` setting by:
-
-#. Changing the setting in a particular process type (*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ).
-#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` )
-
-Overriding a global setting affects all child processes, except those that
-you specifically override.
-
-A typical global setting involves activating authentication. For example:
-
-.. code-block:: ini
-
- [global]
- # Enable authentication between hosts within the cluster.
- auth supported = cephx
-
-
-You can specify settings that apply to a particular type of daemon. When you
-specify settings under ``[osd]``, ``[mon]`` or ``[mds]`` without specifying a
-particular instance, the setting will apply to all OSDs, monitors or metadata
-daemons respectively.
-
-You may specify settings for particular instances of a daemon. You may specify
-an instance by entering its type, delimited by a period (.) and by the
-instance ID. The instance ID for an OSD is always numeric, but it may be
-alphanumeric for monitors and metadata servers.
-
-.. code-block:: ini
-
- [osd.1]
- # settings affect osd.1 only.
-
- [mon.a]
- # settings affect mon.a only.
-
- [mds.b]
- # settings affect mds.b only.
-
-.. _ceph-metavariables:
-
-Metavariables
-=============
-
-Metavariables simplify cluster configuration dramatically. When a metavariable
-is set in a configuration value, Ceph expands the metavariable into a concrete
-value. Metavariables are very powerful when used within the ``[global]``,
-``[osd]``, ``[mon]`` or ``[mds]`` sections of your configuration file. Ceph
-metavariables are similar to Bash shell expansion.
-
-Ceph supports the following metavariables:
-
-
-``$cluster``
-
-:Description: Expands to the cluster name. Useful when running multiple clusters on the same hardware.
-:Example: ``/etc/ceph/$cluster.keyring``
-:Default: ``ceph``
-
-
-``$type``
-
-:Description: Expands to one of ``mds``, ``osd``, or ``mon``, depending on the type of the current daemon.
-:Example: ``/var/lib/ceph/$type``
-
-
-``$id``
-
-:Description: Expands to the daemon identifier. For ``osd.0``, this would be ``0``; for ``mds.a``, it would be ``a``.
-:Example: ``/var/lib/ceph/$type/$cluster-$id``
-
-
-``$host``
-
-:Description: Expands to the host name of the current daemon.
-
-
-``$name``
-
-:Description: Expands to ``$type.$id``.
-:Example: ``/var/run/ceph/$cluster-$name.asok``
-
-.. _ceph-conf-common-settings:
-
-Common Settings
-===============
-
-The `Hardware Recommendations`_ section provides some hardware guidelines for
-configuring the cluster. It is possible for a single host to run multiple
-daemons. For example, a single host with multiple disks or RAIDs may run one
-``ceph-osd`` for each disk or RAID. Additionally, a host may run both a
-``ceph-mon`` and an ``ceph-osd`` daemon on the same host. Ideally, you will have
-a host for a particular type of process. For example, one host may run
-``ceph-osd`` daemons, another host may run a ``ceph-mds`` daemon, and other
-hosts may run ``ceph-mon`` daemons.
-
-Each host has a name identified by the ``host`` setting. Monitors also specify
-a network address and port (i.e., domain name or IP address) identified by the
-``addr`` setting. A basic configuration file will typically specify only
-minimal settings for each instance of a daemon. For example:
-
-.. code-block:: ini
-
- [mon.a]
- host = hostName
- mon addr = 150.140.130.120:6789
-
- [osd.0]
- host = hostName
-
-.. important:: The ``host`` setting is the short name of the host (i.e., not
- an fqdn). It is **NOT** and IP address either. Enter ``hostname -s`` on
- the command line to retrieve the name of the host. Also, this setting is
- **ONLY** for ``mkcephfs`` and manual deployment. It **MUST NOT**
- be used with ``chef`` or ``ceph-deploy``.
-
-.. _Hardware Recommendations: ../../install/hardware-recommendations
-
-.. _ceph-network-config:
-
-Networks
-========
-
-Monitors listen on port 6789 by default, while metadata servers and OSDs listen
-on the first available port beginning at 6800. Ensure that you open port 6789 on
-hosts that run a monitor daemon, and open one port beginning at port 6800 for
-each OSD or metadata server that runs on the host. Ports are host-specific, so
-you don't need to open any more ports open than the number of daemons running on
-that host, other than potentially a few spares. You may consider opening a few
-additional ports in case a daemon fails and restarts without letting go of the
-port such that the restarted daemon binds to a new port. If you set up separate
-public and cluster networks, you may need to make entries for each network.
-For example::
-
- iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:6810 -j ACCEPT
-
-
-In our `hardware recommendations`_ section, we recommend having at least two NIC
-cards, because Ceph can support two networks: a public (front-side) network, and
-a cluster (back-side) network. Ceph functions just fine with a public network
-only. You only need to specify the public and cluster network settings if you
-use both public and cluster networks.
-
-There are several reasons to consider operating two separate networks. First,
-OSDs handle data replication for the clients. When OSDs replicate data more than
-once, the network load between OSDs easily dwarfs the network load between
-clients and the Ceph cluster. This can introduce latency and create a
-performance problem. Second, while most people are generally civil, a very tiny
-segment of the population likes to engage in what's known as a Denial of Service
-(DoS) attack. When traffic between OSDs gets disrupted, placement groups may no
-longer reflect an ``active + clean`` state, which may prevent users from reading
-and writing data. A great way to defeat this type of attack is to maintain a
-completely separate cluster network that doesn't connect directly to the
-internet.
-
-To configure the networks, add the following options to the ``[global]`` section
-of your ``ceph.conf`` file.
-
-.. code-block:: ini
-
- [global]
- public network {public-network-ip-address/netmask}
- cluster network {enter cluster-network-ip-address/netmask}
-
-To configure Ceph hosts to use the networks, you should set the following options
-in the daemon instance sections of your ``ceph.conf`` file.
-
-.. code-block:: ini
-
- [osd.0]
- public addr {host-public-ip-address}
- cluster addr {host-cluster-ip-address}
-
-.. _hardware recommendations: ../../install/hardware-recommendations
-
-
-.. _ceph-monitor-config:
-
-Monitors
-========
-
-Ceph production clusters typically deploy with a minimum 3 monitors to ensure
-high availability should a monitor instance crash. An odd number of monitors (3)
-ensures that the Paxos algorithm can determine which version of the cluster map
-is the most recent from a quorum of monitors.
-
-.. note:: You may deploy Ceph with a single monitor, but if the instance fails,
- the lack of a monitor may interrupt data service availability.
-
-Ceph monitors typically listen on port ``6789``. For example:
-
-.. code-block:: ini
-
- [mon.a]
- host = hostName
- mon addr = 150.140.130.120:6789
-
-By default, Ceph expects that you will store a monitor's data under the following path::
-
- /var/lib/ceph/mon/$cluster-$id
-
-You must create the corresponding directory yourself. With metavariables fully
-expressed and a cluster named "ceph", the foregoing directory would evaluate to::
-
- /var/lib/ceph/mon/ceph-a
-
-You may override this path using the ``mon data`` setting. We don't recommend
-changing the default location. Create the default directory on your new monitor host. ::
-
- ssh {new-mon-host}
- sudo mkdir /var/lib/ceph/mon/ceph-{mon-letter}
-
-
-.. _ceph-osd-config:
-
-OSDs
-====
-
-Ceph production clusters typically deploy OSDs where one host has one OSD daemon
-running a filestore on one data disk. A typical deployment specifies a journal
-size and whether the file store's extended attributes (XATTRs) use an
-object map (i.e., when running on the ``ext4`` filesystem). For example:
-
-.. code-block:: ini
-
- [osd]
- osd journal size = 10000
- filestore xattr use omap = true #enables the object map. Only if running ext4.
-
- [osd.0]
- hostname = {hostname}
-
-
-By default, Ceph expects that you will store an OSD's data with the following path::
-
- /var/lib/ceph/osd/$cluster-$id
-
-You must create the corresponding directory yourself. With metavariables fully
-expressed and a cluster named "ceph", the foregoing directory would evaluate to::
-
- /var/lib/ceph/osd/ceph-0
-
-You may override this path using the ``osd data`` setting. We don't recommend
-changing the default location. Create the default directory on your new OSD host. ::
-
- ssh {new-osd-host}
- sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
-
-The ``osd data`` path ideally leads to a mount point with a hard disk that is
-separate from the hard disk storing and running the operating system and
-daemons. If the OSD is for a disk other than the OS disk, prepare it for
-use with Ceph, and mount it to the directory you just created::
-
- ssh {new-osd-host}
- sudo mkfs -t {fstype} /dev/{disk}
- sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}
-
-We recommend using the ``xfs`` file system or the ``btrfs`` file system when
-running :command:mkfs.
-
-By default, Ceph expects that you will store an OSDs journal with the following path::
-
- /var/lib/ceph/osd/$cluster-$id/journal
-
-Without performance optimization, Ceph stores the journal on the same disk as
-the OSDs data. An OSD optimized for performance may use a separate disk to store
-journal data (e.g., a solid state drive delivers high performance journaling).
-
-Ceph's default ``osd journal size`` is 0, so you will need to set this in your
-``ceph.conf`` file. A journal size should find the product of the ``filestore
-min sync interval`` and the expected throughput, and multiple the product by
-two (2)::
-
- osd journal size = {2 * (expected throughput * filestore min sync interval)}
-
-The expected throughput number should include the expected disk throughput
-(i.e., sustained data transfer rate), and network throughput. For example,
-a 7200 RPM disk will likely have approximately 100 MB/s. Taking the ``min()``
-of the disk and network throughput should provide a reasonable expected
-throughput. Some users just start off with a 10GB journal size. For
-example::
-
- osd journal size = 10000
-
-.. _ceph-logging-and-debugging:
-
-Logs / Debugging
-================
-
-Ceph is still on the leading edge, so you may encounter situations that require
-modifying logging output and using Ceph's debugging. To activate Ceph's
-debugging output (*i.e.*, ``dout()``), you may add ``debug`` settings to your
-configuration. Ceph's logging levels operate on a scale of 1 to 20, where 1 is
-terse and 20 is verbose. Subsystems common to each daemon may be set under
-``[global]`` in your configuration file. Subsystems for particular daemons are
-set under the daemon section in your configuration file (*e.g.*, ``[mon]``,
-``[osd]``, ``[mds]``). For example::
-
- [global]
- debug ms = 1
-
- [mon]
- debug mon = 20
- debug paxos = 20
- debug auth = 20
-
- [osd]
- debug osd = 20
- debug filestore = 20
- debug journal = 20
- debug monc = 20
-
- [mds]
- debug mds = 20
- debug mds balancer = 20
- debug mds log = 20
- debug mds migrator = 20
-
-When your system is running well, choose appropriate logging levels and remove
-unnecessary debugging settings to ensure your cluster runs optimally. Logging
-debug output messages is relatively slow, and a waste of resources when operating
-your cluster.
-
-.. tip: When debug output slows down your system, the latency can hide race conditions.
-
-Each subsystem has a logging level for its output logs, and for its logs
-in-memory. You may set different values for each of these subsystems by setting
-a log file level and a memory level for debug logging. For example::
-
- debug {subsystem} {log-level}/{memory-level}
- #for example
- debug mds log 1/20
-
-+--------------------+-----------+--------------+
-| Subsystem | Log Level | Memory Level |
-+====================+===========+==============+
-| ``default`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``lockdep`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``context`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``crush`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds balancer`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds locker`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds log`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds log expire`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``mds migrator`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``buffer`` | 0 | 0 |
-+--------------------+-----------+--------------+
-| ``timer`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``filer`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``objecter`` | 0 | 0 |
-+--------------------+-----------+--------------+
-| ``rados`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``rbd`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``journaler`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``objectcacher`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``client`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``osd`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``optracker`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``objclass`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``filestore`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``journal`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``ms`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``mon`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``monc`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``paxos`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``tp`` | 0 | 5 |
-+--------------------+-----------+--------------+
-| ``auth`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``finisher`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``heartbeatmap`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``perfcounter`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``rgw`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``hadoop`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``asok`` | 1 | 5 |
-+--------------------+-----------+--------------+
-| ``throttle`` | 1 | 5 |
-+--------------------+-----------+--------------+
-
-
-Example ceph.conf
-=================
-
-.. literalinclude:: demo-ceph.conf
- :language: ini
-
-.. _ceph-runtime-config:
-
-Runtime Changes
-===============
-
-Ceph allows you to make changes to the configuration of an ``ceph-osd``,
-``ceph-mon``, or ``ceph-mds`` daemon at runtime. This capability is quite
-useful for increasing/decreasing logging output, enabling/disabling debug
-settings, and even for runtime optimization. The following reflects runtime
-configuration usage::
-
- ceph {daemon-type} tell {id or *} injectargs '--{name} {value} [--{name} {value}]'
-
-Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply
-the runtime setting to all daemons of a particular type with ``*``, or specify
-a specific daemon's ID (i.e., its number or letter). For example, to increase
-debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following::
-
- ceph osd tell 0 injectargs '--debug-osd 20 --debug-ms 1'
-
-In your ``ceph.conf`` file, you may use spaces when specifying a
-setting name. When specifying a setting name on the command line,
-ensure that you use an underscore or hyphen (``_`` or ``-``) between
-terms (e.g., ``debug osd`` becomes ``debug-osd``).
-
-
-Viewing a Configuration at Runtime
-==================================
-
-If your Ceph cluster is running, and you would like to see the configuration
-settings from a running daemon, execute the following::
-
- ceph --admin-daemon {/path/to/admin/socket} config show | less
-
-The default path for the admin socket for each daemon is::
-
- /var/run/ceph/$cluster-$name.asok
-
-At real time, the metavariables will evaluate to the actual cluster name
-and daemon name. For example, if the cluster name is ``ceph`` (it is by default)
-and you want to retrieve the configuration for ``osd.0``, use the following::
-
- ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less
+++ /dev/null
-=====================
- Deploying with Chef
-=====================
-
-We use Chef cookbooks to deploy Ceph. See `Managing Cookbooks with Knife`_ for details
-on using ``knife``. For Chef installation instructions, see `Installing Chef`_.
-
-.. _clonecbs:
-
-Clone the Required Cookbooks
-----------------------------
-
-To get the cookbooks for Ceph, clone them from git.::
-
- cd ~/chef-cookbooks
- git clone https://github.com/opscode-cookbooks/apache2.git
- git clone https://github.com/ceph/ceph-cookbooks.git ceph
-
-.. _addcbpaths:
-
-Add the Required Cookbook Paths
--------------------------------
-
-If you added a default cookbook path when you installed Chef, ``knife``
-may be able to upload the cookbook you've cloned to your cookbook path
-directory without further configuration. If you used a different path,
-or if the cookbook repository you cloned has a different tree structure,
-add the required cookbook path to your ``knife.rb`` file. The
-``cookbook_path`` setting takes a string or an array of strings.
-For example, you can replace a string path with an array of string paths::
-
- cookbook_path '/home/{user-name}/chef-cookbooks/'
-
-Becomes::
-
- cookbook_path [
- '/home/{user-name}/chef-cookbooks/',
- '/home/{user-name}/chef-cookbooks/{another-directory}/',
- '/some/other/path/to/cookbooks/'
- ]
-
-.. _installcbs:
-
-Install the Cookbooks
----------------------
-
-To install Ceph, you must upload the Ceph cookbooks and the Apache cookbooks
-(for use with RADOSGW) to your Chef server. ::
-
- knife cookbook upload apache2 ceph
-
-.. _configcephenv:
-
-Configure your Ceph Environment
--------------------------------
-
-The Chef server can support installation of software for multiple environments.
-The environment you create for Ceph requires an ``fsid``, the secret for
-your monitor(s) if you are running Ceph with ``cephx`` authentication, and
-the host name (i.e., short name) for your monitor hosts.
-
-.. tip: Open an empty text file to hold the following values until you create
- your Ceph environment.
-
-For the filesystem ID, use ``uuidgen`` from the ``uuid-runtime`` package to
-generate a unique identifier. ::
-
- uuidgen -r
-
-For the monitor(s) secret(s), use ``ceph-authtool`` to generate the secret(s)::
-
- sudo apt-get update
- sudo apt-get install ceph-common
- ceph-authtool /dev/stdout --name=mon. --gen-key
-
-The secret is the value to the right of ``"key ="``, and should look something
-like this::
-
- AQBAMuJPINJgFhAAziXIrLvTvAz4PRo5IK/Log==
-
-To create an environment for Ceph, set a command line editor. For example::
-
- export EDITOR=vim
-
-Then, use ``knife`` to create an environment. ::
-
- knife environment create {env-name}
-
-For example::
-
- knife environment create Ceph
-
-A JSON file will appear. Perform the following steps:
-
-#. Enter a description for the environment.
-#. In ``"default_attributes": {}``, add ``"ceph" : {}``.
-#. Within ``"ceph" : {}``, add ``"monitor-secret":``.
-#. Immediately following ``"monitor-secret":`` add the key you generated within quotes, followed by a comma.
-#. Within ``"ceph":{}`` and following the ``monitor-secret`` key-value pair, add ``"config": {}``
-#. Within ``"config": {}`` add ``"fsid":``.
-#. Immediately following ``"fsid":``, add the unique identifier you generated within quotes, followed by a comma.
-#. Within ``"config": {}`` and following the ``fsid`` key-value pair, add ``"mon_initial_members":``
-#. Immediately following ``"mon_initial_members":``, enter the initial monitor host names.
-
-For example::
-
- "default_attributes" : {
- "ceph": {
- "monitor-secret": "{replace-with-generated-secret}",
- "config": {
- "fsid": "{replace-with-generated-uuid}",
- "mon_initial_members": "{replace-with-monitor-hostname(s)}"
- }
- }
- }
-
-Advanced users (i.e., developers and QA) may also add ``"ceph_branch": "{branch}"``
-to ``default-attributes``, replacing ``{branch}`` with the name of the branch you
-wish to use (e.g., ``master``).
-
-.. configroles:
-
-Configure the Roles
--------------------
-
-Navigate to the Ceph cookbooks directory. ::
-
- cd ~/chef-cookbooks/ceph
-
-Create roles for OSDs, monitors, metadata servers, and RADOS Gateways from
-their respective role files. ::
-
- knife role from file roles/ceph-osd.rb
- knife role from file roles/ceph-mon.rb
- knife role from file roles/ceph-mds.rb
- knife role from file roles/ceph-radosgw.rb
-
-.. _confignodes:
-
-Configure Nodes
----------------
-
-You must configure each node you intend to include in your Ceph cluster.
-Identify nodes for your Ceph cluster. ::
-
- knife node list
-
-.. note: for each host where you installed Chef and executed ``chef-client``,
- the Chef server should have a minimal node configuration. You can create
- additional nodes with ``knife node create {node-name}``.
-
-For each node you intend to use in your Ceph cluster, configure the node
-as follows::
-
- knife node edit {node-name}
-
-The node configuration should appear in your text editor. Change the
-``chef_environment`` value to ``Ceph`` (or whatever name you set for your
-Ceph environment).
-
-In the ``run_list``, add ``"recipe[ceph::apt]",`` to all nodes as
-the first setting, so that Chef can install or update the necessary packages.
-Then, add at least one of::
-
- "role[ceph-mon]"
- "role[ceph-osd]"
- "role[ceph-mds]"
- "role[ceph-radosgw]"
-
-If you add more than one role, separate them with a comma. Run ``hostname``
-on your command line, and replace the ``{hostname}`` setting of the ``name``
-key to the host name for the node. ::
-
- {
- "chef_environment": "Ceph",
- "name": "{hostname}",
- "normal": {
- "tags": [
-
- ]
- },
- "run_list": [
- "recipe[ceph::apt]",
- "role[ceph-mon]",
- "role[ceph-mds]"
- ]
- }
-
-.. _prepdisks:
-
-Prepare OSD Disks
------------------
-
-Configuring a node with an OSD role tells Chef that the node will run at
-least one OSD. However, you may run many OSDs on one host. For example,
-you may run one ``ceph-osd`` daemon for each data disk on the system.
-This step prepares the OSD disk(s) and tells Chef how many OSDs the
-node will be running.
-
-
-For the Ceph 0.48 Argonaut release, install ``gdisk``::
-
- sudo apt-get install gdisk
-
-For the Ceph 0.48 Argonaut release, on each hard disk that will store data for
-an OSD daemon, configure the hard disk for use with Ceph. Replace ``{fsid}``
-with the UUID you generated while using ``uuidgen -r``.
-
-.. important: This procedure will erase all information in ``/dev/{disk}``.
-
-::
-
- sudo sgdisk /dev/{disk} --zap-all --clear --mbrtogpt --largest-new=1 --change-name=1:'ceph data' --typecode=1:{fsid}
-
-Create a file system and allocate the disk to your cluster. Specify a
-filesystem (e.g., ``ext4``, ``xfs``, ``btrfs``). When you execute
-``ceph-disk-prepare``, remember to replace ``{fsid}`` with the UUID you
-generated while using ``uuidgen -r``::
-
- sudo mkfs -t ext4 /dev/{disk}
- sudo mount -o user_xattr /dev/{disk} /mnt
- sudo ceph-disk-prepare --cluster-uuid={fsid} /mnt
- sudo umount /mnt
-
-Finally, simulate a hotplug event. ::
-
- sudo udevadm trigger --subsystem-match=block --action=add
-
-
-.. _runchefclient:
-
-Run ``chef-client`` on each Node
---------------------------------
-
-Once you have completed the preceding steps, you must run ``chef-client``
-on each node. For example::
-
- sudo chef-client
-
-.. _proceedtoops:
-
-Proceed to Operating the Cluster
---------------------------------
-
-Once you complete the deployment, you may begin operating your cluster.
-See `Operating a Cluster`_ for details.
-
-
-.. _Managing Cookbooks with Knife: http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife
-.. _Installing Chef: ../../install/chef
-.. _Operating a Cluster: ../../init/
\ No newline at end of file
+++ /dev/null
-[global]
- auth supported = cephx
-
-
-[osd]
- osd journal size = 1000
- # uncomment the following line if you are mounting with ext4
- # filestore xattr use omap = true
-
-[mon.a]
- host = myserver01
- mon addr = 10.0.0.101:6789
-
-[mon.b]
- host = myserver02
- mon addr = 10.0.0.102:6789
-
-[mon.c]
- host = myserver03
- mon addr = 10.0.0.103:6789
-
-[osd.0]
- host = myserver01
-
-[osd.1]
- host = myserver02
-
-[osd.2]
- host = myserver03
-
-[mds.a]
- host = myserver01
+++ /dev/null
-===========================================
- Hard Disk and File System Recommendations
-===========================================
-
-Hard Disk Prep
-==============
-
-Ceph aims for data safety, which means that when the application receives notice
-that data was written to the disk, that data was actually written to the disk.
-For old kernels (<2.6.33), disable the write cache if the journal is on a raw
-disk. Newer kernels should work fine.
-
-Use ``hdparm`` to disable write caching on the hard disk::
-
- sudo hdparm -W 0 /dev/hda 0
-
-In production environments, we recommend running OSDs with an operating system
-disk, and a separate disk(s) for data. If you run data and an operating system
-on a single disk, create a separate partition for your data before configuring
-your OSD cluster.
-
-
-File Systems
-============
-
-Ceph OSDs rely heavily upon the stability and performance of the
-underlying file system.
-
-.. note:: We currently recommend ``XFS`` for production deployments.
- We recommend ``btrfs`` for testing, development, and any
- non-critical deployments. We believe that ``btrfs`` has the correct
- feature set and roadmap to serve Ceph in the long-term, but ``XFS``
- and ``ext4`` provide the necessary stability for today's deployments.
- ``btrfs`` development is proceeding rapidly: users should be
- comfortable installing the latest released upstream kernels and be
- able to track development activity for critical bug fixes.
-
-Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying
-file system for various forms of internal object state and metadata.
-The underlying file system must provide sufficient capacity for
-XATTRs. ``btrfs`` does not bound the total xattr metadata stored with
-a file. ``XFS`` has a relatively large limit (64 KB) that most
-deployments won't encounter, but the ``ext4`` is too small to be
-usable. To use these file systems, you should add the following like
-to the ``[osd]`` section of your ``ceph.conf`` file.::
-
- filestore xattr use omap = true
-
-FS Background Info
-==================
-
-The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly
-scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``.
-Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that
-they are more robust when recovering from crashes, power outages, etc. These
-filesystems journal all of the changes they will make before performing writes.
-
-``XFS`` was developed for Silicon Graphics, and is a mature and stable
-filesystem. By contrast, ``btrfs`` is a relatively new file system that aims
-to address the long-standing wishes of system administrators working with
-large scale data storage environments. ``btrfs`` has some unique features
-and advantages compared to other Linux filesystems.
-
-``btrfs`` is a `copy-on-write`_ filesystem. It supports file creation
-timestamps and checksums that verify metadata integrity, so it can detect
-bad copies of data and fix them with the good copies. The copy-on-write
-capability means that ``btrfs`` can support snapshots that are writable.
-``btrfs`` supports transparent compression and other features.
-
-``btrfs`` also incorporates multi-device management into the file system,
-which enables you to support heterogeneous disk storage infrastructure,
-data allocation policies. The community also aims to provide ``fsck``,
-deduplication, and data encryption support in the future. This compelling
-list of features makes ``btrfs`` the ideal choice for Ceph clusters.
-
-.. _copy-on-write: http://en.wikipedia.org/wiki/Copy-on-write
-.. _compared: http://en.wikipedia.org/wiki/Comparison_of_file_systems
-.. _journaling file systems: http://en.wikipedia.org/wiki/Journaling_file_system
+++ /dev/null
-============================
- Filestore Config Reference
-============================
-
-
-
-``filestore debug omap check``
-
-:Description: Debugging check on synchronization. Expensive. For debugging only.
-:Type: Boolean
-:Required: No
-:Default: ``0``
-
-
-Extended Attributes
-===================
-
-Extended Attributes (XATTRs) are an imporant aspect in your configuration.
-Some file systems have limits on the number of bytes stored in XATTRS.
-Additionally, in some cases, the filesystem may not be as fast as an alternative
-method of storing XATTRs. The following settings may help improve performance
-by using a method of storing XATTRs that is extrinsic to the underlying filesystem.
-
-
-``filestore xattr use omap``
-
-:Description: Use object map for XATTRS. Set to ``true`` for ``ext4`` file systems.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``filestore max inline xattr size``
-
-:Description: The maximimum size of an XATTR stored in the filesystem (i.e., XFS, btrfs, ext4, etc.) per object. Should not be larger than the filesytem can handle.
-:Type: Unsigned 32-bit Integer
-:Required: No
-:Default: ``512``
-
-
-``filestore max inline xattrs``
-
-:Description: The maximum number of XATTRs stored in the fileystem per object.
-:Type: 32-bit Integer
-:Required: No
-:Default: ``2``
-
-
-Synchronization Intervals
-=========================
-
-Periodically, the filestore needs to quiesce writes and synchronize the filesystem,
-which creates a consistent commit point. It can then free journal entries up to
-the commit point. Synchronizing more frequently tends to reduce the time required
-perform synchronization, and reduces the amount of data that needs to remain in the
-journal. Less frequent synchronization allows the backing filesystem to coalesce
-small writes and metadata updates more optimally--potentially resulting in more
-efficient synchronization.
-
-
-``filestore max sync interval``
-
-:Description: The maximum interval in seconds for synchronizing the filestore.
-:Type: Double
-:Required: No
-:Default: ``5``
-
-
-``filestore min sync interval``
-
-:Description: The minimum interval in seconds for synchronizing the filestore.
-:Type: Double
-:Required: No
-:Default: ``.01``
-
-
-Flusher
-=======
-
-The filestore flusher forces data from large writes to be written out using
-``sync file range`` before the sync in order to (hopefully) reduce the cost of
-the eventual sync. In practice, disabling 'filestore flusher' seems to improve
-performance in some cases.
-
-
-``filestore flusher``
-
-:Description: Enables the filestore flusher.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``filestore flusher max fds``
-
-:Description: Sets the maximum number of file descriptors for the flusher.
-:Type: Integer
-:Required: No
-:Default: ``512``
-
-``filestore sync flush``
-
-:Description: Enables the synchronization flusher.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``filestore fsync flushes journal data``
-
-:Description: Flush journal data during filesystem synchronization.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-Queue
-=====
-
-The following settings provide limits on the size of filestore queue.
-
-``filestore queue max ops``
-
-:Description: Defines the maximum number of in progress operations the file store accepts before blocking on queuing new operations.
-:Type: Integer
-:Required: No. Minimal impact on performance.
-:Default: ``500``
-
-
-``filestore queue max bytes``
-
-:Description: The maximum number of bytes for an operation.
-:Type: Integer
-:Required: No
-:Default: ``100 << 20``
-
-
-``filestore queue committing max ops``
-
-:Description: The maximum number of operations the filestore can commit.
-:Type: Integer
-:Required: No
-:Default: ``500``
-
-
-``filestore queue committing max bytes``
-
-:Description: The maximum number of bytes the filestore can commit.
-:Type: Integer
-:Required: No
-:Default: ``100 << 20``
-
-
-
-Timeouts
-========
-
-
-``filestore op threads``
-
-:Description: The number of filesystem operation threads that execute in parallel.
-:Type: Integer
-:Required: No
-:Default: ``2``
-
-
-``filestore op thread timeout``
-
-:Description: The timeout for a filesystem operation thread (in seconds).
-:Type: Integer
-:Required: No
-:Default: ``60``
-
-
-``filestore op thread suicide timeout``
-
-:Description: The timeout for a commit operation before cancelling the commit (in seconds).
-:Type: Integer
-:Required: No
-:Default: ``180``
-
-
-B-Tree Filesystem
-=================
-
-
-``filestore btrfs snap``
-
-:Description: Enable snapshots for a ``btrfs`` filestore.
-:Type: Boolean
-:Required: No. Only used for ``btrfs``.
-:Default: ``true``
-
-
-``filestore btrfs clone range``
-
-:Description: Enable cloning ranges for a ``btrfs`` filestore.
-:Type: Boolean
-:Required: No. Only used for ``btrfs``.
-:Default: ``true``
-
-Journal
-=======
-
-
-``filestore journal parallel``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``filestore journal writeahead``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``filestore journal trailing``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-Misc
-====
-
-
-``filestore merge threshold``
-
-:Description:
-:Type: Integer
-:Required: No
-:Default: ``10``
-
-
-``filestore split multiple``
-
-:Description:
-:Type: Integer
-:Required: No
-:Default: ``2``
-
-
-``filestore update to``
-
-:Description:
-:Type: Integer
-:Required: No
-:Default: ``1000``
-
-
-``filestore blackhole``
-
-:Description: Drop any new transactions on the floor.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``filestore dump file``
-
-:Description: File onto which store transaction dumps?
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``filestore kill at``
-
-:Description: inject a failure at the n'th opportunity
-:Type: String
-:Required: No
-:Default: ``false``
-
-
-``filestore fail eio``
-
-:Description: Fail/Crash on eio.
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
+++ /dev/null
-==========================
- General Config Reference
-==========================
-
-``auth supported``
-
-.. deprecated:: 0.51
-
-:Description: Indicates the type of authentication used. Currently ``cephx`` only. If not specified, it defaults to ``none``.
-:Type: String
-:Required: No
-:Default: ``none``
-
-
-``auth cluster required``
-
-.. versionadded:: 0.51
-
-:Description: Enables authentication for the cluster. Valid setting is ``cephx``.
-:Type: String
-:Required: No
-:Default: ``none``
-
-
-``auth service required``
-
-.. versionadded:: 0.51
-
-:Description: Enables authentication for the service. Valid setting is ``cephx``.
-:Type: String
-:Required: No
-:Default: ``none``
-
-
-
-``auth client required``
-
-.. versionadded:: 0.51
-
-:Description: Enables authentication for the client. Valid setting is ``cephx``.
-:Type: String
-:Required: No
-:Default: ``none``
-
-
-``keyring``
-
-:Description: The path to the cluster's keyring file.
-:Type: String
-:Required: No
-:Default: ``/etc/ceph/$cluster.$name.keyring,/etc/ceph/$cluster.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin``
-
-
-``fsid``
-
-:Description: The filesystem ID. One per cluster.
-:Type: UUID
-:Required: No.
-:Default: N/A. Generated if not specified.
-
-
-``mon host``
-
-:Description: A list of ``{hostname}:{port}`` entries that clients can use to connect to a Ceph monitor. If not set, Ceph searches ``[mon.*]`` sections.
-:Type: String
-:Required: No
-:Default: N/A
-
-
-``host``
-
-:Description: The hostname. Use this setting for specific daemon instances (e.g., ``[osd.0]``).
-:Type: String
-:Required: Yes, for daemon instances.
-:Default: ``localhost``
-
-.. tip:: Do not use ``localhost``. To get your host name, execute ``hostname -s`` on your command line and use the name of your host (to the first period, not the fully-qualified domain name).
-.. important: You should not specify any value for ``host`` when using a third party deployment system that retrieves the host name for you.
-
-
-``public network``
-
-:Description: The IP address and netmask of the public (front-side) network (e.g., ``10.20.30.40/24``). Set in ``[global]``.
-:Type: ``{ip-address}/{netmask}``
-:Required: No
-:Default: N/A
-
-
-``public addr``
-
-:Description: The IP address for the public (front-side) network. Set for each daemon.
-:Type: IP Address
-:Required: No
-:Default: N/A
-
-
-``cluster network``
-
-:Description: The IP address and netmask of the cluster (back-side) network (e.g., ``10.20.30.41/24``). Set in ``[global]``.
-:Type: ``{ip-address}/{netmask}``
-:Required: No
-:Default: N/A
-
-
-``cluster addr``
-
-:Description: The IP address for the cluster (back-side) network. Set for each daemon.
-:Type: Address
-:Required: No
-:Default: N/A
-
-
-``admin socket``
-
-:Description: The socket for executing administrative commands irrespective of whether Ceph monitors have established a quorum.
-:Type: String
-:Required: No
-:Default: ``/var/run/ceph/$cluster-$name.asok``
-
-
-``pid file``
-
-:Description: Each running Ceph daemon has a running process identifier (PID) file.
-:Type: String
-:Required: No
-:Default: N/A. The default path is ``/var/run/$cluster/$name.pid``. The PID file is generated upon start-up.
-
-
-``chdir``
-
-:Description: The directory Ceph daemons change to once they are up and running. Default ``/`` directory recommended.
-:Type: String
-:Required: No
-:Default: ``/``
-
-
-``max open files``
-
-:Description: If set, when the Ceph service starts, Ceph sets the ``max open fds`` at the OS level (i.e., the max # of file descriptors). It helps prevents OSDs from running out of file descriptors.
-:Type: 64-bit Integer
-:Required: No
-:Default: ``0``
-
-``fatal signal handlers``
-
-:Description: If set, we will install signal handlers for SEGV, ABRT, BUS, ILL, FPE, XCPU, XFSZ, SYS signals to generate a useful log message
-:Type: Boolean
-:Default: ``true``
+++ /dev/null
-===============
- Configuration
-===============
-
-Ceph can run with a cluster containing thousands of Object Storage Devices
-(OSDs). A minimal system will have at least two OSDs for data replication. To
-configure OSD clusters, you must provide settings in the configuration file.
-Ceph provides default values for many settings, which you can override in the
-configuration file. Additionally, you can make runtime modification to the
-configuration using command-line utilities.
-
-When Ceph starts, it activates three daemons:
-
-- ``ceph-osd`` (mandatory)
-- ``ceph-mon`` (mandatory)
-- ``ceph-mds`` (mandatory for cephfs only)
-
-Each process, daemon or utility loads the host's configuration file. A process
-may have information about more than one daemon instance (*i.e.,* multiple
-contexts). A daemon or utility only has information about a single daemon
-instance (a single context).
-
-.. note:: Ceph can run on a single host for evaluation purposes.
-
-
-.. raw:: html
-
- <table cellpadding="10"><colgroup><col width="50%"><col width="50%"></colgroup><tbody valign="top"><tr><td><h3>Cluster Configuration</h3>
-
-For general cluster configuration, refer to the following:
-
-.. toctree::
-
- Disks and Filesystems <file-system-recommendations>
- ceph-conf
-
-
-.. raw:: html
-
- </td><td><h3>Configuration Reference</h3>
-
-To optimize the performance of your cluster, refer to the following:
-
-.. toctree::
-
- General Settings <general-config-ref>
- Monitor Settings <mon-config-ref>
- OSD Settings <osd-config-ref>
- Filestore Settings <filestore-config-ref>
- Journal Settings <journal-ref>
- Metadata Server Settings <mds-config-ref>
- librbd Cache Settings <rbd-config-ref>
- Log / Debug Settings <log-and-debug-ref>
-
-
-.. raw:: html
-
- </td></tr><tr><td><h3>Manual Deployment</h3>
-
-To deploy a cluster manually (this is recommended for testing and development only), refer to the following:
-
-.. toctree::
-
- Deploy with mkcephfs <mkcephfs>
-
-
-.. raw:: html
-
- </td><td><h3>Chef Deployment</h3>
-
-To deploy a cluster with chef, refer to the following:
-
-.. toctree::
-
- Deploy with Chef <chef>
-
-.. raw:: html
-
- </td></tr></tbody></table>
+++ /dev/null
-==========================
- Journal Config Reference
-==========================
-
-Ceph OSDs use a journal for two reasons: speed and consistency.
-
-- **Speed:** The journal enables the OSD to commit small writes quickly.
- Ceph writes small, random i/o to the journal sequentially, which tends to
- speed up bursty workloads by allowing the backing filesystem more time to
- coalesce writes. The OSD journal, however, can lead to spiky performance
- with short spurts of high-speed writes followed by periods without any
- write progress as the filesystem catches up to the journal.
-
-- **Consistency:** Ceph OSDs requires a filesystem interface that guarantees
- atomic compound operations. Ceph OSDs write a description of the operation
- to the journal and apply the operation to the filesystem. This enables
- atomic updates to an object (for example, placement group metadata). Every
- few seconds--between ``filestore max sync interval`` and
- ``filestore min sync interval``--the OSD stops writes and synchronizes the
- journal with the filesystem, allowing OSDs to trim operations from the
- journal and reuse the space. On failure, OSDs replay the journal starting
- after the last synchronization operation.
-
-Ceph OSDs support the following journal settings:
-
-``journal dio``
-
-:Description: Enables direct i/o to the journal. Requires ``journal block align`` set to ``true``.
-:Type: Boolean
-:Required: Yes when using ``aio``.
-:Default: ``true``
-
-
-``journal aio``
-
-:Description: Enables using ``libaio`` for asynchronous writes to the journal. Requires ``journal dio`` set to ``true``.
-:Type: Boolean
-:Required: No.
-:Default: ``false``
-
-
-``journal block align``
-
-:Description: Block aligns writes. Required for ``dio`` and ``aio``.
-:Type: Boolean
-:Required: Yes when using ``dio`` and ``aio``.
-:Default: ``true``
-
-
-``journal max write bytes``
-
-:Description: The maximum number of bytes the journal will write at any one time.
-:Type: Integer
-:Required: No
-:Default: ``10 << 20``
-
-
-``journal max write entries``
-
-:Description: The maximum number of entries the journal will write at any one time.
-:Type: Integer
-:Required: No
-:Default: ``100``
-
-
-``journal queue max ops``
-
-:Description: The maximum number of operations allowed in the queue at any one time.
-:Type: Integer
-:Required: No
-:Default: ``500``
-
-
-``journal queue max bytes``
-
-:Description: The maximum number of bytes allowed in the queue at any one time.
-:Type: Integer
-:Required: No
-:Default: ``10 << 20``
-
-
-``journal align min size``
-
-:Description: Align data payloads greater than the specified minimum.
-:Type: Integer
-:Required: No
-:Default: ``64 << 10``
-
-
-``journal zero on create``
-
-:Description: Causes the file store to overwrite the entire journal with ``0``'s during ``mkfs``.
-:Type: Boolean
-:Required: No
-:Default: ``false``
+++ /dev/null
-========================================
- Logging and Debugging Config Reference
-========================================
-
-Logging and debugging settings are not required, but you may override default settings
-as needed. Ceph supports the following settings:
-
-Logging
-=======
-
-``log file``
-
-:Desription: The location of the logging file for your cluster.
-:Type: String
-:Required: No
-:Default: ``/var/log/ceph/$cluster-$name.log``
-
-
-``log max new``
-
-:Description: The maximum number of new log files.
-:Type: Integer
-:Required: No
-:Default: ``1000``
-
-
-``log max recent``
-
-:Description: The maximum number of recent events to include in a log file.
-:Type: Integer
-:Required: No
-:Default: ``1000000``
-
-
-``log to stderr``
-
-:Description: Determines if logging messages should appear in ``stderr``.
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-``err to stderr``
-
-:Description: Determines if error messages should appear in ``stderr``.
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-``log to syslog``
-
-:Description: Determines if logging messages should appear in ``syslog``.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``err to syslog``
-
-:Description: Determines if error messages should appear in ``syslog``.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``log flush on exit``
-
-:Description: Determines if Ceph should flush the log files after exit.
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-``clog to monitors``
-
-:Description: Determines if ``clog`` messages should be sent to monitors.
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-``clog to syslog``
-
-:Description: Determines if ``clog`` messages should be sent to syslog.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``mon cluster log to syslog``
-
-:Description: Determines if the cluster log should be output to the syslog.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``mon cluster log file``
-
-:Description: The location of the cluster's log file.
-:Type: String
-:Required: No
-:Default: ``/var/log/ceph/$cluster.log``
-
-
-
-OSD
-===
-
-
-``osd debug drop ping probability``
-
-:Description: ?
-:Type: Double
-:Required: No
-:Default: 0
-
-
-``osd debug drop ping duration``
-
-:Description: The duration ?
-:Type: Integer
-:Required: No
-:Default: 0
-
-``osd debug drop pg create probability``
-
-:Description:
-:Type: Integer
-:Required: No
-:Default: 0
-
-``osd debug drop pg create duration``
-
-:Description: ?
-:Type: Double
-:Required: No
-:Default: 1
-
-``osd preserve trimmed log``
-
-:Description: ?
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-``osd tmapput sets uses tmap``
-
-:Description: For debug only. ???
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``osd min pg log entries``
-
-:Description: The minimum number of log entries for placement groups.
-:Type: 32-bit Unsigned Integer
-:Required: No
-:Default: 1000
-
-``osd op log threshold``
-
-:Description: How many op log messages to show up in one pass.
-:Type: Integer
-:Required: No
-:Default: 5
-
-
-
-Filestore
-=========
-
-``filestore debug omap check``
-
-:Description: Checks the ``omap``. This is an expensive operation.
-:Type: Boolean
-:Required: No
-:Default: 0
-
-
-MDS
-===
-
-
-``mds debug scatterstat``
-
-:Description: ?
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``mds debug frag``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``mds debug auth pins``
-
-:Description: ?
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``mds debug subtrees``
-
-:Description: ?
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-
-RADOS Gateway
-=============
-
-
-``rgw log nonexistent bucket``
-
-:Description: Should we log a non-existent buckets?
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``rgw log object name``
-
-:Description: Should an object's name be logged. // man date to see codes (a subset are supported)
-:Type: String
-:Required: No
-:Default: ``%Y-%m-%d-%H-%i-%n``
-
-
-``rgw log object name utc``
-
-:Description: Object log name contains UTC?
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``rgw enable ops log``
-
-:Description: Enables logging of every RGW operation.
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-``rgw enable usage log``
-
-:Description: Enable logging of RGW's bandwidth usage.
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-``rgw usage log flush threshold``
-
-:Description: Threshold to flush pending log data.
-:Type: Integer
-:Required: No
-:Default: ``1024``
-
-
-``rgw usage log tick interval``
-
-:Description: Flush pending log data every ``s`` seconds.
-:Type: Integer
-:Required: No
-:Default: 30
-
-
-``rgw intent log object name``
-
-:Description:
-:Type: String
-:Required: No
-:Default: ``%Y-%m-%d-%i-%n``
-
-
-``rgw intent log object name utc``
-
-:Description: Include a UTC timestamp in the intent log object name.
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-``rgw cluster root pool``
-
-:Description: RADOS pool to store radosgw metadata for this instance
-:Type: String
-:Required: No
-:Default: ``.rgw.root``
-
-``rgw gc max objs``
-
-:Description: Number of objects to collect garbage collection data
-:Type: 32-bit Integer
-:Default: 32
-
-``rgw gc obj min wait``
-
-:Description: Minimum time to wait before object's removal and its processing by the garbage collector
-:Type: 32-bit Integer
-:Default: 2 hours. ``2*60*60``
-
-``rgw gc processor max time``
-
-:Description: Max time for a single garbage collection process run
-:Type: 32-bit Integer
-:Default: 1 hour. ``60*60``
-
-``rgw gc processor max period``
-
-:Description: Max time between the beginning of two consecutive garbage collection processes run
-:Type: 32-bit Integer
-:Default: 1 hour. ``60*60``
-
-
+++ /dev/null
-======================
- MDS Config Reference
-======================
-
-``mds max file size``
-
-:Description:
-:Type: 64-bit Integer Unsigned
-:Default: 1ULL << 40
-
-``mds cache size``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 100000
-
-``mds cache mid``
-
-:Description:
-:Type: Float
-:Default: 0.7
-
-``mds mem max``
-
-:Description: // KB
-:Type: 32-bit Integer
-:Default: 1048576
-
-``mds dir commit ratio``
-
-:Description:
-:Type: Float
-:Default: 0.5
-
-``mds dir max commit size``
-
-:Description: // MB
-:Type: 32-bit Integer
-:Default: 90
-
-``mds decay halflife``
-
-:Description:
-:Type: Float
-:Default: 5
-
-``mds beacon interval``
-
-:Description:
-:Type: Float
-:Default: 4
-
-``mds beacon grace``
-
-:Description:
-:Type: Float
-:Default: 15
-
-``mds blacklist interval``
-
-:Description: // how long to blacklist failed nodes
-:Type: Float
-:Default: 24.0*60.0
-
-``mds session timeout``
-
-:Description: // cap bits and leases time out if client idle
-:Type: Float
-:Default: 60
-
-``mds session autoclose``
-
-:Description: // autoclose idle session
-:Type: Float
-:Default: 300
-
-``mds reconnect timeout``
-
-:Description: // secs to wait for clients during mds restart
-:Type: Float
-:Default: 45
-
-``mds tick interval``
-
-:Description:
-:Type: Float
-:Default: 5
-
-``mds dirstat min interval``
-
-:Description: //try to avoid propagating more often than x
-:Type: Float
-:Default: 1
-
-``mds scatter nudge interval``
-
-:Description: // how quickly dirstat changes propagate up
-:Type: Float
-:Default: 5
-
-``mds client prealloc inos``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 1000
-
-``mds early reply``
-
-:Description:
-:Type: Boolean
-:Default: true
-
-``mds use tmap``
-
-:Description: // use trivialmap for dir updates
-:Type: Boolean
-:Default: true
-
-``mds default dir hash``
-
-:Description: CEPH STR HASH RJENKINS
-:Type: 32-bit Integer
-:Default:
-
-``mds log``
-
-:Description:
-:Type: Boolean
-:Default: true
-
-``mds log skip corrupt events``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds log max events``
-
-:Description:
-:Type: 32-bit Integer
-:Default: -1
-
-``mds log max segments``
-
-:Description: // segment size defined by FileLayout above
-:Type: 32-bit Integer
-:Default: 30
-
-``mds log max expiring``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 20
-
-``mds log eopen size``
-
-:Description: // # open inodes per log entry
-:Type: 32-bit Integer
-:Default: 100
-
-``mds bal sample interval``
-
-:Description: // every 5 seconds
-:Type: Float
-:Default: 3
-
-``mds bal replicate threshold``
-
-:Description:
-:Type: Float
-:Default: 8000
-
-``mds bal unreplicate threshold``
-
-:Description:
-:Type: Float
-:Default: 0
-
-``mds bal frag``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds bal split size``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 10000
-
-``mds bal split rd``
-
-:Description:
-:Type: Float
-:Default: 25000
-
-``mds bal split wr``
-
-:Description:
-:Type: Float
-:Default: 10000
-
-``mds bal split bits``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 3
-
-``mds bal merge size``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 50
-
-``mds bal merge rd``
-
-:Description:
-:Type: Float
-:Default: 1000
-
-``mds bal merge wr``
-
-:Description:
-:Type: Float
-:Default: 1000
-
-``mds bal interval``
-
-:Description: // seconds
-:Type: 32-bit Integer
-:Default: 10
-
-``mds bal fragment interval``
-
-:Description: // seconds
-:Type: 32-bit Integer
-:Default: 5
-
-``mds bal idle threshold``
-
-:Description:
-:Type: Float
-:Default: 0
-
-``mds bal max``
-
-:Description:
-:Type: 32-bit Integer
-:Default: -1
-
-``mds bal max until``
-
-:Description:
-:Type: 32-bit Integer
-:Default: -1
-
-``mds bal mode``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds bal min rebalance``
-
-:Description: // must be x above avg before we export
-:Type: Float
-:Default: 0.1
-
-``mds bal min start``
-
-:Description: // if we need less x. we don't do anything
-:Type: Float
-:Default: 0.2
-
-``mds bal need min``
-
-:Description: // take within this range of what we need
-:Type: Float
-:Default: 0.8
-
-``mds bal need max``
-
-:Description:
-:Type: Float
-:Default: 1.2
-
-``mds bal midchunk``
-
-:Description: // any sub bigger than this taken in full
-:Type: Float
-:Default: 0.3
-
-``mds bal minchunk``
-
-:Description: // never take anything smaller than this
-:Type: Float
-:Default: 0.001
-
-``mds bal target removal min``
-
-:Description: // min bal iters before old target is removed
-:Type: 32-bit Integer
-:Default: 5
-
-``mds bal target removal max``
-
-:Description: // max bal iters before old target is removed
-:Type: 32-bit Integer
-:Default: 10
-
-``mds replay interval``
-
-:Description: // time to wait before starting replay again
-:Type: Float
-:Default: 1
-
-``mds shutdown check``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds thrash exports``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds thrash fragments``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds dump cache on map``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds dump cache after rejoin``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds verify scatter``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds debug scatterstat``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds debug frag``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds debug auth pins``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds debug subtrees``
-
-:Description:
-:Type: Boolean
-:Default: false
-
-``mds kill mdstable at``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds kill export at``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds kill import at``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds kill link at``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds kill rename at``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``mds wipe sessions``
-
-:Description:
-:Type: Boolean
-:Default: 0
-
-``mds wipe ino prealloc``
-
-:Description:
-:Type: Boolean
-:Default: 0
-
-``mds skip ino``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 0
-
-``max mds``
-
-:Description:
-:Type: 32-bit Integer
-:Default: 1
-
-``mds standby for name``
-
-:Description:
-:Type: String
-:Default:
-
-``mds standby for rank``
-
-:Description:
-:Type: 32-bit Integer
-:Default: -1
-
-``mds standby replay``
-
-:Description:
-:Type: Boolean
-:Default: false
\ No newline at end of file
+++ /dev/null
-=============================
- Deploying with ``mkcephfs``
-=============================
-
-To deploy a test or development cluster, you can use the ``mkcephfs`` tool.
-We do not recommend using this tool for production environments.
-
-
-Enable Login to Cluster Hosts as ``root``
-=========================================
-
-To deploy with ``mkcephfs``, you will need to be able to login as ``root``
-on each host without a password. For each host, perform the following::
-
- sudo passwd root
-
-Enter a password for the root user.
-
-On the admin host, generate an ``ssh`` key without specifying a passphrase
-and use the default locations. ::
-
- ssh-keygen
- Generating public/private key pair.
- Enter file in which to save the key (/ceph-admin/.ssh/id_rsa):
- Enter passphrase (empty for no passphrase):
- Enter same passphrase again:
- Your identification has been saved in /ceph-admin/.ssh/id_rsa.
- Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.
-
-You may use RSA or DSA keys. Once you generate your keys, copy them to each
-OSD host. For example::
-
- ssh-copy-id root@myserver01
- ssh-copy-id root@myserver02
-
-Modify your ``~/.ssh/config`` file to login as ``root``, as follows::
-
- Host myserver01
- Hostname myserver01.fully-qualified-domain.com
- User root
- Host myserver02
- Hostname myserver02.fully-qualified-domain.com
- User root
-
-
-Copy Configuration File to All Hosts
-====================================
-
-Ceph's ``mkcephfs`` deployment script does not copy the configuration file you
-created from the Administration host to the OSD Cluster hosts. Copy the
-configuration file you created (*i.e.,* ``mycluster.conf`` in the example below)
-from the Administration host to ``etc/ceph/ceph.conf`` on each OSD Cluster host
-if you are using ``mkcephfs`` to deploy Ceph.
-
-::
-
- ssh myserver01 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
- ssh myserver02 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
- ssh myserver03 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
-
-
-Create the Default Directories
-==============================
-
-The ``mkcephfs`` deployment script does not create the default server
-directories. Create server directories for each instance of a Ceph daemon (if
-you haven't done so already). The ``host`` variables in the ``ceph.conf`` file
-determine which host runs each instance of a Ceph daemon. Using the exemplary
-``ceph.conf`` file, you would perform the following:
-
-On ``myserver01``::
-
- sudo mkdir /var/lib/ceph/osd/ceph-0
- sudo mkdir /var/lib/ceph/mon/ceph-a
-
-On ``myserver02``::
-
- sudo mkdir /var/lib/ceph/osd/ceph-1
- sudo mkdir /var/lib/ceph/mon/ceph-b
-
-On ``myserver03``::
-
- sudo mkdir /var/lib/ceph/osd/ceph-2
- sudo mkdir /var/lib/ceph/mon/ceph-c
- sudo mkdir /var/lib/ceph/mds/ceph-a
-
-
-Mount Disks to the Data Directories
-===================================
-
-If you are running multiple OSDs per host and one hard disk per OSD, you should
-mount the disk under the OSD data directory (if you haven't done so already).
-
-
-Run ``mkcephfs``
-================
-
-Once you have copied your Ceph Configuration to the OSD Cluster hosts
-and created the default directories, you may deploy Ceph with the
-``mkcephfs`` script.
-
-.. note:: ``mkcephfs`` is a quick bootstrapping tool. It does not handle more
- complex operations, such as upgrades.
-
-For production environments, deploy Ceph using Chef cookbooks. To run
-``mkcephfs``, execute the following::
-
- cd /etc/ceph
- sudo mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring
-
-The script adds an admin key to the ``ceph.keyring``, which is analogous to a
-root password. See `Authentication`_ when running with ``cephx`` enabled.
-
-When you start or stop your cluster, you will not have to use ``sudo`` or
-provide passwords. For example::
-
- service ceph -a start
-
-See `Operating a Cluster`_ for details.
-
-
-.. _Authentication: ../authentication
-.. _Operating a Cluster: ../../init/
\ No newline at end of file
+++ /dev/null
-==========================
- Monitor Config Reference
-==========================
-
-``mon data``
-
-:Description: The monitor's data location.
-:Type: String
-:Default: ``/var/lib/ceph/mon/$cluster-$id``
-
-
-``mon initial members``
-
-:Description: The IDs of initial monitors in a cluster during startup. If specified, Ceph requires an odd number of monitors to form an initial quorum.
-:Type: String
-:Default: None
-
-
-``mon sync fs threshold``
-
-:Description: Synchronize with the filesystem when writing the specified number of objects. Set it to ``0`` to disable it.
-:Type: 32-bit Integer
-:Default: ``5``
-
-
-``mon tick interval``
-
-:Description: A monitor's tick interval in seconds.
-:Type: 32-bit Integer
-:Default: ``5``
-
-
-``mon subscribe interval``
-
-:Description: The refresh interval (in seconds) for subscriptions. The subscription mechanism enables obtaining the cluster maps and log informations.
-:Type: Double
-:Default: ``300``
-
-
-``mon osd auto mark in``
-
-:Description: Ceph will mark any booting OSDs as ``in`` the cluster.
-:Type: Boolean
-:Default: ``false``
-
-
-``mon osd auto mark auto out in``
-
-:Description: Ceph will mark booting OSDs auto marked ``out`` of the cluster as ``in`` the cluster.
-:Type: Boolean
-:Default: ``true``
-
-
-``mon osd auto mark new in``
-
-:Description: Ceph will mark booting new OSDs as ``in`` the cluster.
-:Type: Boolean
-:Default: ``true``
-
-
-``mon osd down out interval``
-
-:Description: The number of seconds Ceph waits before marking an OSD ``down`` and ``out`` if it doesn't respond.
-:Type: 32-bit Integer
-:Default: ``300``
-
-
-``mon osd min up ratio``
-
-:Description: The minimum ratio of ``up`` OSDs before Ceph will mark OSDs ``down``.
-:Type: Double
-:Default: ``.3``
-
-
-``mon osd min in ratio``
-
-:Description: The minimum ratio of ``in`` OSDs before Ceph will mark OSDs ``out``.
-:Type: Double
-:Default: ``.3``
-
-
-``mon lease``
-
-:Description: Length (in seconds) of the lease on the monitor's versions.
-:Type: Float
-:Default: ``5``
-
-
-``mon lease renew interval``
-
-:Description: The interval (in seconds) for the Leader to renew the other monitor's leases.
-:Type: Float
-:Default: ``3``
-
-
-``mon lease ack timeout``
-
-:Description: Number of seconds the Leader will wait for the Peons to acknowledge the lease extension.
-:Type: Float
-:Default: ``10.0``
-
-
-``mon clock drift allowed``
-
-:Description: The clock drift in seconds allowed between monitors.
-:Type: Float
-:Default: ``.050``
-
-
-``mon clock drift warn backoff``
-
-:Description: Exponential backoff for clock drift warnings
-:Type: Float
-:Default: ``5``
-
-
-``mon accept timeout``
-
-:Description: Number of seconds the Leader will wait for the Peons to accept a Paxos update. It is also used during the Paxos recovery phase for similar purposes.
-:Type: Float
-:Default: ``10.0``
-
-
-``mon pg create interval``
-
-:Description: Number of seconds between PG creation in the same OSD.
-:Type: Float
-:Default: ``30.0``
-
-
-``mon pg stuck threshold``
-
-:Description: Number of seconds after which PGs can be considered as being stuck.
-:Type: 32-bit Integer
-:Default: ``300``
-
-
-``mon osd full ratio``
-
-:Description: The percentage of disk space used before an OSD is considered ``full``.
-:Type: Float
-:Default: ``.95``
-
-
-``mon osd nearfull ratio``
-
-:Description: The percentage of disk space used before an OSD is considered ``nearfull``.
-:Type: Float
-:Default: ``.85``
-
-
-``mon globalid prealloc``
-
-:Description: The number of global IDs to pre-allocate for the cluster.
-:Type: 32-bit Integer
-:Default: ``100``
-
-
-``mon osd report timeout``
-
-:Description: The grace period in seconds before declaring unresponsive OSDs ``down``.
-:Type: 32-bit Integer
-:Default: ``900``
-
-
-``mon force standby active``
-
-:Description: should mons force standby-replay mds to be active
-:Type: Boolean
-:Default: true
-
-
-``mon min osdmap epochs``
-
-:Description: Minimum number of OSD map epochs to keep at all times.
-:Type: 32-bit Integer
-:Default: ``500``
-
-
-``mon max pgmap epochs``
-
-:Description: Maximum number of PG map epochs the monitor should keep.
-:Type: 32-bit Integer
-:Default: ``500``
-
-
-``mon max log epochs``
-
-:Description: Maximum number of Log epochs the monitor should keep.
-:Type: 32-bit Integer
-:Default: ``500``
-
-
-``mon max osd``
-
-:Description: The maximum number of OSDs allowed in the cluster.
-:Type: 32-bit Integer
-:Default: ``10000``
-
-
-``mon probe timeout``
-
-:Description: Number of seconds the monitor will wait to find peers before bootstrapping.
-:Type: Double
-:Default: ``2.0``
-
-
-``mon slurp timeout``
-
-:Description: Number of seconds the monitor has to recover using slurp before the process is aborted and the monitor bootstraps.
-:Type: Double
-:Default: ``10.0``
-
-
-``mon slurp bytes``
-
-:Description: Limits the slurp messages to the specified number of bytes.
-:Type: 32-bit Integer
-:Default: ``256 * 1024``
-
-
-``mon client bytes``
-
-:Description: The amount of client message data allowed in memory (in bytes).
-:Type: 64-bit Integer Unsigned
-:Default: ``100ul << 20``
-
-
-``mon daemon bytes``
-
-:Description: The message memory cap for metadata server and OSD messages (in bytes).
-:Type: 64-bit Integer Unsigned
-:Default: ``400ul << 20``
-
-
-``mon max log entries per event``
-
-:Description: The maximum number of log entries per event.
-:Type: Integer
-:Default: ``4096``
+++ /dev/null
-===========
- Messaging
-===========
-
-
-``ms tcp nodelay``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-``ms initial backoff``
-
-:Description:
-:Type: Double
-:Required: No
-:Default: ``.2``
-
-
-``ms max backoff``
-
-:Description:
-:Type: Double
-:Required: No
-:Default: ``15.0``
-
-
-``ms nocrc``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``ms die on bad msg``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``ms dispatch throttle bytes``
-
-:Description:
-:Type: 64-bit Unsigned Integer
-:Required: No
-:Default: ``100 << 20``
-
-
-``ms bind ipv6``
-
-:Description:
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``ms rwthread stack bytes``
-
-:Description:
-:Type: 64-bit Unsigned Integer
-:Required: No
-:Default: ``1024 << 10``
-
-
-``ms tcp read timeout``
-
-:Description:
-:Type: 64-bit Unsigned Integer
-:Required: No
-:Default: ``900``
-
-
-``ms inject socket failures``
-
-:Description:
-:Type: 64-bit Unsigned Integer
-:Required: No
-:Default: ``0``
+++ /dev/null
-======================
- OSD Config Reference
-======================
-
-
-
-``osd uuid``
-
-:Description: The universally unique identifier (UUID) for the OSD.
-:Type: UUID
-:Default: None
-
-
-``osd data``
-
-:Description: The path to the OSDs data. You must create the directory. You should mount a data disk at this mount point. We do not recommend changing the default.
-:Type: String
-:Default: ``/var/lib/ceph/osd/$cluster-$id``
-
-
-``osd journal``
-
-:Description: The path to the OSD's journal. This may be a path to a file or a block device (such as a partition of an SSD). If it is a file, you must create the directory to contain it.
-:Type: String
-:Default: ``/var/lib/ceph/osd/$cluster-$id/journal``
-
-
-``osd journal size``
-
-:Description: The size of the journal in megabytes. If this is 0, and the journal is a block device, the entire block device is used. Since v0.54, this is ignored if the journal is a block device, and the entire block device is used.
-:Type: 32-bit Integer
-:Default: ``1024``
-:Recommended: Begin with 1GB. Should at least twice the product of the expected speed multiplied by ``filestore min sync interval``.
-
-
-``osd max write size``
-
-:Description: The maximum size of a write in megabytes.
-:Type: 32-bit Integer
-:Default: ``90``
-
-
-``osd client message size cap``
-
-:Description: The largest client data message allowed in memory.
-:Type: 64-bit Integer Unsigned
-:Default: 500MB default. ``500*1024L*1024L``
-
-
-``osd stat refresh interval``
-
-:Description: The status refresh interval in seconds.
-:Type: 64-bit Integer Unsigned
-:Default: ``.5``
-
-
-``osd pg bits``
-
-:Description: Placement group bits per OSD.
-:Type: 32-bit Integer
-:Default: ``6``
-
-
-``osd pgp bits``
-
-:Description: The number of bits per OSD for PGPs.
-:Type: 32-bit Integer
-:Default: ``4``
-
-
-``osd pg layout``
-
-:Description: Placement group layout.
-:Type: 32-bit Integer
-:Default: ``2``
-
-
-``osd pool default crush rule``
-
-:Description: The default CRUSH rule to use when creating a pool.
-:Type: 32-bit Integer
-:Default: ``0``
-
-
-``osd pool default size``
-
-:Description: The default size of an OSD pool in gigabytes. The default value is the same as ``--size 2`` with ``mkpool``.
-:Type: 32-bit Integer
-:Default: ``2``
-
-
-``osd pool default pg num``
-
-:Description: The default number of placement groups for a pool. The default value is the same as ``pg_num`` with ``mkpool``.
-:Type: 32-bit Integer
-:Default: ``8``
-
-
-``osd pool default pgp num``
-
-:Description: The default number of placement groups for placement for a pool. The default value is the same as ``pgp_num`` with ``mkpool``. PG and PGP should be equal (for now).
-:Type: 32-bit Integer
-:Default: ``8``
-
-
-``osd map dedup``
-
-:Description: Enable removing duplicates in the OSD map.
-:Type: Boolean
-:Default: ``true``
-
-
-``osd map cache size``
-
-:Description: The size of the OSD map cache in megabytes.
-:Type: 32-bit Integer
-:Default: ``500``
-
-
-``osd map cache bl size``
-
-:Description: The size of the in-memory OSD map cache in OSD daemons.
-:Type: 32-bit Integer
-:Default: ``50``
-
-
-``osd map cache bl inc size``
-
-:Description: The size of the in-memory OSD map cache incrementals in OSD daemons.
-:Type: 32-bit Integer
-:Default: ``100``
-
-
-``osd map message max``
-
-:Description: The maximum map entries allowed per MOSDMap message.
-:Type: 32-bit Integer
-:Default: ``100``
-
-
-``osd op threads``
-
-:Description: The number of OSD operation threads. Set to ``0`` to disable it. Increasing the number may increase the request processing rate.
-:Type: 32-bit Integer
-:Default: ``2``
-
-
-``osd op thread timeout``
-
-:Description: The OSD operation thread timeout in seconds.
-:Type: 32-bit Integer
-:Default: ``30``
-
-
-``osd disk threads``
-
-:Description: The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming.
-:Type: 32-bit Integer
-:Default: ``1``
-
-
-``osd recovery threads``
-
-:Description: The number of threads for recovering data.
-:Type: 32-bit Integer
-:Default: ``1``
-
-
-``osd recover clone overlap``
-
-:Description: Preserves clone overlap during recovery and data migration.
-:Type: Boolean
-:Default: ``false``
-
-
-``osd backfill scan min``
-
-:Description: The scan interval in seconds for backfill operations.
-:Type: 32-bit Integer
-:Default: ``64``
-
-
-``osd backfill scan max``
-
-:Description: The maximum scan interval in seconds for backfill operations.
-:Type: 32-bit Integer
-:Default: ``512``
-
-
-``osd backlog thread timeout``
-
-:Description: The maximum time in seconds before timing out a backlog thread.
-:Type: 32-bit Integer
-:Default: ``60*60*1``
-
-
-``osd recovery thread timeout``
-
-:Description: The maximum time in seconds before timing out a recovery thread.
-:Type: 32-bit Integer
-:Default: ``30``
-
-
-``osd snap trim thread timeout``
-
-:Description: The maximum time in seconds before timing out a snap trim thread.
-:Type: 32-bit Integer
-:Default: ``60*60*1``
-
-
-``osd scrub thread timeout``
-
-:Description: The maximum time in seconds before timing out a scrub thread.
-:Type: 32-bit Integer
-:Default: ``60``
-
-
-``osd scrub finalize thread timeout``
-
-:Description: The maximum time in seconds before timing out a scrub finalize thread.
-:Type: 32-bit Integer
-:Default: 60*10
-
-
-``osd remove thread timeout``
-
-:Description: The maximum time in seconds before timing out a remove OSD thread.
-:Type: 32-bit Integer
-:Default: 60*60
-
-
-``osd command thread timeout``
-
-:Description: The maximum time in seconds before timing out a command thread.
-:Type: 32-bit Integer
-:Default: ``10*60``
-
-
-``osd heartbeat address``
-
-:Description: An OSD's network address for heartbeats.
-:Type: Address
-:Default: The host address.
-
-
-``osd heartbeat interval``
-
-:Description: How often an OSD pings its peers (in seconds).
-:Type: 32-bit Integer
-:Default: ``6``
-
-
-``osd heartbeat grace``
-
-:Description: The elapsed time when an OSD hasn't shown a heartbeat that the cluster considers it ``down``.
-:Type: 32-bit Integer
-:Default: ``20``
-
-
-``osd _mon_heartbeat interval``
-
-:Description: How often the OSD pings a monitor if it has no OSD peers.
-:Type: 32-bit Integer
-:Default: ``30``
-
-
-``osd mon report interval max``
-
-:Description: The maximum time in seconds for an OSD to report to a monitor before the monitor considers the OSD ``down``.
-:Type: 32-bit Integer
-:Default: ``120``
-
-
-``osd mon report interval min``
-
-:Description: The number of minutes between reports that include ``pg stats``, ``up thru``, ``boot`` and ``failures``.
-:Type: 32-bit Integer
-:Default: ``5``
-
-
-``osd mon ack timeout``
-
-:Description: The number of seconds to wait for a monitor to acknowledge a request for statistics.
-:Type: 32-bit Integer
-:Default: ``30``
-
-
-``osd min down reporters``
-
-:Description: The minimum number of OSDs required to report a ``down`` OSD.
-:Type: 32-bit Integer
-:Default: ``1``
-
-
-``osd min down reports``
-
-:Description: The minimum number of times an OSD must report that another is ``down``.
-:Type: 32-bit Integer
-:Default: ``3``
-
-
-``osd recovery delay start``
-
-:Description: After peering completes, Ceph will delay for the specified number of seconds before starting to recover objects.
-:Type: Float
-:Default: ``15``
-
-
-``osd recovery max active``
-
-:Description: The number of active recovery requests per OSD at one time. More accelerates recovery, but places an increased load on the cluster.
-:Type: 32-bit Integer
-:Default: ``5``
-
-
-``osd recovery max chunk``
-
-:Description: The maximum size of a recovered chunk of data to push.
-:Type: 64-bit Integer Unsigned
-:Default: ``1 << 20``
-
-
-``osd max scrubs``
-
-:Description: The maximum number of scrub operations for an OSD.
-:Type: 32-bit Int
-:Default: ``1``
-
-
-``osd scrub load threshold``
-
-:Description: The maximum CPU load. Ceph will not scrub when the CPU load is higher than this number. Default is 50%.
-:Type: Float
-:Default: ``0.5``
-
-
-``osd scrub min interval``
-
-:Description: The maximum interval in seconds for scrubbing the OSD.
-:Type: Float
-:Default: 5 minutes. ``300``
-
-
-``osd scrub max interval``
-
-:Description: The maximum interval in seconds for scrubbing the OSD.
-:Type: Float
-:Default: Once per day. ``60*60*24``
-
-``osd deep scrub interval``
-
-:Description: The interval for "deep" scrubbing (fully reading all data)
-:Type: Float
-:Default: Once per week. ``60*60*24*7``
-
-``osd deep scrub stride``
-
-:Description: Read siez when doing a deep scrub
-:Type: 32-bit Int
-:Default: 512 KB. ``524288``
-
-``osd class dir``
-
-:Description: The class path for RADOS class plug-ins.
-:Type: String
-:Default: ``$libdir/rados-classes``
-
-
-``osd check for log corruption``
-
-:Description: Check log files for corruption. Can be computationally expensive.
-:Type: Boolean
-:Default: ``false``
-
-
-``osd default notify timeout``
-
-:Description: The OSD default notification timeout (in seconds).
-:Type: 32-bit Integer Unsigned
-:Default: ``30``
-
-
-``osd min pg log entries``
-
-:Description: The minimum number of placement group logs to maintain when trimming log files.
-:Type: 32-bit Int Unsigned
-:Default: 1000
-
-
-``osd op complaint time``
-
-:Description: An operation becomes complaint worthy after the specified number of seconds have elapsed.
-:Type: Float
-:Default: ``30``
-
-
-``osd command max records``
-
-:Description: Limits the number of lost objects to return.
-:Type: 32-bit Integer
-:Default: ``256``
-
-
-``osd auto upgrade tmap``
-
-:Description: Uses ``tmap`` for ``omap`` on old objects.
-:Type: Boolean
-:Default: ``true``
-
-
-``osd tmapput sets users tmap``
-
-:Description: Uses ``tmap`` for debugging only.
-:Type: Boolean
-:Default: ``false``
-
-
-``osd kill backfill at``
-
-:Description: For debugging only.
-:Type: 32-bit Integer
-:Default: ``0``
+++ /dev/null
-=======================
- librbd Cache Settings
-=======================
-
-See `Block Device`_ for additional details.
-
-.. sidebar:: Kernel Caching
-
- The kernel driver for Ceph block devices can use the Linux page cache to
- improve performance.
-
-The user space implementation of the Ceph block device (i.e., ``librbd``) cannot
-take advantage of the Linux page cache, so it includes its own in-memory
-caching, called "RBD caching." RBD caching behaves just like well-behaved hard
-disk caching. When the OS sends a barrier or a flush request, all dirty data is
-written to the OSDs. This means that using write-back caching is just as safe as
-using a well-behaved physical hard disk with a VM that properly sends flushes
-(i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU)
-algorithm, and in write-back mode it can coalesce contiguous requests for
-better throughput.
-
-.. versionadded:: 0.46
-
-Ceph supports write-back caching for RBD. To enable it, add ``rbd cache =
-true`` to the ``[client]`` section of your ``ceph.conf`` file. By default
-``librbd`` does not perform any caching. Writes and reads go directly to the
-storage cluster, and writes return only when the data is on disk on all
-replicas. With caching enabled, writes return immediately, unless there are more
-than ``rbd cache max dirty`` unflushed bytes. In this case, the write triggers
-writeback and blocks until enough bytes are flushed.
-
-.. versionadded:: 0.47
-
-Ceph supports write-through caching for RBD. You can set the size of
-the cache, and you can set targets and limits to switch from
-write-back caching to write through caching. To enable write-through
-mode, set ``rbd cache max dirty`` to 0. This means writes return only
-when the data is on disk on all replicas, but reads may come from the
-cache. The cache is in memory on the client, and each RBD image has
-its own. Since the cache is local to the client, there's no coherency
-if there are others accesing the image. Running GFS or OCFS on top of
-RBD will not work with caching enabled.
-
-The ``ceph.conf`` file settings for RBD should be set in the ``[client]``
-section of your configuration file. The settings include:
-
-
-``rbd cache``
-
-:Description: Enable caching for RADOS Block Device (RBD).
-:Type: Boolean
-:Required: No
-:Default: ``false``
-
-
-``rbd cache size``
-
-:Description: The RBD cache size in bytes.
-:Type: 64-bit Integer
-:Required: No
-:Default: ``32 MiB``
-
-
-``rbd cache max dirty``
-
-:Description: The ``dirty`` limit in bytes at which the cache triggers write-back. If ``0``, uses write-through caching.
-:Type: 64-bit Integer
-:Required: No
-:Constraint: Must be less than ``rbd cache size``.
-:Default: ``24 MiB``
-
-
-``rbd cache target dirty``
-
-:Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache.
-:Type: 64-bit Integer
-:Required: No
-:Constraint: Must be less than ``rbd cache max dirty``.
-:Default: ``16 MiB``
-
-
-``rbd cache max dirty age``
-
-:Description: The number of seconds dirty data is in the cache before writeback starts.
-:Type: Float
-:Required: No
-:Default: ``1.0``
-
-.. _Block Device: ../../rbd/rbd/
\ No newline at end of file
This should be in the header where the function is declared, and
functions should be grouped into logical categories. The `librados C
API`_ provides a complete example. It is pulled into Sphinx by
-`librados.rst`_, which is rendered at :doc:`/api/librados`.
+`librados.rst`_, which is rendered at :doc:`/rados/api/librados`.
.. _`librados C API`: https://github.com/ceph/ceph/blob/master/src/include/rados/librados.h
.. _`librados.rst`: https://raw.github.com/ceph/ceph/master/doc/api/librados.rst
--- /dev/null
+=================\r
+ Release Process\r
+=================\r
+\r
+Build environment\r
+=================\r
+\r
+Ceph maintains multiple build environments. \r
+\r
+Debian\r
+------\r
+\r
+We build Debian-based packages via ``pbuilder`` for multiple distributions, which identifies the build hosts in the ``deb_hosts`` file and the list of distributions in the ``deb_dist`` file. Each build host will build all distributions. Currently Ceph maintains 1 64-bit build host and 1 32-bit build host.\r
+\r
+RPM\r
+---\r
+\r
+We build RPM-based packages natively. Therefore, we maintain one build host per distribution. We build RPM-based packages via ``pbuilder``, which identifies the build hosts in the ``rpm_hosts`` file.\r
+\r
+Prior to building, it's necessary to update the ``pbuilder`` seed tarballs::\r
+\r
+ ./update_all_pbuilders.sh\r
+\r
+2. Setup keyring for signing packages\r
+=====================================\r
+\r
+::\r
+\r
+ export GNUPGHOME=<path to keyring dir>\r
+\r
+ # verify it's accessible\r
+ gpg --list-keys\r
+\r
+The release key should be present::\r
+\r
+ pub 4096R/17ED316D 2012-05-20\r
+ uid Ceph Release Key <sage@newdream.net>\r
+\r
+\r
+3. Set up build area\r
+====================\r
+\r
+Checkout ceph and ceph-build::\r
+\r
+ git clone http://github.com/ceph/ceph.git\r
+ git clone http://github.com/ceph/ceph-build.git\r
+\r
+Checkout next branch::\r
+\r
+ git checkout next\r
+\r
+Checkout the submodules (only needed to prevent errors in recursive make)::\r
+\r
+ git submodule update --init\r
+\r
+4. Update Build version numbers\r
+================================\r
+\r
+Edit configure.ac and change version number::\r
+\r
+ DEBEMAIL user@host dch -v 0.xx-1\r
+\r
+Commit the changes::\r
+\r
+ git commit -a\r
+\r
+Tag the release::\r
+\r
+ ../ceph-build/tag-release v0.xx\r
+\r
+5. Create Makefiles\r
+===================\r
+\r
+The actual configure options used to build packages are in the\r
+``ceph.spec.in`` and ``debian/rules`` files. At this point we just\r
+need to create a Makefile.::\r
+\r
+ ./do_autogen.sh\r
+\r
+\r
+6. Run the release scripts\r
+==========================\r
+\r
+This creates tarballs and copies them, with other needed files to\r
+the build hosts listed in deb_hosts and rpm_hosts, runs a local build\r
+script, then rsyncs the results back tot the specified release directory.::\r
+\r
+ ../ceph-build/do_release.sh /tmp/release\r
+\r
+7. Create RPM Repo\r
+==================\r
+\r
+Copy the rpms to the destination repo, creates the yum repository\r
+rpm and indexes.::\r
+\r
+ ../ceph-build/push_to_rpm_repo.sh /tmp/release /tmp/repo 0.xx\r
+\r
+8. Create debian repo\r
+=====================\r
+\r
+::\r
+\r
+ mkdir /tmp/debian-repo\r
+ ../ceph-build/gen_reprepro_conf.sh debian-testing\r
+ ../ceph-build/push_to_deb_repo.sh /tmp/release /tmp/debian-repo 0.xx main\r
+\r
+9. Push repos to ceph.org\r
+==========================\r
+\r
+For a development release::\r
+\r
+ rcp ceph-0.xx.tar.bz2 ceph-0.xx.tar.gz \\r
+ ceph_site@ceph.com:ceph.com/downloads/.\r
+ rsync -av /tmp/repo/0.52/ ceph_site@ceph.com:ceph.com/rpm-testing\r
+ rsync -av /tmp/debian-repo/ ceph_site@ceph.com:ceph.com/debian-testing\r
+\r
+For a stable release, replace {CODENAME} with the release codename (e.g., ``argonaut`` or ``bobtail``)::\r
+\r
+ rcp ceph-0.xx.tar.bz2 \\r
+ ceph_site@ceph.com:ceph.com/downloads/ceph-0.xx{CODENAME}.tar.bz2\r
+ rcp ceph-0.xx.tar.gz \\r
+ ceph_site@ceph.com:ceph.com/downloads/ceph-0.xx{CODENAME}.tar.gz\r
+ rsync -av /tmp/repo/0.52/ ceph_site@ceph.com:ceph.com/rpm-{CODENAME}\r
+ rsync -auv /tmp/debian-repo/ ceph_site@ceph.com:ceph.com/debian-{CODENAME}\r
+\r
+10. Update Git\r
+==============\r
+\r
+Development release\r
+-------------------\r
+\r
+For a development release, update tags for ``ceph.git``::\r
+\r
+ git push origin v0.xx\r
+ git push origin HEAD:testing\r
+ git checkout master\r
+ git merge next\r
+ git push origin master\r
+ git push origin HEAD:next\r
+\r
+Similarly, for a development release, for both ``teuthology.git`` and ``ceph-qa-suite.git``::\r
+\r
+ git checkout master\r
+ git reset --hard origin/master\r
+ git branch -f testing origin/next\r
+ git push -f origin testing\r
+ git push -f master:next\r
+\r
+Stable release\r
+--------------\r
+\r
+For ``ceph.git``:\r
+\r
+ git push origin stable\r
start/index
install/index
- config-cluster/index
- cluster-ops/index
+ rados/index
cephfs/index
rbd/rbd
radosgw/index
api/index
- Internals <dev/index>
- man/index
architecture
- faq
+ Development <dev/index>
release-notes
+++ /dev/null
-=================
- Installing Chef
-=================
-
-Chef defines three types of entities:
-
-#. **Chef Nodes:** Run ``chef-client``, which installs and manages software.
-#. **Chef Server:** Interacts with ``chef-client`` on Chef nodes.
-#. **Chef Workstation:** Manages the Chef server.
-
-.. image:: ../images/chef.png
-
-See `Chef Architecture Introduction`_ for details.
-
-.. _createuser:
-
-Create a ``chef`` User
-======================
-
-The ``chef-client`` command requires the proper privileges to install and manage
-installations. On each Chef node, we recommend creating a ``chef`` user with
-full ``root`` privileges. For example::
-
- ssh user@chef-node
- sudo useradd -d /home/chef -m chef
- sudo passwd chef
-
-To provide full privileges, add the following to ``/etc/sudoers.d/chef``. ::
-
- echo "chef ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/chef
- sudo chmod 0440 /etc/sudoers.d/chef
-
-If you are using a version of ``sudo`` that doesn't support includes, you will
-need to add the following to the ``/etc/sudoers`` file::
-
- chef ALL = (root) NOPASSWD:ALL
-
-.. important:: Do not change the file permissions on ``/etc/sudoers``. Use a
- suitable tool such as ``visudo``.
-
-.. _genkeys:
-
-Generate SSH Keys for Chef Clients
-==================================
-
-Chef's ``knife`` tool can run ``ssh``. To streamline deployments, we
-recommend generating an SSH key pair without a passphrase for your
-Chef nodes and copying the public key(s) to your Chef nodes so that you
-can connect to them from your workstation using ``ssh`` from ``knife``
-without having to provide a password. To generate a key pair without
-a passphrase, execute the following on your Chef workstation. ::
-
- ssh-keygen
- Generating public/private key pair.
- Enter file in which to save the key (/ceph-admin/.ssh/id_rsa):
- Enter passphrase (empty for no passphrase):
- Enter same passphrase again:
- Your identification has been saved in /ceph-admin/.ssh/id_rsa.
- Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.
-
-You may use RSA or DSA keys. Once you generate your keys, copy them to each
-OSD host. For example::
-
- ssh-copy-id chef@your-node
-
-Consider modifying your ``~/.ssh/config`` file so that it defaults to
-logging in as ``chef`` when no username is specified. ::
-
- Host myserver01
- Hostname myserver01.fqdn-or-ip-address.com
- User chef
- Host myserver02
- Hostname myserver02.fqdn-or-ip-address.com
- User chef
-
-.. _installruby:
-
-Installing Ruby
-===============
-
-Chef requires you to install Ruby. Use the version applicable to your current
-Linux distribution and install Ruby on all of your hosts. ::
-
- sudo apt-get update
- sudo apt-get install ruby
-
-.. _installchefserver:
-
-Installing Chef and Chef Server on a Server
-===========================================
-
-If you plan on hosting your `Chef Server at Opscode`_ you may skip this step,
-but you must make a note of the the fully qualified domain name or IP address
-of your Chef Server for ``knife`` and ``chef-client``.
-
-First, add Opscode packages to your APT configuration. For example::
-
- sudo tee /etc/apt/sources.list.d/chef.list << EOF
- deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
- deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
- EOF
-
-Next, you must request keys so that APT can verify the packages. Copy
-and paste the following line into your command line::
-
- sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A
-
-The key is only used by ``apt``, so remove it from the ``root`` keyring by
-typing ``Y`` when prompted to delete it.
-
-Install the Opscode keyring, Chef and Chef server on the host designated
-as your Chef Server. ::
-
- sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef chef-server
-
-Enter the fully qualified domain name or IP address for your Chef server. For example::
-
- http://fqdn-or-ip-address.com:4000
-
-The Chef server installer will prompt you to enter a temporary password. Enter
-a temporary password (*e.g.,* ``foo``) and proceed with the installation.
-
-.. tip:: When prompted for a temporary password, you may press **OK**.
- The installer wants you to re-enter the password to confirm it. To
- re-enter the password, you must press the **ESC** key.
-
-Once the installer finishes and activates the Chef server, you may enter the
-fully qualified domain name or IP address in a browser to launch the
-Chef web UI. For example::
-
- http://fqdn-or-ip-address.com:4000
-
-The Chef web UI will prompt you to enter the username and password.
-
-- **login:** ``admin``
-- **password:** ``foo``
-
-Once you have entered the temporary password, the Chef web UI will prompt you
-to enter a new password.
-
-.. _installchef:
-
-Install Chef on all Remaining Hosts
-===================================
-
-Install Chef on all Chef Nodes and on the Chef Workstation (if it is not the
-same host as the Chef Server). See `Installing Chef Client on Ubuntu or Debian`_
-for details.
-
-First, add Opscode packages to your APT configuration. For example::
-
- sudo tee /etc/apt/sources.list.d/chef.list << EOF
- deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
- deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
- EOF
-
-Next, you must request keys so that APT can verify the packages. Copy
-and paste the following line into your command line::
-
- sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A
-
-The key is only used by ``apt``, so remove it from the ``root`` keyring by
-typing ``Y`` when prompted to delete it.
-
-Install the Opscode keyring and Chef on all hosts other than the Chef Server. ::
-
- sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef
-
-Enter the fully qualified domain name or IP address for your Chef server.
-For example::
-
- http://fqdn-or-ip-address.com:4000
-
-.. _configknife:
-
-Configuring Knife
-=================
-
-Once you complete the Chef server installation, install ``knife`` on the your
-Chef Workstation. If the Chef server is a remote host, use ``ssh`` to connect. ::
-
- ssh chef@fqdn-or-ip-address.com
-
-In the ``/home/chef`` directory, create a hidden Chef directory. ::
-
- mkdir -p ~/.chef
-
-The server generates validation and web UI certificates with read/write
-permissions for the user that installed the Chef server. Copy them from the
-``/etc/chef`` directory to the ``~/.chef`` directory. Then, change their
-ownership to the current user. ::
-
- sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef && sudo chown $(id -u):$(id -g) ~/.chef/*.pem
-
-From the current user's home directory, configure ``knife`` with an initial
-API client. ::
-
- knife configure -i
-
-The configuration will prompt you for inputs. Answer accordingly:
-
-*Where should I put the config file? [~/.chef/knife.rb]* Press **Enter**
-to accept the default value.
-
-*Please enter the chef server URL:* If you are installing the
-client on the same host as the server, enter ``http://localhost:4000``.
-Otherwise, enter an appropriate URL for the server.
-
-*Please enter a clientname for the new client:* Press **Enter**
-to accept the default value.
-
-*Please enter the existing admin clientname:* Press **Enter**
-to accept the default value.
-
-*Please enter the location of the existing admin client's private key:*
-Override the default value so that it points to the ``.chef`` directory.
-(*e.g.,* ``/home/chef/.chef/webui.pem``)
-
-*Please enter the validation clientname:* Press **Enter** to accept
-the default value.
-
-*Please enter the location of the validation key:* Override the
-default value so that it points to the ``.chef`` directory.
-(*e.g.,* ``/home/chef/.chef/validation.pem``)
-
-*Please enter the path to a chef repository (or leave blank):*
-Leave the entry field blank and press **Enter**.
-
-.. _addcbpath:
-
-Add a Cookbook Path
-===================
-
-Add ``cookbook_path`` to the ``~/.chef/knife.rb`` configuration file
-on your Chef workstation. For example::
-
- cookbook_path '/home/{user-name}/chef-cookbooks/'
-
-Then create the path if it doesn't already exist. ::
-
- mkdir /home/{user-name}/chef-cookbooks
-
-This is where you will store local copies of cookbooks before uploading
-them to the Chef server.
-
-.. _cpvalpem:
-
-Copy ``validation.pem`` to Nodes
-================================
-
-Copy the ``/etc/chef/validation.pem`` file from your Chef server to
-each Chef Node. In a command line shell on the Chef Server, for each node,
-replace ``{nodename}`` in the following line with the node's host name and
-execute it. ::
-
- sudo cat /etc/chef/validation.pem | ssh {nodename} "exec sudo tee /etc/chef/validation.pem >/dev/null"
-
-.. _runchefcli:
-
-Run ``chef-client`` on each Chef Node
-=====================================
-
-Run the ``chef-client`` on each Chef Node so that the nodes
-register with the Chef server. ::
-
- ssh chef-node
- sudo chef-client
-
-.. _verifynodes:
-
-Verify Nodes
-============
-
-Verify that you have setup all the hosts you want to use as
-Chef nodes. ::
-
- knife node list
-
-A list of the nodes you've configured should appear.
-
-See the `Deploy With Chef <../../config-cluster/chef>`_ section for information
-on using Chef to deploy your Ceph cluster.
-
-.. _Chef Architecture Introduction: http://wiki.opscode.com/display/chef/Architecture+Introduction
-.. _Chef Server at Opscode: http://www.opscode.com/hosted-chef/
-.. _Installing Chef Client on Ubuntu or Debian: http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Ubuntu+or+Debian
-.. _Installing Chef Server on Debian or Ubuntu using Packages: http://wiki.opscode.com/display/chef/Installing+Chef+Server+on+Debian+or+Ubuntu+using+Packages
-.. _Knife Bootstrap: http://wiki.opscode.com/display/chef/Knife+Bootstrap
\ No newline at end of file
If you are deploying a Ceph cluster (that is, not developing Ceph),
install Ceph using our stable release packages. For testing, you
-may install development release and testing packages. If you intend
-to deploy Ceph using OpsCode Chef, review the instructions for
-Chef below.
-
-.. tip:: If you are using Dell Crowbar, or Ceph-Deploy you do not
- need to install OpsCode Chef.
+may install development release and testing packages.
.. toctree::
Installing Debian/Ubuntu Packages <debian>
Installing RPM Packages <rpm>
- Installing OpsCode Chef <chef>
.. raw:: html
--- /dev/null
+=========================
+ RADOS Object Store APIs
+=========================
+
+Ceph's RADOS Object Store has a messaging layer protocol that enables clients
+to interact with Ceph monitors and OSDs. ``librados`` provides this
+functionality to object store clients in the form of a library. All Ceph
+clients either use ``librados`` or the same functionality encapsulated in
+``librados`` to interact with the object store. For example, ``librbd`` and
+``libcephfs`` leverage this functionality. You may use ``librados`` to interact
+with Ceph directly (e.g., an application that talks to Ceph, your own interface
+to Ceph, etc.).
+
+For an overview of where ``librados`` appears in the technology stack,
+see :doc:`../../architecture`.
+
+.. toctree::
+
+ librados (C) <librados>
+ librados (C++) <libradospp>
\ No newline at end of file
.. highlight:: c
-`Librados` provides low-level access to the RADOS service. For an
-overview of RADOS, see :doc:`/architecture`.
+`librados` provides low-level access to the RADOS service. For an
+overview of RADOS, see :doc:`../../architecture`.
Example: connecting and writing an object
--- /dev/null
+==================
+ Configuring Ceph
+==================
+
+When you start the Ceph service, the initialization process activates a series
+of daemons that run in the background. The hosts in a typical Ceph cluster run
+at least one of four daemons:
+
+- Object Storage Device (``ceph-osd``)
+- Monitor (``ceph-mon``)
+- Metadata Server (``ceph-mds``)
+- Ceph Gateway (``radosgw``)
+
+For your convenience, each daemon has a series of default values (*i.e.*, many
+are set by ``ceph/src/common/config_opts.h``). You may override these settings
+with a Ceph configuration file.
+
+.. _ceph-conf-file:
+
+The ceph.conf File
+==================
+
+When you start a Ceph cluster, each daemon looks for a ``ceph.conf`` file that
+provides its configuration settings. For manual deployments, you need to create
+a ``ceph.conf`` file to configure your cluster. For third party tools that
+create configuration files for you (*e.g.*, Chef), you may use the information
+contained herein as a reference. The ``ceph.conf`` file defines:
+
+- Cluster membership
+- Host names
+- Host addresses
+- Paths to keyrings
+- Paths to journals
+- Paths to data
+- Other runtime options
+
+The default ``ceph.conf`` locations in sequential order include:
+
+#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` environment variable)
+#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument)
+#. ``/etc/ceph/ceph.conf``
+#. ``~/.ceph/config``
+#. ``./ceph.conf`` (*i.e.,* in the current working directory)
+
+
+The ``ceph.conf`` file uses an *ini* style syntax. You can add comments to the
+``ceph.conf`` file by preceding comments with a semi-colon (;) or a pound sign
+(#). For example:
+
+.. code-block:: ini
+
+ # <--A number (#) sign precedes a comment.
+ ; A comment may be anything.
+ # Comments always follow a semi-colon (;) or a pound (#) on each line.
+ # The end of the line terminates a comment.
+ # We recommend that you provide comments in your configuration file(s).
+
+.. _ceph-conf-settings:
+
+ceph.conf Settings
+==================
+
+The ``ceph.conf`` file can configure all daemons in a cluster, or all daemons of
+a particular type. To configure a series of daemons, the settings must be
+included under the processes that will receive the configuration as follows:
+
+``[global]``
+
+:Description: Settings under ``[global]`` affect all daemons in a Ceph cluster.
+:Example: ``auth supported = cephx``
+
+``[osd]``
+
+:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in the cluster.
+:Example: ``osd journal size = 1000``
+
+``[mon]``
+
+:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in the cluster.
+:Example: ``mon addr = 10.0.0.101:6789``
+
+
+``[mds]``
+
+:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in the cluster.
+:Example: ``host = myserver01``
+
+
+Global settings affect all instances of all daemon in the cluster. Use the ``[global]``
+setting for values that are common for all daemons in the cluster. You can override each
+``[global]`` setting by:
+
+#. Changing the setting in a particular process type (*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ).
+#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` )
+
+Overriding a global setting affects all child processes, except those that
+you specifically override.
+
+A typical global setting involves activating authentication. For example:
+
+.. code-block:: ini
+
+ [global]
+ # Enable authentication between hosts within the cluster.
+ auth supported = cephx
+
+
+You can specify settings that apply to a particular type of daemon. When you
+specify settings under ``[osd]``, ``[mon]`` or ``[mds]`` without specifying a
+particular instance, the setting will apply to all OSDs, monitors or metadata
+daemons respectively.
+
+You may specify settings for particular instances of a daemon. You may specify
+an instance by entering its type, delimited by a period (.) and by the
+instance ID. The instance ID for an OSD is always numeric, but it may be
+alphanumeric for monitors and metadata servers.
+
+.. code-block:: ini
+
+ [osd.1]
+ # settings affect osd.1 only.
+
+ [mon.a]
+ # settings affect mon.a only.
+
+ [mds.b]
+ # settings affect mds.b only.
+
+.. _ceph-metavariables:
+
+Metavariables
+=============
+
+Metavariables simplify cluster configuration dramatically. When a metavariable
+is set in a configuration value, Ceph expands the metavariable into a concrete
+value. Metavariables are very powerful when used within the ``[global]``,
+``[osd]``, ``[mon]`` or ``[mds]`` sections of your configuration file. Ceph
+metavariables are similar to Bash shell expansion.
+
+Ceph supports the following metavariables:
+
+
+``$cluster``
+
+:Description: Expands to the cluster name. Useful when running multiple clusters on the same hardware.
+:Example: ``/etc/ceph/$cluster.keyring``
+:Default: ``ceph``
+
+
+``$type``
+
+:Description: Expands to one of ``mds``, ``osd``, or ``mon``, depending on the type of the current daemon.
+:Example: ``/var/lib/ceph/$type``
+
+
+``$id``
+
+:Description: Expands to the daemon identifier. For ``osd.0``, this would be ``0``; for ``mds.a``, it would be ``a``.
+:Example: ``/var/lib/ceph/$type/$cluster-$id``
+
+
+``$host``
+
+:Description: Expands to the host name of the current daemon.
+
+
+``$name``
+
+:Description: Expands to ``$type.$id``.
+:Example: ``/var/run/ceph/$cluster-$name.asok``
+
+.. _ceph-conf-common-settings:
+
+Common Settings
+===============
+
+The `Hardware Recommendations`_ section provides some hardware guidelines for
+configuring the cluster. It is possible for a single host to run multiple
+daemons. For example, a single host with multiple disks or RAIDs may run one
+``ceph-osd`` for each disk or RAID. Additionally, a host may run both a
+``ceph-mon`` and an ``ceph-osd`` daemon on the same host. Ideally, you will have
+a host for a particular type of process. For example, one host may run
+``ceph-osd`` daemons, another host may run a ``ceph-mds`` daemon, and other
+hosts may run ``ceph-mon`` daemons.
+
+Each host has a name identified by the ``host`` setting. Monitors also specify
+a network address and port (i.e., domain name or IP address) identified by the
+``addr`` setting. A basic configuration file will typically specify only
+minimal settings for each instance of a daemon. For example:
+
+.. code-block:: ini
+
+ [mon.a]
+ host = hostName
+ mon addr = 150.140.130.120:6789
+
+ [osd.0]
+ host = hostName
+
+.. important:: The ``host`` setting is the short name of the host (i.e., not
+ an fqdn). It is **NOT** and IP address either. Enter ``hostname -s`` on
+ the command line to retrieve the name of the host. Also, this setting is
+ **ONLY** for ``mkcephfs`` and manual deployment. It **MUST NOT**
+ be used with ``chef`` or ``ceph-deploy``.
+
+.. _Hardware Recommendations: ../../install/hardware-recommendations
+
+.. _ceph-network-config:
+
+Networks
+========
+
+Monitors listen on port 6789 by default, while metadata servers and OSDs listen
+on the first available port beginning at 6800. Ensure that you open port 6789 on
+hosts that run a monitor daemon, and open one port beginning at port 6800 for
+each OSD or metadata server that runs on the host. Ports are host-specific, so
+you don't need to open any more ports open than the number of daemons running on
+that host, other than potentially a few spares. You may consider opening a few
+additional ports in case a daemon fails and restarts without letting go of the
+port such that the restarted daemon binds to a new port. If you set up separate
+public and cluster networks, you may need to make entries for each network.
+For example::
+
+ iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:6810 -j ACCEPT
+
+
+In our `hardware recommendations`_ section, we recommend having at least two NIC
+cards, because Ceph can support two networks: a public (front-side) network, and
+a cluster (back-side) network. Ceph functions just fine with a public network
+only. You only need to specify the public and cluster network settings if you
+use both public and cluster networks.
+
+There are several reasons to consider operating two separate networks. First,
+OSDs handle data replication for the clients. When OSDs replicate data more than
+once, the network load between OSDs easily dwarfs the network load between
+clients and the Ceph cluster. This can introduce latency and create a
+performance problem. Second, while most people are generally civil, a very tiny
+segment of the population likes to engage in what's known as a Denial of Service
+(DoS) attack. When traffic between OSDs gets disrupted, placement groups may no
+longer reflect an ``active + clean`` state, which may prevent users from reading
+and writing data. A great way to defeat this type of attack is to maintain a
+completely separate cluster network that doesn't connect directly to the
+internet.
+
+To configure the networks, add the following options to the ``[global]`` section
+of your ``ceph.conf`` file.
+
+.. code-block:: ini
+
+ [global]
+ public network {public-network-ip-address/netmask}
+ cluster network {enter cluster-network-ip-address/netmask}
+
+To configure Ceph hosts to use the networks, you should set the following options
+in the daemon instance sections of your ``ceph.conf`` file.
+
+.. code-block:: ini
+
+ [osd.0]
+ public addr {host-public-ip-address}
+ cluster addr {host-cluster-ip-address}
+
+.. _hardware recommendations: ../../install/hardware-recommendations
+
+
+.. _ceph-monitor-config:
+
+Monitors
+========
+
+Ceph production clusters typically deploy with a minimum 3 monitors to ensure
+high availability should a monitor instance crash. An odd number of monitors (3)
+ensures that the Paxos algorithm can determine which version of the cluster map
+is the most recent from a quorum of monitors.
+
+.. note:: You may deploy Ceph with a single monitor, but if the instance fails,
+ the lack of a monitor may interrupt data service availability.
+
+Ceph monitors typically listen on port ``6789``. For example:
+
+.. code-block:: ini
+
+ [mon.a]
+ host = hostName
+ mon addr = 150.140.130.120:6789
+
+By default, Ceph expects that you will store a monitor's data under the following path::
+
+ /var/lib/ceph/mon/$cluster-$id
+
+You must create the corresponding directory yourself. With metavariables fully
+expressed and a cluster named "ceph", the foregoing directory would evaluate to::
+
+ /var/lib/ceph/mon/ceph-a
+
+You may override this path using the ``mon data`` setting. We don't recommend
+changing the default location. Create the default directory on your new monitor host. ::
+
+ ssh {new-mon-host}
+ sudo mkdir /var/lib/ceph/mon/ceph-{mon-letter}
+
+
+.. _ceph-osd-config:
+
+OSDs
+====
+
+Ceph production clusters typically deploy OSDs where one host has one OSD daemon
+running a filestore on one data disk. A typical deployment specifies a journal
+size and whether the file store's extended attributes (XATTRs) use an
+object map (i.e., when running on the ``ext4`` filesystem). For example:
+
+.. code-block:: ini
+
+ [osd]
+ osd journal size = 10000
+ filestore xattr use omap = true #enables the object map. Only if running ext4.
+
+ [osd.0]
+ hostname = {hostname}
+
+
+By default, Ceph expects that you will store an OSD's data with the following path::
+
+ /var/lib/ceph/osd/$cluster-$id
+
+You must create the corresponding directory yourself. With metavariables fully
+expressed and a cluster named "ceph", the foregoing directory would evaluate to::
+
+ /var/lib/ceph/osd/ceph-0
+
+You may override this path using the ``osd data`` setting. We don't recommend
+changing the default location. Create the default directory on your new OSD host. ::
+
+ ssh {new-osd-host}
+ sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
+
+The ``osd data`` path ideally leads to a mount point with a hard disk that is
+separate from the hard disk storing and running the operating system and
+daemons. If the OSD is for a disk other than the OS disk, prepare it for
+use with Ceph, and mount it to the directory you just created::
+
+ ssh {new-osd-host}
+ sudo mkfs -t {fstype} /dev/{disk}
+ sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}
+
+We recommend using the ``xfs`` file system or the ``btrfs`` file system when
+running :command:mkfs.
+
+By default, Ceph expects that you will store an OSDs journal with the following path::
+
+ /var/lib/ceph/osd/$cluster-$id/journal
+
+Without performance optimization, Ceph stores the journal on the same disk as
+the OSDs data. An OSD optimized for performance may use a separate disk to store
+journal data (e.g., a solid state drive delivers high performance journaling).
+
+Ceph's default ``osd journal size`` is 0, so you will need to set this in your
+``ceph.conf`` file. A journal size should find the product of the ``filestore
+min sync interval`` and the expected throughput, and multiple the product by
+two (2)::
+
+ osd journal size = {2 * (expected throughput * filestore min sync interval)}
+
+The expected throughput number should include the expected disk throughput
+(i.e., sustained data transfer rate), and network throughput. For example,
+a 7200 RPM disk will likely have approximately 100 MB/s. Taking the ``min()``
+of the disk and network throughput should provide a reasonable expected
+throughput. Some users just start off with a 10GB journal size. For
+example::
+
+ osd journal size = 10000
+
+.. _ceph-logging-and-debugging:
+
+Logs / Debugging
+================
+
+Ceph is still on the leading edge, so you may encounter situations that require
+modifying logging output and using Ceph's debugging. To activate Ceph's
+debugging output (*i.e.*, ``dout()``), you may add ``debug`` settings to your
+configuration. Ceph's logging levels operate on a scale of 1 to 20, where 1 is
+terse and 20 is verbose. Subsystems common to each daemon may be set under
+``[global]`` in your configuration file. Subsystems for particular daemons are
+set under the daemon section in your configuration file (*e.g.*, ``[mon]``,
+``[osd]``, ``[mds]``). For example::
+
+ [global]
+ debug ms = 1
+
+ [mon]
+ debug mon = 20
+ debug paxos = 20
+ debug auth = 20
+
+ [osd]
+ debug osd = 20
+ debug filestore = 20
+ debug journal = 20
+ debug monc = 20
+
+ [mds]
+ debug mds = 20
+ debug mds balancer = 20
+ debug mds log = 20
+ debug mds migrator = 20
+
+When your system is running well, choose appropriate logging levels and remove
+unnecessary debugging settings to ensure your cluster runs optimally. Logging
+debug output messages is relatively slow, and a waste of resources when operating
+your cluster.
+
+.. tip: When debug output slows down your system, the latency can hide race conditions.
+
+Each subsystem has a logging level for its output logs, and for its logs
+in-memory. You may set different values for each of these subsystems by setting
+a log file level and a memory level for debug logging. For example::
+
+ debug {subsystem} {log-level}/{memory-level}
+ #for example
+ debug mds log 1/20
+
++--------------------+-----------+--------------+
+| Subsystem | Log Level | Memory Level |
++====================+===========+==============+
+| ``default`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``lockdep`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``context`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``crush`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``mds`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``mds balancer`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``mds locker`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``mds log`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``mds log expire`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``mds migrator`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``buffer`` | 0 | 0 |
++--------------------+-----------+--------------+
+| ``timer`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``filer`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``objecter`` | 0 | 0 |
++--------------------+-----------+--------------+
+| ``rados`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``rbd`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``journaler`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``objectcacher`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``client`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``osd`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``optracker`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``objclass`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``filestore`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``journal`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``ms`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``mon`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``monc`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``paxos`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``tp`` | 0 | 5 |
++--------------------+-----------+--------------+
+| ``auth`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``finisher`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``heartbeatmap`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``perfcounter`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``rgw`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``hadoop`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``asok`` | 1 | 5 |
++--------------------+-----------+--------------+
+| ``throttle`` | 1 | 5 |
++--------------------+-----------+--------------+
+
+
+Example ceph.conf
+=================
+
+.. literalinclude:: demo-ceph.conf
+ :language: ini
+
+.. _ceph-runtime-config:
+
+Runtime Changes
+===============
+
+Ceph allows you to make changes to the configuration of an ``ceph-osd``,
+``ceph-mon``, or ``ceph-mds`` daemon at runtime. This capability is quite
+useful for increasing/decreasing logging output, enabling/disabling debug
+settings, and even for runtime optimization. The following reflects runtime
+configuration usage::
+
+ ceph {daemon-type} tell {id or *} injectargs '--{name} {value} [--{name} {value}]'
+
+Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply
+the runtime setting to all daemons of a particular type with ``*``, or specify
+a specific daemon's ID (i.e., its number or letter). For example, to increase
+debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following::
+
+ ceph osd tell 0 injectargs '--debug-osd 20 --debug-ms 1'
+
+In your ``ceph.conf`` file, you may use spaces when specifying a
+setting name. When specifying a setting name on the command line,
+ensure that you use an underscore or hyphen (``_`` or ``-``) between
+terms (e.g., ``debug osd`` becomes ``debug-osd``).
+
+
+Viewing a Configuration at Runtime
+==================================
+
+If your Ceph cluster is running, and you would like to see the configuration
+settings from a running daemon, execute the following::
+
+ ceph --admin-daemon {/path/to/admin/socket} config show | less
+
+The default path for the admin socket for each daemon is::
+
+ /var/run/ceph/$cluster-$name.asok
+
+At real time, the metavariables will evaluate to the actual cluster name
+and daemon name. For example, if the cluster name is ``ceph`` (it is by default)
+and you want to retrieve the configuration for ``osd.0``, use the following::
+
+ ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less
--- /dev/null
+[global]
+ auth supported = cephx
+
+
+[osd]
+ osd journal size = 1000
+ # uncomment the following line if you are mounting with ext4
+ # filestore xattr use omap = true
+
+[mon.a]
+ host = myserver01
+ mon addr = 10.0.0.101:6789
+
+[mon.b]
+ host = myserver02
+ mon addr = 10.0.0.102:6789
+
+[mon.c]
+ host = myserver03
+ mon addr = 10.0.0.103:6789
+
+[osd.0]
+ host = myserver01
+
+[osd.1]
+ host = myserver02
+
+[osd.2]
+ host = myserver03
+
+[mds.a]
+ host = myserver01
--- /dev/null
+============================
+ Filestore Config Reference
+============================
+
+
+
+``filestore debug omap check``
+
+:Description: Debugging check on synchronization. Expensive. For debugging only.
+:Type: Boolean
+:Required: No
+:Default: ``0``
+
+
+Extended Attributes
+===================
+
+Extended Attributes (XATTRs) are an imporant aspect in your configuration.
+Some file systems have limits on the number of bytes stored in XATTRS.
+Additionally, in some cases, the filesystem may not be as fast as an alternative
+method of storing XATTRs. The following settings may help improve performance
+by using a method of storing XATTRs that is extrinsic to the underlying filesystem.
+
+
+``filestore xattr use omap``
+
+:Description: Use object map for XATTRS. Set to ``true`` for ``ext4`` file systems.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``filestore max inline xattr size``
+
+:Description: The maximimum size of an XATTR stored in the filesystem (i.e., XFS, btrfs, ext4, etc.) per object. Should not be larger than the filesytem can handle.
+:Type: Unsigned 32-bit Integer
+:Required: No
+:Default: ``512``
+
+
+``filestore max inline xattrs``
+
+:Description: The maximum number of XATTRs stored in the fileystem per object.
+:Type: 32-bit Integer
+:Required: No
+:Default: ``2``
+
+
+Synchronization Intervals
+=========================
+
+Periodically, the filestore needs to quiesce writes and synchronize the filesystem,
+which creates a consistent commit point. It can then free journal entries up to
+the commit point. Synchronizing more frequently tends to reduce the time required
+perform synchronization, and reduces the amount of data that needs to remain in the
+journal. Less frequent synchronization allows the backing filesystem to coalesce
+small writes and metadata updates more optimally--potentially resulting in more
+efficient synchronization.
+
+
+``filestore max sync interval``
+
+:Description: The maximum interval in seconds for synchronizing the filestore.
+:Type: Double
+:Required: No
+:Default: ``5``
+
+
+``filestore min sync interval``
+
+:Description: The minimum interval in seconds for synchronizing the filestore.
+:Type: Double
+:Required: No
+:Default: ``.01``
+
+
+Flusher
+=======
+
+The filestore flusher forces data from large writes to be written out using
+``sync file range`` before the sync in order to (hopefully) reduce the cost of
+the eventual sync. In practice, disabling 'filestore flusher' seems to improve
+performance in some cases.
+
+
+``filestore flusher``
+
+:Description: Enables the filestore flusher.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``filestore flusher max fds``
+
+:Description: Sets the maximum number of file descriptors for the flusher.
+:Type: Integer
+:Required: No
+:Default: ``512``
+
+``filestore sync flush``
+
+:Description: Enables the synchronization flusher.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``filestore fsync flushes journal data``
+
+:Description: Flush journal data during filesystem synchronization.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+Queue
+=====
+
+The following settings provide limits on the size of filestore queue.
+
+``filestore queue max ops``
+
+:Description: Defines the maximum number of in progress operations the file store accepts before blocking on queuing new operations.
+:Type: Integer
+:Required: No. Minimal impact on performance.
+:Default: ``500``
+
+
+``filestore queue max bytes``
+
+:Description: The maximum number of bytes for an operation.
+:Type: Integer
+:Required: No
+:Default: ``100 << 20``
+
+
+``filestore queue committing max ops``
+
+:Description: The maximum number of operations the filestore can commit.
+:Type: Integer
+:Required: No
+:Default: ``500``
+
+
+``filestore queue committing max bytes``
+
+:Description: The maximum number of bytes the filestore can commit.
+:Type: Integer
+:Required: No
+:Default: ``100 << 20``
+
+
+
+Timeouts
+========
+
+
+``filestore op threads``
+
+:Description: The number of filesystem operation threads that execute in parallel.
+:Type: Integer
+:Required: No
+:Default: ``2``
+
+
+``filestore op thread timeout``
+
+:Description: The timeout for a filesystem operation thread (in seconds).
+:Type: Integer
+:Required: No
+:Default: ``60``
+
+
+``filestore op thread suicide timeout``
+
+:Description: The timeout for a commit operation before cancelling the commit (in seconds).
+:Type: Integer
+:Required: No
+:Default: ``180``
+
+
+B-Tree Filesystem
+=================
+
+
+``filestore btrfs snap``
+
+:Description: Enable snapshots for a ``btrfs`` filestore.
+:Type: Boolean
+:Required: No. Only used for ``btrfs``.
+:Default: ``true``
+
+
+``filestore btrfs clone range``
+
+:Description: Enable cloning ranges for a ``btrfs`` filestore.
+:Type: Boolean
+:Required: No. Only used for ``btrfs``.
+:Default: ``true``
+
+Journal
+=======
+
+
+``filestore journal parallel``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``filestore journal writeahead``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``filestore journal trailing``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+Misc
+====
+
+
+``filestore merge threshold``
+
+:Description:
+:Type: Integer
+:Required: No
+:Default: ``10``
+
+
+``filestore split multiple``
+
+:Description:
+:Type: Integer
+:Required: No
+:Default: ``2``
+
+
+``filestore update to``
+
+:Description:
+:Type: Integer
+:Required: No
+:Default: ``1000``
+
+
+``filestore blackhole``
+
+:Description: Drop any new transactions on the floor.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``filestore dump file``
+
+:Description: File onto which store transaction dumps?
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``filestore kill at``
+
+:Description: inject a failure at the n'th opportunity
+:Type: String
+:Required: No
+:Default: ``false``
+
+
+``filestore fail eio``
+
+:Description: Fail/Crash on eio.
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
--- /dev/null
+===========================================
+ Hard Disk and File System Recommendations
+===========================================
+
+Hard Disk Prep
+==============
+
+Ceph aims for data safety, which means that when the application receives notice
+that data was written to the disk, that data was actually written to the disk.
+For old kernels (<2.6.33), disable the write cache if the journal is on a raw
+disk. Newer kernels should work fine.
+
+Use ``hdparm`` to disable write caching on the hard disk::
+
+ sudo hdparm -W 0 /dev/hda 0
+
+In production environments, we recommend running OSDs with an operating system
+disk, and a separate disk(s) for data. If you run data and an operating system
+on a single disk, create a separate partition for your data before configuring
+your OSD cluster.
+
+
+File Systems
+============
+
+Ceph OSDs rely heavily upon the stability and performance of the
+underlying file system.
+
+.. note:: We currently recommend ``XFS`` for production deployments.
+ We recommend ``btrfs`` for testing, development, and any
+ non-critical deployments. We believe that ``btrfs`` has the correct
+ feature set and roadmap to serve Ceph in the long-term, but ``XFS``
+ and ``ext4`` provide the necessary stability for today's deployments.
+ ``btrfs`` development is proceeding rapidly: users should be
+ comfortable installing the latest released upstream kernels and be
+ able to track development activity for critical bug fixes.
+
+Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying
+file system for various forms of internal object state and metadata.
+The underlying file system must provide sufficient capacity for
+XATTRs. ``btrfs`` does not bound the total xattr metadata stored with
+a file. ``XFS`` has a relatively large limit (64 KB) that most
+deployments won't encounter, but the ``ext4`` is too small to be
+usable. To use these file systems, you should add the following like
+to the ``[osd]`` section of your ``ceph.conf`` file.::
+
+ filestore xattr use omap = true
+
+FS Background Info
+==================
+
+The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly
+scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``.
+Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that
+they are more robust when recovering from crashes, power outages, etc. These
+filesystems journal all of the changes they will make before performing writes.
+
+``XFS`` was developed for Silicon Graphics, and is a mature and stable
+filesystem. By contrast, ``btrfs`` is a relatively new file system that aims
+to address the long-standing wishes of system administrators working with
+large scale data storage environments. ``btrfs`` has some unique features
+and advantages compared to other Linux filesystems.
+
+``btrfs`` is a `copy-on-write`_ filesystem. It supports file creation
+timestamps and checksums that verify metadata integrity, so it can detect
+bad copies of data and fix them with the good copies. The copy-on-write
+capability means that ``btrfs`` can support snapshots that are writable.
+``btrfs`` supports transparent compression and other features.
+
+``btrfs`` also incorporates multi-device management into the file system,
+which enables you to support heterogeneous disk storage infrastructure,
+data allocation policies. The community also aims to provide ``fsck``,
+deduplication, and data encryption support in the future. This compelling
+list of features makes ``btrfs`` the ideal choice for Ceph clusters.
+
+.. _copy-on-write: http://en.wikipedia.org/wiki/Copy-on-write
+.. _compared: http://en.wikipedia.org/wiki/Comparison_of_file_systems
+.. _journaling file systems: http://en.wikipedia.org/wiki/Journaling_file_system
--- /dev/null
+==========================
+ General Config Reference
+==========================
+
+``auth supported``
+
+.. deprecated:: 0.51
+
+:Description: Indicates the type of authentication used. Currently ``cephx`` only. If not specified, it defaults to ``none``.
+:Type: String
+:Required: No
+:Default: ``none``
+
+
+``auth cluster required``
+
+.. versionadded:: 0.51
+
+:Description: Enables authentication for the cluster. Valid setting is ``cephx``.
+:Type: String
+:Required: No
+:Default: ``none``
+
+
+``auth service required``
+
+.. versionadded:: 0.51
+
+:Description: Enables authentication for the service. Valid setting is ``cephx``.
+:Type: String
+:Required: No
+:Default: ``none``
+
+
+
+``auth client required``
+
+.. versionadded:: 0.51
+
+:Description: Enables authentication for the client. Valid setting is ``cephx``.
+:Type: String
+:Required: No
+:Default: ``none``
+
+
+``keyring``
+
+:Description: The path to the cluster's keyring file.
+:Type: String
+:Required: No
+:Default: ``/etc/ceph/$cluster.$name.keyring,/etc/ceph/$cluster.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin``
+
+
+``fsid``
+
+:Description: The filesystem ID. One per cluster.
+:Type: UUID
+:Required: No.
+:Default: N/A. Generated if not specified.
+
+
+``mon host``
+
+:Description: A list of ``{hostname}:{port}`` entries that clients can use to connect to a Ceph monitor. If not set, Ceph searches ``[mon.*]`` sections.
+:Type: String
+:Required: No
+:Default: N/A
+
+
+``host``
+
+:Description: The hostname. Use this setting for specific daemon instances (e.g., ``[osd.0]``).
+:Type: String
+:Required: Yes, for daemon instances.
+:Default: ``localhost``
+
+.. tip:: Do not use ``localhost``. To get your host name, execute ``hostname -s`` on your command line and use the name of your host (to the first period, not the fully-qualified domain name).
+.. important: You should not specify any value for ``host`` when using a third party deployment system that retrieves the host name for you.
+
+
+``public network``
+
+:Description: The IP address and netmask of the public (front-side) network (e.g., ``10.20.30.40/24``). Set in ``[global]``.
+:Type: ``{ip-address}/{netmask}``
+:Required: No
+:Default: N/A
+
+
+``public addr``
+
+:Description: The IP address for the public (front-side) network. Set for each daemon.
+:Type: IP Address
+:Required: No
+:Default: N/A
+
+
+``cluster network``
+
+:Description: The IP address and netmask of the cluster (back-side) network (e.g., ``10.20.30.41/24``). Set in ``[global]``.
+:Type: ``{ip-address}/{netmask}``
+:Required: No
+:Default: N/A
+
+
+``cluster addr``
+
+:Description: The IP address for the cluster (back-side) network. Set for each daemon.
+:Type: Address
+:Required: No
+:Default: N/A
+
+
+``admin socket``
+
+:Description: The socket for executing administrative commands irrespective of whether Ceph monitors have established a quorum.
+:Type: String
+:Required: No
+:Default: ``/var/run/ceph/$cluster-$name.asok``
+
+
+``pid file``
+
+:Description: Each running Ceph daemon has a running process identifier (PID) file.
+:Type: String
+:Required: No
+:Default: N/A. The default path is ``/var/run/$cluster/$name.pid``. The PID file is generated upon start-up.
+
+
+``chdir``
+
+:Description: The directory Ceph daemons change to once they are up and running. Default ``/`` directory recommended.
+:Type: String
+:Required: No
+:Default: ``/``
+
+
+``max open files``
+
+:Description: If set, when the Ceph service starts, Ceph sets the ``max open fds`` at the OS level (i.e., the max # of file descriptors). It helps prevents OSDs from running out of file descriptors.
+:Type: 64-bit Integer
+:Required: No
+:Default: ``0``
+
+``fatal signal handlers``
+
+:Description: If set, we will install signal handlers for SEGV, ABRT, BUS, ILL, FPE, XCPU, XFSZ, SYS signals to generate a useful log message
+:Type: Boolean
+:Default: ``true``
--- /dev/null
+===============
+ Configuration
+===============
+
+Ceph can run with a cluster containing thousands of Object Storage Devices
+(OSDs). A minimal system will have at least two OSDs for data replication. To
+configure OSD clusters, you must provide settings in the configuration file.
+Ceph provides default values for many settings, which you can override in the
+configuration file. Additionally, you can make runtime modification to the
+configuration using command-line utilities.
+
+When Ceph starts, it activates three daemons:
+
+- ``ceph-osd`` (mandatory)
+- ``ceph-mon`` (mandatory)
+- ``ceph-mds`` (mandatory for cephfs only)
+
+Each process, daemon or utility loads the host's configuration file. A process
+may have information about more than one daemon instance (*i.e.,* multiple
+contexts). A daemon or utility only has information about a single daemon
+instance (a single context).
+
+.. note:: Ceph can run on a single host for evaluation purposes.
+
+
+.. raw:: html
+
+ <table cellpadding="10"><colgroup><col width="50%"><col width="50%"></colgroup><tbody valign="top"><tr><td><h3>Configuring the Object Store</h3>
+
+For general object store configuration, refer to the following:
+
+.. toctree::
+
+ Disks and Filesystems <filesystem-recommendations>
+ ceph-conf
+
+
+.. raw:: html
+
+ </td><td><h3>Reference</h3>
+
+To optimize the performance of your cluster, refer to the following:
+
+.. toctree::
+
+ General Settings <general-config-ref>
+ Monitor Settings <mon-config-ref>
+ OSD Settings <osd-config-ref>
+ Filestore Settings <filestore-config-ref>
+ Journal Settings <journal-ref>
+ Log / Debug Settings <log-and-debug-ref>
+ Messaging Settings <ms-ref>
+
+
+.. raw:: html
+
+ </td></tr></tbody></table>
--- /dev/null
+==========================
+ Journal Config Reference
+==========================
+
+Ceph OSDs use a journal for two reasons: speed and consistency.
+
+- **Speed:** The journal enables the OSD to commit small writes quickly.
+ Ceph writes small, random i/o to the journal sequentially, which tends to
+ speed up bursty workloads by allowing the backing filesystem more time to
+ coalesce writes. The OSD journal, however, can lead to spiky performance
+ with short spurts of high-speed writes followed by periods without any
+ write progress as the filesystem catches up to the journal.
+
+- **Consistency:** Ceph OSDs requires a filesystem interface that guarantees
+ atomic compound operations. Ceph OSDs write a description of the operation
+ to the journal and apply the operation to the filesystem. This enables
+ atomic updates to an object (for example, placement group metadata). Every
+ few seconds--between ``filestore max sync interval`` and
+ ``filestore min sync interval``--the OSD stops writes and synchronizes the
+ journal with the filesystem, allowing OSDs to trim operations from the
+ journal and reuse the space. On failure, OSDs replay the journal starting
+ after the last synchronization operation.
+
+Ceph OSDs support the following journal settings:
+
+``journal dio``
+
+:Description: Enables direct i/o to the journal. Requires ``journal block align`` set to ``true``.
+:Type: Boolean
+:Required: Yes when using ``aio``.
+:Default: ``true``
+
+
+``journal aio``
+
+:Description: Enables using ``libaio`` for asynchronous writes to the journal. Requires ``journal dio`` set to ``true``.
+:Type: Boolean
+:Required: No.
+:Default: ``false``
+
+
+``journal block align``
+
+:Description: Block aligns writes. Required for ``dio`` and ``aio``.
+:Type: Boolean
+:Required: Yes when using ``dio`` and ``aio``.
+:Default: ``true``
+
+
+``journal max write bytes``
+
+:Description: The maximum number of bytes the journal will write at any one time.
+:Type: Integer
+:Required: No
+:Default: ``10 << 20``
+
+
+``journal max write entries``
+
+:Description: The maximum number of entries the journal will write at any one time.
+:Type: Integer
+:Required: No
+:Default: ``100``
+
+
+``journal queue max ops``
+
+:Description: The maximum number of operations allowed in the queue at any one time.
+:Type: Integer
+:Required: No
+:Default: ``500``
+
+
+``journal queue max bytes``
+
+:Description: The maximum number of bytes allowed in the queue at any one time.
+:Type: Integer
+:Required: No
+:Default: ``10 << 20``
+
+
+``journal align min size``
+
+:Description: Align data payloads greater than the specified minimum.
+:Type: Integer
+:Required: No
+:Default: ``64 << 10``
+
+
+``journal zero on create``
+
+:Description: Causes the file store to overwrite the entire journal with ``0``'s during ``mkfs``.
+:Type: Boolean
+:Required: No
+:Default: ``false``
--- /dev/null
+========================================
+ Logging and Debugging Config Reference
+========================================
+
+Logging and debugging settings are not required, but you may override default settings
+as needed. Ceph supports the following settings:
+
+Logging
+=======
+
+``log file``
+
+:Desription: The location of the logging file for your cluster.
+:Type: String
+:Required: No
+:Default: ``/var/log/ceph/$cluster-$name.log``
+
+
+``log max new``
+
+:Description: The maximum number of new log files.
+:Type: Integer
+:Required: No
+:Default: ``1000``
+
+
+``log max recent``
+
+:Description: The maximum number of recent events to include in a log file.
+:Type: Integer
+:Required: No
+:Default: ``1000000``
+
+
+``log to stderr``
+
+:Description: Determines if logging messages should appear in ``stderr``.
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
+
+``err to stderr``
+
+:Description: Determines if error messages should appear in ``stderr``.
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
+
+``log to syslog``
+
+:Description: Determines if logging messages should appear in ``syslog``.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``err to syslog``
+
+:Description: Determines if error messages should appear in ``syslog``.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``log flush on exit``
+
+:Description: Determines if Ceph should flush the log files after exit.
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
+
+``clog to monitors``
+
+:Description: Determines if ``clog`` messages should be sent to monitors.
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
+
+``clog to syslog``
+
+:Description: Determines if ``clog`` messages should be sent to syslog.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``mon cluster log to syslog``
+
+:Description: Determines if the cluster log should be output to the syslog.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``mon cluster log file``
+
+:Description: The location of the cluster's log file.
+:Type: String
+:Required: No
+:Default: ``/var/log/ceph/$cluster.log``
+
+
+
+OSD
+===
+
+
+``osd debug drop ping probability``
+
+:Description: ?
+:Type: Double
+:Required: No
+:Default: 0
+
+
+``osd debug drop ping duration``
+
+:Description: The duration ?
+:Type: Integer
+:Required: No
+:Default: 0
+
+``osd debug drop pg create probability``
+
+:Description:
+:Type: Integer
+:Required: No
+:Default: 0
+
+``osd debug drop pg create duration``
+
+:Description: ?
+:Type: Double
+:Required: No
+:Default: 1
+
+``osd preserve trimmed log``
+
+:Description: ?
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+``osd tmapput sets uses tmap``
+
+:Description: For debug only. ???
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``osd min pg log entries``
+
+:Description: The minimum number of log entries for placement groups.
+:Type: 32-bit Unsigned Integer
+:Required: No
+:Default: 1000
+
+``osd op log threshold``
+
+:Description: How many op log messages to show up in one pass.
+:Type: Integer
+:Required: No
+:Default: 5
+
+
+
+Filestore
+=========
+
+``filestore debug omap check``
+
+:Description: Checks the ``omap``. This is an expensive operation.
+:Type: Boolean
+:Required: No
+:Default: 0
+
+
+MDS
+===
+
+
+``mds debug scatterstat``
+
+:Description: ?
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``mds debug frag``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``mds debug auth pins``
+
+:Description: ?
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``mds debug subtrees``
+
+:Description: ?
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+
+RADOS Gateway
+=============
+
+
+``rgw log nonexistent bucket``
+
+:Description: Should we log a non-existent buckets?
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``rgw log object name``
+
+:Description: Should an object's name be logged. // man date to see codes (a subset are supported)
+:Type: String
+:Required: No
+:Default: ``%Y-%m-%d-%H-%i-%n``
+
+
+``rgw log object name utc``
+
+:Description: Object log name contains UTC?
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``rgw enable ops log``
+
+:Description: Enables logging of every RGW operation.
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
+
+``rgw enable usage log``
+
+:Description: Enable logging of RGW's bandwidth usage.
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
+
+``rgw usage log flush threshold``
+
+:Description: Threshold to flush pending log data.
+:Type: Integer
+:Required: No
+:Default: ``1024``
+
+
+``rgw usage log tick interval``
+
+:Description: Flush pending log data every ``s`` seconds.
+:Type: Integer
+:Required: No
+:Default: 30
+
+
+``rgw intent log object name``
+
+:Description:
+:Type: String
+:Required: No
+:Default: ``%Y-%m-%d-%i-%n``
+
+
+``rgw intent log object name utc``
+
+:Description: Include a UTC timestamp in the intent log object name.
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+``rgw cluster root pool``
+
+:Description: RADOS pool to store radosgw metadata for this instance
+:Type: String
+:Required: No
+:Default: ``.rgw.root``
+
+``rgw gc max objs``
+
+:Description: Number of objects to collect garbage collection data
+:Type: 32-bit Integer
+:Default: 32
+
+``rgw gc obj min wait``
+
+:Description: Minimum time to wait before object's removal and its processing by the garbage collector
+:Type: 32-bit Integer
+:Default: 2 hours. ``2*60*60``
+
+``rgw gc processor max time``
+
+:Description: Max time for a single garbage collection process run
+:Type: 32-bit Integer
+:Default: 1 hour. ``60*60``
+
+``rgw gc processor max period``
+
+:Description: Max time between the beginning of two consecutive garbage collection processes run
+:Type: 32-bit Integer
+:Default: 1 hour. ``60*60``
+
+
--- /dev/null
+==========================
+ Monitor Config Reference
+==========================
+
+``mon data``
+
+:Description: The monitor's data location.
+:Type: String
+:Default: ``/var/lib/ceph/mon/$cluster-$id``
+
+
+``mon initial members``
+
+:Description: The IDs of initial monitors in a cluster during startup. If specified, Ceph requires an odd number of monitors to form an initial quorum.
+:Type: String
+:Default: None
+
+
+``mon sync fs threshold``
+
+:Description: Synchronize with the filesystem when writing the specified number of objects. Set it to ``0`` to disable it.
+:Type: 32-bit Integer
+:Default: ``5``
+
+
+``mon tick interval``
+
+:Description: A monitor's tick interval in seconds.
+:Type: 32-bit Integer
+:Default: ``5``
+
+
+``mon subscribe interval``
+
+:Description: The refresh interval (in seconds) for subscriptions. The subscription mechanism enables obtaining the cluster maps and log informations.
+:Type: Double
+:Default: ``300``
+
+
+``mon osd auto mark in``
+
+:Description: Ceph will mark any booting OSDs as ``in`` the cluster.
+:Type: Boolean
+:Default: ``false``
+
+
+``mon osd auto mark auto out in``
+
+:Description: Ceph will mark booting OSDs auto marked ``out`` of the cluster as ``in`` the cluster.
+:Type: Boolean
+:Default: ``true``
+
+
+``mon osd auto mark new in``
+
+:Description: Ceph will mark booting new OSDs as ``in`` the cluster.
+:Type: Boolean
+:Default: ``true``
+
+
+``mon osd down out interval``
+
+:Description: The number of seconds Ceph waits before marking an OSD ``down`` and ``out`` if it doesn't respond.
+:Type: 32-bit Integer
+:Default: ``300``
+
+
+``mon osd min up ratio``
+
+:Description: The minimum ratio of ``up`` OSDs before Ceph will mark OSDs ``down``.
+:Type: Double
+:Default: ``.3``
+
+
+``mon osd min in ratio``
+
+:Description: The minimum ratio of ``in`` OSDs before Ceph will mark OSDs ``out``.
+:Type: Double
+:Default: ``.3``
+
+
+``mon lease``
+
+:Description: Length (in seconds) of the lease on the monitor's versions.
+:Type: Float
+:Default: ``5``
+
+
+``mon lease renew interval``
+
+:Description: The interval (in seconds) for the Leader to renew the other monitor's leases.
+:Type: Float
+:Default: ``3``
+
+
+``mon lease ack timeout``
+
+:Description: Number of seconds the Leader will wait for the Peons to acknowledge the lease extension.
+:Type: Float
+:Default: ``10.0``
+
+
+``mon clock drift allowed``
+
+:Description: The clock drift in seconds allowed between monitors.
+:Type: Float
+:Default: ``.050``
+
+
+``mon clock drift warn backoff``
+
+:Description: Exponential backoff for clock drift warnings
+:Type: Float
+:Default: ``5``
+
+
+``mon accept timeout``
+
+:Description: Number of seconds the Leader will wait for the Peons to accept a Paxos update. It is also used during the Paxos recovery phase for similar purposes.
+:Type: Float
+:Default: ``10.0``
+
+
+``mon pg create interval``
+
+:Description: Number of seconds between PG creation in the same OSD.
+:Type: Float
+:Default: ``30.0``
+
+
+``mon pg stuck threshold``
+
+:Description: Number of seconds after which PGs can be considered as being stuck.
+:Type: 32-bit Integer
+:Default: ``300``
+
+
+``mon osd full ratio``
+
+:Description: The percentage of disk space used before an OSD is considered ``full``.
+:Type: Float
+:Default: ``.95``
+
+
+``mon osd nearfull ratio``
+
+:Description: The percentage of disk space used before an OSD is considered ``nearfull``.
+:Type: Float
+:Default: ``.85``
+
+
+``mon globalid prealloc``
+
+:Description: The number of global IDs to pre-allocate for the cluster.
+:Type: 32-bit Integer
+:Default: ``100``
+
+
+``mon osd report timeout``
+
+:Description: The grace period in seconds before declaring unresponsive OSDs ``down``.
+:Type: 32-bit Integer
+:Default: ``900``
+
+
+``mon force standby active``
+
+:Description: should mons force standby-replay mds to be active
+:Type: Boolean
+:Default: true
+
+
+``mon min osdmap epochs``
+
+:Description: Minimum number of OSD map epochs to keep at all times.
+:Type: 32-bit Integer
+:Default: ``500``
+
+
+``mon max pgmap epochs``
+
+:Description: Maximum number of PG map epochs the monitor should keep.
+:Type: 32-bit Integer
+:Default: ``500``
+
+
+``mon max log epochs``
+
+:Description: Maximum number of Log epochs the monitor should keep.
+:Type: 32-bit Integer
+:Default: ``500``
+
+
+``mon max osd``
+
+:Description: The maximum number of OSDs allowed in the cluster.
+:Type: 32-bit Integer
+:Default: ``10000``
+
+
+``mon probe timeout``
+
+:Description: Number of seconds the monitor will wait to find peers before bootstrapping.
+:Type: Double
+:Default: ``2.0``
+
+
+``mon slurp timeout``
+
+:Description: Number of seconds the monitor has to recover using slurp before the process is aborted and the monitor bootstraps.
+:Type: Double
+:Default: ``10.0``
+
+
+``mon slurp bytes``
+
+:Description: Limits the slurp messages to the specified number of bytes.
+:Type: 32-bit Integer
+:Default: ``256 * 1024``
+
+
+``mon client bytes``
+
+:Description: The amount of client message data allowed in memory (in bytes).
+:Type: 64-bit Integer Unsigned
+:Default: ``100ul << 20``
+
+
+``mon daemon bytes``
+
+:Description: The message memory cap for metadata server and OSD messages (in bytes).
+:Type: 64-bit Integer Unsigned
+:Default: ``400ul << 20``
+
+
+``mon max log entries per event``
+
+:Description: The maximum number of log entries per event.
+:Type: Integer
+:Default: ``4096``
--- /dev/null
+===========
+ Messaging
+===========
+
+
+``ms tcp nodelay``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``true``
+
+
+``ms initial backoff``
+
+:Description:
+:Type: Double
+:Required: No
+:Default: ``.2``
+
+
+``ms max backoff``
+
+:Description:
+:Type: Double
+:Required: No
+:Default: ``15.0``
+
+
+``ms nocrc``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``ms die on bad msg``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``ms dispatch throttle bytes``
+
+:Description:
+:Type: 64-bit Unsigned Integer
+:Required: No
+:Default: ``100 << 20``
+
+
+``ms bind ipv6``
+
+:Description:
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``ms rwthread stack bytes``
+
+:Description:
+:Type: 64-bit Unsigned Integer
+:Required: No
+:Default: ``1024 << 10``
+
+
+``ms tcp read timeout``
+
+:Description:
+:Type: 64-bit Unsigned Integer
+:Required: No
+:Default: ``900``
+
+
+``ms inject socket failures``
+
+:Description:
+:Type: 64-bit Unsigned Integer
+:Required: No
+:Default: ``0``
--- /dev/null
+======================
+ OSD Config Reference
+======================
+
+
+
+``osd uuid``
+
+:Description: The universally unique identifier (UUID) for the OSD.
+:Type: UUID
+:Default: None
+
+
+``osd data``
+
+:Description: The path to the OSDs data. You must create the directory. You should mount a data disk at this mount point. We do not recommend changing the default.
+:Type: String
+:Default: ``/var/lib/ceph/osd/$cluster-$id``
+
+
+``osd journal``
+
+:Description: The path to the OSD's journal. This may be a path to a file or a block device (such as a partition of an SSD). If it is a file, you must create the directory to contain it.
+:Type: String
+:Default: ``/var/lib/ceph/osd/$cluster-$id/journal``
+
+
+``osd journal size``
+
+:Description: The size of the journal in megabytes. If this is 0, and the journal is a block device, the entire block device is used. Since v0.54, this is ignored if the journal is a block device, and the entire block device is used.
+:Type: 32-bit Integer
+:Default: ``1024``
+:Recommended: Begin with 1GB. Should at least twice the product of the expected speed multiplied by ``filestore min sync interval``.
+
+
+``osd max write size``
+
+:Description: The maximum size of a write in megabytes.
+:Type: 32-bit Integer
+:Default: ``90``
+
+
+``osd client message size cap``
+
+:Description: The largest client data message allowed in memory.
+:Type: 64-bit Integer Unsigned
+:Default: 500MB default. ``500*1024L*1024L``
+
+
+``osd stat refresh interval``
+
+:Description: The status refresh interval in seconds.
+:Type: 64-bit Integer Unsigned
+:Default: ``.5``
+
+
+``osd pg bits``
+
+:Description: Placement group bits per OSD.
+:Type: 32-bit Integer
+:Default: ``6``
+
+
+``osd pgp bits``
+
+:Description: The number of bits per OSD for PGPs.
+:Type: 32-bit Integer
+:Default: ``4``
+
+
+``osd pg layout``
+
+:Description: Placement group layout.
+:Type: 32-bit Integer
+:Default: ``2``
+
+
+``osd pool default crush rule``
+
+:Description: The default CRUSH rule to use when creating a pool.
+:Type: 32-bit Integer
+:Default: ``0``
+
+
+``osd pool default size``
+
+:Description: The default size of an OSD pool in gigabytes. The default value is the same as ``--size 2`` with ``mkpool``.
+:Type: 32-bit Integer
+:Default: ``2``
+
+
+``osd pool default pg num``
+
+:Description: The default number of placement groups for a pool. The default value is the same as ``pg_num`` with ``mkpool``.
+:Type: 32-bit Integer
+:Default: ``8``
+
+
+``osd pool default pgp num``
+
+:Description: The default number of placement groups for placement for a pool. The default value is the same as ``pgp_num`` with ``mkpool``. PG and PGP should be equal (for now).
+:Type: 32-bit Integer
+:Default: ``8``
+
+
+``osd map dedup``
+
+:Description: Enable removing duplicates in the OSD map.
+:Type: Boolean
+:Default: ``true``
+
+
+``osd map cache size``
+
+:Description: The size of the OSD map cache in megabytes.
+:Type: 32-bit Integer
+:Default: ``500``
+
+
+``osd map cache bl size``
+
+:Description: The size of the in-memory OSD map cache in OSD daemons.
+:Type: 32-bit Integer
+:Default: ``50``
+
+
+``osd map cache bl inc size``
+
+:Description: The size of the in-memory OSD map cache incrementals in OSD daemons.
+:Type: 32-bit Integer
+:Default: ``100``
+
+
+``osd map message max``
+
+:Description: The maximum map entries allowed per MOSDMap message.
+:Type: 32-bit Integer
+:Default: ``100``
+
+
+``osd op threads``
+
+:Description: The number of OSD operation threads. Set to ``0`` to disable it. Increasing the number may increase the request processing rate.
+:Type: 32-bit Integer
+:Default: ``2``
+
+
+``osd op thread timeout``
+
+:Description: The OSD operation thread timeout in seconds.
+:Type: 32-bit Integer
+:Default: ``30``
+
+
+``osd disk threads``
+
+:Description: The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming.
+:Type: 32-bit Integer
+:Default: ``1``
+
+
+``osd recovery threads``
+
+:Description: The number of threads for recovering data.
+:Type: 32-bit Integer
+:Default: ``1``
+
+
+``osd recover clone overlap``
+
+:Description: Preserves clone overlap during recovery and data migration.
+:Type: Boolean
+:Default: ``false``
+
+
+``osd backfill scan min``
+
+:Description: The scan interval in seconds for backfill operations.
+:Type: 32-bit Integer
+:Default: ``64``
+
+
+``osd backfill scan max``
+
+:Description: The maximum scan interval in seconds for backfill operations.
+:Type: 32-bit Integer
+:Default: ``512``
+
+
+``osd backlog thread timeout``
+
+:Description: The maximum time in seconds before timing out a backlog thread.
+:Type: 32-bit Integer
+:Default: ``60*60*1``
+
+
+``osd recovery thread timeout``
+
+:Description: The maximum time in seconds before timing out a recovery thread.
+:Type: 32-bit Integer
+:Default: ``30``
+
+
+``osd snap trim thread timeout``
+
+:Description: The maximum time in seconds before timing out a snap trim thread.
+:Type: 32-bit Integer
+:Default: ``60*60*1``
+
+
+``osd scrub thread timeout``
+
+:Description: The maximum time in seconds before timing out a scrub thread.
+:Type: 32-bit Integer
+:Default: ``60``
+
+
+``osd scrub finalize thread timeout``
+
+:Description: The maximum time in seconds before timing out a scrub finalize thread.
+:Type: 32-bit Integer
+:Default: 60*10
+
+
+``osd remove thread timeout``
+
+:Description: The maximum time in seconds before timing out a remove OSD thread.
+:Type: 32-bit Integer
+:Default: 60*60
+
+
+``osd command thread timeout``
+
+:Description: The maximum time in seconds before timing out a command thread.
+:Type: 32-bit Integer
+:Default: ``10*60``
+
+
+``osd heartbeat address``
+
+:Description: An OSD's network address for heartbeats.
+:Type: Address
+:Default: The host address.
+
+
+``osd heartbeat interval``
+
+:Description: How often an OSD pings its peers (in seconds).
+:Type: 32-bit Integer
+:Default: ``6``
+
+
+``osd heartbeat grace``
+
+:Description: The elapsed time when an OSD hasn't shown a heartbeat that the cluster considers it ``down``.
+:Type: 32-bit Integer
+:Default: ``20``
+
+
+``osd _mon_heartbeat interval``
+
+:Description: How often the OSD pings a monitor if it has no OSD peers.
+:Type: 32-bit Integer
+:Default: ``30``
+
+
+``osd mon report interval max``
+
+:Description: The maximum time in seconds for an OSD to report to a monitor before the monitor considers the OSD ``down``.
+:Type: 32-bit Integer
+:Default: ``120``
+
+
+``osd mon report interval min``
+
+:Description: The number of minutes between reports that include ``pg stats``, ``up thru``, ``boot`` and ``failures``.
+:Type: 32-bit Integer
+:Default: ``5``
+
+
+``osd mon ack timeout``
+
+:Description: The number of seconds to wait for a monitor to acknowledge a request for statistics.
+:Type: 32-bit Integer
+:Default: ``30``
+
+
+``osd min down reporters``
+
+:Description: The minimum number of OSDs required to report a ``down`` OSD.
+:Type: 32-bit Integer
+:Default: ``1``
+
+
+``osd min down reports``
+
+:Description: The minimum number of times an OSD must report that another is ``down``.
+:Type: 32-bit Integer
+:Default: ``3``
+
+
+``osd recovery delay start``
+
+:Description: After peering completes, Ceph will delay for the specified number of seconds before starting to recover objects.
+:Type: Float
+:Default: ``15``
+
+
+``osd recovery max active``
+
+:Description: The number of active recovery requests per OSD at one time. More accelerates recovery, but places an increased load on the cluster.
+:Type: 32-bit Integer
+:Default: ``5``
+
+
+``osd recovery max chunk``
+
+:Description: The maximum size of a recovered chunk of data to push.
+:Type: 64-bit Integer Unsigned
+:Default: ``1 << 20``
+
+
+``osd max scrubs``
+
+:Description: The maximum number of scrub operations for an OSD.
+:Type: 32-bit Int
+:Default: ``1``
+
+
+``osd scrub load threshold``
+
+:Description: The maximum CPU load. Ceph will not scrub when the CPU load is higher than this number. Default is 50%.
+:Type: Float
+:Default: ``0.5``
+
+
+``osd scrub min interval``
+
+:Description: The maximum interval in seconds for scrubbing the OSD.
+:Type: Float
+:Default: 5 minutes. ``300``
+
+
+``osd scrub max interval``
+
+:Description: The maximum interval in seconds for scrubbing the OSD.
+:Type: Float
+:Default: Once per day. ``60*60*24``
+
+``osd deep scrub interval``
+
+:Description: The interval for "deep" scrubbing (fully reading all data)
+:Type: Float
+:Default: Once per week. ``60*60*24*7``
+
+``osd deep scrub stride``
+
+:Description: Read siez when doing a deep scrub
+:Type: 32-bit Int
+:Default: 512 KB. ``524288``
+
+``osd class dir``
+
+:Description: The class path for RADOS class plug-ins.
+:Type: String
+:Default: ``$libdir/rados-classes``
+
+
+``osd check for log corruption``
+
+:Description: Check log files for corruption. Can be computationally expensive.
+:Type: Boolean
+:Default: ``false``
+
+
+``osd default notify timeout``
+
+:Description: The OSD default notification timeout (in seconds).
+:Type: 32-bit Integer Unsigned
+:Default: ``30``
+
+
+``osd min pg log entries``
+
+:Description: The minimum number of placement group logs to maintain when trimming log files.
+:Type: 32-bit Int Unsigned
+:Default: 1000
+
+
+``osd op complaint time``
+
+:Description: An operation becomes complaint worthy after the specified number of seconds have elapsed.
+:Type: Float
+:Default: ``30``
+
+
+``osd command max records``
+
+:Description: Limits the number of lost objects to return.
+:Type: 32-bit Integer
+:Default: ``256``
+
+
+``osd auto upgrade tmap``
+
+:Description: Uses ``tmap`` for ``omap`` on old objects.
+:Type: Boolean
+:Default: ``true``
+
+
+``osd tmapput sets users tmap``
+
+:Description: Uses ``tmap`` for debugging only.
+:Type: Boolean
+:Default: ``false``
+
+
+``osd kill backfill at``
+
+:Description: For debugging only.
+:Type: 32-bit Integer
+:Default: ``0``
--- /dev/null
+=============
+ Ceph Deploy
+=============
+
+Coming soon.
\ No newline at end of file
--- /dev/null
+=====================
+ Deploying with Chef
+=====================
+
+We use Chef cookbooks to deploy Ceph. See `Managing Cookbooks with Knife`_ for details
+on using ``knife``. For Chef installation instructions, see `Installing Chef`_.
+
+.. _clonecbs:
+
+Clone the Required Cookbooks
+============================
+
+To get the cookbooks for Ceph, clone them from git.::
+
+ cd ~/chef-cookbooks
+ git clone https://github.com/opscode-cookbooks/apache2.git
+ git clone https://github.com/ceph/ceph-cookbooks.git ceph
+
+.. _addcbpaths:
+
+Add the Required Cookbook Paths
+===============================
+
+If you added a default cookbook path when you installed Chef, ``knife``
+may be able to upload the cookbook you've cloned to your cookbook path
+directory without further configuration. If you used a different path,
+or if the cookbook repository you cloned has a different tree structure,
+add the required cookbook path to your ``knife.rb`` file. The
+``cookbook_path`` setting takes a string or an array of strings.
+For example, you can replace a string path with an array of string paths::
+
+ cookbook_path '/home/{user-name}/chef-cookbooks/'
+
+Becomes::
+
+ cookbook_path [
+ '/home/{user-name}/chef-cookbooks/',
+ '/home/{user-name}/chef-cookbooks/{another-directory}/',
+ '/some/other/path/to/cookbooks/'
+ ]
+
+.. _installcbs:
+
+Install the Cookbooks
+=====================
+
+To install Ceph, you must upload the Ceph cookbooks and the Apache cookbooks
+(for use with RADOSGW) to your Chef server. ::
+
+ knife cookbook upload apache2 ceph
+
+.. _configcephenv:
+
+Configure your Ceph Environment
+===============================
+
+The Chef server can support installation of software for multiple environments.
+The environment you create for Ceph requires an ``fsid``, the secret for
+your monitor(s) if you are running Ceph with ``cephx`` authentication, and
+the host name (i.e., short name) for your monitor hosts.
+
+.. tip: Open an empty text file to hold the following values until you create
+ your Ceph environment.
+
+For the filesystem ID, use ``uuidgen`` from the ``uuid-runtime`` package to
+generate a unique identifier. ::
+
+ uuidgen -r
+
+For the monitor(s) secret(s), use ``ceph-authtool`` to generate the secret(s)::
+
+ sudo apt-get update
+ sudo apt-get install ceph-common
+ ceph-authtool /dev/stdout --name=mon. --gen-key
+
+The secret is the value to the right of ``"key ="``, and should look something
+like this::
+
+ AQBAMuJPINJgFhAAziXIrLvTvAz4PRo5IK/Log==
+
+To create an environment for Ceph, set a command line editor. For example::
+
+ export EDITOR=vim
+
+Then, use ``knife`` to create an environment. ::
+
+ knife environment create {env-name}
+
+For example::
+
+ knife environment create Ceph
+
+A JSON file will appear. Perform the following steps:
+
+#. Enter a description for the environment.
+#. In ``"default_attributes": {}``, add ``"ceph" : {}``.
+#. Within ``"ceph" : {}``, add ``"monitor-secret":``.
+#. Immediately following ``"monitor-secret":`` add the key you generated within quotes, followed by a comma.
+#. Within ``"ceph":{}`` and following the ``monitor-secret`` key-value pair, add ``"config": {}``
+#. Within ``"config": {}`` add ``"fsid":``.
+#. Immediately following ``"fsid":``, add the unique identifier you generated within quotes, followed by a comma.
+#. Within ``"config": {}`` and following the ``fsid`` key-value pair, add ``"mon_initial_members":``
+#. Immediately following ``"mon_initial_members":``, enter the initial monitor host names.
+
+For example::
+
+ "default_attributes" : {
+ "ceph": {
+ "monitor-secret": "{replace-with-generated-secret}",
+ "config": {
+ "fsid": "{replace-with-generated-uuid}",
+ "mon_initial_members": "{replace-with-monitor-hostname(s)}"
+ }
+ }
+ }
+
+Advanced users (i.e., developers and QA) may also add ``"ceph_branch": "{branch}"``
+to ``default-attributes``, replacing ``{branch}`` with the name of the branch you
+wish to use (e.g., ``master``).
+
+.. configroles:
+
+Configure the Roles
+===================
+
+Navigate to the Ceph cookbooks directory. ::
+
+ cd ~/chef-cookbooks/ceph
+
+Create roles for OSDs, monitors, metadata servers, and RADOS Gateways from
+their respective role files. ::
+
+ knife role from file roles/ceph-osd.rb
+ knife role from file roles/ceph-mon.rb
+ knife role from file roles/ceph-mds.rb
+ knife role from file roles/ceph-radosgw.rb
+
+.. _confignodes:
+
+Configure Nodes
+===============
+
+You must configure each node you intend to include in your Ceph cluster.
+Identify nodes for your Ceph cluster. ::
+
+ knife node list
+
+.. note: for each host where you installed Chef and executed ``chef-client``,
+ the Chef server should have a minimal node configuration. You can create
+ additional nodes with ``knife node create {node-name}``.
+
+For each node you intend to use in your Ceph cluster, configure the node
+as follows::
+
+ knife node edit {node-name}
+
+The node configuration should appear in your text editor. Change the
+``chef_environment`` value to ``Ceph`` (or whatever name you set for your
+Ceph environment).
+
+In the ``run_list``, add ``"recipe[ceph::apt]",`` to all nodes as
+the first setting, so that Chef can install or update the necessary packages.
+Then, add at least one of::
+
+ "role[ceph-mon]"
+ "role[ceph-osd]"
+ "role[ceph-mds]"
+ "role[ceph-radosgw]"
+
+If you add more than one role, separate them with a comma. Run ``hostname``
+on your command line, and replace the ``{hostname}`` setting of the ``name``
+key to the host name for the node. ::
+
+ {
+ "chef_environment": "Ceph",
+ "name": "{hostname}",
+ "normal": {
+ "tags": [
+
+ ]
+ },
+ "run_list": [
+ "recipe[ceph::apt]",
+ "role[ceph-mon]",
+ "role[ceph-mds]"
+ ]
+ }
+
+.. _prepdisks:
+
+Prepare OSD Disks
+=================
+
+Configuring a node with an OSD role tells Chef that the node will run at
+least one OSD. However, you may run many OSDs on one host. For example,
+you may run one ``ceph-osd`` daemon for each data disk on the system.
+This step prepares the OSD disk(s) and tells Chef how many OSDs the
+node will be running.
+
+
+For the Ceph 0.48 Argonaut release, install ``gdisk``::
+
+ sudo apt-get install gdisk
+
+For the Ceph 0.48 Argonaut release, on each hard disk that will store data for
+an OSD daemon, configure the hard disk for use with Ceph. Replace ``{fsid}``
+with the UUID you generated while using ``uuidgen -r``.
+
+.. important: This procedure will erase all information in ``/dev/{disk}``.
+
+::
+
+ sudo sgdisk /dev/{disk} --zap-all --clear --mbrtogpt --largest-new=1 --change-name=1:'ceph data' --typecode=1:{fsid}
+
+Create a file system and allocate the disk to your cluster. Specify a
+filesystem (e.g., ``ext4``, ``xfs``, ``btrfs``). When you execute
+``ceph-disk-prepare``, remember to replace ``{fsid}`` with the UUID you
+generated while using ``uuidgen -r``::
+
+ sudo mkfs -t ext4 /dev/{disk}
+ sudo mount -o user_xattr /dev/{disk} /mnt
+ sudo ceph-disk-prepare --cluster-uuid={fsid} /mnt
+ sudo umount /mnt
+
+Finally, simulate a hotplug event. ::
+
+ sudo udevadm trigger --subsystem-match=block --action=add
+
+
+.. _runchefclient:
+
+Run ``chef-client`` on each Node
+================================
+
+Once you have completed the preceding steps, you must run ``chef-client``
+on each node. For example::
+
+ sudo chef-client
+
+.. _proceedtoops:
+
+Proceed to Operating the Cluster
+================================
+
+Once you complete the deployment, you may begin operating your cluster.
+See `Operating a Cluster`_ for details.
+
+
+.. _Managing Cookbooks with Knife: http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife
+.. _Installing Chef: ../../install/chef
+.. _Operating a Cluster: ../../init/
--- /dev/null
+=================
+ Ceph Deployment
+=================
+
+You can deploy Chef using many different deployment systems including Chef, Juju,
+Puppet, and Crowbar. If you are just experimenting, Ceph provides some minimal
+deployment tools that rely only on SSH and DNS to deploy Ceph. You need to set
+up the SSH and DNS settings manually.
+
+
+.. raw:: html
+
+ <table cellpadding="10"><colgroup><col width="33%"><col width="33%"><col width="33%"></colgroup><tbody valign="top"><tr><td><h3>Ceph Deployment Scripts</h3>
+
+We provide light-weight deployment scripts to help you evaluate Ceph. For
+professional deployment, you should consider professional deployment systems
+such as Juju, Puppet, Chef or Crowbar.
+
+.. toctree::
+
+ mkcephfs (Deprecated) <mkcephfs>
+ Ceph Deploy <ceph-deploy>
+
+.. raw:: html
+
+ </td><td><h3>OpsCode Chef</h3>
+
+.. toctree::
+
+ Installing Chef <install-chef>
+ Deploying with Chef <chef>
+
+.. raw:: html
+
+ </td></tr></tbody></table>
--- /dev/null
+=================
+ Installing Chef
+=================
+
+Chef defines three types of entities:
+
+#. **Chef Nodes:** Run ``chef-client``, which installs and manages software.
+#. **Chef Server:** Interacts with ``chef-client`` on Chef nodes.
+#. **Chef Workstation:** Manages the Chef server.
+
+See `Chef Architecture Introduction`_ for details.
+
+.. _createuser:
+
+Create a ``chef`` User
+======================
+
+The ``chef-client`` command requires the proper privileges to install and manage
+installations. On each Chef node, we recommend creating a ``chef`` user with
+full ``root`` privileges. For example::
+
+ ssh user@chef-node
+ sudo useradd -d /home/chef -m chef
+ sudo passwd chef
+
+To provide full privileges, add the following to ``/etc/sudoers.d/chef``. ::
+
+ echo "chef ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/chef
+ sudo chmod 0440 /etc/sudoers.d/chef
+
+If you are using a version of ``sudo`` that doesn't support includes, you will
+need to add the following to the ``/etc/sudoers`` file::
+
+ chef ALL = (root) NOPASSWD:ALL
+
+.. important:: Do not change the file permissions on ``/etc/sudoers``. Use a
+ suitable tool such as ``visudo``.
+
+.. _genkeys:
+
+Generate SSH Keys for Chef Clients
+==================================
+
+Chef's ``knife`` tool can run ``ssh``. To streamline deployments, we
+recommend generating an SSH key pair without a passphrase for your
+Chef nodes and copying the public key(s) to your Chef nodes so that you
+can connect to them from your workstation using ``ssh`` from ``knife``
+without having to provide a password. To generate a key pair without
+a passphrase, execute the following on your Chef workstation. ::
+
+ ssh-keygen
+ Generating public/private key pair.
+ Enter file in which to save the key (/ceph-admin/.ssh/id_rsa):
+ Enter passphrase (empty for no passphrase):
+ Enter same passphrase again:
+ Your identification has been saved in /ceph-admin/.ssh/id_rsa.
+ Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.
+
+You may use RSA or DSA keys. Once you generate your keys, copy them to each
+OSD host. For example::
+
+ ssh-copy-id chef@your-node
+
+Consider modifying your ``~/.ssh/config`` file so that it defaults to
+logging in as ``chef`` when no username is specified. ::
+
+ Host myserver01
+ Hostname myserver01.fqdn-or-ip-address.com
+ User chef
+ Host myserver02
+ Hostname myserver02.fqdn-or-ip-address.com
+ User chef
+
+.. _installruby:
+
+Installing Ruby
+===============
+
+Chef requires you to install Ruby. Use the version applicable to your current
+Linux distribution and install Ruby on all of your hosts. ::
+
+ sudo apt-get update
+ sudo apt-get install ruby
+
+.. _installchefserver:
+
+Installing Chef and Chef Server on a Server
+===========================================
+
+If you plan on hosting your `Chef Server at Opscode`_ you may skip this step,
+but you must make a note of the the fully qualified domain name or IP address
+of your Chef Server for ``knife`` and ``chef-client``.
+
+First, add Opscode packages to your APT configuration. For example::
+
+ sudo tee /etc/apt/sources.list.d/chef.list << EOF
+ deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
+ deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
+ EOF
+
+Next, you must request keys so that APT can verify the packages. Copy
+and paste the following line into your command line::
+
+ sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A
+
+The key is only used by ``apt``, so remove it from the ``root`` keyring by
+typing ``Y`` when prompted to delete it.
+
+Install the Opscode keyring, Chef and Chef server on the host designated
+as your Chef Server. ::
+
+ sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef chef-server
+
+Enter the fully qualified domain name or IP address for your Chef server. For example::
+
+ http://fqdn-or-ip-address.com:4000
+
+The Chef server installer will prompt you to enter a temporary password. Enter
+a temporary password (*e.g.,* ``foo``) and proceed with the installation.
+
+.. tip:: When prompted for a temporary password, you may press **OK**.
+ The installer wants you to re-enter the password to confirm it. To
+ re-enter the password, you must press the **ESC** key.
+
+Once the installer finishes and activates the Chef server, you may enter the
+fully qualified domain name or IP address in a browser to launch the
+Chef web UI. For example::
+
+ http://fqdn-or-ip-address.com:4000
+
+The Chef web UI will prompt you to enter the username and password.
+
+- **login:** ``admin``
+- **password:** ``foo``
+
+Once you have entered the temporary password, the Chef web UI will prompt you
+to enter a new password.
+
+.. _installchef:
+
+Install Chef on all Remaining Hosts
+===================================
+
+Install Chef on all Chef Nodes and on the Chef Workstation (if it is not the
+same host as the Chef Server). See `Installing Chef Client on Ubuntu or Debian`_
+for details.
+
+First, add Opscode packages to your APT configuration. For example::
+
+ sudo tee /etc/apt/sources.list.d/chef.list << EOF
+ deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
+ deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main
+ EOF
+
+Next, you must request keys so that APT can verify the packages. Copy
+and paste the following line into your command line::
+
+ sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A
+
+The key is only used by ``apt``, so remove it from the ``root`` keyring by
+typing ``Y`` when prompted to delete it.
+
+Install the Opscode keyring and Chef on all hosts other than the Chef Server. ::
+
+ sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef
+
+Enter the fully qualified domain name or IP address for your Chef server.
+For example::
+
+ http://fqdn-or-ip-address.com:4000
+
+.. _configknife:
+
+Configuring Knife
+=================
+
+Once you complete the Chef server installation, install ``knife`` on the your
+Chef Workstation. If the Chef server is a remote host, use ``ssh`` to connect. ::
+
+ ssh chef@fqdn-or-ip-address.com
+
+In the ``/home/chef`` directory, create a hidden Chef directory. ::
+
+ mkdir -p ~/.chef
+
+The server generates validation and web UI certificates with read/write
+permissions for the user that installed the Chef server. Copy them from the
+``/etc/chef`` directory to the ``~/.chef`` directory. Then, change their
+ownership to the current user. ::
+
+ sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef && sudo chown $(id -u):$(id -g) ~/.chef/*.pem
+
+From the current user's home directory, configure ``knife`` with an initial
+API client. ::
+
+ knife configure -i
+
+The configuration will prompt you for inputs. Answer accordingly:
+
+*Where should I put the config file? [~/.chef/knife.rb]* Press **Enter**
+to accept the default value.
+
+*Please enter the chef server URL:* If you are installing the
+client on the same host as the server, enter ``http://localhost:4000``.
+Otherwise, enter an appropriate URL for the server.
+
+*Please enter a clientname for the new client:* Press **Enter**
+to accept the default value.
+
+*Please enter the existing admin clientname:* Press **Enter**
+to accept the default value.
+
+*Please enter the location of the existing admin client's private key:*
+Override the default value so that it points to the ``.chef`` directory.
+(*e.g.,* ``/home/chef/.chef/webui.pem``)
+
+*Please enter the validation clientname:* Press **Enter** to accept
+the default value.
+
+*Please enter the location of the validation key:* Override the
+default value so that it points to the ``.chef`` directory.
+(*e.g.,* ``/home/chef/.chef/validation.pem``)
+
+*Please enter the path to a chef repository (or leave blank):*
+Leave the entry field blank and press **Enter**.
+
+.. _addcbpath:
+
+Add a Cookbook Path
+===================
+
+Add ``cookbook_path`` to the ``~/.chef/knife.rb`` configuration file
+on your Chef workstation. For example::
+
+ cookbook_path '/home/{user-name}/chef-cookbooks/'
+
+Then create the path if it doesn't already exist. ::
+
+ mkdir /home/{user-name}/chef-cookbooks
+
+This is where you will store local copies of cookbooks before uploading
+them to the Chef server.
+
+.. _cpvalpem:
+
+Copy ``validation.pem`` to Nodes
+================================
+
+Copy the ``/etc/chef/validation.pem`` file from your Chef server to
+each Chef Node. In a command line shell on the Chef Server, for each node,
+replace ``{nodename}`` in the following line with the node's host name and
+execute it. ::
+
+ sudo cat /etc/chef/validation.pem | ssh {nodename} "exec sudo tee /etc/chef/validation.pem >/dev/null"
+
+.. _runchefcli:
+
+Run ``chef-client`` on each Chef Node
+=====================================
+
+Run the ``chef-client`` on each Chef Node so that the nodes
+register with the Chef server. ::
+
+ ssh chef-node
+ sudo chef-client
+
+.. _verifynodes:
+
+Verify Nodes
+============
+
+Verify that you have setup all the hosts you want to use as
+Chef nodes. ::
+
+ knife node list
+
+A list of the nodes you've configured should appear.
+
+See the `Deploy With Chef <../../config-cluster/chef>`_ section for information
+on using Chef to deploy your Ceph cluster.
+
+.. _Chef Architecture Introduction: http://wiki.opscode.com/display/chef/Architecture+Introduction
+.. _Chef Server at Opscode: http://www.opscode.com/hosted-chef/
+.. _Installing Chef Client on Ubuntu or Debian: http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Ubuntu+or+Debian
+.. _Installing Chef Server on Debian or Ubuntu using Packages: http://wiki.opscode.com/display/chef/Installing+Chef+Server+on+Debian+or+Ubuntu+using+Packages
+.. _Knife Bootstrap: http://wiki.opscode.com/display/chef/Knife+Bootstrap
\ No newline at end of file
--- /dev/null
+=============================
+ Deploying with ``mkcephfs``
+=============================
+
+To deploy a test or development cluster, you can use the ``mkcephfs`` tool.
+We do not recommend using this tool for production environments.
+
+
+Enable Login to Cluster Hosts as ``root``
+=========================================
+
+To deploy with ``mkcephfs``, you will need to be able to login as ``root``
+on each host without a password. For each host, perform the following::
+
+ sudo passwd root
+
+Enter a password for the root user.
+
+On the admin host, generate an ``ssh`` key without specifying a passphrase
+and use the default locations. ::
+
+ ssh-keygen
+ Generating public/private key pair.
+ Enter file in which to save the key (/ceph-admin/.ssh/id_rsa):
+ Enter passphrase (empty for no passphrase):
+ Enter same passphrase again:
+ Your identification has been saved in /ceph-admin/.ssh/id_rsa.
+ Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.
+
+You may use RSA or DSA keys. Once you generate your keys, copy them to each
+OSD host. For example::
+
+ ssh-copy-id root@myserver01
+ ssh-copy-id root@myserver02
+
+Modify your ``~/.ssh/config`` file to login as ``root``, as follows::
+
+ Host myserver01
+ Hostname myserver01.fully-qualified-domain.com
+ User root
+ Host myserver02
+ Hostname myserver02.fully-qualified-domain.com
+ User root
+
+
+Copy Configuration File to All Hosts
+====================================
+
+Ceph's ``mkcephfs`` deployment script does not copy the configuration file you
+created from the Administration host to the OSD Cluster hosts. Copy the
+configuration file you created (*i.e.,* ``mycluster.conf`` in the example below)
+from the Administration host to ``etc/ceph/ceph.conf`` on each OSD Cluster host
+if you are using ``mkcephfs`` to deploy Ceph.
+
+::
+
+ ssh myserver01 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
+ ssh myserver02 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
+ ssh myserver03 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf
+
+
+Create the Default Directories
+==============================
+
+The ``mkcephfs`` deployment script does not create the default server
+directories. Create server directories for each instance of a Ceph daemon (if
+you haven't done so already). The ``host`` variables in the ``ceph.conf`` file
+determine which host runs each instance of a Ceph daemon. Using the exemplary
+``ceph.conf`` file, you would perform the following:
+
+On ``myserver01``::
+
+ sudo mkdir /var/lib/ceph/osd/ceph-0
+ sudo mkdir /var/lib/ceph/mon/ceph-a
+
+On ``myserver02``::
+
+ sudo mkdir /var/lib/ceph/osd/ceph-1
+ sudo mkdir /var/lib/ceph/mon/ceph-b
+
+On ``myserver03``::
+
+ sudo mkdir /var/lib/ceph/osd/ceph-2
+ sudo mkdir /var/lib/ceph/mon/ceph-c
+ sudo mkdir /var/lib/ceph/mds/ceph-a
+
+
+Mount Disks to the Data Directories
+===================================
+
+If you are running multiple OSDs per host and one hard disk per OSD, you should
+mount the disk under the OSD data directory (if you haven't done so already).
+
+
+Run ``mkcephfs``
+================
+
+Once you have copied your Ceph Configuration to the OSD Cluster hosts
+and created the default directories, you may deploy Ceph with the
+``mkcephfs`` script.
+
+.. note:: ``mkcephfs`` is a quick bootstrapping tool. It does not handle more
+ complex operations, such as upgrades.
+
+For production environments, deploy Ceph using Chef cookbooks. To run
+``mkcephfs``, execute the following::
+
+ cd /etc/ceph
+ sudo mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring
+
+The script adds an admin key to the ``ceph.keyring``, which is analogous to a
+root password. See `Authentication`_ when running with ``cephx`` enabled.
+
+When you start or stop your cluster, you will not have to use ``sudo`` or
+provide passwords. For example::
+
+ service ceph -a start
+
+See `Operating a Cluster`_ for details.
+
+
+.. _Authentication: ../authentication
+.. _Operating a Cluster: ../../init/
\ No newline at end of file
--- /dev/null
+====================
+ RADOS Object Store
+====================
+
+Ceph's :abbr:`RADOS (Reliable Autonomic Distributed Object Store)` Object Store
+is the foundation for all Ceph clusters. When you use object store clients such
+as the CephFS filesystem, the RESTful Gateway or Ceph block devices, Ceph reads
+data from and writes data to the object store. Ceph's RADOS Object Stores
+consist of two types of daemons: Object Storage Daemons (OSDs) store data as
+objects on storage nodes; and Monitors maintain a master copy of the cluster
+map. A Ceph cluster may contain thousands of storage nodes. A minimal system
+will have at least two OSDs for data replication.
+
+.. ditaa:: +---------------+ +---------------+
+ | OSDs | | Monitor |
+ +---------------+ +---------------+
+
+.. raw:: html
+
+ <table cellpadding="10"><colgroup><col width="33%"><col width="33%"><col width="33%"></colgroup><tbody valign="top"><tr><td><h3>Config and Deploy</h3>
+
+Once you have installed Ceph packages, you must configure. There are a a few
+required settings, but most configuration settings have default values.
+Following the initial configuration, you must deploy Ceph. Deployment consists
+of creating and initializing data directories, keys, etc.
+
+.. toctree::
+ :maxdepth: 2
+
+ Configuration <configuration/index>
+ Deployment <deployment/index>
+
+.. raw:: html
+
+ </td><td><h3>Operations</h3>
+
+Once you have a deployed Ceph cluster, you may begin operating your cluster.
+
+.. toctree::
+ :maxdepth: 2
+
+
+ Operations <operations/index>
+ Manual Pages <man/index>
+
+
+.. raw:: html
+
+ </td><td><h3>APIs</h3>
+
+Most Ceph deployments use Ceph `block devices`_, the `gateway`_ and/or the
+`CephFS filesystem`_. You may also develop applications that talk directly to
+the Ceph object store.
+
+.. toctree::
+ :maxdepth: 2
+
+ APIs <api/index>
+
+.. raw:: html
+
+ </td></tr></tbody></table>
+
+.. _block devices: ../rbd/rbd
+.. _CephFS filesystem: ../cephfs/
+.. _gateway: ../radosgw/
--- /dev/null
+=======================
+ Object Store Manpages
+=======================
+
+.. toctree::
+ :maxdepth: 1
+
+ ../../man/8/ceph.rst
+ ../../man/8/ceph-authtool.rst
+ ../../man/8/ceph-clsinfo.rst
+ ../../man/8/ceph-conf.rst
+ ../../man/8/ceph-debugpack.rst
+ ../../man/8/ceph-dencoder.rst
+ ../../man/8/ceph-mon.rst
+ ../../man/8/ceph-osd.rst
+ ../../man/8/ceph-run.rst
+ ../../man/8/ceph-syn.rst
+ ../../man/8/crushtool.rst
+ ../../man/8/librados-config.rst
+ ../../man/8/monmaptool.rst
+ ../../man/8/osdmaptool.rst
+ ../../man/8/rados.rst
\ No newline at end of file
For additional information, see our `Cephx Guide`_ and `ceph-authtool manpage`_.
.. _Cephx Guide: ../authentication
-.. _ceph-authtool manpage: ../../man/8/ceph-authtool/
+.. _ceph-authtool manpage: ../../../man/8/ceph-authtool
Ceph Authentication (cephx)
===========================
For additional information, see our `Cephx Intro`_ and `ceph-authtool manpage`_.
.. _Cephx Intro: ../auth-intro
-.. _ceph-authtool manpage: ../../man/8/ceph-authtool/
+.. _ceph-authtool manpage: ../../../man/8/ceph-authtool
Configuring Cephx
See `Enabling Cephx`_ step 2 and 3 for stepwise details to enable ``cephx``.
-.. _ceph-authtool: ../../man/8/ceph-authtool/
+.. _ceph-authtool: ../../man/ceph-authtool/
.. _enable-cephx:
foregoing flag** at the nearest practical time so that you may avail yourself
of the enhanced authentication.
-.. _Ceph configuration: ../../config-cluster/ceph-conf
+.. _Ceph configuration: ../../configuration/ceph-conf
logging, refer to `Ceph Logging and Debugging`_. For additional logging
settings, refer to the `Logging and Debugging Config Reference`_.
-.. _Ceph Logging and Debugging: ../../config-cluster/ceph-conf#ceph-logging-and-debugging
-.. _Logging and Debugging Config Reference: ../../config-cluster/log-and-debug-ref
+.. _Ceph Logging and Debugging: ../../configuration/ceph-conf#ceph-logging-and-debugging
+.. _Logging and Debugging Config Reference: ../../configuration/log-and-debug-ref
You can change the logging settings at runtime so that you don't have to
stop and restart the cluster. Refer to `Ceph Configuration - Runtime Changes`_
Valgrind is computationally expensive, and will slow down your system otherwise.
Valgrind messages are logged to ``stderr``.
-.. _Ceph Configuration - Runtime Changes: ../../config-cluster/ceph-conf#ceph-runtime-config
+.. _Ceph Configuration - Runtime Changes: ../../configuration/ceph-conf#ceph-runtime-config
Swift API <swift/index>
Admin API <admin/index>
troubleshooting
+ Manpage radosgw <../../man/8/radosgw>
+ Manpage radosgw-admin <../../man/8/radosgw-admin>
--- /dev/null
+<disk type='network' device='disk'>
+ <source protocol='rbd' name='poolname/imagename'>
+ <host name='{fqdn}' port='6789'/>
+ <host name='{fgdn}' port='6790'/>
+ <host name='{fqdn}' port='6791'/>
+ </source>
+ <target dev='vda' bus='virtio'/>
+</disk>
\ No newline at end of file
qemu -m 1024 -drive format=raw,file=rbd:data/squeeze:rbd_cache=true
-.. _RBD caching: ../../config-cluster/rbd-config-ref/#rbd-cache-config-settings
+.. _RBD caching: ../rbd-config-ref/#rbd-cache-config-settings
Enabling Discard/TRIM
.. _QEMU Open Source Processor Emulator: http://wiki.qemu.org/Main_Page
.. _QEMU Manual: http://wiki.qemu.org/Manual
-.. _RBD Cache: ../../config-cluster/rbd-config-ref/
\ No newline at end of file
+.. _RBD Cache: ../rbd-config-ref/
\ No newline at end of file
--- /dev/null
+=======================
+ librbd Cache Settings
+=======================
+
+See `Block Device`_ for additional details.
+
+.. sidebar:: Kernel Caching
+
+ The kernel driver for Ceph block devices can use the Linux page cache to
+ improve performance.
+
+The user space implementation of the Ceph block device (i.e., ``librbd``) cannot
+take advantage of the Linux page cache, so it includes its own in-memory
+caching, called "RBD caching." RBD caching behaves just like well-behaved hard
+disk caching. When the OS sends a barrier or a flush request, all dirty data is
+written to the OSDs. This means that using write-back caching is just as safe as
+using a well-behaved physical hard disk with a VM that properly sends flushes
+(i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU)
+algorithm, and in write-back mode it can coalesce contiguous requests for
+better throughput.
+
+.. versionadded:: 0.46
+
+Ceph supports write-back caching for RBD. To enable it, add ``rbd cache =
+true`` to the ``[client]`` section of your ``ceph.conf`` file. By default
+``librbd`` does not perform any caching. Writes and reads go directly to the
+storage cluster, and writes return only when the data is on disk on all
+replicas. With caching enabled, writes return immediately, unless there are more
+than ``rbd cache max dirty`` unflushed bytes. In this case, the write triggers
+writeback and blocks until enough bytes are flushed.
+
+.. versionadded:: 0.47
+
+Ceph supports write-through caching for RBD. You can set the size of
+the cache, and you can set targets and limits to switch from
+write-back caching to write through caching. To enable write-through
+mode, set ``rbd cache max dirty`` to 0. This means writes return only
+when the data is on disk on all replicas, but reads may come from the
+cache. The cache is in memory on the client, and each RBD image has
+its own. Since the cache is local to the client, there's no coherency
+if there are others accesing the image. Running GFS or OCFS on top of
+RBD will not work with caching enabled.
+
+The ``ceph.conf`` file settings for RBD should be set in the ``[client]``
+section of your configuration file. The settings include:
+
+
+``rbd cache``
+
+:Description: Enable caching for RADOS Block Device (RBD).
+:Type: Boolean
+:Required: No
+:Default: ``false``
+
+
+``rbd cache size``
+
+:Description: The RBD cache size in bytes.
+:Type: 64-bit Integer
+:Required: No
+:Default: ``32 MiB``
+
+
+``rbd cache max dirty``
+
+:Description: The ``dirty`` limit in bytes at which the cache triggers write-back. If ``0``, uses write-through caching.
+:Type: 64-bit Integer
+:Required: No
+:Constraint: Must be less than ``rbd cache size``.
+:Default: ``24 MiB``
+
+
+``rbd cache target dirty``
+
+:Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache.
+:Type: 64-bit Integer
+:Required: No
+:Constraint: Must be less than ``rbd cache max dirty``.
+:Default: ``16 MiB``
+
+
+``rbd cache max dirty age``
+
+:Description: The number of seconds dirty data is in the cache before writeback starts.
+:Type: Float
+:Required: No
+:Default: ``1.0``
+
+.. _Block Device: ../../rbd/rbd/
\ No newline at end of file
Snapshots<rbd-snapshot>
QEMU <qemu-rbd>
libvirt <libvirt>
- Cache Settings <../../config-cluster/rbd-config-ref/>
+ Cache Settings <rbd-config-ref/>
OpenStack <rbd-openstack>
CloudStack <rbd-cloudstack>
+ Manpage rbd <../../man/8/rbd>
+ Manpage ceph-rbdnamer <../../man/8/ceph-rbdnamer>
+
.. _RBD Caching: ../../config-cluster/rbd-config-ref/
.. _kernel modules: ../rbd-ko/
projects / ceph.git / commitdiff