From: John Wilkins Date: Wed, 14 Nov 2012 22:57:51 +0000 (-0800) Subject: doc: config-cluser move to new IA. X-Git-Tag: v0.56~200 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=0a2a0c075c32f541c8746530690788adaf048b78;p=ceph.git doc: config-cluser move to new IA. Signed-off-by: John Wilkins --- diff --git a/doc/cephfs/index.rst b/doc/cephfs/index.rst index d419045d274a8..bbcf5833308cd 100644 --- a/doc/cephfs/index.rst +++ b/doc/cephfs/index.rst @@ -9,8 +9,13 @@ with its S3 and Swift APIs, or native bindings. Using Ceph FS requires at least one metadata server in your ``ceph.conf`` configuration file. .. toctree:: + :maxdepth: 1 Mount Ceph FS Mount Ceph FS as FUSE Mount Ceph FS in ``fstab`` Using Ceph with Hadoop + MDS Configuration + Manpage cephfs <../../man/8/cephfs> + Manpage ceph-fuse <../../man/8/ceph-fuse> + Manpage mount.ceph <../../man/8/mount.ceph> diff --git a/doc/cephfs/mds-config-ref.rst b/doc/cephfs/mds-config-ref.rst new file mode 100644 index 0000000000000..8c1fc1e622306 --- /dev/null +++ b/doc/cephfs/mds-config-ref.rst @@ -0,0 +1,447 @@ +====================== + MDS Config Reference +====================== + +``mds max file size`` + +:Description: +:Type: 64-bit Integer Unsigned +:Default: 1ULL << 40 + +``mds cache size`` + +:Description: +:Type: 32-bit Integer +:Default: 100000 + +``mds cache mid`` + +:Description: +:Type: Float +:Default: 0.7 + +``mds mem max`` + +:Description: // KB +:Type: 32-bit Integer +:Default: 1048576 + +``mds dir commit ratio`` + +:Description: +:Type: Float +:Default: 0.5 + +``mds dir max commit size`` + +:Description: // MB +:Type: 32-bit Integer +:Default: 90 + +``mds decay halflife`` + +:Description: +:Type: Float +:Default: 5 + +``mds beacon interval`` + +:Description: +:Type: Float +:Default: 4 + +``mds beacon grace`` + +:Description: +:Type: Float +:Default: 15 + +``mds blacklist interval`` + +:Description: // how long to blacklist failed nodes +:Type: Float +:Default: 24.0*60.0 + +``mds session timeout`` + +:Description: // cap bits and leases time out if client idle +:Type: Float +:Default: 60 + +``mds session autoclose`` + +:Description: // autoclose idle session +:Type: Float +:Default: 300 + +``mds reconnect timeout`` + +:Description: // secs to wait for clients during mds restart +:Type: Float +:Default: 45 + +``mds tick interval`` + +:Description: +:Type: Float +:Default: 5 + +``mds dirstat min interval`` + +:Description: //try to avoid propagating more often than x +:Type: Float +:Default: 1 + +``mds scatter nudge interval`` + +:Description: // how quickly dirstat changes propagate up +:Type: Float +:Default: 5 + +``mds client prealloc inos`` + +:Description: +:Type: 32-bit Integer +:Default: 1000 + +``mds early reply`` + +:Description: +:Type: Boolean +:Default: true + +``mds use tmap`` + +:Description: // use trivialmap for dir updates +:Type: Boolean +:Default: true + +``mds default dir hash`` + +:Description: CEPH STR HASH RJENKINS +:Type: 32-bit Integer +:Default: + +``mds log`` + +:Description: +:Type: Boolean +:Default: true + +``mds log skip corrupt events`` + +:Description: +:Type: Boolean +:Default: false + +``mds log max events`` + +:Description: +:Type: 32-bit Integer +:Default: -1 + +``mds log max segments`` + +:Description: // segment size defined by FileLayout above +:Type: 32-bit Integer +:Default: 30 + +``mds log max expiring`` + +:Description: +:Type: 32-bit Integer +:Default: 20 + +``mds log eopen size`` + +:Description: // # open inodes per log entry +:Type: 32-bit Integer +:Default: 100 + +``mds bal sample interval`` + +:Description: // every 5 seconds +:Type: Float +:Default: 3 + +``mds bal replicate threshold`` + +:Description: +:Type: Float +:Default: 8000 + +``mds bal unreplicate threshold`` + +:Description: +:Type: Float +:Default: 0 + +``mds bal frag`` + +:Description: +:Type: Boolean +:Default: false + +``mds bal split size`` + +:Description: +:Type: 32-bit Integer +:Default: 10000 + +``mds bal split rd`` + +:Description: +:Type: Float +:Default: 25000 + +``mds bal split wr`` + +:Description: +:Type: Float +:Default: 10000 + +``mds bal split bits`` + +:Description: +:Type: 32-bit Integer +:Default: 3 + +``mds bal merge size`` + +:Description: +:Type: 32-bit Integer +:Default: 50 + +``mds bal merge rd`` + +:Description: +:Type: Float +:Default: 1000 + +``mds bal merge wr`` + +:Description: +:Type: Float +:Default: 1000 + +``mds bal interval`` + +:Description: // seconds +:Type: 32-bit Integer +:Default: 10 + +``mds bal fragment interval`` + +:Description: // seconds +:Type: 32-bit Integer +:Default: 5 + +``mds bal idle threshold`` + +:Description: +:Type: Float +:Default: 0 + +``mds bal max`` + +:Description: +:Type: 32-bit Integer +:Default: -1 + +``mds bal max until`` + +:Description: +:Type: 32-bit Integer +:Default: -1 + +``mds bal mode`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds bal min rebalance`` + +:Description: // must be x above avg before we export +:Type: Float +:Default: 0.1 + +``mds bal min start`` + +:Description: // if we need less x. we don't do anything +:Type: Float +:Default: 0.2 + +``mds bal need min`` + +:Description: // take within this range of what we need +:Type: Float +:Default: 0.8 + +``mds bal need max`` + +:Description: +:Type: Float +:Default: 1.2 + +``mds bal midchunk`` + +:Description: // any sub bigger than this taken in full +:Type: Float +:Default: 0.3 + +``mds bal minchunk`` + +:Description: // never take anything smaller than this +:Type: Float +:Default: 0.001 + +``mds bal target removal min`` + +:Description: // min bal iters before old target is removed +:Type: 32-bit Integer +:Default: 5 + +``mds bal target removal max`` + +:Description: // max bal iters before old target is removed +:Type: 32-bit Integer +:Default: 10 + +``mds replay interval`` + +:Description: // time to wait before starting replay again +:Type: Float +:Default: 1 + +``mds shutdown check`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds thrash exports`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds thrash fragments`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds dump cache on map`` + +:Description: +:Type: Boolean +:Default: false + +``mds dump cache after rejoin`` + +:Description: +:Type: Boolean +:Default: false + +``mds verify scatter`` + +:Description: +:Type: Boolean +:Default: false + +``mds debug scatterstat`` + +:Description: +:Type: Boolean +:Default: false + +``mds debug frag`` + +:Description: +:Type: Boolean +:Default: false + +``mds debug auth pins`` + +:Description: +:Type: Boolean +:Default: false + +``mds debug subtrees`` + +:Description: +:Type: Boolean +:Default: false + +``mds kill mdstable at`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds kill export at`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds kill import at`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds kill link at`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds kill rename at`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``mds wipe sessions`` + +:Description: +:Type: Boolean +:Default: 0 + +``mds wipe ino prealloc`` + +:Description: +:Type: Boolean +:Default: 0 + +``mds skip ino`` + +:Description: +:Type: 32-bit Integer +:Default: 0 + +``max mds`` + +:Description: +:Type: 32-bit Integer +:Default: 1 + +``mds standby for name`` + +:Description: +:Type: String +:Default: + +``mds standby for rank`` + +:Description: +:Type: 32-bit Integer +:Default: -1 + +``mds standby replay`` + +:Description: +:Type: Boolean +:Default: false \ No newline at end of file diff --git a/doc/config-cluster/ceph-conf.rst b/doc/config-cluster/ceph-conf.rst deleted file mode 100644 index 3ee71bf24731a..0000000000000 --- a/doc/config-cluster/ceph-conf.rst +++ /dev/null @@ -1,550 +0,0 @@ -================== - Configuring Ceph -================== - -When you start the Ceph service, the initialization process activates a series -of daemons that run in the background. The hosts in a typical Ceph cluster run -at least one of four daemons: - -- Object Storage Device (``ceph-osd``) -- Monitor (``ceph-mon``) -- Metadata Server (``ceph-mds``) -- Ceph Gateway (``radosgw``) - -For your convenience, each daemon has a series of default values (*i.e.*, many -are set by ``ceph/src/common/config_opts.h``). You may override these settings -with a Ceph configuration file. - -.. _ceph-conf-file: - -The ceph.conf File -================== - -When you start a Ceph cluster, each daemon looks for a ``ceph.conf`` file that -provides its configuration settings. For manual deployments, you need to create -a ``ceph.conf`` file to configure your cluster. For third party tools that -create configuration files for you (*e.g.*, Chef), you may use the information -contained herein as a reference. The ``ceph.conf`` file defines: - -- Cluster membership -- Host names -- Host addresses -- Paths to keyrings -- Paths to journals -- Paths to data -- Other runtime options - -The default ``ceph.conf`` locations in sequential order include: - -#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` environment variable) -#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument) -#. ``/etc/ceph/ceph.conf`` -#. ``~/.ceph/config`` -#. ``./ceph.conf`` (*i.e.,* in the current working directory) - - -The ``ceph.conf`` file uses an *ini* style syntax. You can add comments to the -``ceph.conf`` file by preceding comments with a semi-colon (;) or a pound sign -(#). For example: - -.. code-block:: ini - - # <--A number (#) sign precedes a comment. - ; A comment may be anything. - # Comments always follow a semi-colon (;) or a pound (#) on each line. - # The end of the line terminates a comment. - # We recommend that you provide comments in your configuration file(s). - -.. _ceph-conf-settings: - -ceph.conf Settings -================== - -The ``ceph.conf`` file can configure all daemons in a cluster, or all daemons of -a particular type. To configure a series of daemons, the settings must be -included under the processes that will receive the configuration as follows: - -``[global]`` - -:Description: Settings under ``[global]`` affect all daemons in a Ceph cluster. -:Example: ``auth supported = cephx`` - -``[osd]`` - -:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in the cluster. -:Example: ``osd journal size = 1000`` - -``[mon]`` - -:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in the cluster. -:Example: ``mon addr = 10.0.0.101:6789`` - - -``[mds]`` - -:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in the cluster. -:Example: ``host = myserver01`` - - -Global settings affect all instances of all daemon in the cluster. Use the ``[global]`` -setting for values that are common for all daemons in the cluster. You can override each -``[global]`` setting by: - -#. Changing the setting in a particular process type (*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ). -#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` ) - -Overriding a global setting affects all child processes, except those that -you specifically override. - -A typical global setting involves activating authentication. For example: - -.. code-block:: ini - - [global] - # Enable authentication between hosts within the cluster. - auth supported = cephx - - -You can specify settings that apply to a particular type of daemon. When you -specify settings under ``[osd]``, ``[mon]`` or ``[mds]`` without specifying a -particular instance, the setting will apply to all OSDs, monitors or metadata -daemons respectively. - -You may specify settings for particular instances of a daemon. You may specify -an instance by entering its type, delimited by a period (.) and by the -instance ID. The instance ID for an OSD is always numeric, but it may be -alphanumeric for monitors and metadata servers. - -.. code-block:: ini - - [osd.1] - # settings affect osd.1 only. - - [mon.a] - # settings affect mon.a only. - - [mds.b] - # settings affect mds.b only. - -.. _ceph-metavariables: - -Metavariables -============= - -Metavariables simplify cluster configuration dramatically. When a metavariable -is set in a configuration value, Ceph expands the metavariable into a concrete -value. Metavariables are very powerful when used within the ``[global]``, -``[osd]``, ``[mon]`` or ``[mds]`` sections of your configuration file. Ceph -metavariables are similar to Bash shell expansion. - -Ceph supports the following metavariables: - - -``$cluster`` - -:Description: Expands to the cluster name. Useful when running multiple clusters on the same hardware. -:Example: ``/etc/ceph/$cluster.keyring`` -:Default: ``ceph`` - - -``$type`` - -:Description: Expands to one of ``mds``, ``osd``, or ``mon``, depending on the type of the current daemon. -:Example: ``/var/lib/ceph/$type`` - - -``$id`` - -:Description: Expands to the daemon identifier. For ``osd.0``, this would be ``0``; for ``mds.a``, it would be ``a``. -:Example: ``/var/lib/ceph/$type/$cluster-$id`` - - -``$host`` - -:Description: Expands to the host name of the current daemon. - - -``$name`` - -:Description: Expands to ``$type.$id``. -:Example: ``/var/run/ceph/$cluster-$name.asok`` - -.. _ceph-conf-common-settings: - -Common Settings -=============== - -The `Hardware Recommendations`_ section provides some hardware guidelines for -configuring the cluster. It is possible for a single host to run multiple -daemons. For example, a single host with multiple disks or RAIDs may run one -``ceph-osd`` for each disk or RAID. Additionally, a host may run both a -``ceph-mon`` and an ``ceph-osd`` daemon on the same host. Ideally, you will have -a host for a particular type of process. For example, one host may run -``ceph-osd`` daemons, another host may run a ``ceph-mds`` daemon, and other -hosts may run ``ceph-mon`` daemons. - -Each host has a name identified by the ``host`` setting. Monitors also specify -a network address and port (i.e., domain name or IP address) identified by the -``addr`` setting. A basic configuration file will typically specify only -minimal settings for each instance of a daemon. For example: - -.. code-block:: ini - - [mon.a] - host = hostName - mon addr = 150.140.130.120:6789 - - [osd.0] - host = hostName - -.. important:: The ``host`` setting is the short name of the host (i.e., not - an fqdn). It is **NOT** and IP address either. Enter ``hostname -s`` on - the command line to retrieve the name of the host. Also, this setting is - **ONLY** for ``mkcephfs`` and manual deployment. It **MUST NOT** - be used with ``chef`` or ``ceph-deploy``. - -.. _Hardware Recommendations: ../../install/hardware-recommendations - -.. _ceph-network-config: - -Networks -======== - -Monitors listen on port 6789 by default, while metadata servers and OSDs listen -on the first available port beginning at 6800. Ensure that you open port 6789 on -hosts that run a monitor daemon, and open one port beginning at port 6800 for -each OSD or metadata server that runs on the host. Ports are host-specific, so -you don't need to open any more ports open than the number of daemons running on -that host, other than potentially a few spares. You may consider opening a few -additional ports in case a daemon fails and restarts without letting go of the -port such that the restarted daemon binds to a new port. If you set up separate -public and cluster networks, you may need to make entries for each network. -For example:: - - iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:6810 -j ACCEPT - - -In our `hardware recommendations`_ section, we recommend having at least two NIC -cards, because Ceph can support two networks: a public (front-side) network, and -a cluster (back-side) network. Ceph functions just fine with a public network -only. You only need to specify the public and cluster network settings if you -use both public and cluster networks. - -There are several reasons to consider operating two separate networks. First, -OSDs handle data replication for the clients. When OSDs replicate data more than -once, the network load between OSDs easily dwarfs the network load between -clients and the Ceph cluster. This can introduce latency and create a -performance problem. Second, while most people are generally civil, a very tiny -segment of the population likes to engage in what's known as a Denial of Service -(DoS) attack. When traffic between OSDs gets disrupted, placement groups may no -longer reflect an ``active + clean`` state, which may prevent users from reading -and writing data. A great way to defeat this type of attack is to maintain a -completely separate cluster network that doesn't connect directly to the -internet. - -To configure the networks, add the following options to the ``[global]`` section -of your ``ceph.conf`` file. - -.. code-block:: ini - - [global] - public network {public-network-ip-address/netmask} - cluster network {enter cluster-network-ip-address/netmask} - -To configure Ceph hosts to use the networks, you should set the following options -in the daemon instance sections of your ``ceph.conf`` file. - -.. code-block:: ini - - [osd.0] - public addr {host-public-ip-address} - cluster addr {host-cluster-ip-address} - -.. _hardware recommendations: ../../install/hardware-recommendations - - -.. _ceph-monitor-config: - -Monitors -======== - -Ceph production clusters typically deploy with a minimum 3 monitors to ensure -high availability should a monitor instance crash. An odd number of monitors (3) -ensures that the Paxos algorithm can determine which version of the cluster map -is the most recent from a quorum of monitors. - -.. note:: You may deploy Ceph with a single monitor, but if the instance fails, - the lack of a monitor may interrupt data service availability. - -Ceph monitors typically listen on port ``6789``. For example: - -.. code-block:: ini - - [mon.a] - host = hostName - mon addr = 150.140.130.120:6789 - -By default, Ceph expects that you will store a monitor's data under the following path:: - - /var/lib/ceph/mon/$cluster-$id - -You must create the corresponding directory yourself. With metavariables fully -expressed and a cluster named "ceph", the foregoing directory would evaluate to:: - - /var/lib/ceph/mon/ceph-a - -You may override this path using the ``mon data`` setting. We don't recommend -changing the default location. Create the default directory on your new monitor host. :: - - ssh {new-mon-host} - sudo mkdir /var/lib/ceph/mon/ceph-{mon-letter} - - -.. _ceph-osd-config: - -OSDs -==== - -Ceph production clusters typically deploy OSDs where one host has one OSD daemon -running a filestore on one data disk. A typical deployment specifies a journal -size and whether the file store's extended attributes (XATTRs) use an -object map (i.e., when running on the ``ext4`` filesystem). For example: - -.. code-block:: ini - - [osd] - osd journal size = 10000 - filestore xattr use omap = true #enables the object map. Only if running ext4. - - [osd.0] - hostname = {hostname} - - -By default, Ceph expects that you will store an OSD's data with the following path:: - - /var/lib/ceph/osd/$cluster-$id - -You must create the corresponding directory yourself. With metavariables fully -expressed and a cluster named "ceph", the foregoing directory would evaluate to:: - - /var/lib/ceph/osd/ceph-0 - -You may override this path using the ``osd data`` setting. We don't recommend -changing the default location. Create the default directory on your new OSD host. :: - - ssh {new-osd-host} - sudo mkdir /var/lib/ceph/osd/ceph-{osd-number} - -The ``osd data`` path ideally leads to a mount point with a hard disk that is -separate from the hard disk storing and running the operating system and -daemons. If the OSD is for a disk other than the OS disk, prepare it for -use with Ceph, and mount it to the directory you just created:: - - ssh {new-osd-host} - sudo mkfs -t {fstype} /dev/{disk} - sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number} - -We recommend using the ``xfs`` file system or the ``btrfs`` file system when -running :command:mkfs. - -By default, Ceph expects that you will store an OSDs journal with the following path:: - - /var/lib/ceph/osd/$cluster-$id/journal - -Without performance optimization, Ceph stores the journal on the same disk as -the OSDs data. An OSD optimized for performance may use a separate disk to store -journal data (e.g., a solid state drive delivers high performance journaling). - -Ceph's default ``osd journal size`` is 0, so you will need to set this in your -``ceph.conf`` file. A journal size should find the product of the ``filestore -min sync interval`` and the expected throughput, and multiple the product by -two (2):: - - osd journal size = {2 * (expected throughput * filestore min sync interval)} - -The expected throughput number should include the expected disk throughput -(i.e., sustained data transfer rate), and network throughput. For example, -a 7200 RPM disk will likely have approximately 100 MB/s. Taking the ``min()`` -of the disk and network throughput should provide a reasonable expected -throughput. Some users just start off with a 10GB journal size. For -example:: - - osd journal size = 10000 - -.. _ceph-logging-and-debugging: - -Logs / Debugging -================ - -Ceph is still on the leading edge, so you may encounter situations that require -modifying logging output and using Ceph's debugging. To activate Ceph's -debugging output (*i.e.*, ``dout()``), you may add ``debug`` settings to your -configuration. Ceph's logging levels operate on a scale of 1 to 20, where 1 is -terse and 20 is verbose. Subsystems common to each daemon may be set under -``[global]`` in your configuration file. Subsystems for particular daemons are -set under the daemon section in your configuration file (*e.g.*, ``[mon]``, -``[osd]``, ``[mds]``). For example:: - - [global] - debug ms = 1 - - [mon] - debug mon = 20 - debug paxos = 20 - debug auth = 20 - - [osd] - debug osd = 20 - debug filestore = 20 - debug journal = 20 - debug monc = 20 - - [mds] - debug mds = 20 - debug mds balancer = 20 - debug mds log = 20 - debug mds migrator = 20 - -When your system is running well, choose appropriate logging levels and remove -unnecessary debugging settings to ensure your cluster runs optimally. Logging -debug output messages is relatively slow, and a waste of resources when operating -your cluster. - -.. tip: When debug output slows down your system, the latency can hide race conditions. - -Each subsystem has a logging level for its output logs, and for its logs -in-memory. You may set different values for each of these subsystems by setting -a log file level and a memory level for debug logging. For example:: - - debug {subsystem} {log-level}/{memory-level} - #for example - debug mds log 1/20 - -+--------------------+-----------+--------------+ -| Subsystem | Log Level | Memory Level | -+====================+===========+==============+ -| ``default`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``lockdep`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``context`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``crush`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``mds`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``mds balancer`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``mds locker`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``mds log`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``mds log expire`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``mds migrator`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``buffer`` | 0 | 0 | -+--------------------+-----------+--------------+ -| ``timer`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``filer`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``objecter`` | 0 | 0 | -+--------------------+-----------+--------------+ -| ``rados`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``rbd`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``journaler`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``objectcacher`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``client`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``osd`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``optracker`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``objclass`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``filestore`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``journal`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``ms`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``mon`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``monc`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``paxos`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``tp`` | 0 | 5 | -+--------------------+-----------+--------------+ -| ``auth`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``finisher`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``heartbeatmap`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``perfcounter`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``rgw`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``hadoop`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``asok`` | 1 | 5 | -+--------------------+-----------+--------------+ -| ``throttle`` | 1 | 5 | -+--------------------+-----------+--------------+ - - -Example ceph.conf -================= - -.. literalinclude:: demo-ceph.conf - :language: ini - -.. _ceph-runtime-config: - -Runtime Changes -=============== - -Ceph allows you to make changes to the configuration of an ``ceph-osd``, -``ceph-mon``, or ``ceph-mds`` daemon at runtime. This capability is quite -useful for increasing/decreasing logging output, enabling/disabling debug -settings, and even for runtime optimization. The following reflects runtime -configuration usage:: - - ceph {daemon-type} tell {id or *} injectargs '--{name} {value} [--{name} {value}]' - -Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply -the runtime setting to all daemons of a particular type with ``*``, or specify -a specific daemon's ID (i.e., its number or letter). For example, to increase -debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following:: - - ceph osd tell 0 injectargs '--debug-osd 20 --debug-ms 1' - -In your ``ceph.conf`` file, you may use spaces when specifying a -setting name. When specifying a setting name on the command line, -ensure that you use an underscore or hyphen (``_`` or ``-``) between -terms (e.g., ``debug osd`` becomes ``debug-osd``). - - -Viewing a Configuration at Runtime -================================== - -If your Ceph cluster is running, and you would like to see the configuration -settings from a running daemon, execute the following:: - - ceph --admin-daemon {/path/to/admin/socket} config show | less - -The default path for the admin socket for each daemon is:: - - /var/run/ceph/$cluster-$name.asok - -At real time, the metavariables will evaluate to the actual cluster name -and daemon name. For example, if the cluster name is ``ceph`` (it is by default) -and you want to retrieve the configuration for ``osd.0``, use the following:: - - ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less diff --git a/doc/config-cluster/chef.rst b/doc/config-cluster/chef.rst deleted file mode 100644 index 2840391931574..0000000000000 --- a/doc/config-cluster/chef.rst +++ /dev/null @@ -1,251 +0,0 @@ -===================== - Deploying with Chef -===================== - -We use Chef cookbooks to deploy Ceph. See `Managing Cookbooks with Knife`_ for details -on using ``knife``. For Chef installation instructions, see `Installing Chef`_. - -.. _clonecbs: - -Clone the Required Cookbooks ----------------------------- - -To get the cookbooks for Ceph, clone them from git.:: - - cd ~/chef-cookbooks - git clone https://github.com/opscode-cookbooks/apache2.git - git clone https://github.com/ceph/ceph-cookbooks.git ceph - -.. _addcbpaths: - -Add the Required Cookbook Paths -------------------------------- - -If you added a default cookbook path when you installed Chef, ``knife`` -may be able to upload the cookbook you've cloned to your cookbook path -directory without further configuration. If you used a different path, -or if the cookbook repository you cloned has a different tree structure, -add the required cookbook path to your ``knife.rb`` file. The -``cookbook_path`` setting takes a string or an array of strings. -For example, you can replace a string path with an array of string paths:: - - cookbook_path '/home/{user-name}/chef-cookbooks/' - -Becomes:: - - cookbook_path [ - '/home/{user-name}/chef-cookbooks/', - '/home/{user-name}/chef-cookbooks/{another-directory}/', - '/some/other/path/to/cookbooks/' - ] - -.. _installcbs: - -Install the Cookbooks ---------------------- - -To install Ceph, you must upload the Ceph cookbooks and the Apache cookbooks -(for use with RADOSGW) to your Chef server. :: - - knife cookbook upload apache2 ceph - -.. _configcephenv: - -Configure your Ceph Environment -------------------------------- - -The Chef server can support installation of software for multiple environments. -The environment you create for Ceph requires an ``fsid``, the secret for -your monitor(s) if you are running Ceph with ``cephx`` authentication, and -the host name (i.e., short name) for your monitor hosts. - -.. tip: Open an empty text file to hold the following values until you create - your Ceph environment. - -For the filesystem ID, use ``uuidgen`` from the ``uuid-runtime`` package to -generate a unique identifier. :: - - uuidgen -r - -For the monitor(s) secret(s), use ``ceph-authtool`` to generate the secret(s):: - - sudo apt-get update - sudo apt-get install ceph-common - ceph-authtool /dev/stdout --name=mon. --gen-key - -The secret is the value to the right of ``"key ="``, and should look something -like this:: - - AQBAMuJPINJgFhAAziXIrLvTvAz4PRo5IK/Log== - -To create an environment for Ceph, set a command line editor. For example:: - - export EDITOR=vim - -Then, use ``knife`` to create an environment. :: - - knife environment create {env-name} - -For example:: - - knife environment create Ceph - -A JSON file will appear. Perform the following steps: - -#. Enter a description for the environment. -#. In ``"default_attributes": {}``, add ``"ceph" : {}``. -#. Within ``"ceph" : {}``, add ``"monitor-secret":``. -#. Immediately following ``"monitor-secret":`` add the key you generated within quotes, followed by a comma. -#. Within ``"ceph":{}`` and following the ``monitor-secret`` key-value pair, add ``"config": {}`` -#. Within ``"config": {}`` add ``"fsid":``. -#. Immediately following ``"fsid":``, add the unique identifier you generated within quotes, followed by a comma. -#. Within ``"config": {}`` and following the ``fsid`` key-value pair, add ``"mon_initial_members":`` -#. Immediately following ``"mon_initial_members":``, enter the initial monitor host names. - -For example:: - - "default_attributes" : { - "ceph": { - "monitor-secret": "{replace-with-generated-secret}", - "config": { - "fsid": "{replace-with-generated-uuid}", - "mon_initial_members": "{replace-with-monitor-hostname(s)}" - } - } - } - -Advanced users (i.e., developers and QA) may also add ``"ceph_branch": "{branch}"`` -to ``default-attributes``, replacing ``{branch}`` with the name of the branch you -wish to use (e.g., ``master``). - -.. configroles: - -Configure the Roles -------------------- - -Navigate to the Ceph cookbooks directory. :: - - cd ~/chef-cookbooks/ceph - -Create roles for OSDs, monitors, metadata servers, and RADOS Gateways from -their respective role files. :: - - knife role from file roles/ceph-osd.rb - knife role from file roles/ceph-mon.rb - knife role from file roles/ceph-mds.rb - knife role from file roles/ceph-radosgw.rb - -.. _confignodes: - -Configure Nodes ---------------- - -You must configure each node you intend to include in your Ceph cluster. -Identify nodes for your Ceph cluster. :: - - knife node list - -.. note: for each host where you installed Chef and executed ``chef-client``, - the Chef server should have a minimal node configuration. You can create - additional nodes with ``knife node create {node-name}``. - -For each node you intend to use in your Ceph cluster, configure the node -as follows:: - - knife node edit {node-name} - -The node configuration should appear in your text editor. Change the -``chef_environment`` value to ``Ceph`` (or whatever name you set for your -Ceph environment). - -In the ``run_list``, add ``"recipe[ceph::apt]",`` to all nodes as -the first setting, so that Chef can install or update the necessary packages. -Then, add at least one of:: - - "role[ceph-mon]" - "role[ceph-osd]" - "role[ceph-mds]" - "role[ceph-radosgw]" - -If you add more than one role, separate them with a comma. Run ``hostname`` -on your command line, and replace the ``{hostname}`` setting of the ``name`` -key to the host name for the node. :: - - { - "chef_environment": "Ceph", - "name": "{hostname}", - "normal": { - "tags": [ - - ] - }, - "run_list": [ - "recipe[ceph::apt]", - "role[ceph-mon]", - "role[ceph-mds]" - ] - } - -.. _prepdisks: - -Prepare OSD Disks ------------------ - -Configuring a node with an OSD role tells Chef that the node will run at -least one OSD. However, you may run many OSDs on one host. For example, -you may run one ``ceph-osd`` daemon for each data disk on the system. -This step prepares the OSD disk(s) and tells Chef how many OSDs the -node will be running. - - -For the Ceph 0.48 Argonaut release, install ``gdisk``:: - - sudo apt-get install gdisk - -For the Ceph 0.48 Argonaut release, on each hard disk that will store data for -an OSD daemon, configure the hard disk for use with Ceph. Replace ``{fsid}`` -with the UUID you generated while using ``uuidgen -r``. - -.. important: This procedure will erase all information in ``/dev/{disk}``. - -:: - - sudo sgdisk /dev/{disk} --zap-all --clear --mbrtogpt --largest-new=1 --change-name=1:'ceph data' --typecode=1:{fsid} - -Create a file system and allocate the disk to your cluster. Specify a -filesystem (e.g., ``ext4``, ``xfs``, ``btrfs``). When you execute -``ceph-disk-prepare``, remember to replace ``{fsid}`` with the UUID you -generated while using ``uuidgen -r``:: - - sudo mkfs -t ext4 /dev/{disk} - sudo mount -o user_xattr /dev/{disk} /mnt - sudo ceph-disk-prepare --cluster-uuid={fsid} /mnt - sudo umount /mnt - -Finally, simulate a hotplug event. :: - - sudo udevadm trigger --subsystem-match=block --action=add - - -.. _runchefclient: - -Run ``chef-client`` on each Node --------------------------------- - -Once you have completed the preceding steps, you must run ``chef-client`` -on each node. For example:: - - sudo chef-client - -.. _proceedtoops: - -Proceed to Operating the Cluster --------------------------------- - -Once you complete the deployment, you may begin operating your cluster. -See `Operating a Cluster`_ for details. - - -.. _Managing Cookbooks with Knife: http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife -.. _Installing Chef: ../../install/chef -.. _Operating a Cluster: ../../init/ \ No newline at end of file diff --git a/doc/config-cluster/demo-ceph.conf b/doc/config-cluster/demo-ceph.conf deleted file mode 100644 index b96b32cd4a195..0000000000000 --- a/doc/config-cluster/demo-ceph.conf +++ /dev/null @@ -1,32 +0,0 @@ -[global] - auth supported = cephx - - -[osd] - osd journal size = 1000 - # uncomment the following line if you are mounting with ext4 - # filestore xattr use omap = true - -[mon.a] - host = myserver01 - mon addr = 10.0.0.101:6789 - -[mon.b] - host = myserver02 - mon addr = 10.0.0.102:6789 - -[mon.c] - host = myserver03 - mon addr = 10.0.0.103:6789 - -[osd.0] - host = myserver01 - -[osd.1] - host = myserver02 - -[osd.2] - host = myserver03 - -[mds.a] - host = myserver01 diff --git a/doc/config-cluster/file-system-recommendations.rst b/doc/config-cluster/file-system-recommendations.rst deleted file mode 100644 index 17908cc3d19a1..0000000000000 --- a/doc/config-cluster/file-system-recommendations.rst +++ /dev/null @@ -1,78 +0,0 @@ -=========================================== - Hard Disk and File System Recommendations -=========================================== - -Hard Disk Prep -============== - -Ceph aims for data safety, which means that when the application receives notice -that data was written to the disk, that data was actually written to the disk. -For old kernels (<2.6.33), disable the write cache if the journal is on a raw -disk. Newer kernels should work fine. - -Use ``hdparm`` to disable write caching on the hard disk:: - - sudo hdparm -W 0 /dev/hda 0 - -In production environments, we recommend running OSDs with an operating system -disk, and a separate disk(s) for data. If you run data and an operating system -on a single disk, create a separate partition for your data before configuring -your OSD cluster. - - -File Systems -============ - -Ceph OSDs rely heavily upon the stability and performance of the -underlying file system. - -.. note:: We currently recommend ``XFS`` for production deployments. - We recommend ``btrfs`` for testing, development, and any - non-critical deployments. We believe that ``btrfs`` has the correct - feature set and roadmap to serve Ceph in the long-term, but ``XFS`` - and ``ext4`` provide the necessary stability for today's deployments. - ``btrfs`` development is proceeding rapidly: users should be - comfortable installing the latest released upstream kernels and be - able to track development activity for critical bug fixes. - -Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying -file system for various forms of internal object state and metadata. -The underlying file system must provide sufficient capacity for -XATTRs. ``btrfs`` does not bound the total xattr metadata stored with -a file. ``XFS`` has a relatively large limit (64 KB) that most -deployments won't encounter, but the ``ext4`` is too small to be -usable. To use these file systems, you should add the following like -to the ``[osd]`` section of your ``ceph.conf`` file.:: - - filestore xattr use omap = true - -FS Background Info -================== - -The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly -scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``. -Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that -they are more robust when recovering from crashes, power outages, etc. These -filesystems journal all of the changes they will make before performing writes. - -``XFS`` was developed for Silicon Graphics, and is a mature and stable -filesystem. By contrast, ``btrfs`` is a relatively new file system that aims -to address the long-standing wishes of system administrators working with -large scale data storage environments. ``btrfs`` has some unique features -and advantages compared to other Linux filesystems. - -``btrfs`` is a `copy-on-write`_ filesystem. It supports file creation -timestamps and checksums that verify metadata integrity, so it can detect -bad copies of data and fix them with the good copies. The copy-on-write -capability means that ``btrfs`` can support snapshots that are writable. -``btrfs`` supports transparent compression and other features. - -``btrfs`` also incorporates multi-device management into the file system, -which enables you to support heterogeneous disk storage infrastructure, -data allocation policies. The community also aims to provide ``fsck``, -deduplication, and data encryption support in the future. This compelling -list of features makes ``btrfs`` the ideal choice for Ceph clusters. - -.. _copy-on-write: http://en.wikipedia.org/wiki/Copy-on-write -.. _compared: http://en.wikipedia.org/wiki/Comparison_of_file_systems -.. _journaling file systems: http://en.wikipedia.org/wiki/Journaling_file_system diff --git a/doc/config-cluster/filestore-config-ref.rst b/doc/config-cluster/filestore-config-ref.rst deleted file mode 100644 index 0babcd85034bd..0000000000000 --- a/doc/config-cluster/filestore-config-ref.rst +++ /dev/null @@ -1,288 +0,0 @@ -============================ - Filestore Config Reference -============================ - - - -``filestore debug omap check`` - -:Description: Debugging check on synchronization. Expensive. For debugging only. -:Type: Boolean -:Required: No -:Default: ``0`` - - -Extended Attributes -=================== - -Extended Attributes (XATTRs) are an imporant aspect in your configuration. -Some file systems have limits on the number of bytes stored in XATTRS. -Additionally, in some cases, the filesystem may not be as fast as an alternative -method of storing XATTRs. The following settings may help improve performance -by using a method of storing XATTRs that is extrinsic to the underlying filesystem. - - -``filestore xattr use omap`` - -:Description: Use object map for XATTRS. Set to ``true`` for ``ext4`` file systems. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``filestore max inline xattr size`` - -:Description: The maximimum size of an XATTR stored in the filesystem (i.e., XFS, btrfs, ext4, etc.) per object. Should not be larger than the filesytem can handle. -:Type: Unsigned 32-bit Integer -:Required: No -:Default: ``512`` - - -``filestore max inline xattrs`` - -:Description: The maximum number of XATTRs stored in the fileystem per object. -:Type: 32-bit Integer -:Required: No -:Default: ``2`` - - -Synchronization Intervals -========================= - -Periodically, the filestore needs to quiesce writes and synchronize the filesystem, -which creates a consistent commit point. It can then free journal entries up to -the commit point. Synchronizing more frequently tends to reduce the time required -perform synchronization, and reduces the amount of data that needs to remain in the -journal. Less frequent synchronization allows the backing filesystem to coalesce -small writes and metadata updates more optimally--potentially resulting in more -efficient synchronization. - - -``filestore max sync interval`` - -:Description: The maximum interval in seconds for synchronizing the filestore. -:Type: Double -:Required: No -:Default: ``5`` - - -``filestore min sync interval`` - -:Description: The minimum interval in seconds for synchronizing the filestore. -:Type: Double -:Required: No -:Default: ``.01`` - - -Flusher -======= - -The filestore flusher forces data from large writes to be written out using -``sync file range`` before the sync in order to (hopefully) reduce the cost of -the eventual sync. In practice, disabling 'filestore flusher' seems to improve -performance in some cases. - - -``filestore flusher`` - -:Description: Enables the filestore flusher. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``filestore flusher max fds`` - -:Description: Sets the maximum number of file descriptors for the flusher. -:Type: Integer -:Required: No -:Default: ``512`` - -``filestore sync flush`` - -:Description: Enables the synchronization flusher. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``filestore fsync flushes journal data`` - -:Description: Flush journal data during filesystem synchronization. -:Type: Boolean -:Required: No -:Default: ``false`` - - -Queue -===== - -The following settings provide limits on the size of filestore queue. - -``filestore queue max ops`` - -:Description: Defines the maximum number of in progress operations the file store accepts before blocking on queuing new operations. -:Type: Integer -:Required: No. Minimal impact on performance. -:Default: ``500`` - - -``filestore queue max bytes`` - -:Description: The maximum number of bytes for an operation. -:Type: Integer -:Required: No -:Default: ``100 << 20`` - - -``filestore queue committing max ops`` - -:Description: The maximum number of operations the filestore can commit. -:Type: Integer -:Required: No -:Default: ``500`` - - -``filestore queue committing max bytes`` - -:Description: The maximum number of bytes the filestore can commit. -:Type: Integer -:Required: No -:Default: ``100 << 20`` - - - -Timeouts -======== - - -``filestore op threads`` - -:Description: The number of filesystem operation threads that execute in parallel. -:Type: Integer -:Required: No -:Default: ``2`` - - -``filestore op thread timeout`` - -:Description: The timeout for a filesystem operation thread (in seconds). -:Type: Integer -:Required: No -:Default: ``60`` - - -``filestore op thread suicide timeout`` - -:Description: The timeout for a commit operation before cancelling the commit (in seconds). -:Type: Integer -:Required: No -:Default: ``180`` - - -B-Tree Filesystem -================= - - -``filestore btrfs snap`` - -:Description: Enable snapshots for a ``btrfs`` filestore. -:Type: Boolean -:Required: No. Only used for ``btrfs``. -:Default: ``true`` - - -``filestore btrfs clone range`` - -:Description: Enable cloning ranges for a ``btrfs`` filestore. -:Type: Boolean -:Required: No. Only used for ``btrfs``. -:Default: ``true`` - -Journal -======= - - -``filestore journal parallel`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -``filestore journal writeahead`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -``filestore journal trailing`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -Misc -==== - - -``filestore merge threshold`` - -:Description: -:Type: Integer -:Required: No -:Default: ``10`` - - -``filestore split multiple`` - -:Description: -:Type: Integer -:Required: No -:Default: ``2`` - - -``filestore update to`` - -:Description: -:Type: Integer -:Required: No -:Default: ``1000`` - - -``filestore blackhole`` - -:Description: Drop any new transactions on the floor. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``filestore dump file`` - -:Description: File onto which store transaction dumps? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``filestore kill at`` - -:Description: inject a failure at the n'th opportunity -:Type: String -:Required: No -:Default: ``false`` - - -``filestore fail eio`` - -:Description: Fail/Crash on eio. -:Type: Boolean -:Required: No -:Default: ``true`` - diff --git a/doc/config-cluster/general-config-ref.rst b/doc/config-cluster/general-config-ref.rst deleted file mode 100644 index a70b7f776b228..0000000000000 --- a/doc/config-cluster/general-config-ref.rst +++ /dev/null @@ -1,148 +0,0 @@ -========================== - General Config Reference -========================== - -``auth supported`` - -.. deprecated:: 0.51 - -:Description: Indicates the type of authentication used. Currently ``cephx`` only. If not specified, it defaults to ``none``. -:Type: String -:Required: No -:Default: ``none`` - - -``auth cluster required`` - -.. versionadded:: 0.51 - -:Description: Enables authentication for the cluster. Valid setting is ``cephx``. -:Type: String -:Required: No -:Default: ``none`` - - -``auth service required`` - -.. versionadded:: 0.51 - -:Description: Enables authentication for the service. Valid setting is ``cephx``. -:Type: String -:Required: No -:Default: ``none`` - - - -``auth client required`` - -.. versionadded:: 0.51 - -:Description: Enables authentication for the client. Valid setting is ``cephx``. -:Type: String -:Required: No -:Default: ``none`` - - -``keyring`` - -:Description: The path to the cluster's keyring file. -:Type: String -:Required: No -:Default: ``/etc/ceph/$cluster.$name.keyring,/etc/ceph/$cluster.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin`` - - -``fsid`` - -:Description: The filesystem ID. One per cluster. -:Type: UUID -:Required: No. -:Default: N/A. Generated if not specified. - - -``mon host`` - -:Description: A list of ``{hostname}:{port}`` entries that clients can use to connect to a Ceph monitor. If not set, Ceph searches ``[mon.*]`` sections. -:Type: String -:Required: No -:Default: N/A - - -``host`` - -:Description: The hostname. Use this setting for specific daemon instances (e.g., ``[osd.0]``). -:Type: String -:Required: Yes, for daemon instances. -:Default: ``localhost`` - -.. tip:: Do not use ``localhost``. To get your host name, execute ``hostname -s`` on your command line and use the name of your host (to the first period, not the fully-qualified domain name). -.. important: You should not specify any value for ``host`` when using a third party deployment system that retrieves the host name for you. - - -``public network`` - -:Description: The IP address and netmask of the public (front-side) network (e.g., ``10.20.30.40/24``). Set in ``[global]``. -:Type: ``{ip-address}/{netmask}`` -:Required: No -:Default: N/A - - -``public addr`` - -:Description: The IP address for the public (front-side) network. Set for each daemon. -:Type: IP Address -:Required: No -:Default: N/A - - -``cluster network`` - -:Description: The IP address and netmask of the cluster (back-side) network (e.g., ``10.20.30.41/24``). Set in ``[global]``. -:Type: ``{ip-address}/{netmask}`` -:Required: No -:Default: N/A - - -``cluster addr`` - -:Description: The IP address for the cluster (back-side) network. Set for each daemon. -:Type: Address -:Required: No -:Default: N/A - - -``admin socket`` - -:Description: The socket for executing administrative commands irrespective of whether Ceph monitors have established a quorum. -:Type: String -:Required: No -:Default: ``/var/run/ceph/$cluster-$name.asok`` - - -``pid file`` - -:Description: Each running Ceph daemon has a running process identifier (PID) file. -:Type: String -:Required: No -:Default: N/A. The default path is ``/var/run/$cluster/$name.pid``. The PID file is generated upon start-up. - - -``chdir`` - -:Description: The directory Ceph daemons change to once they are up and running. Default ``/`` directory recommended. -:Type: String -:Required: No -:Default: ``/`` - - -``max open files`` - -:Description: If set, when the Ceph service starts, Ceph sets the ``max open fds`` at the OS level (i.e., the max # of file descriptors). It helps prevents OSDs from running out of file descriptors. -:Type: 64-bit Integer -:Required: No -:Default: ``0`` - -``fatal signal handlers`` - -:Description: If set, we will install signal handlers for SEGV, ABRT, BUS, ILL, FPE, XCPU, XFSZ, SYS signals to generate a useful log message -:Type: Boolean -:Default: ``true`` diff --git a/doc/config-cluster/index.rst b/doc/config-cluster/index.rst deleted file mode 100644 index 36ca45786f536..0000000000000 --- a/doc/config-cluster/index.rst +++ /dev/null @@ -1,79 +0,0 @@ -=============== - Configuration -=============== - -Ceph can run with a cluster containing thousands of Object Storage Devices -(OSDs). A minimal system will have at least two OSDs for data replication. To -configure OSD clusters, you must provide settings in the configuration file. -Ceph provides default values for many settings, which you can override in the -configuration file. Additionally, you can make runtime modification to the -configuration using command-line utilities. - -When Ceph starts, it activates three daemons: - -- ``ceph-osd`` (mandatory) -- ``ceph-mon`` (mandatory) -- ``ceph-mds`` (mandatory for cephfs only) - -Each process, daemon or utility loads the host's configuration file. A process -may have information about more than one daemon instance (*i.e.,* multiple -contexts). A daemon or utility only has information about a single daemon -instance (a single context). - -.. note:: Ceph can run on a single host for evaluation purposes. - - -.. raw:: html - -

Cluster Configuration

- -For general cluster configuration, refer to the following: - -.. toctree:: - - Disks and Filesystems - ceph-conf - - -.. raw:: html - -

Configuration Reference

- -To optimize the performance of your cluster, refer to the following: - -.. toctree:: - - General Settings - Monitor Settings - OSD Settings - Filestore Settings - Journal Settings - Metadata Server Settings - librbd Cache Settings - Log / Debug Settings - - -.. raw:: html - -

Manual Deployment

- -To deploy a cluster manually (this is recommended for testing and development only), refer to the following: - -.. toctree:: - - Deploy with mkcephfs - - -.. raw:: html - -

Chef Deployment

- -To deploy a cluster with chef, refer to the following: - -.. toctree:: - - Deploy with Chef - -.. raw:: html - -
diff --git a/doc/config-cluster/journal-ref.rst b/doc/config-cluster/journal-ref.rst deleted file mode 100644 index f1906d12a505b..0000000000000 --- a/doc/config-cluster/journal-ref.rst +++ /dev/null @@ -1,95 +0,0 @@ -========================== - Journal Config Reference -========================== - -Ceph OSDs use a journal for two reasons: speed and consistency. - -- **Speed:** The journal enables the OSD to commit small writes quickly. - Ceph writes small, random i/o to the journal sequentially, which tends to - speed up bursty workloads by allowing the backing filesystem more time to - coalesce writes. The OSD journal, however, can lead to spiky performance - with short spurts of high-speed writes followed by periods without any - write progress as the filesystem catches up to the journal. - -- **Consistency:** Ceph OSDs requires a filesystem interface that guarantees - atomic compound operations. Ceph OSDs write a description of the operation - to the journal and apply the operation to the filesystem. This enables - atomic updates to an object (for example, placement group metadata). Every - few seconds--between ``filestore max sync interval`` and - ``filestore min sync interval``--the OSD stops writes and synchronizes the - journal with the filesystem, allowing OSDs to trim operations from the - journal and reuse the space. On failure, OSDs replay the journal starting - after the last synchronization operation. - -Ceph OSDs support the following journal settings: - -``journal dio`` - -:Description: Enables direct i/o to the journal. Requires ``journal block align`` set to ``true``. -:Type: Boolean -:Required: Yes when using ``aio``. -:Default: ``true`` - - -``journal aio`` - -:Description: Enables using ``libaio`` for asynchronous writes to the journal. Requires ``journal dio`` set to ``true``. -:Type: Boolean -:Required: No. -:Default: ``false`` - - -``journal block align`` - -:Description: Block aligns writes. Required for ``dio`` and ``aio``. -:Type: Boolean -:Required: Yes when using ``dio`` and ``aio``. -:Default: ``true`` - - -``journal max write bytes`` - -:Description: The maximum number of bytes the journal will write at any one time. -:Type: Integer -:Required: No -:Default: ``10 << 20`` - - -``journal max write entries`` - -:Description: The maximum number of entries the journal will write at any one time. -:Type: Integer -:Required: No -:Default: ``100`` - - -``journal queue max ops`` - -:Description: The maximum number of operations allowed in the queue at any one time. -:Type: Integer -:Required: No -:Default: ``500`` - - -``journal queue max bytes`` - -:Description: The maximum number of bytes allowed in the queue at any one time. -:Type: Integer -:Required: No -:Default: ``10 << 20`` - - -``journal align min size`` - -:Description: Align data payloads greater than the specified minimum. -:Type: Integer -:Required: No -:Default: ``64 << 10`` - - -``journal zero on create`` - -:Description: Causes the file store to overwrite the entire journal with ``0``'s during ``mkfs``. -:Type: Boolean -:Required: No -:Default: ``false`` diff --git a/doc/config-cluster/log-and-debug-ref.rst b/doc/config-cluster/log-and-debug-ref.rst deleted file mode 100644 index b0a99975cd1ed..0000000000000 --- a/doc/config-cluster/log-and-debug-ref.rst +++ /dev/null @@ -1,326 +0,0 @@ -======================================== - Logging and Debugging Config Reference -======================================== - -Logging and debugging settings are not required, but you may override default settings -as needed. Ceph supports the following settings: - -Logging -======= - -``log file`` - -:Desription: The location of the logging file for your cluster. -:Type: String -:Required: No -:Default: ``/var/log/ceph/$cluster-$name.log`` - - -``log max new`` - -:Description: The maximum number of new log files. -:Type: Integer -:Required: No -:Default: ``1000`` - - -``log max recent`` - -:Description: The maximum number of recent events to include in a log file. -:Type: Integer -:Required: No -:Default: ``1000000`` - - -``log to stderr`` - -:Description: Determines if logging messages should appear in ``stderr``. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``err to stderr`` - -:Description: Determines if error messages should appear in ``stderr``. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``log to syslog`` - -:Description: Determines if logging messages should appear in ``syslog``. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``err to syslog`` - -:Description: Determines if error messages should appear in ``syslog``. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``log flush on exit`` - -:Description: Determines if Ceph should flush the log files after exit. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``clog to monitors`` - -:Description: Determines if ``clog`` messages should be sent to monitors. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``clog to syslog`` - -:Description: Determines if ``clog`` messages should be sent to syslog. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mon cluster log to syslog`` - -:Description: Determines if the cluster log should be output to the syslog. -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mon cluster log file`` - -:Description: The location of the cluster's log file. -:Type: String -:Required: No -:Default: ``/var/log/ceph/$cluster.log`` - - - -OSD -=== - - -``osd debug drop ping probability`` - -:Description: ? -:Type: Double -:Required: No -:Default: 0 - - -``osd debug drop ping duration`` - -:Description: The duration ? -:Type: Integer -:Required: No -:Default: 0 - -``osd debug drop pg create probability`` - -:Description: -:Type: Integer -:Required: No -:Default: 0 - -``osd debug drop pg create duration`` - -:Description: ? -:Type: Double -:Required: No -:Default: 1 - -``osd preserve trimmed log`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - -``osd tmapput sets uses tmap`` - -:Description: For debug only. ??? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``osd min pg log entries`` - -:Description: The minimum number of log entries for placement groups. -:Type: 32-bit Unsigned Integer -:Required: No -:Default: 1000 - -``osd op log threshold`` - -:Description: How many op log messages to show up in one pass. -:Type: Integer -:Required: No -:Default: 5 - - - -Filestore -========= - -``filestore debug omap check`` - -:Description: Checks the ``omap``. This is an expensive operation. -:Type: Boolean -:Required: No -:Default: 0 - - -MDS -=== - - -``mds debug scatterstat`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mds debug frag`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mds debug auth pins`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``mds debug subtrees`` - -:Description: ? -:Type: Boolean -:Required: No -:Default: ``false`` - - - -RADOS Gateway -============= - - -``rgw log nonexistent bucket`` - -:Description: Should we log a non-existent buckets? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``rgw log object name`` - -:Description: Should an object's name be logged. // man date to see codes (a subset are supported) -:Type: String -:Required: No -:Default: ``%Y-%m-%d-%H-%i-%n`` - - -``rgw log object name utc`` - -:Description: Object log name contains UTC? -:Type: Boolean -:Required: No -:Default: ``false`` - - -``rgw enable ops log`` - -:Description: Enables logging of every RGW operation. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``rgw enable usage log`` - -:Description: Enable logging of RGW's bandwidth usage. -:Type: Boolean -:Required: No -:Default: ``true`` - - -``rgw usage log flush threshold`` - -:Description: Threshold to flush pending log data. -:Type: Integer -:Required: No -:Default: ``1024`` - - -``rgw usage log tick interval`` - -:Description: Flush pending log data every ``s`` seconds. -:Type: Integer -:Required: No -:Default: 30 - - -``rgw intent log object name`` - -:Description: -:Type: String -:Required: No -:Default: ``%Y-%m-%d-%i-%n`` - - -``rgw intent log object name utc`` - -:Description: Include a UTC timestamp in the intent log object name. -:Type: Boolean -:Required: No -:Default: ``false`` - -``rgw cluster root pool`` - -:Description: RADOS pool to store radosgw metadata for this instance -:Type: String -:Required: No -:Default: ``.rgw.root`` - -``rgw gc max objs`` - -:Description: Number of objects to collect garbage collection data -:Type: 32-bit Integer -:Default: 32 - -``rgw gc obj min wait`` - -:Description: Minimum time to wait before object's removal and its processing by the garbage collector -:Type: 32-bit Integer -:Default: 2 hours. ``2*60*60`` - -``rgw gc processor max time`` - -:Description: Max time for a single garbage collection process run -:Type: 32-bit Integer -:Default: 1 hour. ``60*60`` - -``rgw gc processor max period`` - -:Description: Max time between the beginning of two consecutive garbage collection processes run -:Type: 32-bit Integer -:Default: 1 hour. ``60*60`` - - diff --git a/doc/config-cluster/mds-config-ref.rst b/doc/config-cluster/mds-config-ref.rst deleted file mode 100644 index 8c1fc1e622306..0000000000000 --- a/doc/config-cluster/mds-config-ref.rst +++ /dev/null @@ -1,447 +0,0 @@ -====================== - MDS Config Reference -====================== - -``mds max file size`` - -:Description: -:Type: 64-bit Integer Unsigned -:Default: 1ULL << 40 - -``mds cache size`` - -:Description: -:Type: 32-bit Integer -:Default: 100000 - -``mds cache mid`` - -:Description: -:Type: Float -:Default: 0.7 - -``mds mem max`` - -:Description: // KB -:Type: 32-bit Integer -:Default: 1048576 - -``mds dir commit ratio`` - -:Description: -:Type: Float -:Default: 0.5 - -``mds dir max commit size`` - -:Description: // MB -:Type: 32-bit Integer -:Default: 90 - -``mds decay halflife`` - -:Description: -:Type: Float -:Default: 5 - -``mds beacon interval`` - -:Description: -:Type: Float -:Default: 4 - -``mds beacon grace`` - -:Description: -:Type: Float -:Default: 15 - -``mds blacklist interval`` - -:Description: // how long to blacklist failed nodes -:Type: Float -:Default: 24.0*60.0 - -``mds session timeout`` - -:Description: // cap bits and leases time out if client idle -:Type: Float -:Default: 60 - -``mds session autoclose`` - -:Description: // autoclose idle session -:Type: Float -:Default: 300 - -``mds reconnect timeout`` - -:Description: // secs to wait for clients during mds restart -:Type: Float -:Default: 45 - -``mds tick interval`` - -:Description: -:Type: Float -:Default: 5 - -``mds dirstat min interval`` - -:Description: //try to avoid propagating more often than x -:Type: Float -:Default: 1 - -``mds scatter nudge interval`` - -:Description: // how quickly dirstat changes propagate up -:Type: Float -:Default: 5 - -``mds client prealloc inos`` - -:Description: -:Type: 32-bit Integer -:Default: 1000 - -``mds early reply`` - -:Description: -:Type: Boolean -:Default: true - -``mds use tmap`` - -:Description: // use trivialmap for dir updates -:Type: Boolean -:Default: true - -``mds default dir hash`` - -:Description: CEPH STR HASH RJENKINS -:Type: 32-bit Integer -:Default: - -``mds log`` - -:Description: -:Type: Boolean -:Default: true - -``mds log skip corrupt events`` - -:Description: -:Type: Boolean -:Default: false - -``mds log max events`` - -:Description: -:Type: 32-bit Integer -:Default: -1 - -``mds log max segments`` - -:Description: // segment size defined by FileLayout above -:Type: 32-bit Integer -:Default: 30 - -``mds log max expiring`` - -:Description: -:Type: 32-bit Integer -:Default: 20 - -``mds log eopen size`` - -:Description: // # open inodes per log entry -:Type: 32-bit Integer -:Default: 100 - -``mds bal sample interval`` - -:Description: // every 5 seconds -:Type: Float -:Default: 3 - -``mds bal replicate threshold`` - -:Description: -:Type: Float -:Default: 8000 - -``mds bal unreplicate threshold`` - -:Description: -:Type: Float -:Default: 0 - -``mds bal frag`` - -:Description: -:Type: Boolean -:Default: false - -``mds bal split size`` - -:Description: -:Type: 32-bit Integer -:Default: 10000 - -``mds bal split rd`` - -:Description: -:Type: Float -:Default: 25000 - -``mds bal split wr`` - -:Description: -:Type: Float -:Default: 10000 - -``mds bal split bits`` - -:Description: -:Type: 32-bit Integer -:Default: 3 - -``mds bal merge size`` - -:Description: -:Type: 32-bit Integer -:Default: 50 - -``mds bal merge rd`` - -:Description: -:Type: Float -:Default: 1000 - -``mds bal merge wr`` - -:Description: -:Type: Float -:Default: 1000 - -``mds bal interval`` - -:Description: // seconds -:Type: 32-bit Integer -:Default: 10 - -``mds bal fragment interval`` - -:Description: // seconds -:Type: 32-bit Integer -:Default: 5 - -``mds bal idle threshold`` - -:Description: -:Type: Float -:Default: 0 - -``mds bal max`` - -:Description: -:Type: 32-bit Integer -:Default: -1 - -``mds bal max until`` - -:Description: -:Type: 32-bit Integer -:Default: -1 - -``mds bal mode`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds bal min rebalance`` - -:Description: // must be x above avg before we export -:Type: Float -:Default: 0.1 - -``mds bal min start`` - -:Description: // if we need less x. we don't do anything -:Type: Float -:Default: 0.2 - -``mds bal need min`` - -:Description: // take within this range of what we need -:Type: Float -:Default: 0.8 - -``mds bal need max`` - -:Description: -:Type: Float -:Default: 1.2 - -``mds bal midchunk`` - -:Description: // any sub bigger than this taken in full -:Type: Float -:Default: 0.3 - -``mds bal minchunk`` - -:Description: // never take anything smaller than this -:Type: Float -:Default: 0.001 - -``mds bal target removal min`` - -:Description: // min bal iters before old target is removed -:Type: 32-bit Integer -:Default: 5 - -``mds bal target removal max`` - -:Description: // max bal iters before old target is removed -:Type: 32-bit Integer -:Default: 10 - -``mds replay interval`` - -:Description: // time to wait before starting replay again -:Type: Float -:Default: 1 - -``mds shutdown check`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds thrash exports`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds thrash fragments`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds dump cache on map`` - -:Description: -:Type: Boolean -:Default: false - -``mds dump cache after rejoin`` - -:Description: -:Type: Boolean -:Default: false - -``mds verify scatter`` - -:Description: -:Type: Boolean -:Default: false - -``mds debug scatterstat`` - -:Description: -:Type: Boolean -:Default: false - -``mds debug frag`` - -:Description: -:Type: Boolean -:Default: false - -``mds debug auth pins`` - -:Description: -:Type: Boolean -:Default: false - -``mds debug subtrees`` - -:Description: -:Type: Boolean -:Default: false - -``mds kill mdstable at`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds kill export at`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds kill import at`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds kill link at`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds kill rename at`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``mds wipe sessions`` - -:Description: -:Type: Boolean -:Default: 0 - -``mds wipe ino prealloc`` - -:Description: -:Type: Boolean -:Default: 0 - -``mds skip ino`` - -:Description: -:Type: 32-bit Integer -:Default: 0 - -``max mds`` - -:Description: -:Type: 32-bit Integer -:Default: 1 - -``mds standby for name`` - -:Description: -:Type: String -:Default: - -``mds standby for rank`` - -:Description: -:Type: 32-bit Integer -:Default: -1 - -``mds standby replay`` - -:Description: -:Type: Boolean -:Default: false \ No newline at end of file diff --git a/doc/config-cluster/mkcephfs.rst b/doc/config-cluster/mkcephfs.rst deleted file mode 100644 index 721ae43500ca1..0000000000000 --- a/doc/config-cluster/mkcephfs.rst +++ /dev/null @@ -1,123 +0,0 @@ -============================= - Deploying with ``mkcephfs`` -============================= - -To deploy a test or development cluster, you can use the ``mkcephfs`` tool. -We do not recommend using this tool for production environments. - - -Enable Login to Cluster Hosts as ``root`` -========================================= - -To deploy with ``mkcephfs``, you will need to be able to login as ``root`` -on each host without a password. For each host, perform the following:: - - sudo passwd root - -Enter a password for the root user. - -On the admin host, generate an ``ssh`` key without specifying a passphrase -and use the default locations. :: - - ssh-keygen - Generating public/private key pair. - Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /ceph-admin/.ssh/id_rsa. - Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub. - -You may use RSA or DSA keys. Once you generate your keys, copy them to each -OSD host. For example:: - - ssh-copy-id root@myserver01 - ssh-copy-id root@myserver02 - -Modify your ``~/.ssh/config`` file to login as ``root``, as follows:: - - Host myserver01 - Hostname myserver01.fully-qualified-domain.com - User root - Host myserver02 - Hostname myserver02.fully-qualified-domain.com - User root - - -Copy Configuration File to All Hosts -==================================== - -Ceph's ``mkcephfs`` deployment script does not copy the configuration file you -created from the Administration host to the OSD Cluster hosts. Copy the -configuration file you created (*i.e.,* ``mycluster.conf`` in the example below) -from the Administration host to ``etc/ceph/ceph.conf`` on each OSD Cluster host -if you are using ``mkcephfs`` to deploy Ceph. - -:: - - ssh myserver01 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf - ssh myserver02 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf - ssh myserver03 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf - - -Create the Default Directories -============================== - -The ``mkcephfs`` deployment script does not create the default server -directories. Create server directories for each instance of a Ceph daemon (if -you haven't done so already). The ``host`` variables in the ``ceph.conf`` file -determine which host runs each instance of a Ceph daemon. Using the exemplary -``ceph.conf`` file, you would perform the following: - -On ``myserver01``:: - - sudo mkdir /var/lib/ceph/osd/ceph-0 - sudo mkdir /var/lib/ceph/mon/ceph-a - -On ``myserver02``:: - - sudo mkdir /var/lib/ceph/osd/ceph-1 - sudo mkdir /var/lib/ceph/mon/ceph-b - -On ``myserver03``:: - - sudo mkdir /var/lib/ceph/osd/ceph-2 - sudo mkdir /var/lib/ceph/mon/ceph-c - sudo mkdir /var/lib/ceph/mds/ceph-a - - -Mount Disks to the Data Directories -=================================== - -If you are running multiple OSDs per host and one hard disk per OSD, you should -mount the disk under the OSD data directory (if you haven't done so already). - - -Run ``mkcephfs`` -================ - -Once you have copied your Ceph Configuration to the OSD Cluster hosts -and created the default directories, you may deploy Ceph with the -``mkcephfs`` script. - -.. note:: ``mkcephfs`` is a quick bootstrapping tool. It does not handle more - complex operations, such as upgrades. - -For production environments, deploy Ceph using Chef cookbooks. To run -``mkcephfs``, execute the following:: - - cd /etc/ceph - sudo mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring - -The script adds an admin key to the ``ceph.keyring``, which is analogous to a -root password. See `Authentication`_ when running with ``cephx`` enabled. - -When you start or stop your cluster, you will not have to use ``sudo`` or -provide passwords. For example:: - - service ceph -a start - -See `Operating a Cluster`_ for details. - - -.. _Authentication: ../authentication -.. _Operating a Cluster: ../../init/ \ No newline at end of file diff --git a/doc/config-cluster/mon-config-ref.rst b/doc/config-cluster/mon-config-ref.rst deleted file mode 100644 index b8c8c07e03c3c..0000000000000 --- a/doc/config-cluster/mon-config-ref.rst +++ /dev/null @@ -1,240 +0,0 @@ -========================== - Monitor Config Reference -========================== - -``mon data`` - -:Description: The monitor's data location. -:Type: String -:Default: ``/var/lib/ceph/mon/$cluster-$id`` - - -``mon initial members`` - -:Description: The IDs of initial monitors in a cluster during startup. If specified, Ceph requires an odd number of monitors to form an initial quorum. -:Type: String -:Default: None - - -``mon sync fs threshold`` - -:Description: Synchronize with the filesystem when writing the specified number of objects. Set it to ``0`` to disable it. -:Type: 32-bit Integer -:Default: ``5`` - - -``mon tick interval`` - -:Description: A monitor's tick interval in seconds. -:Type: 32-bit Integer -:Default: ``5`` - - -``mon subscribe interval`` - -:Description: The refresh interval (in seconds) for subscriptions. The subscription mechanism enables obtaining the cluster maps and log informations. -:Type: Double -:Default: ``300`` - - -``mon osd auto mark in`` - -:Description: Ceph will mark any booting OSDs as ``in`` the cluster. -:Type: Boolean -:Default: ``false`` - - -``mon osd auto mark auto out in`` - -:Description: Ceph will mark booting OSDs auto marked ``out`` of the cluster as ``in`` the cluster. -:Type: Boolean -:Default: ``true`` - - -``mon osd auto mark new in`` - -:Description: Ceph will mark booting new OSDs as ``in`` the cluster. -:Type: Boolean -:Default: ``true`` - - -``mon osd down out interval`` - -:Description: The number of seconds Ceph waits before marking an OSD ``down`` and ``out`` if it doesn't respond. -:Type: 32-bit Integer -:Default: ``300`` - - -``mon osd min up ratio`` - -:Description: The minimum ratio of ``up`` OSDs before Ceph will mark OSDs ``down``. -:Type: Double -:Default: ``.3`` - - -``mon osd min in ratio`` - -:Description: The minimum ratio of ``in`` OSDs before Ceph will mark OSDs ``out``. -:Type: Double -:Default: ``.3`` - - -``mon lease`` - -:Description: Length (in seconds) of the lease on the monitor's versions. -:Type: Float -:Default: ``5`` - - -``mon lease renew interval`` - -:Description: The interval (in seconds) for the Leader to renew the other monitor's leases. -:Type: Float -:Default: ``3`` - - -``mon lease ack timeout`` - -:Description: Number of seconds the Leader will wait for the Peons to acknowledge the lease extension. -:Type: Float -:Default: ``10.0`` - - -``mon clock drift allowed`` - -:Description: The clock drift in seconds allowed between monitors. -:Type: Float -:Default: ``.050`` - - -``mon clock drift warn backoff`` - -:Description: Exponential backoff for clock drift warnings -:Type: Float -:Default: ``5`` - - -``mon accept timeout`` - -:Description: Number of seconds the Leader will wait for the Peons to accept a Paxos update. It is also used during the Paxos recovery phase for similar purposes. -:Type: Float -:Default: ``10.0`` - - -``mon pg create interval`` - -:Description: Number of seconds between PG creation in the same OSD. -:Type: Float -:Default: ``30.0`` - - -``mon pg stuck threshold`` - -:Description: Number of seconds after which PGs can be considered as being stuck. -:Type: 32-bit Integer -:Default: ``300`` - - -``mon osd full ratio`` - -:Description: The percentage of disk space used before an OSD is considered ``full``. -:Type: Float -:Default: ``.95`` - - -``mon osd nearfull ratio`` - -:Description: The percentage of disk space used before an OSD is considered ``nearfull``. -:Type: Float -:Default: ``.85`` - - -``mon globalid prealloc`` - -:Description: The number of global IDs to pre-allocate for the cluster. -:Type: 32-bit Integer -:Default: ``100`` - - -``mon osd report timeout`` - -:Description: The grace period in seconds before declaring unresponsive OSDs ``down``. -:Type: 32-bit Integer -:Default: ``900`` - - -``mon force standby active`` - -:Description: should mons force standby-replay mds to be active -:Type: Boolean -:Default: true - - -``mon min osdmap epochs`` - -:Description: Minimum number of OSD map epochs to keep at all times. -:Type: 32-bit Integer -:Default: ``500`` - - -``mon max pgmap epochs`` - -:Description: Maximum number of PG map epochs the monitor should keep. -:Type: 32-bit Integer -:Default: ``500`` - - -``mon max log epochs`` - -:Description: Maximum number of Log epochs the monitor should keep. -:Type: 32-bit Integer -:Default: ``500`` - - -``mon max osd`` - -:Description: The maximum number of OSDs allowed in the cluster. -:Type: 32-bit Integer -:Default: ``10000`` - - -``mon probe timeout`` - -:Description: Number of seconds the monitor will wait to find peers before bootstrapping. -:Type: Double -:Default: ``2.0`` - - -``mon slurp timeout`` - -:Description: Number of seconds the monitor has to recover using slurp before the process is aborted and the monitor bootstraps. -:Type: Double -:Default: ``10.0`` - - -``mon slurp bytes`` - -:Description: Limits the slurp messages to the specified number of bytes. -:Type: 32-bit Integer -:Default: ``256 * 1024`` - - -``mon client bytes`` - -:Description: The amount of client message data allowed in memory (in bytes). -:Type: 64-bit Integer Unsigned -:Default: ``100ul << 20`` - - -``mon daemon bytes`` - -:Description: The message memory cap for metadata server and OSD messages (in bytes). -:Type: 64-bit Integer Unsigned -:Default: ``400ul << 20`` - - -``mon max log entries per event`` - -:Description: The maximum number of log entries per event. -:Type: Integer -:Default: ``4096`` diff --git a/doc/config-cluster/ms-ref.rst b/doc/config-cluster/ms-ref.rst deleted file mode 100644 index d76f7f3bb2587..0000000000000 --- a/doc/config-cluster/ms-ref.rst +++ /dev/null @@ -1,83 +0,0 @@ -=========== - Messaging -=========== - - -``ms tcp nodelay`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``true`` - - -``ms initial backoff`` - -:Description: -:Type: Double -:Required: No -:Default: ``.2`` - - -``ms max backoff`` - -:Description: -:Type: Double -:Required: No -:Default: ``15.0`` - - -``ms nocrc`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -``ms die on bad msg`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -``ms dispatch throttle bytes`` - -:Description: -:Type: 64-bit Unsigned Integer -:Required: No -:Default: ``100 << 20`` - - -``ms bind ipv6`` - -:Description: -:Type: Boolean -:Required: No -:Default: ``false`` - - -``ms rwthread stack bytes`` - -:Description: -:Type: 64-bit Unsigned Integer -:Required: No -:Default: ``1024 << 10`` - - -``ms tcp read timeout`` - -:Description: -:Type: 64-bit Unsigned Integer -:Required: No -:Default: ``900`` - - -``ms inject socket failures`` - -:Description: -:Type: 64-bit Unsigned Integer -:Required: No -:Default: ``0`` diff --git a/doc/config-cluster/osd-config-ref.rst b/doc/config-cluster/osd-config-ref.rst deleted file mode 100644 index 05a3369cde148..0000000000000 --- a/doc/config-cluster/osd-config-ref.rst +++ /dev/null @@ -1,422 +0,0 @@ -====================== - OSD Config Reference -====================== - - - -``osd uuid`` - -:Description: The universally unique identifier (UUID) for the OSD. -:Type: UUID -:Default: None - - -``osd data`` - -:Description: The path to the OSDs data. You must create the directory. You should mount a data disk at this mount point. We do not recommend changing the default. -:Type: String -:Default: ``/var/lib/ceph/osd/$cluster-$id`` - - -``osd journal`` - -:Description: The path to the OSD's journal. This may be a path to a file or a block device (such as a partition of an SSD). If it is a file, you must create the directory to contain it. -:Type: String -:Default: ``/var/lib/ceph/osd/$cluster-$id/journal`` - - -``osd journal size`` - -:Description: The size of the journal in megabytes. If this is 0, and the journal is a block device, the entire block device is used. Since v0.54, this is ignored if the journal is a block device, and the entire block device is used. -:Type: 32-bit Integer -:Default: ``1024`` -:Recommended: Begin with 1GB. Should at least twice the product of the expected speed multiplied by ``filestore min sync interval``. - - -``osd max write size`` - -:Description: The maximum size of a write in megabytes. -:Type: 32-bit Integer -:Default: ``90`` - - -``osd client message size cap`` - -:Description: The largest client data message allowed in memory. -:Type: 64-bit Integer Unsigned -:Default: 500MB default. ``500*1024L*1024L`` - - -``osd stat refresh interval`` - -:Description: The status refresh interval in seconds. -:Type: 64-bit Integer Unsigned -:Default: ``.5`` - - -``osd pg bits`` - -:Description: Placement group bits per OSD. -:Type: 32-bit Integer -:Default: ``6`` - - -``osd pgp bits`` - -:Description: The number of bits per OSD for PGPs. -:Type: 32-bit Integer -:Default: ``4`` - - -``osd pg layout`` - -:Description: Placement group layout. -:Type: 32-bit Integer -:Default: ``2`` - - -``osd pool default crush rule`` - -:Description: The default CRUSH rule to use when creating a pool. -:Type: 32-bit Integer -:Default: ``0`` - - -``osd pool default size`` - -:Description: The default size of an OSD pool in gigabytes. The default value is the same as ``--size 2`` with ``mkpool``. -:Type: 32-bit Integer -:Default: ``2`` - - -``osd pool default pg num`` - -:Description: The default number of placement groups for a pool. The default value is the same as ``pg_num`` with ``mkpool``. -:Type: 32-bit Integer -:Default: ``8`` - - -``osd pool default pgp num`` - -:Description: The default number of placement groups for placement for a pool. The default value is the same as ``pgp_num`` with ``mkpool``. PG and PGP should be equal (for now). -:Type: 32-bit Integer -:Default: ``8`` - - -``osd map dedup`` - -:Description: Enable removing duplicates in the OSD map. -:Type: Boolean -:Default: ``true`` - - -``osd map cache size`` - -:Description: The size of the OSD map cache in megabytes. -:Type: 32-bit Integer -:Default: ``500`` - - -``osd map cache bl size`` - -:Description: The size of the in-memory OSD map cache in OSD daemons. -:Type: 32-bit Integer -:Default: ``50`` - - -``osd map cache bl inc size`` - -:Description: The size of the in-memory OSD map cache incrementals in OSD daemons. -:Type: 32-bit Integer -:Default: ``100`` - - -``osd map message max`` - -:Description: The maximum map entries allowed per MOSDMap message. -:Type: 32-bit Integer -:Default: ``100`` - - -``osd op threads`` - -:Description: The number of OSD operation threads. Set to ``0`` to disable it. Increasing the number may increase the request processing rate. -:Type: 32-bit Integer -:Default: ``2`` - - -``osd op thread timeout`` - -:Description: The OSD operation thread timeout in seconds. -:Type: 32-bit Integer -:Default: ``30`` - - -``osd disk threads`` - -:Description: The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming. -:Type: 32-bit Integer -:Default: ``1`` - - -``osd recovery threads`` - -:Description: The number of threads for recovering data. -:Type: 32-bit Integer -:Default: ``1`` - - -``osd recover clone overlap`` - -:Description: Preserves clone overlap during recovery and data migration. -:Type: Boolean -:Default: ``false`` - - -``osd backfill scan min`` - -:Description: The scan interval in seconds for backfill operations. -:Type: 32-bit Integer -:Default: ``64`` - - -``osd backfill scan max`` - -:Description: The maximum scan interval in seconds for backfill operations. -:Type: 32-bit Integer -:Default: ``512`` - - -``osd backlog thread timeout`` - -:Description: The maximum time in seconds before timing out a backlog thread. -:Type: 32-bit Integer -:Default: ``60*60*1`` - - -``osd recovery thread timeout`` - -:Description: The maximum time in seconds before timing out a recovery thread. -:Type: 32-bit Integer -:Default: ``30`` - - -``osd snap trim thread timeout`` - -:Description: The maximum time in seconds before timing out a snap trim thread. -:Type: 32-bit Integer -:Default: ``60*60*1`` - - -``osd scrub thread timeout`` - -:Description: The maximum time in seconds before timing out a scrub thread. -:Type: 32-bit Integer -:Default: ``60`` - - -``osd scrub finalize thread timeout`` - -:Description: The maximum time in seconds before timing out a scrub finalize thread. -:Type: 32-bit Integer -:Default: 60*10 - - -``osd remove thread timeout`` - -:Description: The maximum time in seconds before timing out a remove OSD thread. -:Type: 32-bit Integer -:Default: 60*60 - - -``osd command thread timeout`` - -:Description: The maximum time in seconds before timing out a command thread. -:Type: 32-bit Integer -:Default: ``10*60`` - - -``osd heartbeat address`` - -:Description: An OSD's network address for heartbeats. -:Type: Address -:Default: The host address. - - -``osd heartbeat interval`` - -:Description: How often an OSD pings its peers (in seconds). -:Type: 32-bit Integer -:Default: ``6`` - - -``osd heartbeat grace`` - -:Description: The elapsed time when an OSD hasn't shown a heartbeat that the cluster considers it ``down``. -:Type: 32-bit Integer -:Default: ``20`` - - -``osd _mon_heartbeat interval`` - -:Description: How often the OSD pings a monitor if it has no OSD peers. -:Type: 32-bit Integer -:Default: ``30`` - - -``osd mon report interval max`` - -:Description: The maximum time in seconds for an OSD to report to a monitor before the monitor considers the OSD ``down``. -:Type: 32-bit Integer -:Default: ``120`` - - -``osd mon report interval min`` - -:Description: The number of minutes between reports that include ``pg stats``, ``up thru``, ``boot`` and ``failures``. -:Type: 32-bit Integer -:Default: ``5`` - - -``osd mon ack timeout`` - -:Description: The number of seconds to wait for a monitor to acknowledge a request for statistics. -:Type: 32-bit Integer -:Default: ``30`` - - -``osd min down reporters`` - -:Description: The minimum number of OSDs required to report a ``down`` OSD. -:Type: 32-bit Integer -:Default: ``1`` - - -``osd min down reports`` - -:Description: The minimum number of times an OSD must report that another is ``down``. -:Type: 32-bit Integer -:Default: ``3`` - - -``osd recovery delay start`` - -:Description: After peering completes, Ceph will delay for the specified number of seconds before starting to recover objects. -:Type: Float -:Default: ``15`` - - -``osd recovery max active`` - -:Description: The number of active recovery requests per OSD at one time. More accelerates recovery, but places an increased load on the cluster. -:Type: 32-bit Integer -:Default: ``5`` - - -``osd recovery max chunk`` - -:Description: The maximum size of a recovered chunk of data to push. -:Type: 64-bit Integer Unsigned -:Default: ``1 << 20`` - - -``osd max scrubs`` - -:Description: The maximum number of scrub operations for an OSD. -:Type: 32-bit Int -:Default: ``1`` - - -``osd scrub load threshold`` - -:Description: The maximum CPU load. Ceph will not scrub when the CPU load is higher than this number. Default is 50%. -:Type: Float -:Default: ``0.5`` - - -``osd scrub min interval`` - -:Description: The maximum interval in seconds for scrubbing the OSD. -:Type: Float -:Default: 5 minutes. ``300`` - - -``osd scrub max interval`` - -:Description: The maximum interval in seconds for scrubbing the OSD. -:Type: Float -:Default: Once per day. ``60*60*24`` - -``osd deep scrub interval`` - -:Description: The interval for "deep" scrubbing (fully reading all data) -:Type: Float -:Default: Once per week. ``60*60*24*7`` - -``osd deep scrub stride`` - -:Description: Read siez when doing a deep scrub -:Type: 32-bit Int -:Default: 512 KB. ``524288`` - -``osd class dir`` - -:Description: The class path for RADOS class plug-ins. -:Type: String -:Default: ``$libdir/rados-classes`` - - -``osd check for log corruption`` - -:Description: Check log files for corruption. Can be computationally expensive. -:Type: Boolean -:Default: ``false`` - - -``osd default notify timeout`` - -:Description: The OSD default notification timeout (in seconds). -:Type: 32-bit Integer Unsigned -:Default: ``30`` - - -``osd min pg log entries`` - -:Description: The minimum number of placement group logs to maintain when trimming log files. -:Type: 32-bit Int Unsigned -:Default: 1000 - - -``osd op complaint time`` - -:Description: An operation becomes complaint worthy after the specified number of seconds have elapsed. -:Type: Float -:Default: ``30`` - - -``osd command max records`` - -:Description: Limits the number of lost objects to return. -:Type: 32-bit Integer -:Default: ``256`` - - -``osd auto upgrade tmap`` - -:Description: Uses ``tmap`` for ``omap`` on old objects. -:Type: Boolean -:Default: ``true`` - - -``osd tmapput sets users tmap`` - -:Description: Uses ``tmap`` for debugging only. -:Type: Boolean -:Default: ``false`` - - -``osd kill backfill at`` - -:Description: For debugging only. -:Type: 32-bit Integer -:Default: ``0`` diff --git a/doc/config-cluster/rbd-config-ref.rst b/doc/config-cluster/rbd-config-ref.rst deleted file mode 100644 index dde3dedf257a3..0000000000000 --- a/doc/config-cluster/rbd-config-ref.rst +++ /dev/null @@ -1,89 +0,0 @@ -======================= - librbd Cache Settings -======================= - -See `Block Device`_ for additional details. - -.. sidebar:: Kernel Caching - - The kernel driver for Ceph block devices can use the Linux page cache to - improve performance. - -The user space implementation of the Ceph block device (i.e., ``librbd``) cannot -take advantage of the Linux page cache, so it includes its own in-memory -caching, called "RBD caching." RBD caching behaves just like well-behaved hard -disk caching. When the OS sends a barrier or a flush request, all dirty data is -written to the OSDs. This means that using write-back caching is just as safe as -using a well-behaved physical hard disk with a VM that properly sends flushes -(i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU) -algorithm, and in write-back mode it can coalesce contiguous requests for -better throughput. - -.. versionadded:: 0.46 - -Ceph supports write-back caching for RBD. To enable it, add ``rbd cache = -true`` to the ``[client]`` section of your ``ceph.conf`` file. By default -``librbd`` does not perform any caching. Writes and reads go directly to the -storage cluster, and writes return only when the data is on disk on all -replicas. With caching enabled, writes return immediately, unless there are more -than ``rbd cache max dirty`` unflushed bytes. In this case, the write triggers -writeback and blocks until enough bytes are flushed. - -.. versionadded:: 0.47 - -Ceph supports write-through caching for RBD. You can set the size of -the cache, and you can set targets and limits to switch from -write-back caching to write through caching. To enable write-through -mode, set ``rbd cache max dirty`` to 0. This means writes return only -when the data is on disk on all replicas, but reads may come from the -cache. The cache is in memory on the client, and each RBD image has -its own. Since the cache is local to the client, there's no coherency -if there are others accesing the image. Running GFS or OCFS on top of -RBD will not work with caching enabled. - -The ``ceph.conf`` file settings for RBD should be set in the ``[client]`` -section of your configuration file. The settings include: - - -``rbd cache`` - -:Description: Enable caching for RADOS Block Device (RBD). -:Type: Boolean -:Required: No -:Default: ``false`` - - -``rbd cache size`` - -:Description: The RBD cache size in bytes. -:Type: 64-bit Integer -:Required: No -:Default: ``32 MiB`` - - -``rbd cache max dirty`` - -:Description: The ``dirty`` limit in bytes at which the cache triggers write-back. If ``0``, uses write-through caching. -:Type: 64-bit Integer -:Required: No -:Constraint: Must be less than ``rbd cache size``. -:Default: ``24 MiB`` - - -``rbd cache target dirty`` - -:Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache. -:Type: 64-bit Integer -:Required: No -:Constraint: Must be less than ``rbd cache max dirty``. -:Default: ``16 MiB`` - - -``rbd cache max dirty age`` - -:Description: The number of seconds dirty data is in the cache before writeback starts. -:Type: Float -:Required: No -:Default: ``1.0`` - -.. _Block Device: ../../rbd/rbd/ \ No newline at end of file diff --git a/doc/dev/documenting.rst b/doc/dev/documenting.rst index 2c67273f97357..cc091705f8661 100644 --- a/doc/dev/documenting.rst +++ b/doc/dev/documenting.rst @@ -27,7 +27,7 @@ The general format for function documentation is:: This should be in the header where the function is declared, and functions should be grouped into logical categories. The `librados C API`_ provides a complete example. It is pulled into Sphinx by -`librados.rst`_, which is rendered at :doc:`/api/librados`. +`librados.rst`_, which is rendered at :doc:`/rados/api/librados`. .. _`librados C API`: https://github.com/ceph/ceph/blob/master/src/include/rados/librados.h .. _`librados.rst`: https://raw.github.com/ceph/ceph/master/doc/api/librados.rst diff --git a/doc/dev/release-process2.rst b/doc/dev/release-process2.rst new file mode 100644 index 0000000000000..f7c3b6db8dec1 --- /dev/null +++ b/doc/dev/release-process2.rst @@ -0,0 +1,154 @@ +================= + Release Process +================= + +Build environment +================= + +Ceph maintains multiple build environments. + +Debian +------ + +We build Debian-based packages via ``pbuilder`` for multiple distributions, which identifies the build hosts in the ``deb_hosts`` file and the list of distributions in the ``deb_dist`` file. Each build host will build all distributions. Currently Ceph maintains 1 64-bit build host and 1 32-bit build host. + +RPM +--- + +We build RPM-based packages natively. Therefore, we maintain one build host per distribution. We build RPM-based packages via ``pbuilder``, which identifies the build hosts in the ``rpm_hosts`` file. + +Prior to building, it's necessary to update the ``pbuilder`` seed tarballs:: + + ./update_all_pbuilders.sh + +2. Setup keyring for signing packages +===================================== + +:: + + export GNUPGHOME= + + # verify it's accessible + gpg --list-keys + +The release key should be present:: + + pub 4096R/17ED316D 2012-05-20 + uid Ceph Release Key + + +3. Set up build area +==================== + +Checkout ceph and ceph-build:: + + git clone http://github.com/ceph/ceph.git + git clone http://github.com/ceph/ceph-build.git + +Checkout next branch:: + + git checkout next + +Checkout the submodules (only needed to prevent errors in recursive make):: + + git submodule update --init + +4. Update Build version numbers +================================ + +Edit configure.ac and change version number:: + + DEBEMAIL user@host dch -v 0.xx-1 + +Commit the changes:: + + git commit -a + +Tag the release:: + + ../ceph-build/tag-release v0.xx + +5. Create Makefiles +=================== + +The actual configure options used to build packages are in the +``ceph.spec.in`` and ``debian/rules`` files. At this point we just +need to create a Makefile.:: + + ./do_autogen.sh + + +6. Run the release scripts +========================== + +This creates tarballs and copies them, with other needed files to +the build hosts listed in deb_hosts and rpm_hosts, runs a local build +script, then rsyncs the results back tot the specified release directory.:: + + ../ceph-build/do_release.sh /tmp/release + +7. Create RPM Repo +================== + +Copy the rpms to the destination repo, creates the yum repository +rpm and indexes.:: + + ../ceph-build/push_to_rpm_repo.sh /tmp/release /tmp/repo 0.xx + +8. Create debian repo +===================== + +:: + + mkdir /tmp/debian-repo + ../ceph-build/gen_reprepro_conf.sh debian-testing + ../ceph-build/push_to_deb_repo.sh /tmp/release /tmp/debian-repo 0.xx main + +9. Push repos to ceph.org +========================== + +For a development release:: + + rcp ceph-0.xx.tar.bz2 ceph-0.xx.tar.gz \ + ceph_site@ceph.com:ceph.com/downloads/. + rsync -av /tmp/repo/0.52/ ceph_site@ceph.com:ceph.com/rpm-testing + rsync -av /tmp/debian-repo/ ceph_site@ceph.com:ceph.com/debian-testing + +For a stable release, replace {CODENAME} with the release codename (e.g., ``argonaut`` or ``bobtail``):: + + rcp ceph-0.xx.tar.bz2 \ + ceph_site@ceph.com:ceph.com/downloads/ceph-0.xx{CODENAME}.tar.bz2 + rcp ceph-0.xx.tar.gz \ + ceph_site@ceph.com:ceph.com/downloads/ceph-0.xx{CODENAME}.tar.gz + rsync -av /tmp/repo/0.52/ ceph_site@ceph.com:ceph.com/rpm-{CODENAME} + rsync -auv /tmp/debian-repo/ ceph_site@ceph.com:ceph.com/debian-{CODENAME} + +10. Update Git +============== + +Development release +------------------- + +For a development release, update tags for ``ceph.git``:: + + git push origin v0.xx + git push origin HEAD:testing + git checkout master + git merge next + git push origin master + git push origin HEAD:next + +Similarly, for a development release, for both ``teuthology.git`` and ``ceph-qa-suite.git``:: + + git checkout master + git reset --hard origin/master + git branch -f testing origin/next + git push -f origin testing + git push -f master:next + +Stable release +-------------- + +For ``ceph.git``: + + git push origin stable diff --git a/doc/index.rst b/doc/index.rst index 61aff6a190464..60a613ba14275 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -38,14 +38,11 @@ cluster to ensure that the storage hosts are running smoothly. start/index install/index - config-cluster/index - cluster-ops/index + rados/index cephfs/index rbd/rbd radosgw/index api/index - Internals - man/index architecture - faq + Development release-notes diff --git a/doc/install/chef.rst b/doc/install/chef.rst deleted file mode 100644 index 524fc31b37cc3..0000000000000 --- a/doc/install/chef.rst +++ /dev/null @@ -1,288 +0,0 @@ -================= - Installing Chef -================= - -Chef defines three types of entities: - -#. **Chef Nodes:** Run ``chef-client``, which installs and manages software. -#. **Chef Server:** Interacts with ``chef-client`` on Chef nodes. -#. **Chef Workstation:** Manages the Chef server. - -.. image:: ../images/chef.png - -See `Chef Architecture Introduction`_ for details. - -.. _createuser: - -Create a ``chef`` User -====================== - -The ``chef-client`` command requires the proper privileges to install and manage -installations. On each Chef node, we recommend creating a ``chef`` user with -full ``root`` privileges. For example:: - - ssh user@chef-node - sudo useradd -d /home/chef -m chef - sudo passwd chef - -To provide full privileges, add the following to ``/etc/sudoers.d/chef``. :: - - echo "chef ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/chef - sudo chmod 0440 /etc/sudoers.d/chef - -If you are using a version of ``sudo`` that doesn't support includes, you will -need to add the following to the ``/etc/sudoers`` file:: - - chef ALL = (root) NOPASSWD:ALL - -.. important:: Do not change the file permissions on ``/etc/sudoers``. Use a - suitable tool such as ``visudo``. - -.. _genkeys: - -Generate SSH Keys for Chef Clients -================================== - -Chef's ``knife`` tool can run ``ssh``. To streamline deployments, we -recommend generating an SSH key pair without a passphrase for your -Chef nodes and copying the public key(s) to your Chef nodes so that you -can connect to them from your workstation using ``ssh`` from ``knife`` -without having to provide a password. To generate a key pair without -a passphrase, execute the following on your Chef workstation. :: - - ssh-keygen - Generating public/private key pair. - Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - Your identification has been saved in /ceph-admin/.ssh/id_rsa. - Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub. - -You may use RSA or DSA keys. Once you generate your keys, copy them to each -OSD host. For example:: - - ssh-copy-id chef@your-node - -Consider modifying your ``~/.ssh/config`` file so that it defaults to -logging in as ``chef`` when no username is specified. :: - - Host myserver01 - Hostname myserver01.fqdn-or-ip-address.com - User chef - Host myserver02 - Hostname myserver02.fqdn-or-ip-address.com - User chef - -.. _installruby: - -Installing Ruby -=============== - -Chef requires you to install Ruby. Use the version applicable to your current -Linux distribution and install Ruby on all of your hosts. :: - - sudo apt-get update - sudo apt-get install ruby - -.. _installchefserver: - -Installing Chef and Chef Server on a Server -=========================================== - -If you plan on hosting your `Chef Server at Opscode`_ you may skip this step, -but you must make a note of the the fully qualified domain name or IP address -of your Chef Server for ``knife`` and ``chef-client``. - -First, add Opscode packages to your APT configuration. For example:: - - sudo tee /etc/apt/sources.list.d/chef.list << EOF - deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main - deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main - EOF - -Next, you must request keys so that APT can verify the packages. Copy -and paste the following line into your command line:: - - sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A - -The key is only used by ``apt``, so remove it from the ``root`` keyring by -typing ``Y`` when prompted to delete it. - -Install the Opscode keyring, Chef and Chef server on the host designated -as your Chef Server. :: - - sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef chef-server - -Enter the fully qualified domain name or IP address for your Chef server. For example:: - - http://fqdn-or-ip-address.com:4000 - -The Chef server installer will prompt you to enter a temporary password. Enter -a temporary password (*e.g.,* ``foo``) and proceed with the installation. - -.. tip:: When prompted for a temporary password, you may press **OK**. - The installer wants you to re-enter the password to confirm it. To - re-enter the password, you must press the **ESC** key. - -Once the installer finishes and activates the Chef server, you may enter the -fully qualified domain name or IP address in a browser to launch the -Chef web UI. For example:: - - http://fqdn-or-ip-address.com:4000 - -The Chef web UI will prompt you to enter the username and password. - -- **login:** ``admin`` -- **password:** ``foo`` - -Once you have entered the temporary password, the Chef web UI will prompt you -to enter a new password. - -.. _installchef: - -Install Chef on all Remaining Hosts -=================================== - -Install Chef on all Chef Nodes and on the Chef Workstation (if it is not the -same host as the Chef Server). See `Installing Chef Client on Ubuntu or Debian`_ -for details. - -First, add Opscode packages to your APT configuration. For example:: - - sudo tee /etc/apt/sources.list.d/chef.list << EOF - deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main - deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main - EOF - -Next, you must request keys so that APT can verify the packages. Copy -and paste the following line into your command line:: - - sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A - -The key is only used by ``apt``, so remove it from the ``root`` keyring by -typing ``Y`` when prompted to delete it. - -Install the Opscode keyring and Chef on all hosts other than the Chef Server. :: - - sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef - -Enter the fully qualified domain name or IP address for your Chef server. -For example:: - - http://fqdn-or-ip-address.com:4000 - -.. _configknife: - -Configuring Knife -================= - -Once you complete the Chef server installation, install ``knife`` on the your -Chef Workstation. If the Chef server is a remote host, use ``ssh`` to connect. :: - - ssh chef@fqdn-or-ip-address.com - -In the ``/home/chef`` directory, create a hidden Chef directory. :: - - mkdir -p ~/.chef - -The server generates validation and web UI certificates with read/write -permissions for the user that installed the Chef server. Copy them from the -``/etc/chef`` directory to the ``~/.chef`` directory. Then, change their -ownership to the current user. :: - - sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef && sudo chown $(id -u):$(id -g) ~/.chef/*.pem - -From the current user's home directory, configure ``knife`` with an initial -API client. :: - - knife configure -i - -The configuration will prompt you for inputs. Answer accordingly: - -*Where should I put the config file? [~/.chef/knife.rb]* Press **Enter** -to accept the default value. - -*Please enter the chef server URL:* If you are installing the -client on the same host as the server, enter ``http://localhost:4000``. -Otherwise, enter an appropriate URL for the server. - -*Please enter a clientname for the new client:* Press **Enter** -to accept the default value. - -*Please enter the existing admin clientname:* Press **Enter** -to accept the default value. - -*Please enter the location of the existing admin client's private key:* -Override the default value so that it points to the ``.chef`` directory. -(*e.g.,* ``/home/chef/.chef/webui.pem``) - -*Please enter the validation clientname:* Press **Enter** to accept -the default value. - -*Please enter the location of the validation key:* Override the -default value so that it points to the ``.chef`` directory. -(*e.g.,* ``/home/chef/.chef/validation.pem``) - -*Please enter the path to a chef repository (or leave blank):* -Leave the entry field blank and press **Enter**. - -.. _addcbpath: - -Add a Cookbook Path -=================== - -Add ``cookbook_path`` to the ``~/.chef/knife.rb`` configuration file -on your Chef workstation. For example:: - - cookbook_path '/home/{user-name}/chef-cookbooks/' - -Then create the path if it doesn't already exist. :: - - mkdir /home/{user-name}/chef-cookbooks - -This is where you will store local copies of cookbooks before uploading -them to the Chef server. - -.. _cpvalpem: - -Copy ``validation.pem`` to Nodes -================================ - -Copy the ``/etc/chef/validation.pem`` file from your Chef server to -each Chef Node. In a command line shell on the Chef Server, for each node, -replace ``{nodename}`` in the following line with the node's host name and -execute it. :: - - sudo cat /etc/chef/validation.pem | ssh {nodename} "exec sudo tee /etc/chef/validation.pem >/dev/null" - -.. _runchefcli: - -Run ``chef-client`` on each Chef Node -===================================== - -Run the ``chef-client`` on each Chef Node so that the nodes -register with the Chef server. :: - - ssh chef-node - sudo chef-client - -.. _verifynodes: - -Verify Nodes -============ - -Verify that you have setup all the hosts you want to use as -Chef nodes. :: - - knife node list - -A list of the nodes you've configured should appear. - -See the `Deploy With Chef <../../config-cluster/chef>`_ section for information -on using Chef to deploy your Ceph cluster. - -.. _Chef Architecture Introduction: http://wiki.opscode.com/display/chef/Architecture+Introduction -.. _Chef Server at Opscode: http://www.opscode.com/hosted-chef/ -.. _Installing Chef Client on Ubuntu or Debian: http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Ubuntu+or+Debian -.. _Installing Chef Server on Debian or Ubuntu using Packages: http://wiki.opscode.com/display/chef/Installing+Chef+Server+on+Debian+or+Ubuntu+using+Packages -.. _Knife Bootstrap: http://wiki.opscode.com/display/chef/Knife+Bootstrap \ No newline at end of file diff --git a/doc/install/index.rst b/doc/install/index.rst index 9c17f9849b182..3503c7939a6aa 100644 --- a/doc/install/index.rst +++ b/doc/install/index.rst @@ -31,18 +31,12 @@ questions and how to install Ceph on various distributions. If you are deploying a Ceph cluster (that is, not developing Ceph), install Ceph using our stable release packages. For testing, you -may install development release and testing packages. If you intend -to deploy Ceph using OpsCode Chef, review the instructions for -Chef below. - -.. tip:: If you are using Dell Crowbar, or Ceph-Deploy you do not - need to install OpsCode Chef. +may install development release and testing packages. .. toctree:: Installing Debian/Ubuntu Packages Installing RPM Packages - Installing OpsCode Chef .. raw:: html diff --git a/doc/rados/api/index.rst b/doc/rados/api/index.rst new file mode 100644 index 0000000000000..8cd63e9774e09 --- /dev/null +++ b/doc/rados/api/index.rst @@ -0,0 +1,20 @@ +========================= + RADOS Object Store APIs +========================= + +Ceph's RADOS Object Store has a messaging layer protocol that enables clients +to interact with Ceph monitors and OSDs. ``librados`` provides this +functionality to object store clients in the form of a library. All Ceph +clients either use ``librados`` or the same functionality encapsulated in +``librados`` to interact with the object store. For example, ``librbd`` and +``libcephfs`` leverage this functionality. You may use ``librados`` to interact +with Ceph directly (e.g., an application that talks to Ceph, your own interface +to Ceph, etc.). + +For an overview of where ``librados`` appears in the technology stack, +see :doc:`../../architecture`. + +.. toctree:: + + librados (C) + librados (C++) \ No newline at end of file diff --git a/doc/rados/api/librados.rst b/doc/rados/api/librados.rst index f433b5c412347..c81a09be3f3a0 100644 --- a/doc/rados/api/librados.rst +++ b/doc/rados/api/librados.rst @@ -4,8 +4,8 @@ .. highlight:: c -`Librados` provides low-level access to the RADOS service. For an -overview of RADOS, see :doc:`/architecture`. +`librados` provides low-level access to the RADOS service. For an +overview of RADOS, see :doc:`../../architecture`. Example: connecting and writing an object diff --git a/doc/rados/configuration/ceph-conf.rst b/doc/rados/configuration/ceph-conf.rst new file mode 100644 index 0000000000000..3ee71bf24731a --- /dev/null +++ b/doc/rados/configuration/ceph-conf.rst @@ -0,0 +1,550 @@ +================== + Configuring Ceph +================== + +When you start the Ceph service, the initialization process activates a series +of daemons that run in the background. The hosts in a typical Ceph cluster run +at least one of four daemons: + +- Object Storage Device (``ceph-osd``) +- Monitor (``ceph-mon``) +- Metadata Server (``ceph-mds``) +- Ceph Gateway (``radosgw``) + +For your convenience, each daemon has a series of default values (*i.e.*, many +are set by ``ceph/src/common/config_opts.h``). You may override these settings +with a Ceph configuration file. + +.. _ceph-conf-file: + +The ceph.conf File +================== + +When you start a Ceph cluster, each daemon looks for a ``ceph.conf`` file that +provides its configuration settings. For manual deployments, you need to create +a ``ceph.conf`` file to configure your cluster. For third party tools that +create configuration files for you (*e.g.*, Chef), you may use the information +contained herein as a reference. The ``ceph.conf`` file defines: + +- Cluster membership +- Host names +- Host addresses +- Paths to keyrings +- Paths to journals +- Paths to data +- Other runtime options + +The default ``ceph.conf`` locations in sequential order include: + +#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` environment variable) +#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument) +#. ``/etc/ceph/ceph.conf`` +#. ``~/.ceph/config`` +#. ``./ceph.conf`` (*i.e.,* in the current working directory) + + +The ``ceph.conf`` file uses an *ini* style syntax. You can add comments to the +``ceph.conf`` file by preceding comments with a semi-colon (;) or a pound sign +(#). For example: + +.. code-block:: ini + + # <--A number (#) sign precedes a comment. + ; A comment may be anything. + # Comments always follow a semi-colon (;) or a pound (#) on each line. + # The end of the line terminates a comment. + # We recommend that you provide comments in your configuration file(s). + +.. _ceph-conf-settings: + +ceph.conf Settings +================== + +The ``ceph.conf`` file can configure all daemons in a cluster, or all daemons of +a particular type. To configure a series of daemons, the settings must be +included under the processes that will receive the configuration as follows: + +``[global]`` + +:Description: Settings under ``[global]`` affect all daemons in a Ceph cluster. +:Example: ``auth supported = cephx`` + +``[osd]`` + +:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in the cluster. +:Example: ``osd journal size = 1000`` + +``[mon]`` + +:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in the cluster. +:Example: ``mon addr = 10.0.0.101:6789`` + + +``[mds]`` + +:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in the cluster. +:Example: ``host = myserver01`` + + +Global settings affect all instances of all daemon in the cluster. Use the ``[global]`` +setting for values that are common for all daemons in the cluster. You can override each +``[global]`` setting by: + +#. Changing the setting in a particular process type (*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ). +#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` ) + +Overriding a global setting affects all child processes, except those that +you specifically override. + +A typical global setting involves activating authentication. For example: + +.. code-block:: ini + + [global] + # Enable authentication between hosts within the cluster. + auth supported = cephx + + +You can specify settings that apply to a particular type of daemon. When you +specify settings under ``[osd]``, ``[mon]`` or ``[mds]`` without specifying a +particular instance, the setting will apply to all OSDs, monitors or metadata +daemons respectively. + +You may specify settings for particular instances of a daemon. You may specify +an instance by entering its type, delimited by a period (.) and by the +instance ID. The instance ID for an OSD is always numeric, but it may be +alphanumeric for monitors and metadata servers. + +.. code-block:: ini + + [osd.1] + # settings affect osd.1 only. + + [mon.a] + # settings affect mon.a only. + + [mds.b] + # settings affect mds.b only. + +.. _ceph-metavariables: + +Metavariables +============= + +Metavariables simplify cluster configuration dramatically. When a metavariable +is set in a configuration value, Ceph expands the metavariable into a concrete +value. Metavariables are very powerful when used within the ``[global]``, +``[osd]``, ``[mon]`` or ``[mds]`` sections of your configuration file. Ceph +metavariables are similar to Bash shell expansion. + +Ceph supports the following metavariables: + + +``$cluster`` + +:Description: Expands to the cluster name. Useful when running multiple clusters on the same hardware. +:Example: ``/etc/ceph/$cluster.keyring`` +:Default: ``ceph`` + + +``$type`` + +:Description: Expands to one of ``mds``, ``osd``, or ``mon``, depending on the type of the current daemon. +:Example: ``/var/lib/ceph/$type`` + + +``$id`` + +:Description: Expands to the daemon identifier. For ``osd.0``, this would be ``0``; for ``mds.a``, it would be ``a``. +:Example: ``/var/lib/ceph/$type/$cluster-$id`` + + +``$host`` + +:Description: Expands to the host name of the current daemon. + + +``$name`` + +:Description: Expands to ``$type.$id``. +:Example: ``/var/run/ceph/$cluster-$name.asok`` + +.. _ceph-conf-common-settings: + +Common Settings +=============== + +The `Hardware Recommendations`_ section provides some hardware guidelines for +configuring the cluster. It is possible for a single host to run multiple +daemons. For example, a single host with multiple disks or RAIDs may run one +``ceph-osd`` for each disk or RAID. Additionally, a host may run both a +``ceph-mon`` and an ``ceph-osd`` daemon on the same host. Ideally, you will have +a host for a particular type of process. For example, one host may run +``ceph-osd`` daemons, another host may run a ``ceph-mds`` daemon, and other +hosts may run ``ceph-mon`` daemons. + +Each host has a name identified by the ``host`` setting. Monitors also specify +a network address and port (i.e., domain name or IP address) identified by the +``addr`` setting. A basic configuration file will typically specify only +minimal settings for each instance of a daemon. For example: + +.. code-block:: ini + + [mon.a] + host = hostName + mon addr = 150.140.130.120:6789 + + [osd.0] + host = hostName + +.. important:: The ``host`` setting is the short name of the host (i.e., not + an fqdn). It is **NOT** and IP address either. Enter ``hostname -s`` on + the command line to retrieve the name of the host. Also, this setting is + **ONLY** for ``mkcephfs`` and manual deployment. It **MUST NOT** + be used with ``chef`` or ``ceph-deploy``. + +.. _Hardware Recommendations: ../../install/hardware-recommendations + +.. _ceph-network-config: + +Networks +======== + +Monitors listen on port 6789 by default, while metadata servers and OSDs listen +on the first available port beginning at 6800. Ensure that you open port 6789 on +hosts that run a monitor daemon, and open one port beginning at port 6800 for +each OSD or metadata server that runs on the host. Ports are host-specific, so +you don't need to open any more ports open than the number of daemons running on +that host, other than potentially a few spares. You may consider opening a few +additional ports in case a daemon fails and restarts without letting go of the +port such that the restarted daemon binds to a new port. If you set up separate +public and cluster networks, you may need to make entries for each network. +For example:: + + iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:6810 -j ACCEPT + + +In our `hardware recommendations`_ section, we recommend having at least two NIC +cards, because Ceph can support two networks: a public (front-side) network, and +a cluster (back-side) network. Ceph functions just fine with a public network +only. You only need to specify the public and cluster network settings if you +use both public and cluster networks. + +There are several reasons to consider operating two separate networks. First, +OSDs handle data replication for the clients. When OSDs replicate data more than +once, the network load between OSDs easily dwarfs the network load between +clients and the Ceph cluster. This can introduce latency and create a +performance problem. Second, while most people are generally civil, a very tiny +segment of the population likes to engage in what's known as a Denial of Service +(DoS) attack. When traffic between OSDs gets disrupted, placement groups may no +longer reflect an ``active + clean`` state, which may prevent users from reading +and writing data. A great way to defeat this type of attack is to maintain a +completely separate cluster network that doesn't connect directly to the +internet. + +To configure the networks, add the following options to the ``[global]`` section +of your ``ceph.conf`` file. + +.. code-block:: ini + + [global] + public network {public-network-ip-address/netmask} + cluster network {enter cluster-network-ip-address/netmask} + +To configure Ceph hosts to use the networks, you should set the following options +in the daemon instance sections of your ``ceph.conf`` file. + +.. code-block:: ini + + [osd.0] + public addr {host-public-ip-address} + cluster addr {host-cluster-ip-address} + +.. _hardware recommendations: ../../install/hardware-recommendations + + +.. _ceph-monitor-config: + +Monitors +======== + +Ceph production clusters typically deploy with a minimum 3 monitors to ensure +high availability should a monitor instance crash. An odd number of monitors (3) +ensures that the Paxos algorithm can determine which version of the cluster map +is the most recent from a quorum of monitors. + +.. note:: You may deploy Ceph with a single monitor, but if the instance fails, + the lack of a monitor may interrupt data service availability. + +Ceph monitors typically listen on port ``6789``. For example: + +.. code-block:: ini + + [mon.a] + host = hostName + mon addr = 150.140.130.120:6789 + +By default, Ceph expects that you will store a monitor's data under the following path:: + + /var/lib/ceph/mon/$cluster-$id + +You must create the corresponding directory yourself. With metavariables fully +expressed and a cluster named "ceph", the foregoing directory would evaluate to:: + + /var/lib/ceph/mon/ceph-a + +You may override this path using the ``mon data`` setting. We don't recommend +changing the default location. Create the default directory on your new monitor host. :: + + ssh {new-mon-host} + sudo mkdir /var/lib/ceph/mon/ceph-{mon-letter} + + +.. _ceph-osd-config: + +OSDs +==== + +Ceph production clusters typically deploy OSDs where one host has one OSD daemon +running a filestore on one data disk. A typical deployment specifies a journal +size and whether the file store's extended attributes (XATTRs) use an +object map (i.e., when running on the ``ext4`` filesystem). For example: + +.. code-block:: ini + + [osd] + osd journal size = 10000 + filestore xattr use omap = true #enables the object map. Only if running ext4. + + [osd.0] + hostname = {hostname} + + +By default, Ceph expects that you will store an OSD's data with the following path:: + + /var/lib/ceph/osd/$cluster-$id + +You must create the corresponding directory yourself. With metavariables fully +expressed and a cluster named "ceph", the foregoing directory would evaluate to:: + + /var/lib/ceph/osd/ceph-0 + +You may override this path using the ``osd data`` setting. We don't recommend +changing the default location. Create the default directory on your new OSD host. :: + + ssh {new-osd-host} + sudo mkdir /var/lib/ceph/osd/ceph-{osd-number} + +The ``osd data`` path ideally leads to a mount point with a hard disk that is +separate from the hard disk storing and running the operating system and +daemons. If the OSD is for a disk other than the OS disk, prepare it for +use with Ceph, and mount it to the directory you just created:: + + ssh {new-osd-host} + sudo mkfs -t {fstype} /dev/{disk} + sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number} + +We recommend using the ``xfs`` file system or the ``btrfs`` file system when +running :command:mkfs. + +By default, Ceph expects that you will store an OSDs journal with the following path:: + + /var/lib/ceph/osd/$cluster-$id/journal + +Without performance optimization, Ceph stores the journal on the same disk as +the OSDs data. An OSD optimized for performance may use a separate disk to store +journal data (e.g., a solid state drive delivers high performance journaling). + +Ceph's default ``osd journal size`` is 0, so you will need to set this in your +``ceph.conf`` file. A journal size should find the product of the ``filestore +min sync interval`` and the expected throughput, and multiple the product by +two (2):: + + osd journal size = {2 * (expected throughput * filestore min sync interval)} + +The expected throughput number should include the expected disk throughput +(i.e., sustained data transfer rate), and network throughput. For example, +a 7200 RPM disk will likely have approximately 100 MB/s. Taking the ``min()`` +of the disk and network throughput should provide a reasonable expected +throughput. Some users just start off with a 10GB journal size. For +example:: + + osd journal size = 10000 + +.. _ceph-logging-and-debugging: + +Logs / Debugging +================ + +Ceph is still on the leading edge, so you may encounter situations that require +modifying logging output and using Ceph's debugging. To activate Ceph's +debugging output (*i.e.*, ``dout()``), you may add ``debug`` settings to your +configuration. Ceph's logging levels operate on a scale of 1 to 20, where 1 is +terse and 20 is verbose. Subsystems common to each daemon may be set under +``[global]`` in your configuration file. Subsystems for particular daemons are +set under the daemon section in your configuration file (*e.g.*, ``[mon]``, +``[osd]``, ``[mds]``). For example:: + + [global] + debug ms = 1 + + [mon] + debug mon = 20 + debug paxos = 20 + debug auth = 20 + + [osd] + debug osd = 20 + debug filestore = 20 + debug journal = 20 + debug monc = 20 + + [mds] + debug mds = 20 + debug mds balancer = 20 + debug mds log = 20 + debug mds migrator = 20 + +When your system is running well, choose appropriate logging levels and remove +unnecessary debugging settings to ensure your cluster runs optimally. Logging +debug output messages is relatively slow, and a waste of resources when operating +your cluster. + +.. tip: When debug output slows down your system, the latency can hide race conditions. + +Each subsystem has a logging level for its output logs, and for its logs +in-memory. You may set different values for each of these subsystems by setting +a log file level and a memory level for debug logging. For example:: + + debug {subsystem} {log-level}/{memory-level} + #for example + debug mds log 1/20 + ++--------------------+-----------+--------------+ +| Subsystem | Log Level | Memory Level | ++====================+===========+==============+ +| ``default`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``lockdep`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``context`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``crush`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds balancer`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds locker`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds log`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds log expire`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``mds migrator`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``buffer`` | 0 | 0 | ++--------------------+-----------+--------------+ +| ``timer`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``filer`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``objecter`` | 0 | 0 | ++--------------------+-----------+--------------+ +| ``rados`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``rbd`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``journaler`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``objectcacher`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``client`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``osd`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``optracker`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``objclass`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``filestore`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``journal`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``ms`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``mon`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``monc`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``paxos`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``tp`` | 0 | 5 | ++--------------------+-----------+--------------+ +| ``auth`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``finisher`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``heartbeatmap`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``perfcounter`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``rgw`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``hadoop`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``asok`` | 1 | 5 | ++--------------------+-----------+--------------+ +| ``throttle`` | 1 | 5 | ++--------------------+-----------+--------------+ + + +Example ceph.conf +================= + +.. literalinclude:: demo-ceph.conf + :language: ini + +.. _ceph-runtime-config: + +Runtime Changes +=============== + +Ceph allows you to make changes to the configuration of an ``ceph-osd``, +``ceph-mon``, or ``ceph-mds`` daemon at runtime. This capability is quite +useful for increasing/decreasing logging output, enabling/disabling debug +settings, and even for runtime optimization. The following reflects runtime +configuration usage:: + + ceph {daemon-type} tell {id or *} injectargs '--{name} {value} [--{name} {value}]' + +Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply +the runtime setting to all daemons of a particular type with ``*``, or specify +a specific daemon's ID (i.e., its number or letter). For example, to increase +debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following:: + + ceph osd tell 0 injectargs '--debug-osd 20 --debug-ms 1' + +In your ``ceph.conf`` file, you may use spaces when specifying a +setting name. When specifying a setting name on the command line, +ensure that you use an underscore or hyphen (``_`` or ``-``) between +terms (e.g., ``debug osd`` becomes ``debug-osd``). + + +Viewing a Configuration at Runtime +================================== + +If your Ceph cluster is running, and you would like to see the configuration +settings from a running daemon, execute the following:: + + ceph --admin-daemon {/path/to/admin/socket} config show | less + +The default path for the admin socket for each daemon is:: + + /var/run/ceph/$cluster-$name.asok + +At real time, the metavariables will evaluate to the actual cluster name +and daemon name. For example, if the cluster name is ``ceph`` (it is by default) +and you want to retrieve the configuration for ``osd.0``, use the following:: + + ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | less diff --git a/doc/rados/configuration/demo-ceph.conf b/doc/rados/configuration/demo-ceph.conf new file mode 100644 index 0000000000000..b96b32cd4a195 --- /dev/null +++ b/doc/rados/configuration/demo-ceph.conf @@ -0,0 +1,32 @@ +[global] + auth supported = cephx + + +[osd] + osd journal size = 1000 + # uncomment the following line if you are mounting with ext4 + # filestore xattr use omap = true + +[mon.a] + host = myserver01 + mon addr = 10.0.0.101:6789 + +[mon.b] + host = myserver02 + mon addr = 10.0.0.102:6789 + +[mon.c] + host = myserver03 + mon addr = 10.0.0.103:6789 + +[osd.0] + host = myserver01 + +[osd.1] + host = myserver02 + +[osd.2] + host = myserver03 + +[mds.a] + host = myserver01 diff --git a/doc/rados/configuration/filestore-config-ref.rst b/doc/rados/configuration/filestore-config-ref.rst new file mode 100644 index 0000000000000..0babcd85034bd --- /dev/null +++ b/doc/rados/configuration/filestore-config-ref.rst @@ -0,0 +1,288 @@ +============================ + Filestore Config Reference +============================ + + + +``filestore debug omap check`` + +:Description: Debugging check on synchronization. Expensive. For debugging only. +:Type: Boolean +:Required: No +:Default: ``0`` + + +Extended Attributes +=================== + +Extended Attributes (XATTRs) are an imporant aspect in your configuration. +Some file systems have limits on the number of bytes stored in XATTRS. +Additionally, in some cases, the filesystem may not be as fast as an alternative +method of storing XATTRs. The following settings may help improve performance +by using a method of storing XATTRs that is extrinsic to the underlying filesystem. + + +``filestore xattr use omap`` + +:Description: Use object map for XATTRS. Set to ``true`` for ``ext4`` file systems. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``filestore max inline xattr size`` + +:Description: The maximimum size of an XATTR stored in the filesystem (i.e., XFS, btrfs, ext4, etc.) per object. Should not be larger than the filesytem can handle. +:Type: Unsigned 32-bit Integer +:Required: No +:Default: ``512`` + + +``filestore max inline xattrs`` + +:Description: The maximum number of XATTRs stored in the fileystem per object. +:Type: 32-bit Integer +:Required: No +:Default: ``2`` + + +Synchronization Intervals +========================= + +Periodically, the filestore needs to quiesce writes and synchronize the filesystem, +which creates a consistent commit point. It can then free journal entries up to +the commit point. Synchronizing more frequently tends to reduce the time required +perform synchronization, and reduces the amount of data that needs to remain in the +journal. Less frequent synchronization allows the backing filesystem to coalesce +small writes and metadata updates more optimally--potentially resulting in more +efficient synchronization. + + +``filestore max sync interval`` + +:Description: The maximum interval in seconds for synchronizing the filestore. +:Type: Double +:Required: No +:Default: ``5`` + + +``filestore min sync interval`` + +:Description: The minimum interval in seconds for synchronizing the filestore. +:Type: Double +:Required: No +:Default: ``.01`` + + +Flusher +======= + +The filestore flusher forces data from large writes to be written out using +``sync file range`` before the sync in order to (hopefully) reduce the cost of +the eventual sync. In practice, disabling 'filestore flusher' seems to improve +performance in some cases. + + +``filestore flusher`` + +:Description: Enables the filestore flusher. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``filestore flusher max fds`` + +:Description: Sets the maximum number of file descriptors for the flusher. +:Type: Integer +:Required: No +:Default: ``512`` + +``filestore sync flush`` + +:Description: Enables the synchronization flusher. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``filestore fsync flushes journal data`` + +:Description: Flush journal data during filesystem synchronization. +:Type: Boolean +:Required: No +:Default: ``false`` + + +Queue +===== + +The following settings provide limits on the size of filestore queue. + +``filestore queue max ops`` + +:Description: Defines the maximum number of in progress operations the file store accepts before blocking on queuing new operations. +:Type: Integer +:Required: No. Minimal impact on performance. +:Default: ``500`` + + +``filestore queue max bytes`` + +:Description: The maximum number of bytes for an operation. +:Type: Integer +:Required: No +:Default: ``100 << 20`` + + +``filestore queue committing max ops`` + +:Description: The maximum number of operations the filestore can commit. +:Type: Integer +:Required: No +:Default: ``500`` + + +``filestore queue committing max bytes`` + +:Description: The maximum number of bytes the filestore can commit. +:Type: Integer +:Required: No +:Default: ``100 << 20`` + + + +Timeouts +======== + + +``filestore op threads`` + +:Description: The number of filesystem operation threads that execute in parallel. +:Type: Integer +:Required: No +:Default: ``2`` + + +``filestore op thread timeout`` + +:Description: The timeout for a filesystem operation thread (in seconds). +:Type: Integer +:Required: No +:Default: ``60`` + + +``filestore op thread suicide timeout`` + +:Description: The timeout for a commit operation before cancelling the commit (in seconds). +:Type: Integer +:Required: No +:Default: ``180`` + + +B-Tree Filesystem +================= + + +``filestore btrfs snap`` + +:Description: Enable snapshots for a ``btrfs`` filestore. +:Type: Boolean +:Required: No. Only used for ``btrfs``. +:Default: ``true`` + + +``filestore btrfs clone range`` + +:Description: Enable cloning ranges for a ``btrfs`` filestore. +:Type: Boolean +:Required: No. Only used for ``btrfs``. +:Default: ``true`` + +Journal +======= + + +``filestore journal parallel`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``false`` + + +``filestore journal writeahead`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``false`` + + +``filestore journal trailing`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``false`` + + +Misc +==== + + +``filestore merge threshold`` + +:Description: +:Type: Integer +:Required: No +:Default: ``10`` + + +``filestore split multiple`` + +:Description: +:Type: Integer +:Required: No +:Default: ``2`` + + +``filestore update to`` + +:Description: +:Type: Integer +:Required: No +:Default: ``1000`` + + +``filestore blackhole`` + +:Description: Drop any new transactions on the floor. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``filestore dump file`` + +:Description: File onto which store transaction dumps? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``filestore kill at`` + +:Description: inject a failure at the n'th opportunity +:Type: String +:Required: No +:Default: ``false`` + + +``filestore fail eio`` + +:Description: Fail/Crash on eio. +:Type: Boolean +:Required: No +:Default: ``true`` + diff --git a/doc/rados/configuration/filesystem-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst new file mode 100644 index 0000000000000..17908cc3d19a1 --- /dev/null +++ b/doc/rados/configuration/filesystem-recommendations.rst @@ -0,0 +1,78 @@ +=========================================== + Hard Disk and File System Recommendations +=========================================== + +Hard Disk Prep +============== + +Ceph aims for data safety, which means that when the application receives notice +that data was written to the disk, that data was actually written to the disk. +For old kernels (<2.6.33), disable the write cache if the journal is on a raw +disk. Newer kernels should work fine. + +Use ``hdparm`` to disable write caching on the hard disk:: + + sudo hdparm -W 0 /dev/hda 0 + +In production environments, we recommend running OSDs with an operating system +disk, and a separate disk(s) for data. If you run data and an operating system +on a single disk, create a separate partition for your data before configuring +your OSD cluster. + + +File Systems +============ + +Ceph OSDs rely heavily upon the stability and performance of the +underlying file system. + +.. note:: We currently recommend ``XFS`` for production deployments. + We recommend ``btrfs`` for testing, development, and any + non-critical deployments. We believe that ``btrfs`` has the correct + feature set and roadmap to serve Ceph in the long-term, but ``XFS`` + and ``ext4`` provide the necessary stability for today's deployments. + ``btrfs`` development is proceeding rapidly: users should be + comfortable installing the latest released upstream kernels and be + able to track development activity for critical bug fixes. + +Ceph OSDs depend on the Extended Attributes (XATTRs) of the underlying +file system for various forms of internal object state and metadata. +The underlying file system must provide sufficient capacity for +XATTRs. ``btrfs`` does not bound the total xattr metadata stored with +a file. ``XFS`` has a relatively large limit (64 KB) that most +deployments won't encounter, but the ``ext4`` is too small to be +usable. To use these file systems, you should add the following like +to the ``[osd]`` section of your ``ceph.conf`` file.:: + + filestore xattr use omap = true + +FS Background Info +================== + +The ``XFS`` and ``btrfs`` file systems provide numerous advantages in highly +scaled data storage environments when `compared`_ to ``ext3`` and ``ext4``. +Both ``XFS`` and ``btrfs`` are `journaling file systems`_, which means that +they are more robust when recovering from crashes, power outages, etc. These +filesystems journal all of the changes they will make before performing writes. + +``XFS`` was developed for Silicon Graphics, and is a mature and stable +filesystem. By contrast, ``btrfs`` is a relatively new file system that aims +to address the long-standing wishes of system administrators working with +large scale data storage environments. ``btrfs`` has some unique features +and advantages compared to other Linux filesystems. + +``btrfs`` is a `copy-on-write`_ filesystem. It supports file creation +timestamps and checksums that verify metadata integrity, so it can detect +bad copies of data and fix them with the good copies. The copy-on-write +capability means that ``btrfs`` can support snapshots that are writable. +``btrfs`` supports transparent compression and other features. + +``btrfs`` also incorporates multi-device management into the file system, +which enables you to support heterogeneous disk storage infrastructure, +data allocation policies. The community also aims to provide ``fsck``, +deduplication, and data encryption support in the future. This compelling +list of features makes ``btrfs`` the ideal choice for Ceph clusters. + +.. _copy-on-write: http://en.wikipedia.org/wiki/Copy-on-write +.. _compared: http://en.wikipedia.org/wiki/Comparison_of_file_systems +.. _journaling file systems: http://en.wikipedia.org/wiki/Journaling_file_system diff --git a/doc/rados/configuration/general-config-ref.rst b/doc/rados/configuration/general-config-ref.rst new file mode 100644 index 0000000000000..a70b7f776b228 --- /dev/null +++ b/doc/rados/configuration/general-config-ref.rst @@ -0,0 +1,148 @@ +========================== + General Config Reference +========================== + +``auth supported`` + +.. deprecated:: 0.51 + +:Description: Indicates the type of authentication used. Currently ``cephx`` only. If not specified, it defaults to ``none``. +:Type: String +:Required: No +:Default: ``none`` + + +``auth cluster required`` + +.. versionadded:: 0.51 + +:Description: Enables authentication for the cluster. Valid setting is ``cephx``. +:Type: String +:Required: No +:Default: ``none`` + + +``auth service required`` + +.. versionadded:: 0.51 + +:Description: Enables authentication for the service. Valid setting is ``cephx``. +:Type: String +:Required: No +:Default: ``none`` + + + +``auth client required`` + +.. versionadded:: 0.51 + +:Description: Enables authentication for the client. Valid setting is ``cephx``. +:Type: String +:Required: No +:Default: ``none`` + + +``keyring`` + +:Description: The path to the cluster's keyring file. +:Type: String +:Required: No +:Default: ``/etc/ceph/$cluster.$name.keyring,/etc/ceph/$cluster.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin`` + + +``fsid`` + +:Description: The filesystem ID. One per cluster. +:Type: UUID +:Required: No. +:Default: N/A. Generated if not specified. + + +``mon host`` + +:Description: A list of ``{hostname}:{port}`` entries that clients can use to connect to a Ceph monitor. If not set, Ceph searches ``[mon.*]`` sections. +:Type: String +:Required: No +:Default: N/A + + +``host`` + +:Description: The hostname. Use this setting for specific daemon instances (e.g., ``[osd.0]``). +:Type: String +:Required: Yes, for daemon instances. +:Default: ``localhost`` + +.. tip:: Do not use ``localhost``. To get your host name, execute ``hostname -s`` on your command line and use the name of your host (to the first period, not the fully-qualified domain name). +.. important: You should not specify any value for ``host`` when using a third party deployment system that retrieves the host name for you. + + +``public network`` + +:Description: The IP address and netmask of the public (front-side) network (e.g., ``10.20.30.40/24``). Set in ``[global]``. +:Type: ``{ip-address}/{netmask}`` +:Required: No +:Default: N/A + + +``public addr`` + +:Description: The IP address for the public (front-side) network. Set for each daemon. +:Type: IP Address +:Required: No +:Default: N/A + + +``cluster network`` + +:Description: The IP address and netmask of the cluster (back-side) network (e.g., ``10.20.30.41/24``). Set in ``[global]``. +:Type: ``{ip-address}/{netmask}`` +:Required: No +:Default: N/A + + +``cluster addr`` + +:Description: The IP address for the cluster (back-side) network. Set for each daemon. +:Type: Address +:Required: No +:Default: N/A + + +``admin socket`` + +:Description: The socket for executing administrative commands irrespective of whether Ceph monitors have established a quorum. +:Type: String +:Required: No +:Default: ``/var/run/ceph/$cluster-$name.asok`` + + +``pid file`` + +:Description: Each running Ceph daemon has a running process identifier (PID) file. +:Type: String +:Required: No +:Default: N/A. The default path is ``/var/run/$cluster/$name.pid``. The PID file is generated upon start-up. + + +``chdir`` + +:Description: The directory Ceph daemons change to once they are up and running. Default ``/`` directory recommended. +:Type: String +:Required: No +:Default: ``/`` + + +``max open files`` + +:Description: If set, when the Ceph service starts, Ceph sets the ``max open fds`` at the OS level (i.e., the max # of file descriptors). It helps prevents OSDs from running out of file descriptors. +:Type: 64-bit Integer +:Required: No +:Default: ``0`` + +``fatal signal handlers`` + +:Description: If set, we will install signal handlers for SEGV, ABRT, BUS, ILL, FPE, XCPU, XFSZ, SYS signals to generate a useful log message +:Type: Boolean +:Default: ``true`` diff --git a/doc/rados/configuration/index.rst b/doc/rados/configuration/index.rst new file mode 100644 index 0000000000000..8fa30136c7190 --- /dev/null +++ b/doc/rados/configuration/index.rst @@ -0,0 +1,57 @@ +=============== + Configuration +=============== + +Ceph can run with a cluster containing thousands of Object Storage Devices +(OSDs). A minimal system will have at least two OSDs for data replication. To +configure OSD clusters, you must provide settings in the configuration file. +Ceph provides default values for many settings, which you can override in the +configuration file. Additionally, you can make runtime modification to the +configuration using command-line utilities. + +When Ceph starts, it activates three daemons: + +- ``ceph-osd`` (mandatory) +- ``ceph-mon`` (mandatory) +- ``ceph-mds`` (mandatory for cephfs only) + +Each process, daemon or utility loads the host's configuration file. A process +may have information about more than one daemon instance (*i.e.,* multiple +contexts). A daemon or utility only has information about a single daemon +instance (a single context). + +.. note:: Ceph can run on a single host for evaluation purposes. + + +.. raw:: html + +

Configuring the Object Store

+ +For general object store configuration, refer to the following: + +.. toctree:: + + Disks and Filesystems + ceph-conf + + +.. raw:: html + +

Reference

+ +To optimize the performance of your cluster, refer to the following: + +.. toctree:: + + General Settings + Monitor Settings + OSD Settings + Filestore Settings + Journal Settings + Log / Debug Settings + Messaging Settings + + +.. raw:: html + +
diff --git a/doc/rados/configuration/journal-ref.rst b/doc/rados/configuration/journal-ref.rst new file mode 100644 index 0000000000000..f1906d12a505b --- /dev/null +++ b/doc/rados/configuration/journal-ref.rst @@ -0,0 +1,95 @@ +========================== + Journal Config Reference +========================== + +Ceph OSDs use a journal for two reasons: speed and consistency. + +- **Speed:** The journal enables the OSD to commit small writes quickly. + Ceph writes small, random i/o to the journal sequentially, which tends to + speed up bursty workloads by allowing the backing filesystem more time to + coalesce writes. The OSD journal, however, can lead to spiky performance + with short spurts of high-speed writes followed by periods without any + write progress as the filesystem catches up to the journal. + +- **Consistency:** Ceph OSDs requires a filesystem interface that guarantees + atomic compound operations. Ceph OSDs write a description of the operation + to the journal and apply the operation to the filesystem. This enables + atomic updates to an object (for example, placement group metadata). Every + few seconds--between ``filestore max sync interval`` and + ``filestore min sync interval``--the OSD stops writes and synchronizes the + journal with the filesystem, allowing OSDs to trim operations from the + journal and reuse the space. On failure, OSDs replay the journal starting + after the last synchronization operation. + +Ceph OSDs support the following journal settings: + +``journal dio`` + +:Description: Enables direct i/o to the journal. Requires ``journal block align`` set to ``true``. +:Type: Boolean +:Required: Yes when using ``aio``. +:Default: ``true`` + + +``journal aio`` + +:Description: Enables using ``libaio`` for asynchronous writes to the journal. Requires ``journal dio`` set to ``true``. +:Type: Boolean +:Required: No. +:Default: ``false`` + + +``journal block align`` + +:Description: Block aligns writes. Required for ``dio`` and ``aio``. +:Type: Boolean +:Required: Yes when using ``dio`` and ``aio``. +:Default: ``true`` + + +``journal max write bytes`` + +:Description: The maximum number of bytes the journal will write at any one time. +:Type: Integer +:Required: No +:Default: ``10 << 20`` + + +``journal max write entries`` + +:Description: The maximum number of entries the journal will write at any one time. +:Type: Integer +:Required: No +:Default: ``100`` + + +``journal queue max ops`` + +:Description: The maximum number of operations allowed in the queue at any one time. +:Type: Integer +:Required: No +:Default: ``500`` + + +``journal queue max bytes`` + +:Description: The maximum number of bytes allowed in the queue at any one time. +:Type: Integer +:Required: No +:Default: ``10 << 20`` + + +``journal align min size`` + +:Description: Align data payloads greater than the specified minimum. +:Type: Integer +:Required: No +:Default: ``64 << 10`` + + +``journal zero on create`` + +:Description: Causes the file store to overwrite the entire journal with ``0``'s during ``mkfs``. +:Type: Boolean +:Required: No +:Default: ``false`` diff --git a/doc/rados/configuration/log-and-debug-ref.rst b/doc/rados/configuration/log-and-debug-ref.rst new file mode 100644 index 0000000000000..b0a99975cd1ed --- /dev/null +++ b/doc/rados/configuration/log-and-debug-ref.rst @@ -0,0 +1,326 @@ +======================================== + Logging and Debugging Config Reference +======================================== + +Logging and debugging settings are not required, but you may override default settings +as needed. Ceph supports the following settings: + +Logging +======= + +``log file`` + +:Desription: The location of the logging file for your cluster. +:Type: String +:Required: No +:Default: ``/var/log/ceph/$cluster-$name.log`` + + +``log max new`` + +:Description: The maximum number of new log files. +:Type: Integer +:Required: No +:Default: ``1000`` + + +``log max recent`` + +:Description: The maximum number of recent events to include in a log file. +:Type: Integer +:Required: No +:Default: ``1000000`` + + +``log to stderr`` + +:Description: Determines if logging messages should appear in ``stderr``. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``err to stderr`` + +:Description: Determines if error messages should appear in ``stderr``. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``log to syslog`` + +:Description: Determines if logging messages should appear in ``syslog``. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``err to syslog`` + +:Description: Determines if error messages should appear in ``syslog``. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``log flush on exit`` + +:Description: Determines if Ceph should flush the log files after exit. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``clog to monitors`` + +:Description: Determines if ``clog`` messages should be sent to monitors. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``clog to syslog`` + +:Description: Determines if ``clog`` messages should be sent to syslog. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mon cluster log to syslog`` + +:Description: Determines if the cluster log should be output to the syslog. +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mon cluster log file`` + +:Description: The location of the cluster's log file. +:Type: String +:Required: No +:Default: ``/var/log/ceph/$cluster.log`` + + + +OSD +=== + + +``osd debug drop ping probability`` + +:Description: ? +:Type: Double +:Required: No +:Default: 0 + + +``osd debug drop ping duration`` + +:Description: The duration ? +:Type: Integer +:Required: No +:Default: 0 + +``osd debug drop pg create probability`` + +:Description: +:Type: Integer +:Required: No +:Default: 0 + +``osd debug drop pg create duration`` + +:Description: ? +:Type: Double +:Required: No +:Default: 1 + +``osd preserve trimmed log`` + +:Description: ? +:Type: Boolean +:Required: No +:Default: ``false`` + +``osd tmapput sets uses tmap`` + +:Description: For debug only. ??? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``osd min pg log entries`` + +:Description: The minimum number of log entries for placement groups. +:Type: 32-bit Unsigned Integer +:Required: No +:Default: 1000 + +``osd op log threshold`` + +:Description: How many op log messages to show up in one pass. +:Type: Integer +:Required: No +:Default: 5 + + + +Filestore +========= + +``filestore debug omap check`` + +:Description: Checks the ``omap``. This is an expensive operation. +:Type: Boolean +:Required: No +:Default: 0 + + +MDS +=== + + +``mds debug scatterstat`` + +:Description: ? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mds debug frag`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mds debug auth pins`` + +:Description: ? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``mds debug subtrees`` + +:Description: ? +:Type: Boolean +:Required: No +:Default: ``false`` + + + +RADOS Gateway +============= + + +``rgw log nonexistent bucket`` + +:Description: Should we log a non-existent buckets? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``rgw log object name`` + +:Description: Should an object's name be logged. // man date to see codes (a subset are supported) +:Type: String +:Required: No +:Default: ``%Y-%m-%d-%H-%i-%n`` + + +``rgw log object name utc`` + +:Description: Object log name contains UTC? +:Type: Boolean +:Required: No +:Default: ``false`` + + +``rgw enable ops log`` + +:Description: Enables logging of every RGW operation. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``rgw enable usage log`` + +:Description: Enable logging of RGW's bandwidth usage. +:Type: Boolean +:Required: No +:Default: ``true`` + + +``rgw usage log flush threshold`` + +:Description: Threshold to flush pending log data. +:Type: Integer +:Required: No +:Default: ``1024`` + + +``rgw usage log tick interval`` + +:Description: Flush pending log data every ``s`` seconds. +:Type: Integer +:Required: No +:Default: 30 + + +``rgw intent log object name`` + +:Description: +:Type: String +:Required: No +:Default: ``%Y-%m-%d-%i-%n`` + + +``rgw intent log object name utc`` + +:Description: Include a UTC timestamp in the intent log object name. +:Type: Boolean +:Required: No +:Default: ``false`` + +``rgw cluster root pool`` + +:Description: RADOS pool to store radosgw metadata for this instance +:Type: String +:Required: No +:Default: ``.rgw.root`` + +``rgw gc max objs`` + +:Description: Number of objects to collect garbage collection data +:Type: 32-bit Integer +:Default: 32 + +``rgw gc obj min wait`` + +:Description: Minimum time to wait before object's removal and its processing by the garbage collector +:Type: 32-bit Integer +:Default: 2 hours. ``2*60*60`` + +``rgw gc processor max time`` + +:Description: Max time for a single garbage collection process run +:Type: 32-bit Integer +:Default: 1 hour. ``60*60`` + +``rgw gc processor max period`` + +:Description: Max time between the beginning of two consecutive garbage collection processes run +:Type: 32-bit Integer +:Default: 1 hour. ``60*60`` + + diff --git a/doc/rados/configuration/mon-config-ref.rst b/doc/rados/configuration/mon-config-ref.rst new file mode 100644 index 0000000000000..b8c8c07e03c3c --- /dev/null +++ b/doc/rados/configuration/mon-config-ref.rst @@ -0,0 +1,240 @@ +========================== + Monitor Config Reference +========================== + +``mon data`` + +:Description: The monitor's data location. +:Type: String +:Default: ``/var/lib/ceph/mon/$cluster-$id`` + + +``mon initial members`` + +:Description: The IDs of initial monitors in a cluster during startup. If specified, Ceph requires an odd number of monitors to form an initial quorum. +:Type: String +:Default: None + + +``mon sync fs threshold`` + +:Description: Synchronize with the filesystem when writing the specified number of objects. Set it to ``0`` to disable it. +:Type: 32-bit Integer +:Default: ``5`` + + +``mon tick interval`` + +:Description: A monitor's tick interval in seconds. +:Type: 32-bit Integer +:Default: ``5`` + + +``mon subscribe interval`` + +:Description: The refresh interval (in seconds) for subscriptions. The subscription mechanism enables obtaining the cluster maps and log informations. +:Type: Double +:Default: ``300`` + + +``mon osd auto mark in`` + +:Description: Ceph will mark any booting OSDs as ``in`` the cluster. +:Type: Boolean +:Default: ``false`` + + +``mon osd auto mark auto out in`` + +:Description: Ceph will mark booting OSDs auto marked ``out`` of the cluster as ``in`` the cluster. +:Type: Boolean +:Default: ``true`` + + +``mon osd auto mark new in`` + +:Description: Ceph will mark booting new OSDs as ``in`` the cluster. +:Type: Boolean +:Default: ``true`` + + +``mon osd down out interval`` + +:Description: The number of seconds Ceph waits before marking an OSD ``down`` and ``out`` if it doesn't respond. +:Type: 32-bit Integer +:Default: ``300`` + + +``mon osd min up ratio`` + +:Description: The minimum ratio of ``up`` OSDs before Ceph will mark OSDs ``down``. +:Type: Double +:Default: ``.3`` + + +``mon osd min in ratio`` + +:Description: The minimum ratio of ``in`` OSDs before Ceph will mark OSDs ``out``. +:Type: Double +:Default: ``.3`` + + +``mon lease`` + +:Description: Length (in seconds) of the lease on the monitor's versions. +:Type: Float +:Default: ``5`` + + +``mon lease renew interval`` + +:Description: The interval (in seconds) for the Leader to renew the other monitor's leases. +:Type: Float +:Default: ``3`` + + +``mon lease ack timeout`` + +:Description: Number of seconds the Leader will wait for the Peons to acknowledge the lease extension. +:Type: Float +:Default: ``10.0`` + + +``mon clock drift allowed`` + +:Description: The clock drift in seconds allowed between monitors. +:Type: Float +:Default: ``.050`` + + +``mon clock drift warn backoff`` + +:Description: Exponential backoff for clock drift warnings +:Type: Float +:Default: ``5`` + + +``mon accept timeout`` + +:Description: Number of seconds the Leader will wait for the Peons to accept a Paxos update. It is also used during the Paxos recovery phase for similar purposes. +:Type: Float +:Default: ``10.0`` + + +``mon pg create interval`` + +:Description: Number of seconds between PG creation in the same OSD. +:Type: Float +:Default: ``30.0`` + + +``mon pg stuck threshold`` + +:Description: Number of seconds after which PGs can be considered as being stuck. +:Type: 32-bit Integer +:Default: ``300`` + + +``mon osd full ratio`` + +:Description: The percentage of disk space used before an OSD is considered ``full``. +:Type: Float +:Default: ``.95`` + + +``mon osd nearfull ratio`` + +:Description: The percentage of disk space used before an OSD is considered ``nearfull``. +:Type: Float +:Default: ``.85`` + + +``mon globalid prealloc`` + +:Description: The number of global IDs to pre-allocate for the cluster. +:Type: 32-bit Integer +:Default: ``100`` + + +``mon osd report timeout`` + +:Description: The grace period in seconds before declaring unresponsive OSDs ``down``. +:Type: 32-bit Integer +:Default: ``900`` + + +``mon force standby active`` + +:Description: should mons force standby-replay mds to be active +:Type: Boolean +:Default: true + + +``mon min osdmap epochs`` + +:Description: Minimum number of OSD map epochs to keep at all times. +:Type: 32-bit Integer +:Default: ``500`` + + +``mon max pgmap epochs`` + +:Description: Maximum number of PG map epochs the monitor should keep. +:Type: 32-bit Integer +:Default: ``500`` + + +``mon max log epochs`` + +:Description: Maximum number of Log epochs the monitor should keep. +:Type: 32-bit Integer +:Default: ``500`` + + +``mon max osd`` + +:Description: The maximum number of OSDs allowed in the cluster. +:Type: 32-bit Integer +:Default: ``10000`` + + +``mon probe timeout`` + +:Description: Number of seconds the monitor will wait to find peers before bootstrapping. +:Type: Double +:Default: ``2.0`` + + +``mon slurp timeout`` + +:Description: Number of seconds the monitor has to recover using slurp before the process is aborted and the monitor bootstraps. +:Type: Double +:Default: ``10.0`` + + +``mon slurp bytes`` + +:Description: Limits the slurp messages to the specified number of bytes. +:Type: 32-bit Integer +:Default: ``256 * 1024`` + + +``mon client bytes`` + +:Description: The amount of client message data allowed in memory (in bytes). +:Type: 64-bit Integer Unsigned +:Default: ``100ul << 20`` + + +``mon daemon bytes`` + +:Description: The message memory cap for metadata server and OSD messages (in bytes). +:Type: 64-bit Integer Unsigned +:Default: ``400ul << 20`` + + +``mon max log entries per event`` + +:Description: The maximum number of log entries per event. +:Type: Integer +:Default: ``4096`` diff --git a/doc/rados/configuration/ms-ref.rst b/doc/rados/configuration/ms-ref.rst new file mode 100644 index 0000000000000..d76f7f3bb2587 --- /dev/null +++ b/doc/rados/configuration/ms-ref.rst @@ -0,0 +1,83 @@ +=========== + Messaging +=========== + + +``ms tcp nodelay`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``true`` + + +``ms initial backoff`` + +:Description: +:Type: Double +:Required: No +:Default: ``.2`` + + +``ms max backoff`` + +:Description: +:Type: Double +:Required: No +:Default: ``15.0`` + + +``ms nocrc`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``false`` + + +``ms die on bad msg`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``false`` + + +``ms dispatch throttle bytes`` + +:Description: +:Type: 64-bit Unsigned Integer +:Required: No +:Default: ``100 << 20`` + + +``ms bind ipv6`` + +:Description: +:Type: Boolean +:Required: No +:Default: ``false`` + + +``ms rwthread stack bytes`` + +:Description: +:Type: 64-bit Unsigned Integer +:Required: No +:Default: ``1024 << 10`` + + +``ms tcp read timeout`` + +:Description: +:Type: 64-bit Unsigned Integer +:Required: No +:Default: ``900`` + + +``ms inject socket failures`` + +:Description: +:Type: 64-bit Unsigned Integer +:Required: No +:Default: ``0`` diff --git a/doc/rados/configuration/osd-config-ref.rst b/doc/rados/configuration/osd-config-ref.rst new file mode 100644 index 0000000000000..05a3369cde148 --- /dev/null +++ b/doc/rados/configuration/osd-config-ref.rst @@ -0,0 +1,422 @@ +====================== + OSD Config Reference +====================== + + + +``osd uuid`` + +:Description: The universally unique identifier (UUID) for the OSD. +:Type: UUID +:Default: None + + +``osd data`` + +:Description: The path to the OSDs data. You must create the directory. You should mount a data disk at this mount point. We do not recommend changing the default. +:Type: String +:Default: ``/var/lib/ceph/osd/$cluster-$id`` + + +``osd journal`` + +:Description: The path to the OSD's journal. This may be a path to a file or a block device (such as a partition of an SSD). If it is a file, you must create the directory to contain it. +:Type: String +:Default: ``/var/lib/ceph/osd/$cluster-$id/journal`` + + +``osd journal size`` + +:Description: The size of the journal in megabytes. If this is 0, and the journal is a block device, the entire block device is used. Since v0.54, this is ignored if the journal is a block device, and the entire block device is used. +:Type: 32-bit Integer +:Default: ``1024`` +:Recommended: Begin with 1GB. Should at least twice the product of the expected speed multiplied by ``filestore min sync interval``. + + +``osd max write size`` + +:Description: The maximum size of a write in megabytes. +:Type: 32-bit Integer +:Default: ``90`` + + +``osd client message size cap`` + +:Description: The largest client data message allowed in memory. +:Type: 64-bit Integer Unsigned +:Default: 500MB default. ``500*1024L*1024L`` + + +``osd stat refresh interval`` + +:Description: The status refresh interval in seconds. +:Type: 64-bit Integer Unsigned +:Default: ``.5`` + + +``osd pg bits`` + +:Description: Placement group bits per OSD. +:Type: 32-bit Integer +:Default: ``6`` + + +``osd pgp bits`` + +:Description: The number of bits per OSD for PGPs. +:Type: 32-bit Integer +:Default: ``4`` + + +``osd pg layout`` + +:Description: Placement group layout. +:Type: 32-bit Integer +:Default: ``2`` + + +``osd pool default crush rule`` + +:Description: The default CRUSH rule to use when creating a pool. +:Type: 32-bit Integer +:Default: ``0`` + + +``osd pool default size`` + +:Description: The default size of an OSD pool in gigabytes. The default value is the same as ``--size 2`` with ``mkpool``. +:Type: 32-bit Integer +:Default: ``2`` + + +``osd pool default pg num`` + +:Description: The default number of placement groups for a pool. The default value is the same as ``pg_num`` with ``mkpool``. +:Type: 32-bit Integer +:Default: ``8`` + + +``osd pool default pgp num`` + +:Description: The default number of placement groups for placement for a pool. The default value is the same as ``pgp_num`` with ``mkpool``. PG and PGP should be equal (for now). +:Type: 32-bit Integer +:Default: ``8`` + + +``osd map dedup`` + +:Description: Enable removing duplicates in the OSD map. +:Type: Boolean +:Default: ``true`` + + +``osd map cache size`` + +:Description: The size of the OSD map cache in megabytes. +:Type: 32-bit Integer +:Default: ``500`` + + +``osd map cache bl size`` + +:Description: The size of the in-memory OSD map cache in OSD daemons. +:Type: 32-bit Integer +:Default: ``50`` + + +``osd map cache bl inc size`` + +:Description: The size of the in-memory OSD map cache incrementals in OSD daemons. +:Type: 32-bit Integer +:Default: ``100`` + + +``osd map message max`` + +:Description: The maximum map entries allowed per MOSDMap message. +:Type: 32-bit Integer +:Default: ``100`` + + +``osd op threads`` + +:Description: The number of OSD operation threads. Set to ``0`` to disable it. Increasing the number may increase the request processing rate. +:Type: 32-bit Integer +:Default: ``2`` + + +``osd op thread timeout`` + +:Description: The OSD operation thread timeout in seconds. +:Type: 32-bit Integer +:Default: ``30`` + + +``osd disk threads`` + +:Description: The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming. +:Type: 32-bit Integer +:Default: ``1`` + + +``osd recovery threads`` + +:Description: The number of threads for recovering data. +:Type: 32-bit Integer +:Default: ``1`` + + +``osd recover clone overlap`` + +:Description: Preserves clone overlap during recovery and data migration. +:Type: Boolean +:Default: ``false`` + + +``osd backfill scan min`` + +:Description: The scan interval in seconds for backfill operations. +:Type: 32-bit Integer +:Default: ``64`` + + +``osd backfill scan max`` + +:Description: The maximum scan interval in seconds for backfill operations. +:Type: 32-bit Integer +:Default: ``512`` + + +``osd backlog thread timeout`` + +:Description: The maximum time in seconds before timing out a backlog thread. +:Type: 32-bit Integer +:Default: ``60*60*1`` + + +``osd recovery thread timeout`` + +:Description: The maximum time in seconds before timing out a recovery thread. +:Type: 32-bit Integer +:Default: ``30`` + + +``osd snap trim thread timeout`` + +:Description: The maximum time in seconds before timing out a snap trim thread. +:Type: 32-bit Integer +:Default: ``60*60*1`` + + +``osd scrub thread timeout`` + +:Description: The maximum time in seconds before timing out a scrub thread. +:Type: 32-bit Integer +:Default: ``60`` + + +``osd scrub finalize thread timeout`` + +:Description: The maximum time in seconds before timing out a scrub finalize thread. +:Type: 32-bit Integer +:Default: 60*10 + + +``osd remove thread timeout`` + +:Description: The maximum time in seconds before timing out a remove OSD thread. +:Type: 32-bit Integer +:Default: 60*60 + + +``osd command thread timeout`` + +:Description: The maximum time in seconds before timing out a command thread. +:Type: 32-bit Integer +:Default: ``10*60`` + + +``osd heartbeat address`` + +:Description: An OSD's network address for heartbeats. +:Type: Address +:Default: The host address. + + +``osd heartbeat interval`` + +:Description: How often an OSD pings its peers (in seconds). +:Type: 32-bit Integer +:Default: ``6`` + + +``osd heartbeat grace`` + +:Description: The elapsed time when an OSD hasn't shown a heartbeat that the cluster considers it ``down``. +:Type: 32-bit Integer +:Default: ``20`` + + +``osd _mon_heartbeat interval`` + +:Description: How often the OSD pings a monitor if it has no OSD peers. +:Type: 32-bit Integer +:Default: ``30`` + + +``osd mon report interval max`` + +:Description: The maximum time in seconds for an OSD to report to a monitor before the monitor considers the OSD ``down``. +:Type: 32-bit Integer +:Default: ``120`` + + +``osd mon report interval min`` + +:Description: The number of minutes between reports that include ``pg stats``, ``up thru``, ``boot`` and ``failures``. +:Type: 32-bit Integer +:Default: ``5`` + + +``osd mon ack timeout`` + +:Description: The number of seconds to wait for a monitor to acknowledge a request for statistics. +:Type: 32-bit Integer +:Default: ``30`` + + +``osd min down reporters`` + +:Description: The minimum number of OSDs required to report a ``down`` OSD. +:Type: 32-bit Integer +:Default: ``1`` + + +``osd min down reports`` + +:Description: The minimum number of times an OSD must report that another is ``down``. +:Type: 32-bit Integer +:Default: ``3`` + + +``osd recovery delay start`` + +:Description: After peering completes, Ceph will delay for the specified number of seconds before starting to recover objects. +:Type: Float +:Default: ``15`` + + +``osd recovery max active`` + +:Description: The number of active recovery requests per OSD at one time. More accelerates recovery, but places an increased load on the cluster. +:Type: 32-bit Integer +:Default: ``5`` + + +``osd recovery max chunk`` + +:Description: The maximum size of a recovered chunk of data to push. +:Type: 64-bit Integer Unsigned +:Default: ``1 << 20`` + + +``osd max scrubs`` + +:Description: The maximum number of scrub operations for an OSD. +:Type: 32-bit Int +:Default: ``1`` + + +``osd scrub load threshold`` + +:Description: The maximum CPU load. Ceph will not scrub when the CPU load is higher than this number. Default is 50%. +:Type: Float +:Default: ``0.5`` + + +``osd scrub min interval`` + +:Description: The maximum interval in seconds for scrubbing the OSD. +:Type: Float +:Default: 5 minutes. ``300`` + + +``osd scrub max interval`` + +:Description: The maximum interval in seconds for scrubbing the OSD. +:Type: Float +:Default: Once per day. ``60*60*24`` + +``osd deep scrub interval`` + +:Description: The interval for "deep" scrubbing (fully reading all data) +:Type: Float +:Default: Once per week. ``60*60*24*7`` + +``osd deep scrub stride`` + +:Description: Read siez when doing a deep scrub +:Type: 32-bit Int +:Default: 512 KB. ``524288`` + +``osd class dir`` + +:Description: The class path for RADOS class plug-ins. +:Type: String +:Default: ``$libdir/rados-classes`` + + +``osd check for log corruption`` + +:Description: Check log files for corruption. Can be computationally expensive. +:Type: Boolean +:Default: ``false`` + + +``osd default notify timeout`` + +:Description: The OSD default notification timeout (in seconds). +:Type: 32-bit Integer Unsigned +:Default: ``30`` + + +``osd min pg log entries`` + +:Description: The minimum number of placement group logs to maintain when trimming log files. +:Type: 32-bit Int Unsigned +:Default: 1000 + + +``osd op complaint time`` + +:Description: An operation becomes complaint worthy after the specified number of seconds have elapsed. +:Type: Float +:Default: ``30`` + + +``osd command max records`` + +:Description: Limits the number of lost objects to return. +:Type: 32-bit Integer +:Default: ``256`` + + +``osd auto upgrade tmap`` + +:Description: Uses ``tmap`` for ``omap`` on old objects. +:Type: Boolean +:Default: ``true`` + + +``osd tmapput sets users tmap`` + +:Description: Uses ``tmap`` for debugging only. +:Type: Boolean +:Default: ``false`` + + +``osd kill backfill at`` + +:Description: For debugging only. +:Type: 32-bit Integer +:Default: ``0`` diff --git a/doc/rados/deployment/ceph-deploy.rst b/doc/rados/deployment/ceph-deploy.rst new file mode 100644 index 0000000000000..989f086e4b327 --- /dev/null +++ b/doc/rados/deployment/ceph-deploy.rst @@ -0,0 +1,5 @@ +============= + Ceph Deploy +============= + +Coming soon. \ No newline at end of file diff --git a/doc/rados/deployment/chef.rst b/doc/rados/deployment/chef.rst new file mode 100644 index 0000000000000..c13ecc89fb73d --- /dev/null +++ b/doc/rados/deployment/chef.rst @@ -0,0 +1,251 @@ +===================== + Deploying with Chef +===================== + +We use Chef cookbooks to deploy Ceph. See `Managing Cookbooks with Knife`_ for details +on using ``knife``. For Chef installation instructions, see `Installing Chef`_. + +.. _clonecbs: + +Clone the Required Cookbooks +============================ + +To get the cookbooks for Ceph, clone them from git.:: + + cd ~/chef-cookbooks + git clone https://github.com/opscode-cookbooks/apache2.git + git clone https://github.com/ceph/ceph-cookbooks.git ceph + +.. _addcbpaths: + +Add the Required Cookbook Paths +=============================== + +If you added a default cookbook path when you installed Chef, ``knife`` +may be able to upload the cookbook you've cloned to your cookbook path +directory without further configuration. If you used a different path, +or if the cookbook repository you cloned has a different tree structure, +add the required cookbook path to your ``knife.rb`` file. The +``cookbook_path`` setting takes a string or an array of strings. +For example, you can replace a string path with an array of string paths:: + + cookbook_path '/home/{user-name}/chef-cookbooks/' + +Becomes:: + + cookbook_path [ + '/home/{user-name}/chef-cookbooks/', + '/home/{user-name}/chef-cookbooks/{another-directory}/', + '/some/other/path/to/cookbooks/' + ] + +.. _installcbs: + +Install the Cookbooks +===================== + +To install Ceph, you must upload the Ceph cookbooks and the Apache cookbooks +(for use with RADOSGW) to your Chef server. :: + + knife cookbook upload apache2 ceph + +.. _configcephenv: + +Configure your Ceph Environment +=============================== + +The Chef server can support installation of software for multiple environments. +The environment you create for Ceph requires an ``fsid``, the secret for +your monitor(s) if you are running Ceph with ``cephx`` authentication, and +the host name (i.e., short name) for your monitor hosts. + +.. tip: Open an empty text file to hold the following values until you create + your Ceph environment. + +For the filesystem ID, use ``uuidgen`` from the ``uuid-runtime`` package to +generate a unique identifier. :: + + uuidgen -r + +For the monitor(s) secret(s), use ``ceph-authtool`` to generate the secret(s):: + + sudo apt-get update + sudo apt-get install ceph-common + ceph-authtool /dev/stdout --name=mon. --gen-key + +The secret is the value to the right of ``"key ="``, and should look something +like this:: + + AQBAMuJPINJgFhAAziXIrLvTvAz4PRo5IK/Log== + +To create an environment for Ceph, set a command line editor. For example:: + + export EDITOR=vim + +Then, use ``knife`` to create an environment. :: + + knife environment create {env-name} + +For example:: + + knife environment create Ceph + +A JSON file will appear. Perform the following steps: + +#. Enter a description for the environment. +#. In ``"default_attributes": {}``, add ``"ceph" : {}``. +#. Within ``"ceph" : {}``, add ``"monitor-secret":``. +#. Immediately following ``"monitor-secret":`` add the key you generated within quotes, followed by a comma. +#. Within ``"ceph":{}`` and following the ``monitor-secret`` key-value pair, add ``"config": {}`` +#. Within ``"config": {}`` add ``"fsid":``. +#. Immediately following ``"fsid":``, add the unique identifier you generated within quotes, followed by a comma. +#. Within ``"config": {}`` and following the ``fsid`` key-value pair, add ``"mon_initial_members":`` +#. Immediately following ``"mon_initial_members":``, enter the initial monitor host names. + +For example:: + + "default_attributes" : { + "ceph": { + "monitor-secret": "{replace-with-generated-secret}", + "config": { + "fsid": "{replace-with-generated-uuid}", + "mon_initial_members": "{replace-with-monitor-hostname(s)}" + } + } + } + +Advanced users (i.e., developers and QA) may also add ``"ceph_branch": "{branch}"`` +to ``default-attributes``, replacing ``{branch}`` with the name of the branch you +wish to use (e.g., ``master``). + +.. configroles: + +Configure the Roles +=================== + +Navigate to the Ceph cookbooks directory. :: + + cd ~/chef-cookbooks/ceph + +Create roles for OSDs, monitors, metadata servers, and RADOS Gateways from +their respective role files. :: + + knife role from file roles/ceph-osd.rb + knife role from file roles/ceph-mon.rb + knife role from file roles/ceph-mds.rb + knife role from file roles/ceph-radosgw.rb + +.. _confignodes: + +Configure Nodes +=============== + +You must configure each node you intend to include in your Ceph cluster. +Identify nodes for your Ceph cluster. :: + + knife node list + +.. note: for each host where you installed Chef and executed ``chef-client``, + the Chef server should have a minimal node configuration. You can create + additional nodes with ``knife node create {node-name}``. + +For each node you intend to use in your Ceph cluster, configure the node +as follows:: + + knife node edit {node-name} + +The node configuration should appear in your text editor. Change the +``chef_environment`` value to ``Ceph`` (or whatever name you set for your +Ceph environment). + +In the ``run_list``, add ``"recipe[ceph::apt]",`` to all nodes as +the first setting, so that Chef can install or update the necessary packages. +Then, add at least one of:: + + "role[ceph-mon]" + "role[ceph-osd]" + "role[ceph-mds]" + "role[ceph-radosgw]" + +If you add more than one role, separate them with a comma. Run ``hostname`` +on your command line, and replace the ``{hostname}`` setting of the ``name`` +key to the host name for the node. :: + + { + "chef_environment": "Ceph", + "name": "{hostname}", + "normal": { + "tags": [ + + ] + }, + "run_list": [ + "recipe[ceph::apt]", + "role[ceph-mon]", + "role[ceph-mds]" + ] + } + +.. _prepdisks: + +Prepare OSD Disks +================= + +Configuring a node with an OSD role tells Chef that the node will run at +least one OSD. However, you may run many OSDs on one host. For example, +you may run one ``ceph-osd`` daemon for each data disk on the system. +This step prepares the OSD disk(s) and tells Chef how many OSDs the +node will be running. + + +For the Ceph 0.48 Argonaut release, install ``gdisk``:: + + sudo apt-get install gdisk + +For the Ceph 0.48 Argonaut release, on each hard disk that will store data for +an OSD daemon, configure the hard disk for use with Ceph. Replace ``{fsid}`` +with the UUID you generated while using ``uuidgen -r``. + +.. important: This procedure will erase all information in ``/dev/{disk}``. + +:: + + sudo sgdisk /dev/{disk} --zap-all --clear --mbrtogpt --largest-new=1 --change-name=1:'ceph data' --typecode=1:{fsid} + +Create a file system and allocate the disk to your cluster. Specify a +filesystem (e.g., ``ext4``, ``xfs``, ``btrfs``). When you execute +``ceph-disk-prepare``, remember to replace ``{fsid}`` with the UUID you +generated while using ``uuidgen -r``:: + + sudo mkfs -t ext4 /dev/{disk} + sudo mount -o user_xattr /dev/{disk} /mnt + sudo ceph-disk-prepare --cluster-uuid={fsid} /mnt + sudo umount /mnt + +Finally, simulate a hotplug event. :: + + sudo udevadm trigger --subsystem-match=block --action=add + + +.. _runchefclient: + +Run ``chef-client`` on each Node +================================ + +Once you have completed the preceding steps, you must run ``chef-client`` +on each node. For example:: + + sudo chef-client + +.. _proceedtoops: + +Proceed to Operating the Cluster +================================ + +Once you complete the deployment, you may begin operating your cluster. +See `Operating a Cluster`_ for details. + + +.. _Managing Cookbooks with Knife: http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife +.. _Installing Chef: ../../install/chef +.. _Operating a Cluster: ../../init/ diff --git a/doc/rados/deployment/index.rst b/doc/rados/deployment/index.rst new file mode 100644 index 0000000000000..997dc2f9a0a96 --- /dev/null +++ b/doc/rados/deployment/index.rst @@ -0,0 +1,35 @@ +================= + Ceph Deployment +================= + +You can deploy Chef using many different deployment systems including Chef, Juju, +Puppet, and Crowbar. If you are just experimenting, Ceph provides some minimal +deployment tools that rely only on SSH and DNS to deploy Ceph. You need to set +up the SSH and DNS settings manually. + + +.. raw:: html + +

Ceph Deployment Scripts

+ +We provide light-weight deployment scripts to help you evaluate Ceph. For +professional deployment, you should consider professional deployment systems +such as Juju, Puppet, Chef or Crowbar. + +.. toctree:: + + mkcephfs (Deprecated) + Ceph Deploy + +.. raw:: html + +

OpsCode Chef

+ +.. toctree:: + + Installing Chef + Deploying with Chef + +.. raw:: html + +
diff --git a/doc/rados/deployment/install-chef.rst b/doc/rados/deployment/install-chef.rst new file mode 100644 index 0000000000000..5989c56c4ff66 --- /dev/null +++ b/doc/rados/deployment/install-chef.rst @@ -0,0 +1,286 @@ +================= + Installing Chef +================= + +Chef defines three types of entities: + +#. **Chef Nodes:** Run ``chef-client``, which installs and manages software. +#. **Chef Server:** Interacts with ``chef-client`` on Chef nodes. +#. **Chef Workstation:** Manages the Chef server. + +See `Chef Architecture Introduction`_ for details. + +.. _createuser: + +Create a ``chef`` User +====================== + +The ``chef-client`` command requires the proper privileges to install and manage +installations. On each Chef node, we recommend creating a ``chef`` user with +full ``root`` privileges. For example:: + + ssh user@chef-node + sudo useradd -d /home/chef -m chef + sudo passwd chef + +To provide full privileges, add the following to ``/etc/sudoers.d/chef``. :: + + echo "chef ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/chef + sudo chmod 0440 /etc/sudoers.d/chef + +If you are using a version of ``sudo`` that doesn't support includes, you will +need to add the following to the ``/etc/sudoers`` file:: + + chef ALL = (root) NOPASSWD:ALL + +.. important:: Do not change the file permissions on ``/etc/sudoers``. Use a + suitable tool such as ``visudo``. + +.. _genkeys: + +Generate SSH Keys for Chef Clients +================================== + +Chef's ``knife`` tool can run ``ssh``. To streamline deployments, we +recommend generating an SSH key pair without a passphrase for your +Chef nodes and copying the public key(s) to your Chef nodes so that you +can connect to them from your workstation using ``ssh`` from ``knife`` +without having to provide a password. To generate a key pair without +a passphrase, execute the following on your Chef workstation. :: + + ssh-keygen + Generating public/private key pair. + Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /ceph-admin/.ssh/id_rsa. + Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub. + +You may use RSA or DSA keys. Once you generate your keys, copy them to each +OSD host. For example:: + + ssh-copy-id chef@your-node + +Consider modifying your ``~/.ssh/config`` file so that it defaults to +logging in as ``chef`` when no username is specified. :: + + Host myserver01 + Hostname myserver01.fqdn-or-ip-address.com + User chef + Host myserver02 + Hostname myserver02.fqdn-or-ip-address.com + User chef + +.. _installruby: + +Installing Ruby +=============== + +Chef requires you to install Ruby. Use the version applicable to your current +Linux distribution and install Ruby on all of your hosts. :: + + sudo apt-get update + sudo apt-get install ruby + +.. _installchefserver: + +Installing Chef and Chef Server on a Server +=========================================== + +If you plan on hosting your `Chef Server at Opscode`_ you may skip this step, +but you must make a note of the the fully qualified domain name or IP address +of your Chef Server for ``knife`` and ``chef-client``. + +First, add Opscode packages to your APT configuration. For example:: + + sudo tee /etc/apt/sources.list.d/chef.list << EOF + deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main + deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main + EOF + +Next, you must request keys so that APT can verify the packages. Copy +and paste the following line into your command line:: + + sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A + +The key is only used by ``apt``, so remove it from the ``root`` keyring by +typing ``Y`` when prompted to delete it. + +Install the Opscode keyring, Chef and Chef server on the host designated +as your Chef Server. :: + + sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef chef-server + +Enter the fully qualified domain name or IP address for your Chef server. For example:: + + http://fqdn-or-ip-address.com:4000 + +The Chef server installer will prompt you to enter a temporary password. Enter +a temporary password (*e.g.,* ``foo``) and proceed with the installation. + +.. tip:: When prompted for a temporary password, you may press **OK**. + The installer wants you to re-enter the password to confirm it. To + re-enter the password, you must press the **ESC** key. + +Once the installer finishes and activates the Chef server, you may enter the +fully qualified domain name or IP address in a browser to launch the +Chef web UI. For example:: + + http://fqdn-or-ip-address.com:4000 + +The Chef web UI will prompt you to enter the username and password. + +- **login:** ``admin`` +- **password:** ``foo`` + +Once you have entered the temporary password, the Chef web UI will prompt you +to enter a new password. + +.. _installchef: + +Install Chef on all Remaining Hosts +=================================== + +Install Chef on all Chef Nodes and on the Chef Workstation (if it is not the +same host as the Chef Server). See `Installing Chef Client on Ubuntu or Debian`_ +for details. + +First, add Opscode packages to your APT configuration. For example:: + + sudo tee /etc/apt/sources.list.d/chef.list << EOF + deb http://apt.opscode.com/ $(lsb_release -cs)-0.10 main + deb-src http://apt.opscode.com/ $(lsb_release -cs)-0.10 main + EOF + +Next, you must request keys so that APT can verify the packages. Copy +and paste the following line into your command line:: + + sudo touch /etc/apt/trusted.gpg.d/opscode-keyring.gpg && sudo gpg --fetch-key http://apt.opscode.com/packages@opscode.com.gpg.key && sudo gpg --export 83EF826A | sudo apt-key --keyring /etc/apt/trusted.gpg.d/opscode-keyring.gpg add - && sudo gpg --yes --delete-key 83EF826A + +The key is only used by ``apt``, so remove it from the ``root`` keyring by +typing ``Y`` when prompted to delete it. + +Install the Opscode keyring and Chef on all hosts other than the Chef Server. :: + + sudo apt-get update && sudo apt-get upgrade && sudo apt-get install opscode-keyring chef + +Enter the fully qualified domain name or IP address for your Chef server. +For example:: + + http://fqdn-or-ip-address.com:4000 + +.. _configknife: + +Configuring Knife +================= + +Once you complete the Chef server installation, install ``knife`` on the your +Chef Workstation. If the Chef server is a remote host, use ``ssh`` to connect. :: + + ssh chef@fqdn-or-ip-address.com + +In the ``/home/chef`` directory, create a hidden Chef directory. :: + + mkdir -p ~/.chef + +The server generates validation and web UI certificates with read/write +permissions for the user that installed the Chef server. Copy them from the +``/etc/chef`` directory to the ``~/.chef`` directory. Then, change their +ownership to the current user. :: + + sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef && sudo chown $(id -u):$(id -g) ~/.chef/*.pem + +From the current user's home directory, configure ``knife`` with an initial +API client. :: + + knife configure -i + +The configuration will prompt you for inputs. Answer accordingly: + +*Where should I put the config file? [~/.chef/knife.rb]* Press **Enter** +to accept the default value. + +*Please enter the chef server URL:* If you are installing the +client on the same host as the server, enter ``http://localhost:4000``. +Otherwise, enter an appropriate URL for the server. + +*Please enter a clientname for the new client:* Press **Enter** +to accept the default value. + +*Please enter the existing admin clientname:* Press **Enter** +to accept the default value. + +*Please enter the location of the existing admin client's private key:* +Override the default value so that it points to the ``.chef`` directory. +(*e.g.,* ``/home/chef/.chef/webui.pem``) + +*Please enter the validation clientname:* Press **Enter** to accept +the default value. + +*Please enter the location of the validation key:* Override the +default value so that it points to the ``.chef`` directory. +(*e.g.,* ``/home/chef/.chef/validation.pem``) + +*Please enter the path to a chef repository (or leave blank):* +Leave the entry field blank and press **Enter**. + +.. _addcbpath: + +Add a Cookbook Path +=================== + +Add ``cookbook_path`` to the ``~/.chef/knife.rb`` configuration file +on your Chef workstation. For example:: + + cookbook_path '/home/{user-name}/chef-cookbooks/' + +Then create the path if it doesn't already exist. :: + + mkdir /home/{user-name}/chef-cookbooks + +This is where you will store local copies of cookbooks before uploading +them to the Chef server. + +.. _cpvalpem: + +Copy ``validation.pem`` to Nodes +================================ + +Copy the ``/etc/chef/validation.pem`` file from your Chef server to +each Chef Node. In a command line shell on the Chef Server, for each node, +replace ``{nodename}`` in the following line with the node's host name and +execute it. :: + + sudo cat /etc/chef/validation.pem | ssh {nodename} "exec sudo tee /etc/chef/validation.pem >/dev/null" + +.. _runchefcli: + +Run ``chef-client`` on each Chef Node +===================================== + +Run the ``chef-client`` on each Chef Node so that the nodes +register with the Chef server. :: + + ssh chef-node + sudo chef-client + +.. _verifynodes: + +Verify Nodes +============ + +Verify that you have setup all the hosts you want to use as +Chef nodes. :: + + knife node list + +A list of the nodes you've configured should appear. + +See the `Deploy With Chef <../../config-cluster/chef>`_ section for information +on using Chef to deploy your Ceph cluster. + +.. _Chef Architecture Introduction: http://wiki.opscode.com/display/chef/Architecture+Introduction +.. _Chef Server at Opscode: http://www.opscode.com/hosted-chef/ +.. _Installing Chef Client on Ubuntu or Debian: http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Ubuntu+or+Debian +.. _Installing Chef Server on Debian or Ubuntu using Packages: http://wiki.opscode.com/display/chef/Installing+Chef+Server+on+Debian+or+Ubuntu+using+Packages +.. _Knife Bootstrap: http://wiki.opscode.com/display/chef/Knife+Bootstrap \ No newline at end of file diff --git a/doc/rados/deployment/mkcephfs.rst b/doc/rados/deployment/mkcephfs.rst new file mode 100644 index 0000000000000..721ae43500ca1 --- /dev/null +++ b/doc/rados/deployment/mkcephfs.rst @@ -0,0 +1,123 @@ +============================= + Deploying with ``mkcephfs`` +============================= + +To deploy a test or development cluster, you can use the ``mkcephfs`` tool. +We do not recommend using this tool for production environments. + + +Enable Login to Cluster Hosts as ``root`` +========================================= + +To deploy with ``mkcephfs``, you will need to be able to login as ``root`` +on each host without a password. For each host, perform the following:: + + sudo passwd root + +Enter a password for the root user. + +On the admin host, generate an ``ssh`` key without specifying a passphrase +and use the default locations. :: + + ssh-keygen + Generating public/private key pair. + Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /ceph-admin/.ssh/id_rsa. + Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub. + +You may use RSA or DSA keys. Once you generate your keys, copy them to each +OSD host. For example:: + + ssh-copy-id root@myserver01 + ssh-copy-id root@myserver02 + +Modify your ``~/.ssh/config`` file to login as ``root``, as follows:: + + Host myserver01 + Hostname myserver01.fully-qualified-domain.com + User root + Host myserver02 + Hostname myserver02.fully-qualified-domain.com + User root + + +Copy Configuration File to All Hosts +==================================== + +Ceph's ``mkcephfs`` deployment script does not copy the configuration file you +created from the Administration host to the OSD Cluster hosts. Copy the +configuration file you created (*i.e.,* ``mycluster.conf`` in the example below) +from the Administration host to ``etc/ceph/ceph.conf`` on each OSD Cluster host +if you are using ``mkcephfs`` to deploy Ceph. + +:: + + ssh myserver01 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf + ssh myserver02 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf + ssh myserver03 sudo tee /etc/ceph/ceph.conf < /etc/ceph/ceph.conf + + +Create the Default Directories +============================== + +The ``mkcephfs`` deployment script does not create the default server +directories. Create server directories for each instance of a Ceph daemon (if +you haven't done so already). The ``host`` variables in the ``ceph.conf`` file +determine which host runs each instance of a Ceph daemon. Using the exemplary +``ceph.conf`` file, you would perform the following: + +On ``myserver01``:: + + sudo mkdir /var/lib/ceph/osd/ceph-0 + sudo mkdir /var/lib/ceph/mon/ceph-a + +On ``myserver02``:: + + sudo mkdir /var/lib/ceph/osd/ceph-1 + sudo mkdir /var/lib/ceph/mon/ceph-b + +On ``myserver03``:: + + sudo mkdir /var/lib/ceph/osd/ceph-2 + sudo mkdir /var/lib/ceph/mon/ceph-c + sudo mkdir /var/lib/ceph/mds/ceph-a + + +Mount Disks to the Data Directories +=================================== + +If you are running multiple OSDs per host and one hard disk per OSD, you should +mount the disk under the OSD data directory (if you haven't done so already). + + +Run ``mkcephfs`` +================ + +Once you have copied your Ceph Configuration to the OSD Cluster hosts +and created the default directories, you may deploy Ceph with the +``mkcephfs`` script. + +.. note:: ``mkcephfs`` is a quick bootstrapping tool. It does not handle more + complex operations, such as upgrades. + +For production environments, deploy Ceph using Chef cookbooks. To run +``mkcephfs``, execute the following:: + + cd /etc/ceph + sudo mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring + +The script adds an admin key to the ``ceph.keyring``, which is analogous to a +root password. See `Authentication`_ when running with ``cephx`` enabled. + +When you start or stop your cluster, you will not have to use ``sudo`` or +provide passwords. For example:: + + service ceph -a start + +See `Operating a Cluster`_ for details. + + +.. _Authentication: ../authentication +.. _Operating a Cluster: ../../init/ \ No newline at end of file diff --git a/doc/rados/index.rst b/doc/rados/index.rst new file mode 100644 index 0000000000000..8e858b1caf4bd --- /dev/null +++ b/doc/rados/index.rst @@ -0,0 +1,66 @@ +==================== + RADOS Object Store +==================== + +Ceph's :abbr:`RADOS (Reliable Autonomic Distributed Object Store)` Object Store +is the foundation for all Ceph clusters. When you use object store clients such +as the CephFS filesystem, the RESTful Gateway or Ceph block devices, Ceph reads +data from and writes data to the object store. Ceph's RADOS Object Stores +consist of two types of daemons: Object Storage Daemons (OSDs) store data as +objects on storage nodes; and Monitors maintain a master copy of the cluster +map. A Ceph cluster may contain thousands of storage nodes. A minimal system +will have at least two OSDs for data replication. + +.. ditaa:: +---------------+ +---------------+ + | OSDs | | Monitor | + +---------------+ +---------------+ + +.. raw:: html + +

Config and Deploy

+ +Once you have installed Ceph packages, you must configure. There are a a few +required settings, but most configuration settings have default values. +Following the initial configuration, you must deploy Ceph. Deployment consists +of creating and initializing data directories, keys, etc. + +.. toctree:: + :maxdepth: 2 + + Configuration + Deployment + +.. raw:: html + +

Operations

+ +Once you have a deployed Ceph cluster, you may begin operating your cluster. + +.. toctree:: + :maxdepth: 2 + + + Operations + Manual Pages + + +.. raw:: html + +

APIs

+ +Most Ceph deployments use Ceph `block devices`_, the `gateway`_ and/or the +`CephFS filesystem`_. You may also develop applications that talk directly to +the Ceph object store. + +.. toctree:: + :maxdepth: 2 + + APIs + +.. raw:: html + +
+ +.. _block devices: ../rbd/rbd +.. _CephFS filesystem: ../cephfs/ +.. _gateway: ../radosgw/ diff --git a/doc/rados/man/index.rst b/doc/rados/man/index.rst new file mode 100644 index 0000000000000..82645287f51c0 --- /dev/null +++ b/doc/rados/man/index.rst @@ -0,0 +1,22 @@ +======================= + Object Store Manpages +======================= + +.. toctree:: + :maxdepth: 1 + + ../../man/8/ceph.rst + ../../man/8/ceph-authtool.rst + ../../man/8/ceph-clsinfo.rst + ../../man/8/ceph-conf.rst + ../../man/8/ceph-debugpack.rst + ../../man/8/ceph-dencoder.rst + ../../man/8/ceph-mon.rst + ../../man/8/ceph-osd.rst + ../../man/8/ceph-run.rst + ../../man/8/ceph-syn.rst + ../../man/8/crushtool.rst + ../../man/8/librados-config.rst + ../../man/8/monmaptool.rst + ../../man/8/osdmaptool.rst + ../../man/8/rados.rst \ No newline at end of file diff --git a/doc/rados/operations/auth-intro.rst b/doc/rados/operations/auth-intro.rst index ca8277e69d8a7..30e015de6c597 100644 --- a/doc/rados/operations/auth-intro.rst +++ b/doc/rados/operations/auth-intro.rst @@ -39,7 +39,7 @@ use Ceph clients to interact with Ceph server daemons. For additional information, see our `Cephx Guide`_ and `ceph-authtool manpage`_. .. _Cephx Guide: ../authentication -.. _ceph-authtool manpage: ../../man/8/ceph-authtool/ +.. _ceph-authtool manpage: ../../../man/8/ceph-authtool Ceph Authentication (cephx) =========================== diff --git a/doc/rados/operations/authentication.rst b/doc/rados/operations/authentication.rst index 02fae3f323363..74330aed1a6d6 100644 --- a/doc/rados/operations/authentication.rst +++ b/doc/rados/operations/authentication.rst @@ -17,7 +17,7 @@ do not need to generate the keys again. For additional information, see our `Cephx Intro`_ and `ceph-authtool manpage`_. .. _Cephx Intro: ../auth-intro -.. _ceph-authtool manpage: ../../man/8/ceph-authtool/ +.. _ceph-authtool manpage: ../../../man/8/ceph-authtool Configuring Cephx @@ -70,7 +70,7 @@ following directory:: See `Enabling Cephx`_ step 2 and 3 for stepwise details to enable ``cephx``. -.. _ceph-authtool: ../../man/8/ceph-authtool/ +.. _ceph-authtool: ../../man/ceph-authtool/ .. _enable-cephx: @@ -382,4 +382,4 @@ for authentication:: foregoing flag** at the nearest practical time so that you may avail yourself of the enhanced authentication. -.. _Ceph configuration: ../../config-cluster/ceph-conf +.. _Ceph configuration: ../../configuration/ceph-conf diff --git a/doc/rados/operations/debug.rst b/doc/rados/operations/debug.rst index 66ecbd011e25b..caa0e0f664db9 100644 --- a/doc/rados/operations/debug.rst +++ b/doc/rados/operations/debug.rst @@ -9,8 +9,8 @@ using Ceph's debugging and logging. To activate and configure Ceph's debug logging, refer to `Ceph Logging and Debugging`_. For additional logging settings, refer to the `Logging and Debugging Config Reference`_. -.. _Ceph Logging and Debugging: ../../config-cluster/ceph-conf#ceph-logging-and-debugging -.. _Logging and Debugging Config Reference: ../../config-cluster/log-and-debug-ref +.. _Ceph Logging and Debugging: ../../configuration/ceph-conf#ceph-logging-and-debugging +.. _Logging and Debugging Config Reference: ../../configuration/log-and-debug-ref You can change the logging settings at runtime so that you don't have to stop and restart the cluster. Refer to `Ceph Configuration - Runtime Changes`_ @@ -22,4 +22,4 @@ Valgrind. You should only use Valgrind when developing or debugging Ceph. Valgrind is computationally expensive, and will slow down your system otherwise. Valgrind messages are logged to ``stderr``. -.. _Ceph Configuration - Runtime Changes: ../../config-cluster/ceph-conf#ceph-runtime-config +.. _Ceph Configuration - Runtime Changes: ../../configuration/ceph-conf#ceph-runtime-config diff --git a/doc/radosgw/index.rst b/doc/radosgw/index.rst index dd28676b9a989..d9c2e3579bd37 100644 --- a/doc/radosgw/index.rst +++ b/doc/radosgw/index.rst @@ -43,4 +43,6 @@ one API and retrieve it with the other. Swift API Admin API troubleshooting + Manpage radosgw <../../man/8/radosgw> + Manpage radosgw-admin <../../man/8/radosgw-admin> diff --git a/doc/rbd/disk.conf b/doc/rbd/disk.conf new file mode 100644 index 0000000000000..5dba7b64460fb --- /dev/null +++ b/doc/rbd/disk.conf @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/doc/rbd/qemu-rbd.rst b/doc/rbd/qemu-rbd.rst index 74d412277b2f9..d111b39d3edb6 100644 --- a/doc/rbd/qemu-rbd.rst +++ b/doc/rbd/qemu-rbd.rst @@ -98,7 +98,7 @@ configuration (like any Ceph configuration option) as part of the qemu -m 1024 -drive format=raw,file=rbd:data/squeeze:rbd_cache=true -.. _RBD caching: ../../config-cluster/rbd-config-ref/#rbd-cache-config-settings +.. _RBD caching: ../rbd-config-ref/#rbd-cache-config-settings Enabling Discard/TRIM @@ -147,4 +147,4 @@ QEMU command line settings override the Ceph configuration file settings. .. _QEMU Open Source Processor Emulator: http://wiki.qemu.org/Main_Page .. _QEMU Manual: http://wiki.qemu.org/Manual -.. _RBD Cache: ../../config-cluster/rbd-config-ref/ \ No newline at end of file +.. _RBD Cache: ../rbd-config-ref/ \ No newline at end of file diff --git a/doc/rbd/rbd-config-ref.rst b/doc/rbd/rbd-config-ref.rst new file mode 100644 index 0000000000000..dde3dedf257a3 --- /dev/null +++ b/doc/rbd/rbd-config-ref.rst @@ -0,0 +1,89 @@ +======================= + librbd Cache Settings +======================= + +See `Block Device`_ for additional details. + +.. sidebar:: Kernel Caching + + The kernel driver for Ceph block devices can use the Linux page cache to + improve performance. + +The user space implementation of the Ceph block device (i.e., ``librbd``) cannot +take advantage of the Linux page cache, so it includes its own in-memory +caching, called "RBD caching." RBD caching behaves just like well-behaved hard +disk caching. When the OS sends a barrier or a flush request, all dirty data is +written to the OSDs. This means that using write-back caching is just as safe as +using a well-behaved physical hard disk with a VM that properly sends flushes +(i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU) +algorithm, and in write-back mode it can coalesce contiguous requests for +better throughput. + +.. versionadded:: 0.46 + +Ceph supports write-back caching for RBD. To enable it, add ``rbd cache = +true`` to the ``[client]`` section of your ``ceph.conf`` file. By default +``librbd`` does not perform any caching. Writes and reads go directly to the +storage cluster, and writes return only when the data is on disk on all +replicas. With caching enabled, writes return immediately, unless there are more +than ``rbd cache max dirty`` unflushed bytes. In this case, the write triggers +writeback and blocks until enough bytes are flushed. + +.. versionadded:: 0.47 + +Ceph supports write-through caching for RBD. You can set the size of +the cache, and you can set targets and limits to switch from +write-back caching to write through caching. To enable write-through +mode, set ``rbd cache max dirty`` to 0. This means writes return only +when the data is on disk on all replicas, but reads may come from the +cache. The cache is in memory on the client, and each RBD image has +its own. Since the cache is local to the client, there's no coherency +if there are others accesing the image. Running GFS or OCFS on top of +RBD will not work with caching enabled. + +The ``ceph.conf`` file settings for RBD should be set in the ``[client]`` +section of your configuration file. The settings include: + + +``rbd cache`` + +:Description: Enable caching for RADOS Block Device (RBD). +:Type: Boolean +:Required: No +:Default: ``false`` + + +``rbd cache size`` + +:Description: The RBD cache size in bytes. +:Type: 64-bit Integer +:Required: No +:Default: ``32 MiB`` + + +``rbd cache max dirty`` + +:Description: The ``dirty`` limit in bytes at which the cache triggers write-back. If ``0``, uses write-through caching. +:Type: 64-bit Integer +:Required: No +:Constraint: Must be less than ``rbd cache size``. +:Default: ``24 MiB`` + + +``rbd cache target dirty`` + +:Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache. +:Type: 64-bit Integer +:Required: No +:Constraint: Must be less than ``rbd cache max dirty``. +:Default: ``16 MiB`` + + +``rbd cache max dirty age`` + +:Description: The number of seconds dirty data is in the cache before writeback starts. +:Type: Float +:Required: No +:Default: ``1.0`` + +.. _Block Device: ../../rbd/rbd/ \ No newline at end of file diff --git a/doc/rbd/rbd.rst b/doc/rbd/rbd.rst index 59c7bde7f0307..b9cb49e071fb2 100644 --- a/doc/rbd/rbd.rst +++ b/doc/rbd/rbd.rst @@ -43,9 +43,12 @@ devices simultaneously. Snapshots QEMU libvirt - Cache Settings <../../config-cluster/rbd-config-ref/> + Cache Settings OpenStack CloudStack + Manpage rbd <../../man/8/rbd> + Manpage ceph-rbdnamer <../../man/8/ceph-rbdnamer> + .. _RBD Caching: ../../config-cluster/rbd-config-ref/ .. _kernel modules: ../rbd-ko/