OSD Config Reference
======================
+You can configure OSDs in the Ceph configuration file, but OSDs can use the
+default values and a very minimal configuration. A minimal OSD configuration
+sets ``osd journal size`` and ``osd host``, and uses default values for
+nearly everything else.
+
+
+General Settings
+================
+
+The following settings provide an OSD's ID, and determine paths to data and
+journals. Ceph deployment scripts typically generate the UUID automatically. We
+**DO NOT** recommend changing the default paths for data or journals, as it
+makes it more problematic to troubleshoot Ceph later.
+
+The journal size should be at least twice the product of the expected drive
+speed multiplied by ``filestore max sync interval``. However, the most common
+practice is to partition the journal drive (often an SSD), and mount it such
+that Ceph uses the entire partition for the journal.
``osd uuid``
:Description: The universally unique identifier (UUID) for the OSD.
:Type: UUID
-:Default: None
+:Default: The UUID.
``osd data``
-:Description: The path to the OSDs data. You must create the directory. You should mount a data disk at this mount point. We do not recommend changing the default.
+:Description: The path to the OSDs data. You must create the directory when
+ deploying Ceph. You should mount a drive for OSD data at this
+ mount point. We do not recommend changing the default.
+
:Type: String
:Default: ``/var/lib/ceph/osd/$cluster-$id``
``osd journal``
-:Description: The path to the OSD's journal. This may be a path to a file or a block device (such as a partition of an SSD). If it is a file, you must create the directory to contain it.
+:Description: The path to the OSD's journal. This may be a path to a file or a
+ block device (such as a partition of an SSD). If it is a file,
+ you must create the directory to contain it. We recommend using a
+ drive separate from the ``osd data`` drive.
+
:Type: String
:Default: ``/var/lib/ceph/osd/$cluster-$id/journal``
``osd journal size``
-:Description: The size of the journal in megabytes. If this is 0, and the journal is a block device, the entire block device is used. Since v0.54, this is ignored if the journal is a block device, and the entire block device is used.
+:Description: The size of the journal in megabytes. If this is 0, and the
+ journal is a block device, the entire block device is used.
+ Since v0.54, this is ignored if the journal is a block device,
+ and the entire block device is used.
+
:Type: 32-bit Integer
-:Default: ``1024``
-:Recommended: Begin with 1GB. Should be at least twice the product of the expected speed multiplied by ``filestore max sync interval``.
+:Default: ``5120``
+:Recommended: Begin with 1GB. Should be at least twice the product of the
+ expected speed multiplied by ``filestore max sync interval``.
``osd max write size``
:Default: 500MB default. ``500*1024L*1024L``
-``osd stat refresh interval``
+``osd class dir``
-:Description: The status refresh interval in seconds.
-:Type: 64-bit Integer Unsigned
-:Default: ``.5``
+:Description: The class path for RADOS class plug-ins.
+:Type: String
+:Default: ``$libdir/rados-classes``
-``osd pg bits``
+Monitor OSD Interaction
+=======================
-:Description: Placement group bits per OSD.
+OSDs check each other's heartbeats and report to monitors periodically. Ceph can
+use default values in many cases. However, if your network has latency issues,
+you may need to adopt longer intervals. See `Configuring Monitor/OSD
+Interaction`_ for a detailed discussion of heartbeats.
+
+
+``osd heartbeat address``
+
+:Description: An OSD's network address for heartbeats.
+:Type: Address
+:Default: The host address.
+
+
+``osd heartbeat interval``
+
+:Description: How often an OSD pings its peers (in seconds).
:Type: 32-bit Integer
-:Default: ``6``
+:Default: ``6``
-``osd pgp bits``
+``osd heartbeat grace``
-:Description: The number of bits per OSD for PGPs.
+:Description: The elapsed time when an OSD hasn't shown a heartbeat that the
+ cluster considers it ``down``.
+
:Type: 32-bit Integer
-:Default: ``4``
+:Default: ``20``
-``osd pg layout``
+``osd mon heartbeat interval``
-:Description: Placement group layout.
+:Description: How often the OSD pings a monitor if it has no OSD peers.
:Type: 32-bit Integer
-:Default: ``2``
+:Default: ``30``
-``osd pool default crush rule``
+``osd mon report interval max``
+
+:Description: The maximum time in seconds for an OSD to report to a monitor
+ before the monitor considers the OSD ``down``.
-:Description: The default CRUSH rule to use when creating a pool.
:Type: 32-bit Integer
-:Default: ``0``
+:Default: ``120``
-``osd pool default size``
+``osd mon report interval min``
+
+:Description: The minimum number of seconds for an OSD to report to a monitor
+ to avoid the monitor considering the OSD ``down``.
-:Description: The default size of an OSD pool in gigabytes. The default value is the same as ``--size 2`` with ``mkpool``.
:Type: 32-bit Integer
-:Default: ``2``
+:Default: ``5``
+:Valid Range: Should be less than ``osd mon report interval max``
-``osd pool default pg num``
+``osd mon ack timeout``
+
+:Description: The number of seconds to wait for a monitor to acknowledge a
+ request for statistics.
-:Description: The default number of placement groups for a pool. The default value is the same as ``pg_num`` with ``mkpool``.
:Type: 32-bit Integer
-:Default: ``8``
+:Default: ``30``
-``osd pool default pgp num``
+``osd min down reporters``
-:Description: The default number of placement groups for placement for a pool. The default value is the same as ``pgp_num`` with ``mkpool``. PG and PGP should be equal (for now).
+:Description: The minimum number of OSDs required to report a ``down`` OSD.
:Type: 32-bit Integer
-:Default: ``8``
+:Default: ``1``
-``osd map dedup``
+``osd min down reports``
-:Description: Enable removing duplicates in the OSD map.
-:Type: Boolean
-:Default: ``true``
+:Description: The minimum number of times an OSD must report that another
+ is ``down``.
+:Type: 32-bit Integer
+:Default: ``3``
-``osd map cache size``
-:Description: The size of the OSD map cache in megabytes.
+Data Placement
+==============
+
+When you create pools and set the number of placement groups for the pool, Ceph
+uses default values when you don't specifically override the defaults. **We
+recommend** overridding some of the defaults. Specifically, we recommend setting
+a pools replica size and overriding the default number of placement groups. You
+can specifically set these values when running `pool`_ commands. You can also
+override the defaults by adding new ones in the ``[osd]`` section of your Ceph
+configuration file.
+
+
+.. literalinclude:: pool-pg.conf
+ :language: ini
+
+
+``osd pg bits``
+
+:Description: Placement group bits per OSD.
:Type: 32-bit Integer
-:Default: ``500``
+:Default: ``6``
-``osd map cache bl size``
+``osd pgp bits``
-:Description: The size of the in-memory OSD map cache in OSD daemons.
+:Description: The number of bits per OSD for PGPs.
:Type: 32-bit Integer
-:Default: ``50``
+:Default: ``6``
-``osd map cache bl inc size``
+``osd crush chooseleaf type``
+
+:Description: The bucket type to use for ``chooseleaf`` in a CRUSH rule. Uses
+ ordinal rank rather than name.
-:Description: The size of the in-memory OSD map cache incrementals in OSD daemons.
:Type: 32-bit Integer
-:Default: ``100``
+:Default: ``1``. Typically a host containing one or more OSDs.
-``osd map message max``
+``osd min rep``
-:Description: The maximum map entries allowed per MOSDMap message.
+:Description: The minimum number of replicas for a ruleset.
:Type: 32-bit Integer
-:Default: ``100``
+:Default: ``1``
-``osd op threads``
+``osd max rep``
-:Description: The number of OSD operation threads. Set to ``0`` to disable it. Increasing the number may increase the request processing rate.
+:Description: The maximum number of replicas for a ruleset.
:Type: 32-bit Integer
-:Default: ``2``
+:Default: ``10``
-``osd op thread timeout``
+``osd pool default crush rule``
-:Description: The OSD operation thread timeout in seconds.
+:Description: The default CRUSH ruleset to use when creating a pool.
:Type: 32-bit Integer
-:Default: ``30``
+:Default: ``0``
-``osd disk threads``
+``osd pool default size``
+
+:Description: Sets the number of replicas for objects in the pool. The default
+ value is the same as
+ ``ceph osd pool set {pool-name} size {size}``.
-:Description: The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming.
:Type: 32-bit Integer
-:Default: ``1``
+:Default: ``2``
-``osd recovery threads``
+``osd pool default min size``
+
+:Descrption: Sets the minimum number of written replicas for objects in the
+ pool in order to acknowledge a write operation to the client.
+ If minimum is not met, Ceph will not acknowledge the write to the
+ client. This setting ensures a minimum number of replicas when
+ operating in ``degraded`` mode.
-:Description: The number of threads for recovering data.
:Type: 32-bit Integer
-:Default: ``1``
+:Default: ``0``, which means no particular minimum. If ``0``,
+ minimum is ``size - (size / 2)``.
-``osd backfill scan min``
+``osd pool default pg num``
+
+:Description: The default number of placement groups for a pool. The default
+ value is the same as ``pg_num`` with ``mkpool``.
-:Description: The scan interval in seconds for backfill operations.
:Type: 32-bit Integer
-:Default: ``64``
+:Default: ``8``
-``osd backfill scan max``
+``osd pool default pgp num``
+
+:Description: The default number of placement groups for placement for a pool.
+ The default value is the same as ``pgp_num`` with ``mkpool``.
+ PG and PGP should be equal (for now).
-:Description: The maximum scan interval in seconds for backfill operations.
:Type: 32-bit Integer
-:Default: ``512``
+:Default: ``8``
-``osd backlog thread timeout``
+``osd pool default flags``
-:Description: The maximum time in seconds before timing out a backlog thread.
+:Description: The default flags for new pools.
:Type: 32-bit Integer
-:Default: ``60*60*1``
+:Default: ``0``
-``osd max backfills``
+``osd max pgls``
-:Description: The maximum number of backfills allowed to or from a single OSD.
-:Type: 64-bit Unsigned Integer
-:Default: 10
+:Description: The maximum number of placement groups to list. A client
+ requesting a large number can tie up the OSD.
+:Type: Unsigned 64-bit Integer
+:Default: ``1024``
+:Note: Default should be fine.
-``osd backfill full ratio``
-:Description: Refuse to accept backfill requests when the OSD's full ratio is above this value.
-:Type: Float
-:Default: ``0.85``
+``osd min pg log entries``
+:Description: The minimum number of placement group logs to maintain
+ when trimming log files.
-``osd backfill retry interval``
+:Type: 32-bit Int Unsigned
+:Default: ``1000``
-:Description: The number of seconds to wait before retrying backfill requests.
-:Type: Double
-:Default: ``10.0``
+``osd default data pool replay window``
-``osd recovery thread timeout``
+:Description: The time (in seconds) for an OSD to wait for a client to replay
+ a request.
-:Description: The maximum time in seconds before timing out a recovery thread.
:Type: 32-bit Integer
-:Default: ``30``
+:Default: ``45``
-``osd snap trim thread timeout``
+Scrubbing
+=========
+
+In addition to making multiple copies of objects, Ceph insures data integrity by
+scrubbing placement groups. Ceph scrubbing is analogous to ``fsck`` on the
+object storage layer. For each placement group, Ceph generates a catalog of all
+objects and compares each primary object and its replicas to ensure that no
+objects are missing or mismatched. Light scrubbing (daily) checks the object
+size and attributes. Deep scrubbing (weekly) reads the data and uses checksums
+to ensure data integrity.
+
+Scrubbing is important for maintaining data integrity, but it can reduce
+performance. You can adjust the following settings to increase or decrease
+scrubbing operations.
-:Description: The maximum time in seconds before timing out a snap trim thread.
-:Type: 32-bit Integer
-:Default: ``60*60*1``
+
+``osd max scrubs``
+
+:Description: The maximum number of scrub operations for an OSD.
+:Type: 32-bit Int
+:Default: ``1``
``osd scrub thread timeout``
``osd scrub finalize thread timeout``
-:Description: The maximum time in seconds before timing out a scrub finalize thread.
+:Description: The maximum time in seconds before timing out a scrub finalize
+ thread.
+
:Type: 32-bit Integer
-:Default: 60*10
+:Default: ``60*10``
-``osd remove thread timeout``
+``osd scrub load threshold``
+
+:Description: The maximum CPU load. Ceph will not scrub when the CPU load is
+ higher than this number. Default is 50%.
+
+:Type: Float
+:Default: ``0.5``
+
+
+``osd scrub min interval``
+
+:Description: The maximum interval in seconds for scrubbing the OSD when the
+ cluster load is low.
+
+:Type: Float
+:Default: 5 minutes. ``300``
+
+
+``osd scrub max interval``
+
+:Description: The maximum interval in seconds for scrubbing the OSD
+ irrespective of cluster load.
+
+:Type: Float
+:Default: Once per day. ``60*60*24``
+
+
+``osd deep scrub interval``
+
+:Description: The interval for "deep" scrubbing (fully reading all data).
+:Type: Float
+:Default: Once per week. ``60*60*24*7``
+
+
+``osd deep scrub stride``
+
+:Description: Read size when doing a deep scrub.
+:Type: 32-bit Int
+:Default: 512 KB. ``524288``
+
+
+Operations
+==========
+
+Operations settings allow you to configure the number of threads for servicing
+requests. If you set ``osd op threads`` to ``0``, it disables multi-threading.
+By default, Ceph uses two threads with a 30 second timeout and a 30 second
+complaint time if an operation doesn't complete within those time parameters.
+You can set operations priority weights between client operations and
+recovery operations to ensure optimal performance during recovery.
+
+
+``osd op threads``
+
+:Description: The number of threads to service OSD operations. Set to ``0`` to
+ disable it. Increasing the number may increase the request
+ processing rate.
-:Description: The maximum time in seconds before timing out a remove OSD thread.
:Type: 32-bit Integer
-:Default: 60*60
+:Default: ``2``
-``osd command thread timeout``
+``osd client op priority``
+
+:Description: The priority set for client operations. It is relative to
+ ``osd recovery op priority``.
-:Description: The maximum time in seconds before timing out a command thread.
:Type: 32-bit Integer
-:Default: ``10*60``
+:Default: ``63``
+:Valid Range: 1-63
-``osd heartbeat address``
+``osd recovery op priority``
-:Description: An OSD's network address for heartbeats.
-:Type: Address
-:Default: The host address.
+:Description: The priority set for recovery operations. It is relative to
+ ``osd client op priority``.
+
+:Type: 32-bit Integer
+:Default: ``10``
+:Valid Range: 1-63
-``osd heartbeat interval``
+``osd op thread timeout``
-:Description: How often an OSD pings its peers (in seconds).
+:Description: The OSD operation thread timeout in seconds.
:Type: 32-bit Integer
-:Default: ``6``
+:Default: ``30``
-``osd heartbeat grace``
+``osd op complaint time``
+
+:Description: An operation becomes complaint worthy after the specified number
+ of seconds have elapsed.
+
+:Type: Float
+:Default: ``30``
+
+
+``osd disk threads``
+
+:Description: The number of disk threads, which are used to perform background
+ disk intensive OSD operations such as scrubbing and snap
+ trimming.
-:Description: The elapsed time when an OSD hasn't shown a heartbeat that the cluster considers it ``down``.
:Type: 32-bit Integer
+:Default: ``1``
+
+
+``osd op history size``
+
+:Description: The maximum number of completed operations to track.
+:Type: 32-bit Unsigned Integer
:Default: ``20``
-``osd mon heartbeat interval``
+``osd op history duration``
-:Description: How often the OSD pings a monitor if it has no OSD peers.
+:Description: The oldest completed operation to track.
+:Type: 32-bit Unsigned Integer
+:Default: ``600``
+
+
+``osd op log threshold``
+
+:Description: How many operations logs to display at once.
:Type: 32-bit Integer
-:Default: ``30``
+:Default: ``5``
-``osd mon report interval max``
-:Description: The maximum time in seconds for an OSD to report to a monitor before the monitor considers the OSD ``down``.
+Backfilling
+===========
+
+When you add or remove OSDs to a cluster, the CRUSH algorithm will want to
+rebalance the cluster by moving placement groups to or from OSDs to restore the
+balance. The process of migrating placement groups and the objects they contain
+can reduce the cluster's operational performance considerably. To maintain
+operational performance, Ceph performs this migration with 'backfilling', which
+allows Ceph to set backfill operations to a lower priority than requests to read
+or write data.
+
+
+``osd max backfills``
+
+:Description: The maximum number of backfills allowed to or from a single OSD.
+:Type: 64-bit Unsigned Integer
+:Default: ``10``
+
+
+``osd backfill scan min``
+
+:Description: The scan interval in seconds for backfill operations when cluster
+ load is low.
:Type: 32-bit Integer
-:Default: ``120``
+:Default: ``64``
-``osd mon report interval min``
+``osd backfill scan max``
+
+:Description: The maximum scan interval in seconds for backfill operations
+ irrespective of cluster load.
-:Description: The minimum number of seconds for an OSD to report to a monitor to avoid the monitor considering the OSD ``down``.
:Type: 32-bit Integer
-:Default: ``5``
-:Valid Range: Should be less than ``osd mon report interval max``
+:Default: ``512``
-``osd mon ack timeout``
+``osd backfill full ratio``
+
+:Description: Refuse to accept backfill requests when the OSD's full ratio is
+ above this value.
+:Type: Float
+:Default: ``0.85``
-:Description: The number of seconds to wait for a monitor to acknowledge a request for statistics.
+
+``osd backfill retry interval``
+
+:Description: The number of seconds to wait before retrying backfill requests.
+:Type: Double
+:Default: ``10.0``
+
+
+
+
+OSD Map
+=======
+
+OSD maps reflect the OSD daemons operating in the cluster. Over time, the
+number of map epochs increases. Ceph provides some settings to ensure that
+Ceph performs well as the OSD map grows larger.
+
+
+``osd map dedup``
+
+:Description: Enable removing duplicates in the OSD map.
+:Type: Boolean
+:Default: ``true``
+
+
+``osd map cache size``
+
+:Description: The size of the OSD map cache in megabytes.
:Type: 32-bit Integer
-:Default: ``30``
+:Default: ``500``
-``osd min down reporters``
+``osd map cache bl size``
-:Description: The minimum number of OSDs required to report a ``down`` OSD.
+:Description: The size of the in-memory OSD map cache in OSD daemons.
:Type: 32-bit Integer
-:Default: ``1``
+:Default: ``50``
-``osd min down reports``
+``osd map cache bl inc size``
+
+:Description: The size of the in-memory OSD map cache incrementals in
+ OSD daemons.
-:Description: The minimum number of times an OSD must report that another is ``down``.
:Type: 32-bit Integer
-:Default: ``3``
+:Default: ``100``
+
+
+``osd map message max``
+
+:Description: The maximum map entries allowed per MOSDMap message.
+:Type: 32-bit Integer
+:Default: ``100``
+
+
+
+Recovery
+========
+
+When the cluster starts or when an OSD crashes and restarts, the OSD begins
+peering with other OSDs before writes can occur. See `Monitoring OSDs and PGs`_
+for details.
+
+If an OSD crashed and comes back online, usually it will be out of sync with
+other OSDs containing more recent versions of objects in the placement groups.
+When this happens, the OSD goes into recovery mode and seeks to get the latest
+copy of the data and bring its map back up to date. Depending upon how long the
+OSD was down, the OSD's objects and placement groups may be significantly out of
+date. Also, if a failure domain went down (e.g., a rack), more than one OSD may
+come back online at the same time. This can make the recovery process time
+consuming and resource intensive.
+
+To maintain operational performance, Ceph performs recovery with limitations on
+the number recovery requests, threads and object chunk sizes which allows Ceph
+perform well in a degraded state.
``osd recovery delay start``
-:Description: After peering completes, Ceph will delay for the specified number of seconds before starting to recover objects.
+:Description: After peering completes, Ceph will delay for the specified number
+ of seconds before starting to recover objects.
+
:Type: Float
:Default: ``15``
``osd recovery max active``
-:Description: The number of active recovery requests per OSD at one time. More accelerates recovery, but places an increased load on the cluster.
+:Description: The number of active recovery requests per OSD at one time. More
+ requests will accelerate recovery, but the requests places an
+ increased load on the cluster.
+
:Type: 32-bit Integer
:Default: ``5``
:Default: ``1 << 20``
-``osd max scrubs``
+``osd recovery threads``
-:Description: The maximum number of scrub operations for an OSD.
-:Type: 32-bit Int
-:Default: ``1``
+:Description: The number of threads for recovering data.
+:Type: 32-bit Integer
+:Default: ``1``
-``osd scrub load threshold``
+``osd recovery thread timeout``
-:Description: The maximum CPU load. Ceph will not scrub when the CPU load is higher than this number. Default is 50%.
-:Type: Float
-:Default: ``0.5``
+:Description: The maximum time in seconds before timing out a recovery thread.
+:Type: 32-bit Integer
+:Default: ``30``
-``osd scrub min interval``
+``osd recover clone overlap``
-:Description: The maximum interval in seconds for scrubbing the OSD.
-:Type: Float
-:Default: 5 minutes. ``300``
+:Description: Preserves clone overlap during recovery. Should always be set
+ to ``true``.
+:Type: Boolean
+:Default: ``true``
-``osd scrub max interval``
-:Description: The maximum interval in seconds for scrubbing the OSD.
-:Type: Float
-:Default: Once per day. ``60*60*24``
+Miscellaneous
+=============
-``osd deep scrub interval``
-:Description: The interval for "deep" scrubbing (fully reading all data).
-:Type: Float
-:Default: Once per week. ``60*60*24*7``
+``osd snap trim thread timeout``
+:Description: The maximum time in seconds before timing out a snap trim thread.
+:Type: 32-bit Integer
+:Default: ``60*60*1``
-``osd deep scrub stride``
-:Description: Read size when doing a deep scrub.
-:Type: 32-bit Int
-:Default: 512 KB. ``524288``
+``osd backlog thread timeout``
+
+:Description: The maximum time in seconds before timing out a backlog thread.
+:Type: 32-bit Integer
+:Default: ``60*60*1``
-``osd class dir``
+``osd default notify timeout``
-:Description: The class path for RADOS class plug-ins.
-:Type: String
-:Default: ``$libdir/rados-classes``
+:Description: The OSD default notification timeout (in seconds).
+:Type: 32-bit Integer Unsigned
+:Default: ``30``
``osd check for log corruption``
:Default: ``false``
-``osd default notify timeout``
-
-:Description: The OSD default notification timeout (in seconds).
-:Type: 32-bit Integer Unsigned
-:Default: ``30``
-
-
-``osd min pg log entries``
+``osd remove thread timeout``
-:Description: The minimum number of placement group logs to maintain when trimming log files.
-:Type: 32-bit Int Unsigned
-:Default: 1000
+:Description: The maximum time in seconds before timing out a remove OSD thread.
+:Type: 32-bit Integer
+:Default: ``60*60``
-``osd op complaint time``
+``osd command thread timeout``
-:Description: An operation becomes complaint worthy after the specified number of seconds have elapsed.
-:Type: Float
-:Default: ``30``
+:Description: The maximum time in seconds before timing out a command thread.
+:Type: 32-bit Integer
+:Default: ``10*60``
``osd command max records``
:Type: Boolean
:Default: ``false``
+
+``osd preserve trimmed log``
+
+:Description: Preserves trimmed log files, but uses more disk space.
+:Type: Boolean
+:Default: ``false``
+
+
+
+.. _pool: ../../operations/pools
+.. _Configuring Monitor/OSD Interaction: ../mon-osd-interaction
+.. _Monitoring OSDs and PGs: ../../operations/monitoring-osd-pg#peering
\ No newline at end of file
projects / ceph.git / commitdiff