Ceph config option names may use spaces, underscores, or by one reference hyphens
as interstitial separators. Most usage within the doc tree uses underscores,
though example conf files and especially structured lists of options mostly
use spaces. Mostly. Underscores help differentiate the config names from
surrounding text, and moreover facilitate scripting, grep, awk, etc and match
their form in src/common/options.cc.
This PR conforms these occurrences of option names to use interstitial underscores instead of spaces.
Fixes: https://tracker.ceph.com/issues/48301
Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>
Client Config Reference
------------------------
-``client acl type``
+``client_acl_type``
:Description: Set the ACL type. Currently, only possible value is ``"posix_acl"`` to enable POSIX ACL, or an empty string. This option only takes effect when the ``fuse_default_permissions`` is set to ``false``.
:Type: String
:Default: ``""`` (no ACL enforcement)
-``client cache mid``
+``client_cache_mid``
:Description: Set client cache midpoint. The midpoint splits the least recently used lists into a hot and warm list.
:Type: Float
:Default: ``0.75``
-``client cache size``
+``client_cache_size``
:Description: Set the number of inodes that the client keeps in the metadata cache.
:Type: Integer
:Default: ``16384``
-``client caps release delay``
+``client_caps_release_delay``
:Description: Set the delay between capability releases in seconds. The delay sets how many seconds a client waits to release capabilities that it no longer needs in case the capabilities are needed for another user space operation.
:Type: Integer
:Default: ``5`` (seconds)
-``client debug force sync read``
+``client_debug_force_sync_read``
:Description: If set to ``true``, clients read data directly from OSDs instead of using a local page cache.
:Type: Boolean
:Default: ``false``
-``client dirsize rbytes``
+``client_dirsize_rbytes``
:Description: If set to ``true``, use the recursive size of a directory (that is, total of all descendants).
:Type: Boolean
:Default: ``true``
-``client max inline size``
+``client_max_inline_size``
:Description: Set the maximum size of inlined data stored in a file inode rather than in a separate data object in RADOS. This setting only applies if the ``inline_data`` flag is set on the MDS map.
:Type: Integer
:Default: ``4096``
-``client metadata``
+``client_metadata``
:Description: Comma-delimited strings for client metadata sent to each MDS, in addition to the automatically generated version, host name, and other metadata.
:Type: String
:Default: ``""`` (no additional metadata)
-``client mount gid``
+``client_mount_gid``
:Description: Set the group ID of CephFS mount.
:Type: Integer
:Default: ``-1``
-``client mount timeout``
+``client_mount_timeout``
:Description: Set the timeout for CephFS mount in seconds.
:Type: Float
:Default: ``300.0``
-``client mount uid``
+``client_mount_uid``
:Description: Set the user ID of CephFS mount.
:Type: Integer
:Default: ``-1``
-``client mountpoint``
+``client_mountpoint``
:Description: Directory to mount on the CephFS file system. An alternative to the ``-r`` option of the ``ceph-fuse`` command.
:Type: String
:Default: ``"/"``
-``client oc``
+``client_oc``
:Description: Enable object caching.
:Type: Boolean
:Default: ``true``
-``client oc max dirty``
+``client_oc_max_dirty``
:Description: Set the maximum number of dirty bytes in the object cache.
:Type: Integer
:Default: ``104857600`` (100MB)
-``client oc max dirty age``
+``client_oc_max_dirty_age``
:Description: Set the maximum age in seconds of dirty data in the object cache before writeback.
:Type: Float
:Default: ``5.0`` (seconds)
-``client oc max objects``
+``client_oc_max_objects``
:Description: Set the maximum number of objects in the object cache.
:Type: Integer
:Default: ``1000``
-``client oc size``
+``client_oc_size``
:Description: Set how many bytes of data will the client cache.
:Type: Integer
:Default: ``209715200`` (200 MB)
-``client oc target dirty``
+``client_oc_target_dirty``
:Description: Set the target size of dirty data. We recommend to keep this number low.
:Type: Integer
:Default: ``8388608`` (8MB)
-``client permissions``
+``client_permissions``
:Description: Check client permissions on all I/O operations.
:Type: Boolean
:Default: ``true``
-``client quota``
+``client_quota``
:Description: Enable client quota checking if set to ``true``.
:Type: Boolean
:Default: ``true``
-``client quota df``
+``client_quota_df``
:Description: Report root directory quota for the ``statfs`` operation.
:Type: Boolean
:Default: ``true``
-``client readahead max bytes``
+``client_readahead_max_bytes``
:Description: Set the maximum number of bytes that the client reads ahead for future read operations. Overridden by the ``client_readahead_max_periods`` setting.
:Type: Integer
:Default: ``0`` (unlimited)
-``client readahead max periods``
+``client_readahead_max_periods``
:Description: Set the number of file layout periods (object size * number of stripes) that the client reads ahead. Overrides the ``client_readahead_max_bytes`` setting.
:Type: Integer
:Default: ``4``
-``client readahead min``
+``client_readahead_min``
:Description: Set the minimum number bytes that the client reads ahead.
:Type: Integer
:Default: ``131072`` (128KB)
-``client reconnect stale``
+``client_reconnect_stale``
:Description: Automatically reconnect stale session.
:Type: Boolean
:Default: ``false``
-``client snapdir``
+``client_snapdir``
:Description: Set the snapshot directory name.
:Type: String
:Default: ``".snap"``
-``client tick interval``
+``client_tick_interval``
:Description: Set the interval in seconds between capability renewal and other upkeep.
:Type: Float
:Default: ``1.0`` (seconds)
-``client use random mds``
+``client_use_random_mds``
:Description: Choose random MDS for each request.
:Type: Boolean
:Default: ``false``
-``fuse default permissions``
+``fuse_default_permissions``
:Description: When set to ``false``, ``ceph-fuse`` utility checks does its own permissions checking, instead of relying on the permissions enforcement in FUSE. Set to ``false`` together with the ``client acl type=posix_acl`` option to enable POSIX ACL.
:Type: Boolean
:Default: ``true``
-``fuse max write``
+``fuse_max_write``
-:Description: Set the maximum number of bytes in a single write operation. Because the FUSE default is 128kbytes, SO fuse_max_write default set to 0(The default does not take effect)
+:Description: Set the maximum number of bytes in a single write operation.
+ A value of 0 indicates no change; the
+ FUSE default of 128 kbytes remains in force.
:Type: Integer
:Default: ``0``
-``fuse disable pagecache``
+``fuse_disable_pagecache``
-:Description: If set to ``true``, kernel page cache is disabled for ceph-fuse mount. Because when multiple clients read/write to a file at the same time, reader may get stale data from page cache. (Due to limitation of fuse, ceph-fuse can't disable page cache dynamically)
+:Description: If set to ``true``, kernel page cache is disabled for ``ceph-fuse``
+ mounts. When multiple clients read/write to a file at the same
+ time, readers may get stale data from page cache. Due to
+ limitations of FUSE, ``ceph-fuse`` can't disable page cache dynamically.
:Type: Boolean
:Default: ``false``
.. important:: These options are internal. They are listed here only to complete the list of options.
-``client debug getattr caps``
+``client_debug_getattr_caps``
:Description: Check if the reply from the MDS contains required capabilities.
:Type: Boolean
:Default: ``false``
-``client debug inject tick delay``
+``client_debug_inject_tick_delay``
:Description: Add artificial delay between client ticks.
:Type: Integer
:Default: ``0``
-``client inject fixed oldest tid``
+``client_inject_fixed_oldest_tid``
:Description:
:Type: Boolean
:Default: ``false``
-``client inject release failure``
+``client_inject_release_failure``
:Description:
:Type: Boolean
:Default: ``false``
-``client trace``
+``client_trace``
:Description: The path to the trace file for all file operations. The output is designed to be used by the Ceph `synthetic client <../../man/8/ceph-syn>`_.
:Type: String
MDS Config Reference
======================
-``mds cache memory limit``
+``mds_cache_memory_limit``
:Description: The memory limit the MDS should enforce for its cache.
:Type: 64-bit Integer Unsigned
:Default: ``4G``
-``mds cache reservation``
+``mds_cache_reservation``
:Description: The cache reservation (memory or inodes) for the MDS cache to maintain.
Once the MDS begins dipping into its reservation, it will recall
:Default: ``0.05``
-``mds cache mid``
+``mds_cache_mid``
:Description: The insertion point for new items in the cache LRU
(from the top).
:Default: ``0.7``
-``mds dir commit ratio``
+``mds_dir_commit_ratio``
:Description: The fraction of directory that is dirty before Ceph commits using
a full update (instead of partial update).
:Default: ``0.5``
-``mds dir max commit size``
+``mds_dir_max_commit_size``
:Description: The maximum size of a directory update before Ceph breaks it into
smaller transactions (MB).
:Default: ``10``
-``mds decay halflife``
+``mds_decay_halflife``
:Description: The half-life of MDS cache temperature.
:Type: Float
:Default: ``5``
-``mds beacon interval``
+``mds_beacon_interval``
:Description: The frequency (in seconds) of beacon messages sent
to the monitor.
:Default: ``4``
-``mds beacon grace``
+``mds_beacon_grace``
:Description: The interval without beacons before Ceph declares an MDS laggy
(and possibly replace it).
:Default: ``15``
-``mds blocklist interval``
+``mds_blocklist_interval``
:Description: The blocklist duration for failed MDSs in the OSD map. Note,
this controls how long failed MDS daemons will stay in the
:Default: ``24.0*60.0``
-``mds reconnect timeout``
+``mds_reconnect_timeout``
:Description: The interval (in seconds) to wait for clients to reconnect
during MDS restart.
:Default: ``45``
-``mds tick interval``
+``mds_tick_interval``
:Description: How frequently the MDS performs internal periodic tasks.
:Type: Float
:Default: ``5``
-``mds dirstat min interval``
+``mds_dirstat_min_interval``
:Description: The minimum interval (in seconds) to try to avoid propagating
recursive stats up the tree.
:Type: Float
:Default: ``1``
-``mds scatter nudge interval``
+``mds_scatter_nudge_interval``
:Description: How quickly dirstat changes propagate up.
:Type: Float
:Default: ``5``
-``mds client prealloc inos``
+``mds_client_prealloc_inos``
:Description: The number of inode numbers to preallocate per client session.
:Type: 32-bit Integer
:Default: ``1000``
-``mds early reply``
+``mds_early_reply``
:Description: Determines whether the MDS should allow clients to see request
results before they commit to the journal.
:Default: ``true``
-``mds default dir hash``
+``mds_default_dir_hash``
:Description: The function to use for hashing files across directory fragments.
:Type: 32-bit Integer
:Default: ``2`` (i.e., rjenkins)
-``mds log skip corrupt events``
+``mds_log_skip_corrupt_events``
:Description: Determines whether the MDS should try to skip corrupt journal
events during journal replay.
:Default: ``false``
-``mds log max events``
+``mds_log_max_events``
:Description: The maximum events in the journal before we initiate trimming.
Set to ``-1`` to disable limits.
:Default: ``-1``
-``mds log max segments``
+``mds_log_max_segments``
:Description: The maximum number of segments (objects) in the journal before
we initiate trimming. Set to ``-1`` to disable limits.
:Default: ``128``
-``mds bal sample interval``
+``mds_bal_sample_interval``
:Description: Determines how frequently to sample directory temperature
(for fragmentation decisions).
:Default: ``3``
-``mds bal replicate threshold``
+``mds_bal_replicate_threshold``
:Description: The maximum temperature before Ceph attempts to replicate
metadata to other nodes.
:Default: ``8000``
-``mds bal unreplicate threshold``
+``mds_bal_unreplicate_threshold``
:Description: The minimum temperature before Ceph stops replicating
metadata to other nodes.
:Default: ``0``
-``mds bal split size``
+``mds_bal_split_size``
:Description: The maximum directory size before the MDS will split a directory
fragment into smaller bits.
:Default: ``10000``
-``mds bal split rd``
+``mds_bal_split_rd``
:Description: The maximum directory read temperature before Ceph splits
a directory fragment.
:Default: ``25000``
-``mds bal split wr``
+``mds_bal_split_wr``
:Description: The maximum directory write temperature before Ceph splits
a directory fragment.
:Default: ``10000``
-``mds bal split bits``
+``mds_bal_split_bits``
:Description: The number of bits by which to split a directory fragment.
:Type: 32-bit Integer
:Default: ``3``
-``mds bal merge size``
+``mds_bal_merge_size``
:Description: The minimum directory size before Ceph tries to merge
adjacent directory fragments.
:Default: ``50``
-``mds bal interval``
+``mds_bal_interval``
:Description: The frequency (in seconds) of workload exchanges between MDSs.
:Type: 32-bit Integer
:Default: ``10``
-``mds bal fragment interval``
+``mds_bal_fragment_interval``
:Description: The delay (in seconds) between a fragment being eligible for split
or merge and executing the fragmentation change.
:Default: ``5``
-``mds bal fragment fast factor``
+``mds_bal_fragment_fast_factor``
:Description: The ratio by which frags may exceed the split size before
a split is executed immediately (skipping the fragment interval)
:Type: Float
:Default: ``1.5``
-``mds bal fragment size max``
+``mds_bal_fragment_size_max``
:Description: The maximum size of a fragment before any new entries
are rejected with ENOSPC.
:Type: 32-bit Integer
:Default: ``100000``
-``mds bal idle threshold``
+``mds_bal_idle_threshold``
:Description: The minimum temperature before Ceph migrates a subtree
back to its parent.
:Default: ``0``
-``mds bal max``
+``mds_bal_max``
:Description: The number of iterations to run balancer before Ceph stops.
(used for testing purposes only)
:Default: ``-1``
-``mds bal max until``
+``mds_bal_max_until``
:Description: The number of seconds to run balancer before Ceph stops.
(used for testing purposes only)
:Default: ``-1``
-``mds bal mode``
+``mds_bal_mode``
:Description: The method for calculating MDS load.
:Default: ``0``
-``mds bal min rebalance``
+``mds_bal_min_rebalance``
:Description: The minimum subtree temperature before Ceph migrates.
:Type: Float
:Default: ``0.1``
-``mds bal min start``
+``mds_bal_min_start``
:Description: The minimum subtree temperature before Ceph searches a subtree.
:Type: Float
:Default: ``0.2``
-``mds bal need min``
+``mds_bal_need_min``
:Description: The minimum fraction of target subtree size to accept.
:Type: Float
:Default: ``0.8``
-``mds bal need max``
+``mds_bal_need_max``
:Description: The maximum fraction of target subtree size to accept.
:Type: Float
:Default: ``1.2``
-``mds bal midchunk``
+``mds_bal_midchunk``
:Description: Ceph will migrate any subtree that is larger than this fraction
of the target subtree size.
:Default: ``0.3``
-``mds bal minchunk``
+``mds_bal_minchunk``
:Description: Ceph will ignore any subtree that is smaller than this fraction
of the target subtree size.
:Default: ``0.001``
-``mds bal target removal min``
+``mds_bal_target_removal_min``
:Description: The minimum number of balancer iterations before Ceph removes
an old MDS target from the MDS map.
:Default: ``5``
-``mds bal target removal max``
+``mds_bal_target_removal_max``
-:Description: The maximum number of balancer iteration before Ceph removes
+:Description: The maximum number of balancer iterations before Ceph removes
an old MDS target from the MDS map.
:Type: 32-bit Integer
:Default: ``10``
-``mds replay interval``
+``mds_replay_interval``
:Description: The journal poll interval when in standby-replay mode.
("hot standby")
:Default: ``1``
-``mds shutdown check``
+``mds_shutdown_check``
:Description: The interval for polling the cache during MDS shutdown.
:Type: 32-bit Integer
:Default: ``0``
-``mds thrash exports``
+``mds_thrash_exports``
:Description: Ceph will randomly export subtrees between nodes (testing only).
:Type: 32-bit Integer
:Default: ``0``
-``mds thrash fragments``
+``mds_thrash_fragments``
:Description: Ceph will randomly fragment or merge directories.
:Type: 32-bit Integer
:Default: ``0``
-``mds dump cache on map``
+``mds_dump_cache_on_map``
:Description: Ceph will dump the MDS cache contents to a file on each MDSMap.
:Type: Boolean
:Default: ``false``
-``mds dump cache after rejoin``
+``mds_dump_cache_after_rejoin``
:Description: Ceph will dump MDS cache contents to a file after
rejoining the cache (during recovery).
:Default: ``false``
-``mds verify scatter``
+``mds_verify_scatter``
:Description: Ceph will assert that various scatter/gather invariants
are ``true`` (developers only).
:Default: ``false``
-``mds debug scatterstat``
+``mds_debug_scatterstat``
:Description: Ceph will assert that various recursive stat invariants
are ``true`` (for developers only).
:Default: ``false``
-``mds debug frag``
+``mds_debug_frag``
:Description: Ceph will verify directory fragmentation invariants
when convenient (developers only).
:Default: ``false``
-``mds debug auth pins``
+``mds_debug_auth_pins``
:Description: The debug auth pin invariants (for developers only).
:Type: Boolean
:Default: ``false``
-``mds debug subtrees``
+``mds_debug_subtrees``
:Description: The debug subtree invariants (for developers only).
:Type: Boolean
:Default: ``false``
-``mds kill mdstable at``
+``mds_kill_mdstable_at``
:Description: Ceph will inject MDS failure in MDSTable code
(for developers only).
:Default: ``0``
-``mds kill export at``
+``mds_kill_export_at``
:Description: Ceph will inject MDS failure in the subtree export code
(for developers only).
:Default: ``0``
-``mds kill import at``
+``mds_kill_import_at``
:Description: Ceph will inject MDS failure in the subtree import code
(for developers only).
:Default: ``0``
-``mds kill link at``
+``mds_kill_link_at``
:Description: Ceph will inject MDS failure in hard link code
(for developers only).
:Default: ``0``
-``mds kill rename at``
+``mds_kill_rename_at``
:Description: Ceph will inject MDS failure in the rename code
(for developers only).
:Default: ``0``
-``mds wipe sessions``
+``mds_wipe_sessions``
:Description: Ceph will delete all client sessions on startup
(for testing only).
:Default: ``false``
-``mds wipe ino prealloc``
+``mds_wipe_ino_prealloc``
:Description: Ceph will delete ino preallocation metadata on startup
(for testing only).
:Default: ``false``
-``mds skip ino``
+``mds_skip_ino``
:Description: The number of inode numbers to skip on startup
(for testing only).
:Default: ``0``
-``mds min caps per client``
+``mds_min_caps_per_client``
:Description: Set the minimum number of capabilities a client may hold.
:Type: Integer
:Default: ``100``
-``mds max ratio caps per client``
+``mds_max_ratio_caps_per_client``
:Description: Set the maximum ratio of current caps that may be recalled during MDS cache pressure.
:Type: Float
::
[mon]
- mgr initial modules = dashboard balancer
+ mgr_initial_modules = dashboard balancer
Calling module commands
-----------------------
Configuration
-------------
-``mgr module path``
+``mgr_module_path``
:Description: Path to load modules from
:Type: String
:Default: ``"<library dir>/mgr"``
-``mgr data``
+``mgr_data``
:Description: Path to load daemon data (such as keyring)
:Type: String
:Default: ``"/var/lib/ceph/mgr/$cluster-$id"``
-``mgr tick period``
+``mgr_tick_period``
:Description: How many seconds between mgr beacons to monitors, and other
periodic checks.
:Type: Integer
:Default: ``5``
-``mon mgr beacon grace``
+``mon_mgr_beacon_grace``
:Description: How long after last beacon should a mgr be considered failed
:Type: Integer
The *localpool* module can automatically create RADOS pools that are
localized to a subset of the overall cluster. For example, by default, it will
-create a pool for each distinct rack in the cluster. This can be useful for some
-deployments that want to distribute some data locally as well as globally across the cluster .
+create a pool for each distinct ``rack`` in the cluster. This can be useful for
+deployments where it is desirable to distribute some data locally and other data
+globally across the cluster. One use-case is measuring performance and testing
+behavior of specific drive, NIC, or chassis models in isolation.
Enabling
--------
.. code-block:: ini
- auth cluster required = cephx
- auth service required = cephx
- auth client required = cephx
+ auth_cluster_required = cephx
+ auth_service_required = cephx
+ auth_client_required = cephx
#. Start or restart the Ceph cluster. See `Operating a Cluster`_ for details.
.. code-block:: ini
- auth cluster required = none
- auth service required = none
- auth client required = none
+ auth_cluster_required = none
+ auth_service_required = none
+ auth_client_required = none
#. Start or restart the Ceph cluster. See `Operating a Cluster`_ for details.
----------
-``auth cluster required``
+``auth_cluster_required``
:Description: If enabled, the Ceph Storage Cluster daemons (i.e., ``ceph-mon``,
``ceph-osd``, ``ceph-mds`` and ``ceph-mgr``) must authenticate with
:Default: ``cephx``.
-``auth service required``
+``auth_service_required``
:Description: If enabled, the Ceph Storage Cluster daemons require Ceph Clients
to authenticate with the Ceph Storage Cluster in order to access
:Default: ``cephx``.
-``auth client required``
+``auth_client_required``
:Description: If enabled, the Ceph Client requires the Ceph Storage Cluster to
authenticate with the Ceph Client. Valid settings are ``cephx``
middle" attack).
Like other parts of Ceph authentication, Ceph provides fine-grained control so
-you can enable/disable signatures for service messages between the client and
-Ceph, and you can enable/disable signatures for messages between Ceph daemons.
+you can enable/disable signatures for service messages between clients and
+Ceph, and so you can enable/disable signatures for messages between Ceph daemons.
Note that even with signatures enabled data is not encrypted in
flight.
-``cephx require signatures``
+``cephx_require_signatures``
:Description: If set to ``true``, Ceph requires signatures on all message
traffic between the Ceph Client and the Ceph Storage Cluster, and
:Default: ``false``
-``cephx cluster require signatures``
+``cephx_cluster_require_signatures``
:Description: If set to ``true``, Ceph requires signatures on all message
traffic between Ceph daemons comprising the Ceph Storage Cluster.
:Default: ``false``
-``cephx service require signatures``
+``cephx_service_require_signatures``
:Description: If set to ``true``, Ceph requires signatures on all message
traffic between Ceph Clients and the Ceph Storage Cluster.
:Default: ``false``
-``cephx sign messages``
+``cephx_sign_messages``
:Description: If the Ceph version supports message signing, Ceph will sign
all messages so they are more difficult to spoof.
Time to Live
------------
-``auth service ticket ttl``
+``auth_service_ticket_ttl``
:Description: When the Ceph Storage Cluster sends a Ceph Client a ticket for
authentication, the Ceph Storage Cluster assigns the ticket a
ceph osd pool set <pool-name> compression_min_blob_size <size>
ceph osd pool set <pool-name> compression_max_blob_size <size>
-``bluestore compression algorithm``
+``bluestore_compression_algorithm``
:Description: The default compressor to use (if any) if the per-pool property
- ``compression_algorithm`` is not set. Note that zstd is *not*
- recommended for bluestore due to high CPU overhead when
+ ``compression_algorithm`` is not set. Note that ``zstd`` is *not*
+ recommended for BlueStore due to high CPU overhead when
compressing small amounts of data.
:Type: String
:Required: No
:Valid Settings: ``lz4``, ``snappy``, ``zlib``, ``zstd``
:Default: ``snappy``
-``bluestore compression mode``
+``bluestore_compression_mode``
:Description: The default policy for using compression if the per-pool property
``compression_mode`` is not set. ``none`` means never use
:Valid Settings: ``none``, ``passive``, ``aggressive``, ``force``
:Default: ``none``
-``bluestore compression required ratio``
+``bluestore_compression_required_ratio``
:Description: The ratio of the size of the data chunk after
compression relative to the original size must be at
:Required: No
:Default: .875
-``bluestore compression min blob size``
+``bluestore_compression_min_blob_size``
:Description: Chunks smaller than this are never compressed.
The per-pool property ``compression_min_blob_size`` overrides
:Required: No
:Default: 0
-``bluestore compression min blob size hdd``
+``bluestore_compression_min_blob_size_hdd``
:Description: Default value of ``bluestore compression min blob size``
for rotational media.
:Required: No
:Default: 128K
-``bluestore compression min blob size ssd``
+``bluestore_compression_min_blob_size_ssd``
:Description: Default value of ``bluestore compression min blob size``
for non-rotational (solid state) media.
:Required: No
:Default: 8K
-``bluestore compression max blob size``
+``bluestore_compression_max_blob_size``
-:Description: Chunks larger than this are broken into smaller blobs sizing
- ``bluestore compression max blob size`` before being compressed.
+:Description: Chunks larger than this value are broken into smaller blobs of at most
+ ``bluestore_compression_max_blob_size`` bytes before being compressed.
The per-pool property ``compression_max_blob_size`` overrides
this setting.
:Required: No
:Default: 0
-``bluestore compression max blob size hdd``
+``bluestore_compression_max_blob_size_hdd``
:Description: Default value of ``bluestore compression max blob size``
for rotational media.
:Required: No
:Default: 512K
-``bluestore compression max blob size ssd``
+``bluestore_compression_max_blob_size_ssd``
:Description: Default value of ``bluestore compression max blob size``
- for non-rotational (solid state) media.
+ for non-rotational (SSD, NVMe) media.
:Type: Unsigned Integer
:Required: No
SPDK Usage
==================
-If you want to use SPDK driver for NVME SSD, you need to ready your system.
-Please refer to `SPDK document`__ for more details.
+If you want to use the SPDK driver for NVMe devices, you must prepare your system.
+Refer to `SPDK document`__ for more details.
.. __: http://www.spdk.io/doc/getting_started.html#getting_started_examples
$ sudo src/spdk/scripts/setup.sh
-Then you need to specify NVMe device's device selector here with "spdk:" prefix for
-``bluestore_block_path``.
+You will need to specify the subject NVMe device's device selector with
+the "spdk:" prefix for ``bluestore_block_path``.
-For example, users can find the device selector of an Intel PCIe SSD with::
+For example, you can find the device selector of an Intel PCIe SSD with::
$ lspci -mm -n -D -d 8086:0953
and then set::
- bluestore block path = spdk:0000:01:00.0
+ bluestore_block_path = spdk:0000:01:00.0
Where ``0000:01:00.0`` is the device selector found in the output of ``lspci``
command above.
-If you want to run multiple SPDK instances per node, you must specify the
-amount of dpdk memory size in MB each instance will use, to make sure each
+To run multiple SPDK instances per node, you must specify the
+amount of dpdk memory in MB that each instance will use, to make sure each
instance uses its own dpdk memory
-In most cases, we only need one device to serve as data, db, db wal purposes.
-We need to make sure configurations below to make sure all IOs issued under
-SPDK.::
+In most cases, a single device can be used for data, DB, and WAL. We describe
+this strategy as *colocating* these components. Be sure to enter the below
+settings to ensure that all IOs are issued through SPDK.::
bluestore_block_db_path = ""
bluestore_block_db_size = 0
bluestore_block_wal_path = ""
bluestore_block_wal_size = 0
-Otherwise, the current implementation will setup symbol file to kernel
-file system location and uses kernel driver to issue DB/WAL IO.
+Otherwise, the current implementation will populate the SPDK map files with
+kernel file system symbols and will use the kernel driver to issue DB/WAL IO.
Configuring Ceph
==================
-When you start the Ceph service, the initialization process activates a series
+When Ceph services start, the initialization process activates a series
of daemons that run in the background. A :term:`Ceph Storage Cluster` runs
-three types of daemons:
+at a minimum three types of daemons:
- :term:`Ceph Monitor` (``ceph-mon``)
- :term:`Ceph Manager` (``ceph-mgr``)
- :term:`Ceph OSD Daemon` (``ceph-osd``)
-Ceph Storage Clusters that support the :term:`Ceph File System` run at
+Ceph Storage Clusters that support the :term:`Ceph File System` also run at
least one :term:`Ceph Metadata Server` (``ceph-mds``). Clusters that
-support :term:`Ceph Object Storage` run Ceph Gateway daemons
-(``radosgw``).
+support :term:`Ceph Object Storage` run Ceph RADOS Gateway daemons
+(``radosgw``) as well.
-Each daemon has a series of configuration options, each of which has a
-default values. You may adjust the behavior of the system by changing these
-configuration options.
+Each daemon has a number of configuration options, each of which has a
+default value. You may adjust the behavior of the system by changing these
+configuration options. Be careful to understand the consequences before
+overriding default values, as it is possible to significantly degrade the
+performance and stability of your cluster. Also note that default values
+sometimes change between releases, so it is best to review the version of
+this documentation that aligns with your Ceph release.
Option names
============
``--mon-host`` is equivalent to ``--mon_host``).
When option names appear in configuration files, spaces can also be
-used in place of underscore or dash.
+used in place of underscore or dash. We suggest, though, that for
+clarity and convenience you consistently use underscores, as we do
+throughout this documentation.
Config sources
==============
where ``$cluster`` is the cluster's name (default ``ceph``).
-The Ceph configuration file uses an *ini* style syntax. You can add comments
-by preceding comments with a pound sign (#) or a semi-colon (;). For example:
+The Ceph configuration file uses an *ini* style syntax. You can add comment
+text after a pound sign (#) or a semi-colon (;). For example:
.. code-block:: ini
.. code-block:: ini
[global]
- debug ms = 0
+ debug_ms = 0
[osd]
- debug ms = 1
+ debug_ms = 1
[osd.1]
- debug ms = 10
+ debug_ms = 10
[osd.2]
- debug ms = 10
+ debug_ms = 10
Config file option values
.. code-block:: ini
[global]
- obscure one = difficult to explain # I will try harder in next release
- simpler one = nothing to explain
+ obscure_one = difficult to explain # I will try harder in next release
+ simpler_one = nothing to explain
In the example above, the value of "``obscure one``" would be "``difficult to explain``";
and the value of "``simpler one`` would be "``nothing to explain``".
-Changes since nautilus
+Changes since Nautilus
======================
-We changed the way the configuration file is parsed in Octopus. The changes are
-listed as follows:
+With the Octopus release We changed the way the configuration file is parsed.
+These changes are as follows:
-- repeated configuration options are allowed, and no warnings will be printed.
- The value of the last one wins. Before this change, we would print out warning
- messages at seeing lines with duplicated values, like::
+- Repeated configuration options are allowed, and no warnings will be printed.
+ The value of the last one is used, which means that the setting last in the file
+ is the one that takes effect. Before this change, we would print warning messages
+ when lines with duplicated options were encountered, like::
warning line 42: 'foo' in section 'bar' redefined
-- invalid UTF-8 options are ignored with warning messages. But since Octopus,
+
+- Invalid UTF-8 options were ignored with warning messages. But since Octopus,
they are treated as fatal errors.
-- backslash ``\`` is used as the line continuation marker to combine the next
- line with current one. Before Octopus, it was required to follow backslash with
- non-empty line. But in Octopus, empty line following backslash is now allowed.
+
+- Backslash ``\`` is used as the line continuation marker to combine the next
+ line with current one. Before Octopus, it was required to follow a backslash with
+ a non-empty line. But in Octopus, an empty line following a backslash is now allowed.
+
- In the configuration file, each line specifies an individual configuration
- option. The option's name and its value are separated with ``=``. And the
- value can be quoted using single or double quotes. But if an invalid
+ option. The option's name and its value are separated with ``=``, and the
+ value may be quoted using single or double quotes. If an invalid
configuration is specified, we will treat it as an invalid configuration
file ::
bad option ==== bad value
+
- Before Octopus, if no section name was specified in the configuration file,
- all options would be grouped into the section of ``global``. But this is
- discouraged now. Since Octopus, only a single option is allowed for
+ all options would be set as though they were within the ``global`` section. This is
+ now discouraged. Since Octopus, only a single option is allowed for
configuration files without a section name.
Monitors
========
-Ceph production clusters typically deploy with a minimum 3 :term:`Ceph Monitor`
-daemons to ensure high availability should a monitor instance crash. At least
-three (3) monitors ensures that the Paxos algorithm can determine which version
+Production Ceph clusters typically provision a minimum of three :term:`Ceph Monitor`
+daemons to ensure availability should a monitor instance crash. A minimum of
+three ensures that the Paxos algorithm can determine which version
of the :term:`Ceph Cluster Map` is the most recent from a majority of Ceph
Monitors in the quorum.
Ceph Monitors normally listen on port ``3300`` for the new v2 protocol, and ``6789`` for the old v1 protocol.
-By default, Ceph expects that you will store a monitor's data under the
+By default, Ceph expects to store monitor data under the
following path::
/var/lib/ceph/mon/$cluster-$id
.. code-block:: ini
- auth cluster required = cephx
- auth service required = cephx
- auth client required = cephx
+ auth_cluster_required = cephx
+ auth_service_required = cephx
+ auth_client_required = cephx
Additionally, you should enable message signing. See `Cephx Config Reference`_ for details.
====
Ceph production clusters typically deploy :term:`Ceph OSD Daemons` where one node
-has one OSD daemon running a filestore on one storage drive. A typical
-deployment specifies a journal size. For example:
+has one OSD daemon running a Filestore on one storage device. The BlueStore back
+end is now default, but when using Filestore you specify a journal size. For example:
.. code-block:: ini
[osd]
- osd journal size = 10000
+ osd_journal_size = 10000
[osd.0]
host = {hostname} #manual deployments only.
-By default, Ceph expects that you will store a Ceph OSD Daemon's data with the
+By default, Ceph expects to store a Ceph OSD Daemon's data at the
following path::
/var/lib/ceph/osd/$cluster-$id
You or a deployment tool (e.g., ``cephadm``) must create the corresponding
-directory. With metavariables fully expressed and a cluster named "ceph", the
-foregoing directory would evaluate to::
+directory. With metavariables fully expressed and a cluster named "ceph", this
+example would evaluate to::
/var/lib/ceph/osd/ceph-0
-You may override this path using the ``osd data`` setting. We don't recommend
+You may override this path using the ``osd_data`` setting. We recommend not
changing the default location. Create the default directory on your OSD host.
.. prompt:: bash $
ssh {osd-host}
sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
-The ``osd data`` path ideally leads to a mount point with a hard disk that is
-separate from the hard disk storing and running the operating system and
-daemons. If the OSD is for a disk other than the OS disk, prepare it for
+The ``osd_data`` path ideally leads to a mount point with a device that is
+separate from the device that contains the operating system and
+daemons. If an OSD is to use a device other than the OS device, prepare it for
use with Ceph, and mount it to the directory you just created
.. prompt:: bash $
sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}
We recommend using the ``xfs`` file system when running
-:command:`mkfs`. (``btrfs`` and ``ext4`` are not recommended and no
+:command:`mkfs`. (``btrfs`` and ``ext4`` are not recommended and are no
longer tested.)
See the `OSD Config Reference`_ for additional configuration details.
[global]
fsid = {cluster-id}
-mon initial members = {hostname}[, {hostname}]
-mon host = {ip-address}[, {ip-address}]
+mon_initial_ members = {hostname}[, {hostname}]
+mon_host = {ip-address}[, {ip-address}]
#All clusters have a front-side public network.
-#If you have two NICs, you can configure a back side cluster
-#network for OSD object replication, heart beats, backfilling,
+#If you have two network interfaces, you can configure a private / cluster
+#network for RADOS object replication, heartbeats, backfill,
#recovery, etc.
-public network = {network}[, {network}]
-#cluster network = {network}[, {network}]
+public_network = {network}[, {network}]
+#cluster_network = {network}[, {network}]
#Clusters require authentication by default.
-auth cluster required = cephx
-auth service required = cephx
-auth client required = cephx
+auth_cluster_required = cephx
+auth_service_required = cephx
+auth_client_required = cephx
-#Choose reasonable numbers for your journals, number of replicas
+#Choose reasonable numbers for journals, number of replicas
#and placement groups.
-osd journal size = {n}
-osd pool default size = {n} # Write an object n times.
-osd pool default min size = {n} # Allow writing n copy in a degraded state.
-osd pool default pg num = {n}
-osd pool default pgp num = {n}
+osd_journal_size = {n}
+osd_pool_default_size = {n} # Write an object n times.
+osd_pool_default_min size = {n} # Allow writing n copy in a degraded state.
+osd_pool_default_pg num = {n}
+osd_pool_default_pgp num = {n}
#Choose a reasonable crush leaf type.
#0 for a 1-node cluster.
#1 for a multi node cluster in a single rack
#2 for a multi node, multi chassis cluster with multiple hosts in a chassis
#3 for a multi node cluster with hosts across racks, etc.
-osd crush chooseleaf type = {n}
\ No newline at end of file
+osd_crush_chooseleaf_type = {n}
\ No newline at end of file
Filestore Config Reference
============================
+The Filestore back end is no longer the default when creating new OSDs,
+though Filestore OSDs are still supported.
``filestore debug omap check``
Extended Attributes
===================
-Extended Attributes (XATTRs) are an important aspect in your configuration.
-Some file systems have limits on the number of bytes stored in XATTRS.
+Extended Attributes (XATTRs) are important for Filestore OSDs.
+Some file systems have limits on the number of bytes that can be stored in XATTRs.
Additionally, in some cases, the file system may not be as fast as an alternative
method of storing XATTRs. The following settings may help improve performance
by using a method of storing XATTRs that is extrinsic to the underlying file system.
Ceph XATTRs are stored as ``inline xattr``, using the XATTRs provided
by the underlying file system, if it does not impose a size limit. If
there is a size limit (4KB total on ext4, for instance), some Ceph
-XATTRs will be stored in an key/value database when either the
-``filestore max inline xattr size`` or ``filestore max inline
-xattrs`` threshold is reached.
+XATTRs will be stored in a key/value database when either the
+``filestore_max_inline_xattr_size`` or ``filestore_max_inline_xattrs``
+threshold is reached.
-``filestore max inline xattr size``
+``filestore_max_inline_xattr_size``
:Description: The maximum size of an XATTR stored in the file system (i.e., XFS,
- btrfs, ext4, etc.) per object. Should not be larger than the
+ Btrfs, EXT4, etc.) per object. Should not be larger than the
file system can handle. Default value of 0 means to use the value
specific to the underlying file system.
:Type: Unsigned 32-bit Integer
:Default: ``0``
-``filestore max inline xattr size xfs``
+``filestore_max_inline_xattr_size_xfs``
:Description: The maximum size of an XATTR stored in the XFS file system.
- Only used if ``filestore max inline xattr size`` == 0.
+ Only used if ``filestore_max_inline_xattr_size`` == 0.
:Type: Unsigned 32-bit Integer
:Required: No
:Default: ``65536``
-``filestore max inline xattr size btrfs``
+``filestore_max_inline_xattr_size_btrfs``
-:Description: The maximum size of an XATTR stored in the btrfs file system.
- Only used if ``filestore max inline xattr size`` == 0.
+:Description: The maximum size of an XATTR stored in the Btrfs file system.
+ Only used if ``filestore_max_inline_xattr_size`` == 0.
:Type: Unsigned 32-bit Integer
:Required: No
:Default: ``2048``
-``filestore max inline xattr size other``
+``filestore_max_inline_xattr_size_other``
:Description: The maximum size of an XATTR stored in other file systems.
- Only used if ``filestore max inline xattr size`` == 0.
+ Only used if ``filestore_max_inline_xattr_size`` == 0.
:Type: Unsigned 32-bit Integer
:Required: No
:Default: ``512``
-``filestore max inline xattrs``
+``filestore_max_inline_xattrs``
:Description: The maximum number of XATTRs stored in the file system per object.
Default value of 0 means to use the value specific to the
:Default: ``0``
-``filestore max inline xattrs xfs``
+``filestore_max_inline_xattrs_xfs``
:Description: The maximum number of XATTRs stored in the XFS file system per object.
- Only used if ``filestore max inline xattrs`` == 0.
+ Only used if ``filestore_max_inline_xattrs`` == 0.
:Type: 32-bit Integer
:Required: No
:Default: ``10``
-``filestore max inline xattrs btrfs``
+``filestore_max_inline_xattrs_btrfs``
-:Description: The maximum number of XATTRs stored in the btrfs file system per object.
- Only used if ``filestore max inline xattrs`` == 0.
+:Description: The maximum number of XATTRs stored in the Btrfs file system per object.
+ Only used if ``filestore_max_inline_xattrs`` == 0.
:Type: 32-bit Integer
:Required: No
:Default: ``10``
-``filestore max inline xattrs other``
+``filestore_max_inline_xattrs_other``
:Description: The maximum number of XATTRs stored in other file systems per object.
- Only used if ``filestore max inline xattrs`` == 0.
+ Only used if ``filestore_max_inline_xattrs`` == 0.
:Type: 32-bit Integer
:Required: No
:Default: ``2``
Synchronization Intervals
=========================
-Periodically, the filestore needs to quiesce writes and synchronize the
+Filestore needs to periodically quiesce writes and synchronize the
file system, which creates a consistent commit point. It can then free journal
entries up to the commit point. Synchronizing more frequently tends to reduce
the time required to perform synchronization, and reduces the amount of data
that needs to remain in the journal. Less frequent synchronization allows the
-backing file system to coalesce small writes and metadata updates more
-optimally--potentially resulting in more efficient synchronization.
+backing file system to coalesce small writes and metadata updates more
+optimally, potentially resulting in more efficient synchronization at the
+expense of potentially increasing tail latency.
+``filestore_max_sync_interval``
-``filestore max sync interval``
-
-:Description: The maximum interval in seconds for synchronizing the filestore.
+:Description: The maximum interval in seconds for synchronizing Filestore.
:Type: Double
:Required: No
:Default: ``5``
-``filestore min sync interval``
+``filestore_min_sync_interval``
-:Description: The minimum interval in seconds for synchronizing the filestore.
+:Description: The minimum interval in seconds for synchronizing Filestore.
:Type: Double
:Required: No
:Default: ``.01``
Flusher
=======
-The filestore flusher forces data from large writes to be written out using
-``sync file range`` before the sync in order to (hopefully) reduce the cost of
-the eventual sync. In practice, disabling 'filestore flusher' seems to improve
+The Filestore flusher forces data from large writes to be written out using
+``sync_file_range`` before the sync in order to (hopefully) reduce the cost of
+the eventual sync. In practice, disabling 'filestore_flusher' seems to improve
performance in some cases.
-``filestore flusher``
+``filestore_flusher``
:Description: Enables the filestore flusher.
:Type: Boolean
.. deprecated:: v.65
-``filestore flusher max fds``
+``filestore_flusher_max_fds``
:Description: Sets the maximum number of file descriptors for the flusher.
:Type: Integer
.. deprecated:: v.65
-``filestore sync flush``
+``filestore_sync_flush``
:Description: Enables the synchronization flusher.
:Type: Boolean
.. deprecated:: v.65
-``filestore fsync flushes journal data``
+``filestore_fsync_flushes_journal_data``
:Description: Flush journal data during file system synchronization.
:Type: Boolean
Queue
=====
-The following settings provide limits on the size of filestore queue.
+The following settings provide limits on the size of the Filestore queue.
-``filestore queue max ops``
+``filestore_queue_max_ops``
:Description: Defines the maximum number of in progress operations the file store accepts before blocking on queuing new operations.
:Type: Integer
:Default: ``50``
-``filestore queue max bytes``
+``filestore_queue_max_bytes``
:Description: The maximum number of bytes for an operation.
:Type: Integer
========
-``filestore op threads``
+``filestore_op_threads``
:Description: The number of file system operation threads that execute in parallel.
:Type: Integer
:Default: ``2``
-``filestore op thread timeout``
+``filestore_op_thread_timeout``
:Description: The timeout for a file system operation thread (in seconds).
:Type: Integer
:Default: ``60``
-``filestore op thread suicide timeout``
+``filestore_op_thread_suicide_timeout``
:Description: The timeout for a commit operation before cancelling the commit (in seconds).
:Type: Integer
=================
-``filestore btrfs snap``
+``filestore_btrfs_snap``
:Description: Enable snapshots for a ``btrfs`` filestore.
:Type: Boolean
:Default: ``true``
-``filestore btrfs clone range``
+``filestore_btrfs_clone_range``
:Description: Enable cloning ranges for a ``btrfs`` filestore.
:Type: Boolean
=======
-``filestore journal parallel``
+``filestore_journal_parallel``
-:Description: Enables parallel journaling, default for btrfs.
+:Description: Enables parallel journaling, default for Btrfs.
:Type: Boolean
:Required: No
:Default: ``false``
-``filestore journal writeahead``
+``filestore_journal_writeahead``
-:Description: Enables writeahead journaling, default for xfs.
+:Description: Enables writeahead journaling, default for XFS.
:Type: Boolean
:Required: No
:Default: ``false``
-``filestore journal trailing``
+``filestore_journal_trailing``
:Description: Deprecated, never use.
:Type: Boolean
====
-``filestore merge threshold``
+``filestore_merge_threshold``
:Description: Min number of files in a subdir before merging into parent
NOTE: A negative value means to disable subdir merging
:Default: ``-10``
-``filestore split multiple``
+``filestore_split_multiple``
:Description: ``(filestore_split_multiple * abs(filestore_merge_threshold) + (rand() % filestore_split_rand_factor)) * 16``
is the maximum number of files in a subdirectory before
:Default: ``2``
-``filestore split rand factor``
+``filestore_split_rand_factor``
:Description: A random factor added to the split threshold to avoid
- too many filestore splits occurring at once. See
- ``filestore split multiple`` for details.
- This can only be changed for an existing osd offline,
- via ceph-objectstore-tool's apply-layout-settings command.
+ too many (expensive) Filestore splits occurring at once. See
+ ``filestore_split_multiple`` for details.
+ This can only be changed offline for an existing OSD,
+ via the ``ceph-objectstore-tool apply-layout-settings`` command.
:Type: Unsigned 32-bit Integer
:Required: No
:Default: ``20``
-``filestore update to``
+``filestore_update_to``
-:Description: Limits filestore auto upgrade to specified version.
+:Description: Limits Filestore auto upgrade to specified version.
:Type: Integer
:Required: No
:Default: ``1000``
-``filestore blackhole``
+``filestore_blackhole``
:Description: Drop any new transactions on the floor.
:Type: Boolean
:Default: ``false``
-``filestore dump file``
+``filestore_dump_file``
:Description: File onto which store transaction dumps.
:Type: Boolean
:Default: ``false``
-``filestore kill at``
+``filestore_kill_at``
:Description: inject a failure at the n'th opportunity
:Type: String
:Default: ``false``
-``filestore fail eio``
+``filestore_fail_eio``
:Description: Fail/Crash on eio.
:Type: Boolean
:Default: N/A. Usually generated by deployment tools.
-``admin socket``
+``admin_socket``
:Description: The socket for executing administrative commands on a daemon,
irrespective of whether Ceph Monitors have established a quorum.
:Default: ``/var/run/ceph/$cluster-$name.asok``
-``pid file``
+``pid_file``
:Description: The file in which the mon, osd or mds will write its
PID. For instance, ``/var/run/$cluster/$type.$id.pid``
:Default: ``/``
-``max open files``
+``max_open_files``
:Description: If set, when the :term:`Ceph Storage Cluster` starts, Ceph sets
- the ``max open fds`` at the OS level (i.e., the max # of file
- descriptors). It helps prevents Ceph OSD Daemons from running out
+ the max open FDs at the OS level (i.e., the max # of file
+ descriptors). A suitably large value prevents Ceph Daemons from running out
of file descriptors.
:Type: 64-bit Integer
:Default: ``0``
-``fatal signal handlers``
+``fatal_signal_handlers``
:Description: If set, we will install signal handlers for SEGV, ABRT, BUS, ILL,
FPE, XCPU, XFSZ, SYS signals to generate a useful log message
Configuration
===============
-Each Ceph process, daemon or utility draws its configuration from
+Each Ceph process, daemon, or utility draws its configuration from
several sources on startup, include a local configuration, the
monitors, the command line, or environment variables. Configuration
options may be set globally such that they apply to all daemons, to
.. index:: journal; journal configuration
-Ceph OSDs use a journal for two reasons: speed and consistency.
+Filestore OSDs use a journal for two reasons: speed and consistency. Note
+that since Luminous, the BlueStore OSD back end has been preferred and default.
+This information is provided for pre-existing OSDs and for rare situations where
+Filestore is preferred for new deployments.
- **Speed:** The journal enables the Ceph OSD Daemon to commit small writes
quickly. Ceph writes small, random i/o to the journal sequentially, which
OSD Daemons replay the journal starting after the last synchronization
operation.
-Ceph OSD Daemons support the following journal settings:
+Ceph OSD Daemons recognize the following journal settings:
-``journal dio``
+``journal_dio``
:Description: Enables direct i/o to the journal. Requires ``journal block
align`` set to ``true``.
-``journal aio``
+``journal_aio``
.. versionchanged:: 0.61 Cuttlefish
:Default: Version 0.61 and later, ``true``. Version 0.60 and earlier, ``false``.
-``journal block align``
+``journal_block_align``
:Description: Block aligns write operations. Required for ``dio`` and ``aio``.
:Type: Boolean
:Default: ``true``
-``journal max write bytes``
+``journal_max_write_bytes``
:Description: The maximum number of bytes the journal will write at
any one time.
:Default: ``10 << 20``
-``journal max write entries``
+``journal_max_write_entries``
:Description: The maximum number of entries the journal will write at
any one time.
:Default: ``100``
-``journal queue max ops``
+``journal_queue_max_ops``
:Description: The maximum number of operations allowed in the queue at
any one time.
:Default: ``500``
-``journal queue max bytes``
+``journal_queue_max_bytes``
:Description: The maximum number of bytes allowed in the queue at
any one time.
:Default: ``10 << 20``
-``journal align min size``
+``journal_align_min_size``
:Description: Align data payloads greater than the specified minimum.
:Type: Integer
:Default: ``64 << 10``
-``journal zero on create``
+``journal_zero_on_create``
:Description: Causes the file store to overwrite the entire journal with
``0``'s during ``mkfs``.
Understanding how to configure a :term:`Ceph Monitor` is an important part of
building a reliable :term:`Ceph Storage Cluster`. **All Ceph Storage Clusters
-have at least one monitor**. A monitor configuration usually remains fairly
+have at least one monitor**. The monitor complement usually remains fairly
consistent, but you can add, remove or replace a monitor in a cluster. See
`Adding/Removing a Monitor`_ for details.
Monitor that has an older version of the monmap to catch up with the current
state of the Ceph Storage Cluster.
-If Ceph Monitors discovered each other through the Ceph configuration file
-instead of through the monmap, it would introduce additional risks because the
+If Ceph Monitors were to discover each other through the Ceph configuration file
+instead of through the monmap, additional risks would be introduced because
Ceph configuration files are not updated and distributed automatically. Ceph
Monitors might inadvertently use an older Ceph configuration file, fail to
recognize a Ceph Monitor, fall out of a quorum, or develop a situation where
Bootstrapping Monitors
----------------------
-In most configuration and deployment cases, tools that deploy Ceph may help
+In most configuration and deployment cases, tools that deploy Ceph help
bootstrap the Ceph Monitors by generating a monitor map for you (e.g.,
``cephadm``, etc). A Ceph Monitor requires a few explicit
settings:
---------------------
The bare minimum monitor settings for a Ceph monitor via the Ceph configuration
-file include a hostname and a monitor address for each monitor. You can configure
+file include a hostname and a network address for each monitor. You can configure
these under ``[mon]`` or under the entry for a specific monitor.
.. code-block:: ini
.. note:: This minimum configuration for monitors assumes that a deployment
tool generates the ``fsid`` and the ``mon.`` key for you.
-Once you deploy a Ceph cluster, you **SHOULD NOT** change the IP address of
-the monitors. However, if you decide to change the monitor's IP address, you
+Once you deploy a Ceph cluster, you **SHOULD NOT** change the IP addresses of
+monitors. However, if you decide to change the monitor's IP address, you
must follow a specific procedure. See `Changing a Monitor's IP Address`_ for
details.
-Monitors can also be found by clients using DNS SRV records. See `Monitor lookup through DNS`_ for details.
+Monitors can also be found by clients by using DNS SRV records. See `Monitor lookup through DNS`_ for details.
Cluster ID
----------
.. code-block:: ini
[mon]
- mon initial members = a,b,c
+ mon_initial_members = a,b,c
-``mon initial members``
+``mon_initial_members``
:Description: The IDs of initial monitors in a cluster during startup. If
specified, Ceph requires an odd number of monitors to form an
Ceph provides a default path where Ceph Monitors store data. For optimal
performance in a production Ceph Storage Cluster, we recommend running Ceph
-Monitors on separate hosts and drives from Ceph OSD Daemons. As leveldb is using
+Monitors on separate hosts and drives from Ceph OSD Daemons. As leveldb uses
``mmap()`` for writing the data, Ceph Monitors flush their data from memory to disk
very often, which can interfere with Ceph OSD Daemon workloads if the data
store is co-located with the OSD Daemons.
-In Ceph versions 0.58 and earlier, Ceph Monitors store their data in files. This
+In Ceph versions 0.58 and earlier, Ceph Monitors store their data in plain files. This
approach allows users to inspect monitor data with common tools like ``ls``
-and ``cat``. However, it doesn't provide strong consistency.
+and ``cat``. However, this approach didn't provide strong consistency.
In Ceph versions 0.59 and later, Ceph Monitors store their data as key/value
pairs. Ceph Monitors require `ACID`_ transactions. Using a data store prevents
by setting it in the ``[mon]`` section of the configuration file.
-``mon data``
+``mon_data``
:Description: The monitor's data location.
:Type: String
:Default: ``/var/lib/ceph/mon/$cluster-$id``
-``mon data size warn``
+``mon_data_size_warn``
-:Description: Issue a ``HEALTH_WARN`` in cluster log when the monitor's data
- store goes over 15GB.
+:Description: Raise ``HEALTH_WARN`` status when a monitor's data
+ store grows to be larger than this size, 15GB by default.
:Type: Integer
:Default: ``15*1024*1024*1024``
-``mon data avail warn``
+``mon_data_avail_warn``
-:Description: Issue a ``HEALTH_WARN`` in cluster log when the available disk
- space of monitor's data store is lower or equal to this
- percentage.
+:Description: Raise ``HEALTH_WARN`` status when the filesystem that houses a
+ monitor's data store reports that its available capacity is
+ less than or equal to this percentage .
:Type: Integer
:Default: ``30``
-``mon data avail crit``
+``mon_data_avail_crit``
-:Description: Issue a ``HEALTH_ERR`` in cluster log when the available disk
- space of monitor's data store is lower or equal to this
- percentage.
+:Description: Raise ``HEALTH_ERR`` status when the filesystem that houses a
+ monitor's data store reports that its available capacity is
+ less than or equal to this percentage.
:Type: Integer
:Default: ``5``
+``mon_warn_on_cache_pools_without_hit_sets``
-``mon warn on cache pools without hit sets``
-
-:Description: Issue a ``HEALTH_WARN`` in cluster log if a cache pool does not
+:Description: Raise ``HEALTH_WARN`` when a cache pool does not
have the ``hit_set_type`` value configured.
See :ref:`hit_set_type <hit_set_type>` for more
details.
:Type: Boolean
:Default: ``True``
+``mon_warn_on_crush_straw_calc_version_zero``
-``mon warn on crush straw calc version zero``
-
-:Description: Issue a ``HEALTH_WARN`` in cluster log if the CRUSH's
+:Description: Raise ``HEALTH_WARN`` when the CRUSH
``straw_calc_version`` is zero. See
:ref:`CRUSH map tunables <crush-map-tunables>` for
details.
:Default: ``True``
-``mon warn on legacy crush tunables``
+``mon_warn_on_legacy_crush_tunables``
-:Description: Issue a ``HEALTH_WARN`` in cluster log if
+:Description: Raise ``HEALTH_WARN`` when
CRUSH tunables are too old (older than ``mon_min_crush_required_version``)
:Type: Boolean
:Default: ``True``
-``mon crush min required version``
+``mon_crush_min_required_version``
-:Description: The minimum tunable profile version required by the cluster.
+:Description: The minimum tunable profile required by the cluster.
See
:ref:`CRUSH map tunables <crush-map-tunables>` for
details.
:Default: ``hammer``
-``mon warn on osd down out interval zero``
+``mon_warn_on_osd_down_out_interval_zero``
-:Description: Issue a ``HEALTH_WARN`` in cluster log if
- ``mon osd down out interval`` is zero. Having this option set to
+:Description: Raise ``HEALTH_WARN`` when
+ ``mon_osd_down_out_interval`` is zero. Having this option set to
zero on the leader acts much like the ``noout`` flag. It's hard
to figure out what's going wrong with clusters without the
``noout`` flag set but acting like that just the same, so we
:Default: ``True``
-``mon warn on slow ping ratio``
+``mon_warn_on_slow_ping_ratio``
-:Description: Issue a ``HEALTH_WARN`` in cluster log if any heartbeat
- between OSDs exceeds ``mon warn on slow ping ratio``
- of ``osd heartbeat grace``. The default is 5%.
+:Description: Raise ``HEALTH_WARN`` when any heartbeat
+ between OSDs exceeds ``mon_warn_on_slow_ping_ratio``
+ of ``osd_heartbeat_grace``. The default is 5%.
:Type: Float
:Default: ``0.05``
-``mon warn on slow ping time``
+``mon_warn_on_slow_ping_time``
-:Description: Override ``mon warn on slow ping ratio`` with a specific value.
- Issue a ``HEALTH_WARN`` in cluster log if any heartbeat
- between OSDs exceeds ``mon warn on slow ping time``
+:Description: Override ``mon_warn_on_slow_ping_ratio`` with a specific value.
+ Raise ``HEALTH_WARN`` if any heartbeat
+ between OSDs exceeds ``mon_warn_on_slow_ping_time``
milliseconds. The default is 0 (disabled).
:Type: Integer
:Default: ``0``
-``mon warn on pool no redundancy``
+``mon_warn_on_pool_no_redundancy``
-:Description: Issue a ``HEALTH_WARN`` in cluster log if any pool is
+:Description: Raise ``HEALTH_WARN`` if any pool is
configured with no replicas.
:Type: Boolean
:Default: ``True``
-``mon cache target full warn ratio``
+``mon_cache_target_full_warn_ratio``
:Description: Position between pool's ``cache_target_full`` and
``target_max_object`` where we start warning
:Default: ``0.66``
-``mon health to clog``
+``mon_health_to_clog``
-:Description: Enable sending health summary to cluster log periodically.
+:Description: Enable sending a health summary to the cluster log periodically.
:Type: Boolean
:Default: ``True``
-``mon health to clog tick interval``
+``mon_health_to_clog_tick_interval``
-:Description: How often (in seconds) the monitor send health summary to cluster
- log (a non-positive number disables it). If current health summary
+:Description: How often (in seconds) the monitor sends a health summary to the cluster
+ log (a non-positive number disables). If current health summary
is empty or identical to the last time, monitor will not send it
to cluster log.
:Default: ``60.0``
-``mon health to clog interval``
+``mon_health_to_clog_interval``
-:Description: How often (in seconds) the monitor send health summary to cluster
- log (a non-positive number disables it). Monitor will always
- send the summary to cluster log no matter if the summary changes
- or not.
+:Description: How often (in seconds) the monitor sends a health summary to the cluster
+ log (a non-positive number disables). Monitors will always
+ send a summary to the cluster log whether or not it differs from
+ the previous summary.
:Type: Integer
:Default: ``3600``
Storage Capacity
----------------
-When a Ceph Storage Cluster gets close to its maximum capacity (i.e., ``mon osd
-full ratio``), Ceph prevents you from writing to or reading from Ceph OSD
-Daemons as a safety measure to prevent data loss. Therefore, letting a
+When a Ceph Storage Cluster gets close to its maximum capacity
+(see``mon_osd_full ratio``), Ceph prevents you from writing to or reading from OSDs
+as a safety measure to prevent data loss. Therefore, letting a
production Ceph Storage Cluster approach its full ratio is not a good practice,
because it sacrifices high availability. The default full ratio is ``.95``, or
95% of capacity. This a very aggressive setting for a test cluster with a small
in a temporary service disruption if one or more OSDs fails. Consider adding
more OSDs to increase storage capacity.
-A common scenario for test clusters involves a system administrator removing a
-Ceph OSD Daemon from the Ceph Storage Cluster to watch the cluster rebalance;
-then, removing another Ceph OSD Daemon, and so on until the Ceph Storage Cluster
-eventually reaches the full ratio and locks up. We recommend a bit of capacity
+A common scenario for test clusters involves a system administrator removing an
+OSD from the Ceph Storage Cluster, watching the cluster rebalance, then removing
+another OSD, and another, until at least one OSD eventually reaches the full
+ratio and the cluster locks up. We recommend a bit of capacity
planning even with a test cluster. Planning enables you to gauge how much spare
capacity you will need in order to maintain high availability. Ideally, you want
to plan for a series of Ceph OSD Daemon failures where the cluster can recover
-to an ``active + clean`` state without replacing those Ceph OSD Daemons
-immediately. You can run a cluster in an ``active + degraded`` state, but this
-is not ideal for normal operating conditions.
+to an ``active+clean`` state without replacing those OSDs
+immediately. Cluster operation continues in the ``active+degraded`` state, but this
+is not ideal for normal operation and should be addressed promptly.
The following diagram depicts a simplistic Ceph Storage Cluster containing 33
-Ceph Nodes with one Ceph OSD Daemon per host, each Ceph OSD Daemon reading from
+Ceph Nodes with one OSD per host, each OSD reading from
and writing to a 3TB drive. So this exemplary Ceph Storage Cluster has a maximum
actual capacity of 99TB. With a ``mon osd full ratio`` of ``0.95``, if the Ceph
Storage Cluster falls to 5TB of remaining capacity, the cluster will not allow
a reasonable number for a near full ratio.
The following settings only apply on cluster creation and are then stored in
-the OSDMap.
+the OSDMap. To clarify, in normal operation the values that are used by OSDs
+are those found in the OSDMap, not those in the configuration file or central
+config store.
.. code-block:: ini
[global]
-
- mon osd full ratio = .80
- mon osd backfillfull ratio = .75
- mon osd nearfull ratio = .70
+ mon_osd_full_ratio = .80
+ mon_osd_backfillfull_ratio = .75
+ mon_osd_nearfull_ratio = .70
-``mon osd full ratio``
+``mon_osd_full_ratio``
-:Description: The percentage of disk space used before an OSD is
+:Description: The threshold percentage of device space utilized before an OSD is
considered ``full``.
:Type: Float
:Default: ``0.95``
-``mon osd backfillfull ratio``
+``mon_osd_backfillfull_ratio``
-:Description: The percentage of disk space used before an OSD is
+:Description: The threshold percentage of device space utilized before an OSD is
considered too ``full`` to backfill.
:Type: Float
:Default: ``0.90``
-``mon osd nearfull ratio``
+``mon_osd_nearfull_ratio``
-:Description: The percentage of disk space used before an OSD is
+:Description: The threshold percentage of device space used before an OSD is
considered ``nearfull``.
:Type: Float
.. tip:: If some OSDs are nearfull, but others have plenty of capacity, you
- may have a problem with the CRUSH weight for the nearfull OSDs.
+ may have an inaccurate CRUSH weight set for the nearfull OSDs.
.. tip:: These settings only apply during cluster creation. Afterwards they need
to be changed in the OSDMap using ``ceph osd set-nearfull-ratio`` and
another. If this happens while synchronizing (e.g., a provider falls behind the
leader), the provider can terminate synchronization with a requester.
-Once synchronization is complete, Ceph requires trimming across the cluster.
-Trimming requires that the placement groups are ``active + clean``.
+Once synchronization is complete, Ceph performs trimming across the cluster.
+Trimming requires that the placement groups are ``active+clean``.
-``mon sync timeout``
+``mon_sync_timeout``
:Description: Number of seconds the monitor will wait for the next update
message from its sync provider before it gives up and bootstrap
:Default: ``60.0``
-``mon sync max payload size``
+``mon_sync_max_payload_size``
:Description: The maximum size for a sync payload (in bytes).
:Type: 32-bit Integer
:Default: ``1048576``
-``paxos max join drift``
+``paxos_max_join_drift``
:Description: The maximum Paxos iterations before we must first sync the
monitor data stores. When a monitor finds that its peer is too
:Default: ``10``
-``paxos stash full interval``
+``paxos_stash_full_interval``
:Description: How often (in commits) to stash a full copy of the PaxosService state.
Current this setting only affects ``mds``, ``mon``, ``auth`` and ``mgr``
:Default: ``25``
-``paxos propose interval``
+``paxos_propose_interval``
:Description: Gather updates for this time interval before proposing
a map update.
:Default: ``1.0``
-``paxos min``
+``paxos_min``
-:Description: The minimum number of paxos states to keep around
+:Description: The minimum number of Paxos states to keep around
:Type: Integer
:Default: ``500``
-``paxos min wait``
+``paxos_min_wait``
:Description: The minimum amount of time to gather updates after a period of
inactivity.
:Default: ``0.05``
-``paxos trim min``
+``paxos_trim_min``
:Description: Number of extra proposals tolerated before trimming
:Type: Integer
:Default: ``250``
-``paxos trim max``
+``paxos_trim_max``
:Description: The maximum number of extra proposals to trim at a time
:Type: Integer
:Default: ``500``
-``paxos service trim min``
+``paxos_service_trim_min``
:Description: The minimum amount of versions to trigger a trim (0 disables it)
:Type: Integer
:Default: ``250``
-``paxos service trim max``
+``paxos_service_trim_max``
:Description: The maximum amount of versions to trim during a single proposal (0 disables it)
:Type: Integer
:Default: ``500``
-``mon mds force trim to``
+``mon_mds_force_trim_to``
:Description: Force monitor to trim mdsmaps to this point (0 disables it.
dangerous, use with care)
:Default: ``0``
-``mon osd force trim to``
+``mon_osd_force_trim_to``
:Description: Force monitor to trim osdmaps to this point, even if there is
PGs not clean at the specified epoch (0 disables it. dangerous,
:Default: ``0``
-``mon osd cache size``
+``mon_osd_cache_size``
:Description: The size of osdmaps cache, not to rely on underlying store's cache
:Type: Integer
:Default: ``500``
-``mon election timeout``
+``mon_election_timeout``
:Description: On election proposer, maximum waiting time for all ACKs in seconds.
:Type: Float
:Default: ``5.00``
-``mon lease``
+``mon_lease``
:Description: The length (in seconds) of the lease on the monitor's versions.
:Type: Float
:Default: ``5.00``
-``mon lease renew interval factor``
+``mon_lease_renew_interval_factor``
-:Description: ``mon lease`` \* ``mon lease renew interval factor`` will be the
+:Description: ``mon_lease`` \* ``mon_lease_renew_interval_factor`` will be the
interval for the Leader to renew the other monitor's leases. The
factor should be less than ``1.0``.
:Default: ``0.60``
-``mon lease ack timeout factor``
+``mon_lease_ack_timeout_factor``
-:Description: The Leader will wait ``mon lease`` \* ``mon lease ack timeout factor``
+:Description: The Leader will wait ``mon_lease`` \* ``mon_lease_ack_timeout_factor``
for the Providers to acknowledge the lease extension.
:Type: Float
:Default: ``2.00``
-``mon accept timeout factor``
+``mon_accept_timeout_factor``
-:Description: The Leader will wait ``mon lease`` \* ``mon accept timeout factor``
+:Description: The Leader will wait ``mon_lease`` \* ``mon_accept_timeout_factor``
for the Requester(s) to accept a Paxos update. It is also used
during the Paxos recovery phase for similar purposes.
:Default: ``2.00``
-``mon min osdmap epochs``
+``mon_min_osdmap_epochs``
:Description: Minimum number of OSD map epochs to keep at all times.
:Type: 32-bit Integer
:Default: ``500``
-``mon max log epochs``
+``mon_max_log_epochs``
:Description: Maximum number of Log epochs the monitor should keep.
:Type: 32-bit Integer
See `Monitor Store Synchronization`_ for details.
-.. tip:: You SHOULD install NTP on your Ceph monitor hosts to
+.. tip:: You must configure NTP or PTP daemons on your Ceph monitor hosts to
ensure that the monitor cluster operates with synchronized clocks.
+ It can be advantageous to have monitor hosts sync with each other
+ as well as with multiple quality upstream time sources.
Clock drift may still be noticeable with NTP even though the discrepancy is not
yet harmful. Ceph's clock drift / clock skew warnings may get triggered even
acceptable values.
-``mon tick interval``
+``mon_tick_interval``
:Description: A monitor's tick interval in seconds.
:Type: 32-bit Integer
:Default: ``5``
-``mon clock drift allowed``
+``mon_clock_drift_allowed``
:Description: The clock drift in seconds allowed between monitors.
:Type: Float
:Default: ``0.05``
-``mon clock drift warn backoff``
+``mon_clock_drift_warn_backoff``
:Description: Exponential backoff for clock drift warnings
:Type: Float
:Default: ``5.00``
-``mon timecheck interval``
+``mon_timecheck_interval``
:Description: The time check interval (clock drift check) in seconds
for the Leader.
:Default: ``300.00``
-``mon timecheck skew interval``
+``mon_timecheck_skew_interval``
:Description: The time check interval (clock drift check) in seconds when in
presence of a skew in seconds for the Leader.
Client
------
-``mon client hunt interval``
+``mon_client_hunt_interval``
:Description: The client will try a new monitor every ``N`` seconds until it
establishes a connection.
:Default: ``3.00``
-``mon client ping interval``
+``mon_client_ping_interval``
:Description: The client will ping the monitor every ``N`` seconds.
:Type: Double
:Default: ``10.00``
-``mon client max log entries per message``
+``mon_client_max_log_entries_per_message``
:Description: The maximum number of log entries a monitor will generate
per client message.
:Default: ``1000``
-``mon client bytes``
+``mon_client_bytes``
:Description: The amount of client message data allowed in memory (in bytes).
:Type: 64-bit Integer Unsigned
=============
Since version v0.94 there is support for pool flags which allow or disallow changes to be made to pools.
+Monitors can also disallow removal of pools if appropriately configured. The inconvenience of this guardrail
+is far outweighed by the number of accidental pool (and thus data) deletions it prevents.
-Monitors can also disallow removal of pools if configured that way.
+``mon_allow_pool_delete``
-``mon allow pool delete``
+:Description: Should monitors allow pools to be removed, regardless of what the pool flags say?
-:Description: If the monitors should allow pools to be removed. Regardless of what the pool flags say.
:Type: Boolean
:Default: ``false``
-``osd pool default ec fast read``
+``osd_pool_default_ec_fast_read``
:Description: Whether to turn on fast read on the pool or not. It will be used as
the default setting of newly created erasure coded pools if ``fast_read``
:Default: ``false``
-``osd pool default flag hashpspool``
+``osd_pool_default_flag_hashpspool``
:Description: Set the hashpspool flag on new pools
:Type: Boolean
:Default: ``true``
-``osd pool default flag nodelete``
+``osd_pool_default_flag_nodelete``
-:Description: Set the nodelete flag on new pools. Prevents allow pool removal with this flag in any way.
+:Description: Set the ``nodelete`` flag on new pools, which prevents pool removal.
:Type: Boolean
:Default: ``false``
-``osd pool default flag nopgchange``
+``osd_pool_default_flag_nopgchange``
-:Description: Set the nopgchange flag on new pools. Does not allow the number of PGs to be changed for a pool.
+:Description: Set the ``nopgchange`` flag on new pools. Does not allow the number of PGs to be changed.
:Type: Boolean
:Default: ``false``
-``osd pool default flag nosizechange``
+``osd_pool_default_flag_nosizechange``
-:Description: Set the nosizechange flag on new pools. Does not allow the size to be changed of pool.
+:Description: Set the ``nosizechange`` flag on new pools. Does not allow the ``size`` to be changed.
:Type: Boolean
:Default: ``false``
Miscellaneous
=============
-``mon max osd``
+``mon_max_osd``
:Description: The maximum number of OSDs allowed in the cluster.
:Type: 32-bit Integer
:Default: ``10000``
-``mon globalid prealloc``
+``mon_globalid_prealloc``
:Description: The number of global IDs to pre-allocate for clients and daemons in the cluster.
:Type: 32-bit Integer
:Default: ``10000``
-``mon subscribe interval``
+``mon_subscribe_interval``
:Description: The refresh interval (in seconds) for subscriptions. The
- subscription mechanism enables obtaining the cluster maps
+ subscription mechanism enables obtaining cluster maps
and log information.
:Type: Double
:Default: ``86400.00``
-``mon stat smooth intervals``
+``mon_stat_smooth_intervals``
:Description: Ceph will smooth statistics over the last ``N`` PG maps.
:Type: Integer
:Default: ``6``
-``mon probe timeout``
+``mon_probe_timeout``
:Description: Number of seconds the monitor will wait to find peers before bootstrapping.
:Type: Double
:Default: ``2.00``
-``mon daemon bytes``
+``mon_daemon_bytes``
:Description: The message memory cap for metadata server and OSD messages (in bytes).
:Type: 64-bit Integer Unsigned
:Default: ``400ul << 20``
-``mon max log entries per event``
+``mon_max_log_entries_per_event``
:Description: The maximum number of log entries per event.
:Type: Integer
:Default: ``4096``
-``mon osd prime pg temp``
+``mon_osd_prime_pg_temp``
-:Description: Enables or disable priming the PGMap with the previous OSDs when an out
- OSD comes back into the cluster. With the ``true`` setting the clients
- will continue to use the previous OSDs until the newly in OSDs as that
- PG peered.
+:Description: Enables or disables priming the PGMap with the previous OSDs when an ``out``
+ OSD comes back into the cluster. With the ``true`` setting, clients
+ will continue to use the previous OSDs until the newly ``in`` OSDs for
+ a PG have peered.
:Type: Boolean
:Default: ``true``
-``mon osd prime pg temp max time``
+``mon_osd_prime pg temp max time``
:Description: How much time in seconds the monitor should spend trying to prime the
PGMap when an out OSD comes back into the cluster.
:Default: ``0.50``
-``mon osd prime pg temp max time estimate``
+``mon_osd_prime_pg_temp_max_time_estimate``
:Description: Maximum estimate of time spent on each PG before we prime all PGs
in parallel.
:Default: ``0.25``
-``mon mds skip sanity``
+``mon_mds_skip_sanity``
:Description: Skip safety assertions on FSMap (in case of bugs where we want to
continue anyway). Monitor terminates if the FSMap sanity check
:Default: ``False``
-``mon max mdsmap epochs``
+``mon_max_mdsmap_epochs``
-:Description: The maximum amount of mdsmap epochs to trim during a single proposal.
+:Description: The maximum number of mdsmap epochs to trim during a single proposal.
:Type: Integer
:Default: ``500``
-``mon config key max entry size``
+``mon_config_key_max_entry_size``
:Description: The maximum size of config-key entry (in bytes)
:Type: Integer
:Default: ``65536``
-``mon scrub interval``
+``mon_scrub_interval``
-:Description: How often the monitor scrub its store by comparing
- the stored checksums with the computed ones of all the stored
+:Description: How often the monitor scrubs its store by comparing
+ the stored checksums with the computed ones for all stored
keys. (0 disables it. dangerous, use with care)
:Type: Seconds
:Default: ``1 day``
-``mon scrub max keys``
+``mon_scrub_max_keys``
:Description: The maximum number of keys to scrub each time.
:Type: Integer
:Default: ``100``
-``mon compact on start``
+``mon_compact_on_start``
:Description: Compact the database used as Ceph Monitor store on
``ceph-mon`` start. A manual compaction helps to shrink the
:Default: ``False``
-``mon compact on bootstrap``
+``mon_compact_on_bootstrap``
-:Description: Compact the database used as Ceph Monitor store on
- on bootstrap. Monitor starts probing each other for creating
- a quorum after bootstrap. If it times out before joining the
- quorum, it will start over and bootstrap itself again.
+:Description: Compact the database used as Ceph Monitor store
+ on bootstrap. Monitors probe each other to establish
+ a quorum after bootstrap. If a monitor times out before joining the
+ quorum, it will start over and bootstrap again.
:Type: Boolean
:Default: ``False``
-``mon compact on trim``
+``mon_compact_on_trim``
:Description: Compact a certain prefix (including paxos) when we trim its old states.
:Type: Boolean
:Default: ``True``
-``mon cpu threads``
+``mon_cpu_threads``
:Description: Number of threads for performing CPU intensive work on monitor.
:Type: Integer
:Default: ``4``
-``mon osd mapping pgs per chunk``
+``mon_osd_mapping_pgs_per_chunk``
:Description: We calculate the mapping from placement group to OSDs in chunks.
This option specifies the number of placement groups per chunk.
:Default: ``4096``
-``mon session timeout``
+``mon_session_timeout``
:Description: Monitor will terminate inactive sessions stay idle over this
time limit.
:Default: ``300``
-``mon osd cache size min``
+``mon_osd_cache_size_min``
:Description: The minimum amount of bytes to be kept mapped in memory for osd
monitor caches.
:Default: ``134217728``
-``mon memory target``
+``mon_memory_target``
-:Description: The amount of bytes pertaining to osd monitor caches and kv cache
+:Description: The amount of bytes pertaining to OSD monitor caches and KV cache
to be kept mapped in memory with cache auto-tuning enabled.
:Type: 64-bit Integer
:Default: ``2147483648``
-``mon memory autotune``
+``mon_memory_autotune``
-:Description: Autotune the cache memory being used for osd monitors and kv
+:Description: Autotune the cache memory used for OSD monitors and KV
database.
:Type: Boolean
:Default: ``2``
-``mon osd reporter subtree level``
+``mon_osd_reporter_subtree_level``
:Description: In which level of parent bucket the reporters are counted. The OSDs
- send failure reports to monitor if they find its peer is not responsive.
- And monitor mark the reported OSD out and then down after a grace period.
+ send failure reports to monitors if they find a peer that is not responsive.
+ Monitors mark the reported ``OSD`` out and then ``down`` after a grace period.
:Type: String
:Default: ``host``
OSD Settings
------------
-``osd heartbeat address``
+``osd_heartbeat_address``
:Description: An Ceph OSD Daemon's network address for heartbeats.
:Type: Address
:Default: The host address.
-``osd heartbeat interval``
+``osd_heartbeat_interval``
:Description: How often an Ceph OSD Daemon pings its peers (in seconds).
:Type: 32-bit Integer
:Default: ``6``
-``osd heartbeat grace``
+``osd_heartbeat_grace``
:Description: The elapsed time when a Ceph OSD Daemon hasn't shown a heartbeat
that the Ceph Storage Cluster considers it ``down``.
- This setting has to be set in both the [mon] and [osd] or [global]
- section so that it is read by both the MON and OSD daemons.
+ This setting must be set in both the [mon] and [osd] or [global]
+ sections so that it is read by both monitor and OSD daemons.
:Type: 32-bit Integer
:Default: ``20``
-``osd mon heartbeat interval``
+``osd_mon_heartbeat_interval``
:Description: How often the Ceph OSD Daemon pings a Ceph Monitor if it has no
Ceph OSD Daemon peers.
:Default: ``30``
-``osd mon heartbeat stat stale``
+``osd_mon_heartbeat_stat_stale``
:Description: Stop reporting on heartbeat ping times which haven't been updated for
this many seconds. Set to zero to disable this action.
:Default: ``3600``
-``osd mon report interval``
+``osd_mon_report_interval``
:Description: The number of seconds a Ceph OSD Daemon may wait
from startup or another reportable event before reporting
:Default: ``5``
-``osd mon ack timeout``
+``osd_mon_ack_timeout``
:Description: The number of seconds to wait for a Ceph Monitor to acknowledge a
request for statistics.
General Settings
================
-``ms tcp nodelay``
+``ms_tcp_nodelay``
-:Description: Disables nagle's algorithm on messenger tcp sessions.
+:Description: Disables Nagle's algorithm on messenger TCP sessions.
:Type: Boolean
:Required: No
:Default: ``true``
-``ms initial backoff``
+``ms_initial_backoff``
:Description: The initial time to wait before reconnecting on a fault.
:Type: Double
:Default: ``.2``
-``ms max backoff``
+``ms_max_backoff``
:Description: The maximum time to wait before reconnecting on a fault.
:Type: Double
:Default: ``15.0``
-``ms nocrc``
+``ms_nocrc``
-:Description: Disables crc on network messages. May increase performance if cpu limited.
+:Description: Disables CRC on network messages. May increase performance if CPU limited.
:Type: Boolean
:Required: No
:Default: ``false``
-``ms die on bad msg``
+``ms_die_on_bad_msg``
:Description: Debug option; do not configure.
:Type: Boolean
:Default: ``false``
-``ms dispatch throttle bytes``
+``ms_dispatch_throttle_bytes``
:Description: Throttles total size of messages waiting to be dispatched.
:Type: 64-bit Unsigned Integer
:Default: ``100 << 20``
-``ms bind ipv6``
+``ms_bind_ipv6``
-:Description: Enable if you want your daemons to bind to IPv6 address instead of IPv4 ones. (Not required if you specify a daemon or cluster IP.)
+:Description: Enable to bind daemons to IPv6 addresses instead of IPv4. Not required if you specify a daemon or cluster IP.
:Type: Boolean
:Required: No
:Default: ``false``
-``ms rwthread stack bytes``
+``ms_rwthread_stack_bytes``
:Description: Debug option for stack size; do not configure.
:Type: 64-bit Unsigned Integer
:Default: ``1024 << 10``
-``ms tcp read timeout``
+``ms_tcp_read_timeout``
:Description: Controls how long (in seconds) the messenger will wait before closing an idle connection.
:Type: 64-bit Unsigned Integer
:Default: ``900``
-``ms inject socket failures``
+``ms_inject_socket_failures``
:Description: Debug option; do not configure.
:Type: 64-bit Unsigned Integer
=======================
-``ms async transport type``
+``ms_async_transport_type``
:Description: Transport type used by Async Messenger. Can be ``posix``, ``dpdk``
or ``rdma``. Posix uses standard TCP/IP networking and is default.
:Default: ``posix``
-``ms async op threads``
+``ms_async_op_threads``
:Description: Initial number of worker threads used by each Async Messenger instance.
Should be at least equal to highest number of replicas, but you can
:Default: ``3``
-``ms async max op threads``
+``ms_async_max_op_threads``
:Description: Maximum number of worker threads used by each Async Messenger instance.
Set to lower values when your machine has limited CPU count, and increase
:Default: ``5``
-``ms async send inline``
+``ms_async_send_inline``
:Description: Send messages directly from the thread that generated them instead of
queuing and sending from Async Messenger thread. This option is known
Address formats
---------------
-Prior to nautilus, all network addresses were rendered like
+Prior to Nautilus, all network addresses were rendered like
``1.2.3.4:567/89012`` where there was an IP address, a port, and a
nonce to uniquely identify a client or daemon on the network.
-Starting with nautilus, we now have three different address types:
+Starting with Nautilus, we now have three different address types:
* **v2**: ``v2:1.2.3.4:578/89012`` identifies a daemon binding to a
port speaking the new v2 protocol
improvement with a second "cluster" network in a large cluster.
It is possible to run a Ceph Storage Cluster with two networks: a public
-(front-side) network and a cluster (back-side) network. However, this approach
+(client, front-side) network and a cluster (private, replication, back-side)
+network. However, this approach
complicates network configuration (both hardware and software) and does not usually
have a significant impact on overall performance. For this reason, we recommend
that for resilience and capacity dual-NIC systems either active/active bond
these interfaces or implemebnt a layer 3 multipath strategy with eg. FRR.
If, despite the complexity, one still wishes to use two networks, each
-:term:`Ceph Node` will need to have more than one NIC. See `Hardware
+:term:`Ceph Node` will need to have more than one network interface or VLAN. See `Hardware
Recommendations - Networks`_ for additional details.
.. ditaa::
[global]
# ... elided configuration
- public network = {public-network/netmask}
+ public_network = {public-network/netmask}
Cluster Network
[global]
# ... elided configuration
- cluster network = {cluster-network/netmask}
+ cluster_network = {cluster-network/netmask}
We prefer that the cluster network is **NOT** reachable from the public network
or the Internet for added security.
[global]
# ... elided configuration
- public network = {IPv4 public-network/netmask}, {IPv6 public-network/netmask}
+ public_network = {IPv4 public-network/netmask}, {IPv6 public-network/netmask}
-This is so ceph can find a valid IP address for both address families.
+This is so that Ceph can find a valid IP address for both address families.
If you want just an IPv4 or an IPv6 stack environment, then make sure you set the `ms bind`
options correctly.
Ceph Daemons
============
-The monitor daemons are each configured to bind to a specific IP address. These addresses are normally configured by your deployment tool. Other components in the Ceph system discover the monitors via the ``mon host`` configuration option, normally specified in the ``[global]`` section of the ``ceph.conf`` file.
+Monitor daemons are each configured to bind to a specific IP address. These
+addresses are normally configured by your deployment tool. Other components
+in the Ceph cluster discover the monitors via the ``mon host`` configuration
+option, normally specified in the ``[global]`` section of the ``ceph.conf`` file.
.. code-block:: ini
[global]
- mon host = 10.0.0.2, 10.0.0.3, 10.0.0.4
+ mon_host = 10.0.0.2, 10.0.0.3, 10.0.0.4
-The ``mon host`` value can be a list of IP addresses or a name that is
+The ``mon_host`` value can be a list of IP addresses or a name that is
looked up via DNS. In the case of a DNS name with multiple A or AAAA
records, all records are probed in order to discover a monitor. Once
one monitor is reached, all other current monitors are discovered, so
.. topic:: One NIC OSD in a Two Network Cluster
- Generally, we do not recommend deploying an OSD host with a single NIC in a
+ Generally, we do not recommend deploying an OSD host with a single network interface in a
cluster with two networks. However, you may accomplish this by forcing the
- OSD host to operate on the public network by adding a ``public addr`` entry
+ OSD host to operate on the public network by adding a ``public_addr`` entry
to the ``[osd.n]`` section of the Ceph configuration file, where ``n``
- refers to the number of the OSD with one NIC. Additionally, the public
+ refers to the ID of the OSD with one network interface. Additionally, the public
network and cluster network must be able to route traffic to each other,
which we don't recommend for security reasons.
The public network configuration allows you specifically define IP addresses
and subnets for the public network. You may specifically assign static IP
-addresses or override ``public network`` settings using the ``public addr``
+addresses or override ``public_network`` settings using the ``public_addr``
setting for a specific daemon.
-``public network``
+``public_network``
:Description: The IP address and netmask of the public (front-side) network
(e.g., ``192.168.0.0/24``). Set in ``[global]``. You may specify
:Default: N/A
-``public addr``
+``public_addr``
:Description: The IP address for the public (front-side) network.
Set for each daemon.
The cluster network configuration allows you to declare a cluster network, and
specifically define IP addresses and subnets for the cluster network. You may
-specifically assign static IP addresses or override ``cluster network``
-settings using the ``cluster addr`` setting for specific OSD daemons.
+specifically assign static IP addresses or override ``cluster_network``
+settings using the ``cluster_addr`` setting for specific OSD daemons.
-``cluster network``
+``cluster_network``
:Description: The IP address and netmask of the cluster (back-side) network
(e.g., ``10.0.0.0/24``). Set in ``[global]``. You may specify
:Default: N/A
-``cluster addr``
+``cluster_addr``
:Description: The IP address for the cluster (back-side) network.
Set for each daemon.
addresses.
-``ms bind port min``
+``ms_bind_port_min``
:Description: The minimum port number to which an OSD or MDS daemon will bind.
:Type: 32-bit Integer
:Required: No
-``ms bind port max``
+``ms_bind_port_max``
:Description: The maximum port number to which an OSD or MDS daemon will bind.
:Type: 32-bit Integer
:Default: ``7300``
:Required: No.
-``ms bind ipv4``
+``ms_bind_ipv4``
:Description: Enables Ceph daemons to bind to IPv4 addresses.
:Type: Boolean
:Default: ``true``
:Required: No
-``ms bind ipv6``
+``ms_bind_ipv6``
:Description: Enables Ceph daemons to bind to IPv6 addresses.
:Type: Boolean
:Default: ``false``
:Required: No
-``public bind addr``
+``public_bind_addr``
:Description: In some dynamic deployments the Ceph MON daemon might bind
- to an IP address locally that is different from the ``public addr``
+ to an IP address locally that is different from the ``public_addr``
advertised to other peers in the network. The environment must ensure
- that routing rules are set correctly. If ``public bind addr`` is set
- the Ceph MON daemon will bind to it locally and use ``public addr``
+ that routing rules are set correctly. If ``public_bind_addr`` is set
+ the Ceph Monitor daemon will bind to it locally and use ``public_addr``
in the monmaps to advertise its address to peers. This behavior is limited
- to the MON daemon.
+ to the Monitor daemon.
:Type: IP Address
:Required: No
Ceph disables TCP buffering by default.
-``ms tcp nodelay``
+``ms_tcp_nodelay``
-:Description: Ceph enables ``ms tcp nodelay`` so that each request is sent
+:Description: Ceph enables ``ms_tcp_nodelay`` so that each request is sent
immediately (no buffering). Disabling `Nagle's algorithm`_
increases network traffic, which can introduce latency. If you
experience large numbers of small packets, you may try
- disabling ``ms tcp nodelay``.
+ disabling ``ms_tcp_nodelay``.
:Type: Boolean
:Required: No
:Default: ``true``
-
-``ms tcp rcvbuf``
+``ms_tcp_rcvbuf``
:Description: The size of the socket buffer on the receiving end of a network
connection. Disable by default.
:Default: ``0``
-
-``ms tcp read timeout``
+``ms_tcp_read_timeout``
:Description: If a client or daemon makes a request to another Ceph daemon and
does not drop an unused connection, the ``ms tcp read timeout``
.. index:: OSD; configuration
-You can configure Ceph OSD Daemons in the Ceph configuration file, but Ceph OSD
+You can configure Ceph OSD Daemons in the Ceph configuration file (or in recent
+releases, the central config store), but Ceph OSD
Daemons can use the default values and a very minimal configuration. A minimal
-Ceph OSD Daemon configuration sets ``osd journal size`` and ``host``, and
+Ceph OSD Daemon configuration sets ``osd journal size`` (for Filestore), ``host``, and
uses default values for nearly everything else.
Ceph OSD Daemons are numerically identified in incremental fashion, beginning
.. code-block:: ini
[osd]
- osd journal size = 5120
+ osd_journal_size = 5120
[osd.0]
host = osd-host-a
.. warning:: **DO NOT** change the default paths for data or journals, as it
makes it more problematic to troubleshoot Ceph later.
-The journal size should be at least twice the product of the expected drive
-speed multiplied by ``filestore max sync interval``. However, the most common
+When using Filestore, the journal size should be at least twice the product of the expected drive
+speed multiplied by ``filestore_max_sync_interval``. However, the most common
practice is to partition the journal drive (often an SSD), and mount it such
that Ceph uses the entire partition for the journal.
-``osd uuid``
+``osd_uuid``
:Description: The universally unique identifier (UUID) for the Ceph OSD Daemon.
:Type: UUID
:Default: The UUID.
-:Note: The ``osd uuid`` applies to a single Ceph OSD Daemon. The ``fsid``
+:Note: The ``osd_uuid`` applies to a single Ceph OSD Daemon. The ``fsid``
applies to the entire cluster.
-``osd data``
+``osd_data``
:Description: The path to the OSDs data. You must create the directory when
deploying Ceph. You should mount a drive for OSD data at this
:Default: ``/var/lib/ceph/osd/$cluster-$id``
-``osd max write size``
+``osd_max_write_size``
:Description: The maximum size of a write in megabytes.
:Type: 32-bit Integer
:Default: ``90``
-``osd max object size``
+``osd_max_object_size``
:Description: The maximum size of a RADOS object in bytes.
:Type: 32-bit Unsigned Integer
:Default: 128MB
-``osd client message size cap``
+``osd_client_message_size_cap``
:Description: The largest client data message allowed in memory.
:Type: 64-bit Unsigned Integer
:Default: 500MB default. ``500*1024L*1024L``
-``osd class dir``
+``osd_class_dir``
:Description: The class path for RADOS class plug-ins.
:Type: String
====================
Ceph builds and mounts file systems which are used for Ceph OSDs.
-``osd mkfs options {fs-type}``
+``osd_mkfs_options {fs-type}``
-:Description: Options used when creating a new Ceph OSD of type {fs-type}.
+:Description: Options used when creating a new Ceph Filestore OSD of type {fs-type}.
:Type: String
:Default for xfs: ``-f -i 2048``
:Default for other file systems: {empty string}
For example::
- ``osd mkfs options xfs = -f -d agcount=24``
+ ``osd_mkfs_options_xfs = -f -d agcount=24``
-``osd mount options {fs-type}``
+``osd_mount_options {fs-type}``
-:Description: Options used when mounting a Ceph OSD of type {fs-type}.
+:Description: Options used when mounting a Ceph Filestore OSD of type {fs-type}.
:Type: String
:Default for xfs: ``rw,noatime,inode64``
:Default for other file systems: ``rw, noatime``
For example::
- ``osd mount options xfs = rw, noatime, inode64, logbufs=8``
+ ``osd_mount_options_xfs = rw, noatime, inode64, logbufs=8``
.. index:: OSD; journal settings
Journal Settings
================
-By default, Ceph expects that you will store an Ceph OSD Daemons journal with
-the following path::
+This section applies only to the older Filestore OSD back end. Since Luminous
+BlueStore has been default and preferred.
+
+By default, Ceph expects that you will provision a Ceph OSD Daemon's journal at
+the following path, which is usually a symlink to a device or partition::
/var/lib/ceph/osd/$cluster-$id/journal
drives) it makes sense to place the journal on the faster device, while
``data`` occupies the slower device fully.
-The default ``osd journal size`` value is 5120 (5 gigabytes), but it can be
-larger, in which case it will need to be set in the ``ceph.conf`` file::
-
+The default ``osd_journal_size`` value is 5120 (5 gigabytes), but it can be
+larger, in which case it will need to be set in the ``ceph.conf`` file.
+A value of 10 gigabytes is common in practice::
- osd journal size = 10240
+ osd_journal_size = 10240
-``osd journal``
+``osd_journal``
:Description: The path to the OSD's journal. This may be a path to a file or a
block device (such as a partition of an SSD). If it is a file,
you must create the directory to contain it. We recommend using a
- drive separate from the ``osd data`` drive.
+ separate fast device when the ``osd_data`` drive is an HDD.
:Type: String
:Default: ``/var/lib/ceph/osd/$cluster-$id/journal``
-``osd journal size``
+``osd_journal_size``
:Description: The size of the journal in megabytes.
Ceph OSD Daemons check each other's heartbeats and report to monitors
periodically. Ceph can use default values in many cases. However, if your
-network has latency issues, you may need to adopt longer intervals. See
+network has latency issues, you may need to adopt longer intervals. See
`Configuring Monitor/OSD Interaction`_ for a detailed discussion of heartbeats.
scrubbing operations.
-``osd max scrubs``
+``osd_max_scrubs``
:Description: The maximum number of simultaneous scrub operations for
a Ceph OSD Daemon.
:Type: 32-bit Int
:Default: ``1``
-``osd scrub begin hour``
+``osd_scrub_begin_hour``
:Description: This restricts scrubbing to this hour of the day or later.
- Use ``osd scrub begin hour = 0`` and ``osd scrub end hour = 0``
- to allow scrubbing the entire day. Along with ``osd scrub end hour``, they define a time
+ Use ``osd_scrub_begin_hour = 0`` and ``osd_scrub_end_hour = 0``
+ to allow scrubbing the entire day. Along with ``osd_scrub_end_hour``, they define a time
window, in which the scrubs can happen.
But a scrub will be performed
no matter whether the time window allows or not, as long as the placement
- group's scrub interval exceeds ``osd scrub max interval``.
+ group's scrub interval exceeds ``osd_scrub_max_interval``.
:Type: Integer in the range of 0 to 23
:Default: ``0``
-``osd scrub end hour``
+``osd_scrub_end_hour``
:Description: This restricts scrubbing to the hour earlier than this.
- Use ``osd scrub begin hour = 0`` and ``osd scrub end hour = 0`` to allow scrubbing
- for the entire day. Along with ``osd scrub begin hour``, they define a time
+ Use ``osd_scrub_begin_hour = 0`` and ``osd_scrub_end_hour = 0`` to allow scrubbing
+ for the entire day. Along with ``osd_scrub_begin_hour``, they define a time
window, in which the scrubs can happen. But a scrub will be performed
no matter whether the time window allows or not, as long as the placement
- group's scrub interval exceeds ``osd scrub max interval``.
+ group's scrub interval exceeds ``osd_scrub_max_interval``.
:Type: Integer in the range of 0 to 23
:Default: ``0``
-``osd scrub begin week day``
+``osd_scrub_begin_week_day``
:Description: This restricts scrubbing to this day of the week or later.
- 0 = Sunday, 1 = Monday, etc. Use ``osd scrub begin week day = 0``
- and ``osd scrub end week day = 0`` to allow scrubbing for the entire week.
- Along with ``osd scrub end week day``, they define a time window, in which
- the scrubs can happen. But a scrub will be performed
- no matter whether the time window allows or not, as long as the placement
- group's scrub interval exceeds ``osd scrub max interval``.
+ 0 = Sunday, 1 = Monday, etc. Use ``osd_scrub_begin_week_day = 0``
+ and ``osd_scrub_end_week_day = 0`` to allow scrubbing for the entire week.
+ Along with ``osd_scrub_end_week_day``, they define a time window in which
+ scrubs can happen. But a scrub will be performed
+ no matter whether the time window allows or not, when the PG's
+ scrub interval exceeds ``osd_scrub_max_interval``.
:Type: Integer in the range of 0 to 6
:Default: ``0``
-``osd scrub end week day``
+``osd_scrub_end_week_day``
:Description: This restricts scrubbing to days of the week earlier than this.
- 0 = Sunday, 1 = Monday, etc. Use ``osd scrub begin week day = 0``
- and ``osd scrub end week day = 0`` to allow scrubbing for the entire week.
- Along with ``osd scrub begin week day``, they define a time
+ 0 = Sunday, 1 = Monday, etc. Use ``osd_scrub_begin_week_day = 0``
+ and ``osd_scrub_end_week_day = 0`` to allow scrubbing for the entire week.
+ Along with ``osd_scrub_begin_week_day``, they define a time
window, in which the scrubs can happen. But a scrub will be performed
no matter whether the time window allows or not, as long as the placement
- group's scrub interval exceeds ``osd scrub max interval``.
+ group's scrub interval exceeds ``osd_scrub_max_interval``.
:Type: Integer in the range of 0 to 6
:Default: ``0``
:Default: ``false``
-``osd scrub thread timeout``
+``osd_scrub_thread_timeout``
:Description: The maximum time in seconds before timing out a scrub thread.
:Type: 32-bit Integer
:Default: ``60``
-``osd scrub finalize thread timeout``
+``osd_scrub_finalize_thread_timeout``
:Description: The maximum time in seconds before timing out a scrub finalize
thread.
:Default: ``10*60``
-``osd scrub load threshold``
+``osd_scrub_load_threshold``
:Description: The normalized maximum load. Ceph will not scrub when the system load
- (as defined by ``getloadavg() / number of online cpus``) is higher than this number.
+ (as defined by ``getloadavg() / number of online CPUs``) is higher than this number.
Default is ``0.5``.
:Type: Float
:Default: ``0.5``
-``osd scrub min interval``
+``osd_scrub_min_interval``
:Description: The minimal interval in seconds for scrubbing the Ceph OSD Daemon
when the Ceph Storage Cluster load is low.
.. _osd_scrub_max_interval:
-``osd scrub max interval``
+``osd_scrub_max_interval``
:Description: The maximum interval in seconds for scrubbing the Ceph OSD Daemon
irrespective of cluster load.
:Default: Once per week. ``7*24*60*60``
-``osd scrub chunk min``
+``osd_scrub_chunk_min``
:Description: The minimal number of object store chunks to scrub during single operation.
Ceph blocks writes to single chunk during scrub.
:Default: 5
-``osd scrub chunk max``
+``osd_scrub_chunk_max``
:Description: The maximum number of object store chunks to scrub during single operation.
:Default: 25
-``osd scrub sleep``
+``osd_scrub_sleep``
-:Description: Time to sleep before scrubbing next group of chunks. Increasing this value will slow
- down whole scrub operation while client operations will be less impacted.
+:Description: Time to sleep before scrubbing the next group of chunks. Increasing this value will slow
+ down the overall rate of scrubbing so that client operations will be less impacted.
:Type: Float
:Default: 0
-``osd deep scrub interval``
+``osd_deep_scrub_interval``
:Description: The interval for "deep" scrubbing (fully reading all data). The
- ``osd scrub load threshold`` does not affect this setting.
+ ``osd_scrub_load_threshold`` does not affect this setting.
:Type: Float
:Default: Once per week. ``7*24*60*60``
-``osd scrub interval randomize ratio``
+``osd_scrub_interval_randomize_ratio``
-:Description: Add a random delay to ``osd scrub min interval`` when scheduling
- the next scrub job for a placement group. The delay is a random
- value less than ``osd scrub min interval`` \*
- ``osd scrub interval randomized ratio``. So the default setting
- practically randomly spreads the scrubs out in the allowed time
- window of ``[1, 1.5]`` \* ``osd scrub min interval``.
+:Description: Add a random delay to ``osd_scrub_min_interval`` when scheduling
+ the next scrub job for a PG. The delay is a random
+ value less than ``osd_scrub_min_interval`` \*
+ ``osd_scrub_interval_randomized_ratio``. The default setting
+ spreads scrubs throughout the allowed time
+ window of ``[1, 1.5]`` \* ``osd_scrub_min_interval``.
:Type: Float
:Default: ``0.5``
-``osd deep scrub stride``
+``osd_deep_scrub_stride``
:Description: Read size when doing a deep scrub.
:Type: 32-bit Integer
:Default: 512 KB. ``524288``
-``osd scrub auto repair``
+``osd_scrub_auto_repair``
-:Description: Setting this to ``true`` will enable automatic pg repair when errors
- are found in scrub or deep-scrub. However, if more than
- ``osd scrub auto repair num errors`` errors are found a repair is NOT performed.
+:Description: Setting this to ``true`` will enable automatic PG repair when errors
+ are found by scrubs or deep-scrubs. However, if more than
+ ``osd_scrub_auto_repair_num_errors`` errors are found a repair is NOT performed.
:Type: Boolean
:Default: ``false``
-``osd scrub auto repair num errors``
+``osd_scrub_auto_repair_num_errors``
:Description: Auto repair will not occur if more than this many errors are found.
:Type: 32-bit Integer
Operations
==========
-``osd op queue``
+ ``osd_op_queue``
:Description: This sets the type of queue to be used for prioritizing ops
- in the OSDs. Both queues feature a strict sub-queue which is
+ within each OSD. Both queues feature a strict sub-queue which is
dequeued before the normal queue. The normal queue is different
between implementations. The WeightedPriorityQueue (``wpq``)
dequeues operations in relation to their priorities to prevent
:Default: ``wpq``
-``osd op queue cut off``
+``osd_op_queue_cut_off``
:Description: This selects which priority ops will be sent to the strict
queue verses the normal queue. The ``low`` setting sends all
option sends only replication acknowledgment ops and higher to
the strict queue. Setting this to ``high`` should help when a few
OSDs in the cluster are very busy especially when combined with
- ``wpq`` in the ``osd op queue`` setting. OSDs that are very busy
+ ``wpq`` in the ``osd_op_queue`` setting. OSDs that are very busy
handling replication traffic could starve primary client traffic
on these OSDs without these settings. Requires a restart.
:Default: ``high``
-``osd client op priority``
+``osd_client_op_priority``
:Description: The priority set for client operations. This value is relative
- to that of ``osd recovery op priority`` below. The default
+ to that of ``osd_recovery_op_priority`` below. The default
strongly favors client ops over recovery.
:Type: 32-bit Integer
:Valid Range: 1-63
-``osd recovery op priority``
+``osd_recovery_op_priority``
:Description: The priority of recovery operations vs client operations, if not specified by the
pool's ``recovery_op_priority``. The default value prioritizes client
:Valid Range: 1-63
-``osd scrub priority``
+``osd_scrub_priority``
-:Description: The default priority set for a scheduled scrub work queue when the
+:Description: The default work queue priority for scheduled scrubs when the
pool doesn't specify a value of ``scrub_priority``. This can be
- boosted to the value of ``osd client op priority`` when scrub is
+ boosted to the value of ``osd_client_op_priority`` when scrubs are
blocking client operations.
:Type: 32-bit Integer
:Valid Range: 1-63
-``osd requested scrub priority``
+``osd_requested_scrub_priority``
:Description: The priority set for user requested scrub on the work queue. If
- this value were to be smaller than ``osd client op priority`` it
- can be boosted to the value of ``osd client op priority`` when
+ this value were to be smaller than ``osd_client_op_priority`` it
+ can be boosted to the value of ``osd_client_op_priority`` when
scrub is blocking client operations.
:Type: 32-bit Integer
:Default: ``120``
-``osd snap trim priority``
+``osd_snap_trim_priority``
:Description: The priority set for the snap trim work queue.
:Default: ``5``
:Valid Range: 1-63
-``osd snap trim sleep``
+``osd_snap_trim_sleep``
:Description: Time in seconds to sleep before next snap trim op.
Increasing this value will slow down snap trimming.
:Default: ``0``
-``osd snap trim sleep hdd``
+``osd_snap_trim_sleep_hdd``
:Description: Time in seconds to sleep before next snap trim op
for HDDs.
:Default: ``5``
-``osd snap trim sleep ssd``
+``osd_snap_trim_sleep_ssd``
:Description: Time in seconds to sleep before next snap trim op
- for SSDs.
+ for SSD OSDs (including NVMe).
:Type: Float
:Default: ``0``
-``osd snap trim sleep hybrid``
+``osd_snap_trim_sleep_hybrid``
:Description: Time in seconds to sleep before next snap trim op
- when osd data is on HDD and osd journal is on SSD.
+ when OSD data is on an HDD and the OSD journal or WAL+DB is on an SSD.
:Type: Float
:Default: ``2``
-``osd op thread timeout``
+``osd_op_thread_timeout``
:Description: The Ceph OSD Daemon operation thread timeout in seconds.
:Type: 32-bit Integer
:Default: ``15``
-``osd op complaint time``
+``osd_op_complaint_time``
:Description: An operation becomes complaint worthy after the specified number
of seconds have elapsed.
:Default: ``30``
-``osd op history size``
+``osd_op_history_size``
:Description: The maximum number of completed operations to track.
:Type: 32-bit Unsigned Integer
:Default: ``20``
-``osd op history duration``
+``osd_op_history_duration``
:Description: The oldest completed operation to track.
:Type: 32-bit Unsigned Integer
:Default: ``600``
-``osd op log threshold``
+``osd_op_log_threshold``
:Description: How many operations logs to display at once.
:Type: 32-bit Integer
QoS Based on mClock
-------------------
-Ceph's use of mClock is currently in the experimental phase and should
+Ceph's use of mClock is currently experimental and should
be approached with an exploratory mindset.
Core Concepts
`````````````
-The QoS support of Ceph is implemented using a queueing scheduler
+Ceph's QoS support is implemented using a queueing scheduler
based on `the dmClock algorithm`_. This algorithm allocates the I/O
resources of the Ceph cluster in proportion to weights, and enforces
the constraints of minimum reservation and maximum limitation, so that
Various organizations and individuals are currently experimenting with
mClock as it exists in this code base along with their modifications
to the code base. We hope you'll share you're experiences with your
-mClock and dmClock experiments in the ceph-devel mailing list.
+mClock and dmClock experiments on the ``ceph-devel`` mailing list.
-``osd push per object cost``
+``osd_push_per_object_cost``
:Description: the overhead for serving a push op
:Type: Unsigned Integer
:Default: 1000
-``osd recovery max chunk``
+
+``osd_recovery_max_chunk``
:Description: the maximum total size of data chunks a recovery op can carry.
:Default: 8 MiB
-``osd mclock scheduler client res``
+``osd_mclock_scheduler_client_res``
:Description: IO proportion reserved for each client (default).
:Default: 1
-``osd mclock scheduler client wgt``
+``osd_mclock_scheduler_client_wgt``
:Description: IO share for each client (default) over reservation.
:Default: 1
-``osd mclock scheduler client lim``
+``osd_mclock_scheduler_client_lim``
:Description: IO limit for each client (default) over reservation.
:Default: 999999
-``osd mclock scheduler background recovery res``
+``osd_mclock_scheduler_background_recovery_res``
:Description: IO proportion reserved for background recovery (default).
:Default: 1
-``osd mclock scheduler background recovery wgt``
+``osd_mclock_scheduler_background_recovery_wgt``
:Description: IO share for each background recovery over reservation.
:Default: 1
-``osd mclock scheduler background recovery lim``
+``osd_mclock_scheduler_background_recovery_lim``
:Description: IO limit for background recovery over reservation.
:Default: 999999
-``osd mclock scheduler background best effort res``
+``osd_mclock_scheduler_background_best_effort_res``
:Description: IO proportion reserved for background best_effort (default).
:Default: 1
-``osd mclock scheduler background best effort wgt``
+``osd_mclock_scheduler_background_best_effort_wgt``
:Description: IO share for each background best_effort over reservation.
:Default: 1
-``osd mclock scheduler background best effort lim``
+``osd_mclock_scheduler_background_best_effort_lim``
:Description: IO limit for background best_effort over reservation.
Backfilling
===========
-When you add or remove Ceph OSD Daemons to a cluster, the CRUSH algorithm will
-want to rebalance the cluster by moving placement groups to or from Ceph OSD
-Daemons to restore the balance. The process of migrating placement groups and
+When you add or remove Ceph OSD Daemons to a cluster, CRUSH will
+rebalance the cluster by moving placement groups to or from Ceph OSDs
+to restore balanced utilization. The process of migrating placement groups and
the objects they contain can reduce the cluster's operational performance
considerably. To maintain operational performance, Ceph performs this migration
with 'backfilling', which allows Ceph to set backfill operations to a lower
priority than requests to read or write data.
-``osd max backfills``
+``osd_max_backfills``
:Description: The maximum number of backfills allowed to or from a single OSD.
+ Note that this is applied separately for read and write operations.
:Type: 64-bit Unsigned Integer
:Default: ``1``
-``osd backfill scan min``
+``osd_backfill_scan_min``
:Description: The minimum number of objects per backfill scan.
:Default: ``64``
-``osd backfill scan max``
+``osd_backfill_scan_max``
:Description: The maximum number of objects per backfill scan.
:Default: ``512``
-``osd backfill retry interval``
+``osd_backfill_retry_interval``
:Description: The number of seconds to wait before retrying backfill requests.
:Type: Double
Ceph performs well as the OSD map grows larger.
-``osd map dedup``
+``osd_map_dedup``
:Description: Enable removing duplicates in the OSD map.
:Type: Boolean
:Default: ``true``
-``osd map cache size``
+``osd_map_cache_size``
:Description: The number of OSD maps to keep cached.
:Type: 32-bit Integer
:Default: ``50``
-``osd map message max``
+``osd_map_message_max``
:Description: The maximum map entries allowed per MOSDMap message.
:Type: 32-bit Integer
perform well in a degraded state.
-``osd recovery delay start``
+``osd_recovery_delay_start``
:Description: After peering completes, Ceph will delay for the specified number
- of seconds before starting to recover objects.
+ of seconds before starting to recover RADOS objects.
:Type: Float
:Default: ``0``
-``osd recovery max active``
+``osd_recovery_max_active``
:Description: The number of active recovery requests per OSD at one time. More
requests will accelerate recovery, but the requests places an
:Type: 32-bit Integer
:Default: ``0``
-``osd recovery max active hdd``
+``osd_recovery_max_active_hdd``
:Description: The number of active recovery requests per OSD at one time, if the
primary device is rotational.
:Type: 32-bit Integer
:Default: ``3``
-``osd recovery max active ssd``
+``osd_recovery_max_active_ssd``
:Description: The number of active recovery requests per OSD at one time, if the
primary device is non-rotational (i.e., an SSD).
:Default: ``10``
-``osd recovery max chunk``
+``osd_recovery_max_chunk``
:Description: The maximum size of a recovered chunk of data to push.
:Type: 64-bit Unsigned Integer
:Default: ``8 << 20``
-``osd recovery max single start``
+``osd_recovery_max_single_start``
:Description: The maximum number of recovery operations per OSD that will be
newly started when an OSD is recovering.
:Default: ``1``
-``osd recovery thread timeout``
+``osd_recovery_thread_timeout``
:Description: The maximum time in seconds before timing out a recovery thread.
:Type: 32-bit Integer
:Default: ``30``
-``osd recover clone overlap``
+``osd_recover_clone_overlap``
:Description: Preserves clone overlap during recovery. Should always be set
to ``true``.
:Default: ``true``
-``osd recovery sleep``
+``osd_recovery_sleep``
-:Description: Time in seconds to sleep before next recovery or backfill op.
+:Description: Time in seconds to sleep before the next recovery or backfill op.
Increasing this value will slow down recovery operation while
client operations will be less impacted.
:Default: ``0``
-``osd recovery sleep hdd``
+``osd_recovery_sleep_hdd``
:Description: Time in seconds to sleep before next recovery or backfill op
for HDDs.
:Default: ``0.1``
-``osd recovery sleep ssd``
+``osd_recovery_sleep_ssd``
-:Description: Time in seconds to sleep before next recovery or backfill op
+:Description: Time in seconds to sleep before the next recovery or backfill op
for SSDs.
:Type: Float
:Default: ``0``
-``osd recovery sleep hybrid``
+``osd_recovery_sleep_hybrid``
-:Description: Time in seconds to sleep before next recovery or backfill op
- when osd data is on HDD and osd journal is on SSD.
+:Description: Time in seconds to sleep before the next recovery or backfill op
+ when OSD data is on HDD and OSD journal / WAL+DB is on SSD.
:Type: Float
:Default: ``0.025``
-``osd recovery priority``
+``osd_recovery_priority``
:Description: The default priority set for recovery work queue. Not
related to a pool's ``recovery_priority``.
Tiering
=======
-``osd agent max ops``
+``osd_agent_max_ops``
:Description: The maximum number of simultaneous flushing ops per tiering agent
in the high speed mode.
:Default: ``4``
-``osd agent max low ops``
+``osd_agent_max_low_ops``
:Description: The maximum number of simultaneous flushing ops per tiering agent
in the low speed mode.
=============
-``osd snap trim thread timeout``
+``osd_snap_trim_thread_timeout``
:Description: The maximum time in seconds before timing out a snap trim thread.
:Type: 32-bit Integer
:Default: ``1*60*60``
-``osd backlog thread timeout``
+``osd_backlog_thread_timeout``
:Description: The maximum time in seconds before timing out a backlog thread.
:Type: 32-bit Integer
:Default: ``1*60*60``
-``osd default notify timeout``
+``osd_default_notify_timeout``
:Description: The OSD default notification timeout (in seconds).
:Type: 32-bit Unsigned Integer
:Default: ``30``
-``osd check for log corruption``
+``osd_check_for_log_corruption``
:Description: Check log files for corruption. Can be computationally expensive.
:Type: Boolean
:Default: ``false``
-``osd remove thread timeout``
+``osd_remove_thread_timeout``
:Description: The maximum time in seconds before timing out a remove OSD thread.
:Type: 32-bit Integer
:Default: ``60*60``
-``osd command thread timeout``
+``osd_command_thread_timeout``
:Description: The maximum time in seconds before timing out a command thread.
:Type: 32-bit Integer
:Default: ``10*60``
-``osd delete sleep``
+``osd_delete_sleep``
-:Description: Time in seconds to sleep before next removal transaction. This
- helps to throttle the pg deletion process.
+:Description: Time in seconds to sleep before the next removal transaction. This
+ throttles the PG deletion process.
:Type: Float
:Default: ``0``
-``osd delete sleep hdd``
+``osd_delete_sleep_hdd``
-:Description: Time in seconds to sleep before next removal transaction
+:Description: Time in seconds to sleep before the next removal transaction
for HDDs.
:Type: Float
:Default: ``5``
-``osd delete sleep ssd``
+``osd_delete_sleep_ssd``
-:Description: Time in seconds to sleep before next removal transaction
+:Description: Time in seconds to sleep before the next removal transaction
for SSDs.
:Type: Float
:Default: ``0``
-``osd delete sleep hybrid``
+``osd_delete_sleep_hybrid``
-:Description: Time in seconds to sleep before next removal transaction
- when osd data is on HDD and osd journal is on SSD.
+:Description: Time in seconds to sleep before the next removal transaction
+ when OSD data is on HDD and OSD journal or WAL+DB is on SSD.
:Type: Float
:Default: ``1``
-``osd command max records``
+``osd_command_max_records``
:Description: Limits the number of lost objects to return.
:Type: 32-bit Integer
:Default: ``256``
-``osd fast fail on connection refused``
+``osd_fast_fail_on_connection_refused``
:Description: If this option is enabled, crashed OSDs are marked down
immediately by connected peers and MONs (assuming that the
.. index:: pools; configuration
-When you create pools and set the number of placement groups for the pool, Ceph
+When you create pools and set the number of placement groups (PGs) for each, Ceph
uses default values when you don't specifically override the defaults. **We
recommend** overriding some of the defaults. Specifically, we recommend setting
a pool's replica size and overriding the default number of placement groups. You
:language: ini
-
-``mon max pool pg num``
+``mon_max_pool_pg_num``
:Description: The maximum number of placement groups per pool.
:Type: Integer
:Default: ``65536``
-``mon pg create interval``
+``mon_pg_create_interval``
:Description: Number of seconds between PG creation in the same
Ceph OSD Daemon.
:Default: ``30.0``
-``mon pg stuck threshold``
+``mon_pg_stuck_threshold``
:Description: Number of seconds after which PGs can be considered as
being stuck.
:Type: 32-bit Integer
:Default: ``300``
-``mon pg min inactive``
+``mon_pg_min_inactive``
-:Description: Issue a ``HEALTH_ERR`` in cluster log if the number of PGs stay
- inactive longer than ``mon_pg_stuck_threshold`` exceeds this
+:Description: Raise ``HEALTH_ERR`` if the count of PGs that have been
+ inactive longer than the ``mon_pg_stuck_threshold`` exceeds this
setting. A non-positive number means disabled, never go into ERR.
:Type: Integer
:Default: ``1``
-``mon pg warn min per osd``
+``mon_pg_warn_min_per_osd``
-:Description: Issue a ``HEALTH_WARN`` in cluster log if the average number
- of PGs per (in) OSD is under this number. (a non-positive number
- disables this)
+:Description: Raise ``HEALTH_WARN`` if the average number
+ of PGs per ``in`` OSD is under this number. A non-positive number
+ disables this.
:Type: Integer
:Default: ``30``
-``mon pg warn min objects``
+``mon_pg_warn_min_objects``
-:Description: Do not warn if the total number of objects in cluster is below
+:Description: Do not warn if the total number of RADOS objects in cluster is below
this number
:Type: Integer
:Default: ``1000``
-``mon pg warn min pool objects``
+``mon_pg_warn_min_pool_objects``
-:Description: Do not warn on pools whose object number is below this number
+:Description: Do not warn on pools whose RADOS object count is below this number
:Type: Integer
:Default: ``1000``
-``mon pg check down all threshold``
+``mon_pg_check_down_all_threshold``
-:Description: Threshold of down OSDs percentage after which we check all PGs
+:Description: Percentage threshold of ``down`` OSDs above which we check all PGs
for stale ones.
:Type: Float
:Default: ``0.5``
-``mon pg warn max object skew``
+``mon_pg_warn_max_object_skew``
-:Description: Issue a ``HEALTH_WARN`` in cluster log if the average object number
- of a certain pool is greater than ``mon pg warn max object skew`` times
- the average object number of the whole pool. (zero or a non-positive
- number disables this). Note that this option applies to the managers.
+:Description: Raise ``HEALTH_WARN`` if the average RADOS object count per PG
+ of any pool is greater than ``mon_pg_warn_max_object_skew`` times
+ the average RADOS object count per PG of all pools. Zero or a non-positive
+ number disables this. Note that this option applies to ``ceph-mgr`` daemons.
:Type: Float
:Default: ``10``
-``mon delta reset interval``
+``mon_delta_reset_interval``
-:Description: Seconds of inactivity before we reset the pg delta to 0. We keep
+:Description: Seconds of inactivity before we reset the PG delta to 0. We keep
track of the delta of the used space of each pool, so, for
example, it would be easier for us to understand the progress of
recovery or the performance of cache tier. But if there's no
:Default: ``10``
-``mon osd max op age``
+``mon_osd_max_op_age``
:Description: Maximum op age before we get concerned (make it a power of 2).
- A ``HEALTH_WARN`` will be issued if a request has been blocked longer
+ ``HEALTH_WARN`` will be raised if a request has been blocked longer
than this limit.
:Type: Float
:Default: ``32.0``
-``osd pg bits``
+``osd_pg_bits``
:Description: Placement group bits per Ceph OSD Daemon.
:Type: 32-bit Integer
:Default: ``6``
-``osd pgp bits``
+``osd_pgp_bits``
:Description: The number of bits per Ceph OSD Daemon for PGPs.
:Type: 32-bit Integer
:Default: ``6``
-``osd crush chooseleaf type``
+``osd_crush_chooseleaf_type``
:Description: The bucket type to use for ``chooseleaf`` in a CRUSH rule. Uses
ordinal rank rather than name.
:Default: ``1``. Typically a host containing one or more Ceph OSD Daemons.
-``osd crush initial weight``
+``osd_crush_initial_weight``
-:Description: The initial crush weight for newly added osds into crushmap.
+:Description: The initial CRUSH weight for newly added OSDs.
:Type: Double
-:Default: ``the size of newly added osd in TB``. By default, the initial crush
- weight for the newly added osd is set to its volume size in TB.
+:Default: ``the size of a newly added OSD in TB``. By default, the initial CRUSH
+ weight for a newly added OSD is set to its device size in TB.
See `Weighting Bucket Items`_ for details.
-``osd pool default crush rule``
+``osd_pool_default_crush_rule``
:Description: The default CRUSH rule to use when creating a replicated pool.
:Type: 8-bit Integer
use that". This is to make pool creation work in the absence of rule 0.
-``osd pool erasure code stripe unit``
+``osd_pool_erasure_code_stripe_unit``
:Description: Sets the default size, in bytes, of a chunk of an object
stripe for erasure coded pools. Every object of size S
:Default: ``4096``
-``osd pool default size``
+``osd_pool_default_size``
:Description: Sets the number of replicas for objects in the pool. The default
value is the same as
:Default: ``3``
-``osd pool default min size``
+``osd_pool_default_min_size``
:Description: Sets the minimum number of written replicas for objects in the
pool in order to acknowledge a write operation to the client. If
minimum is ``size - (size / 2)``.
-``osd pool default pg num``
+``osd_pool_default_pg_num``
:Description: The default number of placement groups for a pool. The default
value is the same as ``pg_num`` with ``mkpool``.
:Default: ``32``
-``osd pool default pgp num``
+``osd_pool_default_pgp_num``
:Description: The default number of placement groups for placement for a pool.
The default value is the same as ``pgp_num`` with ``mkpool``.
:Default: ``8``
-``osd pool default flags``
+``osd_pool_default_flags``
:Description: The default flags for new pools.
:Type: 32-bit Integer
:Default: ``0``
-``osd max pgls``
+``osd_max_pgls``
:Description: The maximum number of placement groups to list. A client
requesting a large number can tie up the Ceph OSD Daemon.
:Note: Default should be fine.
-``osd min pg log entries``
+``osd_min_pg_log_entries``
:Description: The minimum number of placement group logs to maintain
when trimming log files.
:Default: ``250``
-``osd max pg log entries``
+``osd_max_pg_log_entries``
:Description: The maximum number of placement group logs to maintain
when trimming log files.
:Default: ``10000``
-``osd default data pool replay window``
+``osd_default_data_pool_replay_window``
:Description: The time (in seconds) for an OSD to wait for a client to replay
a request.
:Type: 32-bit Integer
:Default: ``45``
-``osd max pg per osd hard ratio``
+``osd_max_pg_per_osd_hard_ratio``
-:Description: The ratio of number of PGs per OSD allowed by the cluster before
- OSD refuses to create new PGs. OSD stops creating new PGs if the number
+:Description: The ratio of number of PGs per OSD allowed by the cluster before the
+ OSD refuses to create new PGs. An OSD stops creating new PGs if the number
of PGs it serves exceeds
- ``osd max pg per osd hard ratio`` \* ``mon max pg per osd``.
+ ``osd_max_pg_per_osd_hard_ratio`` \* ``mon_max_pg_per_osd``.
:Type: Float
:Default: ``2``
-``osd recovery priority``
+``osd_recovery_priority``
:Description: Priority of recovery in the work queue.
:Type: Integer
:Default: ``5``
-``osd recovery op priority``
+``osd_recovery_op_priority``
:Description: Default priority used for recovery operations if pool doesn't override.
[global]
- # By default, Ceph makes 3 replicas of objects. If you want to make four
+ # By default, Ceph makes 3 replicas of RADOS objects. If you want to maintain four
# copies of an object the default value--a primary copy and three replica
- # copies--reset the default values as shown in 'osd pool default size'.
+ # copies--reset the default values as shown in 'osd_pool_default_size'.
# If you want to allow Ceph to write a lesser number of copies in a degraded
- # state, set 'osd pool default min size' to a number less than the
- # 'osd pool default size' value.
+ # state, set 'osd_pool_default_min_size' to a number less than the
+ # 'osd_pool_default_size' value.
- osd pool default size = 3 # Write an object 3 times.
- osd pool default min size = 2 # Allow writing two copies in a degraded state.
+ osd_pool_default_size = 3 # Write an object 3 times.
+ osd_pool_default_min_size = 2 # Allow writing two copies in a degraded state.
# Ensure you have a realistic number of placement groups. We recommend
# approximately 100 per OSD. E.g., total number of OSDs multiplied by 100
# (100 * 10) / 4 = 250.
# always use the nearest power of 2
- osd pool default pg num = 256
- osd pool default pgp num = 256
+ osd_pool_default_pg_num = 256
+ osd_pool_default_pgp_num = 256
Storage Devices
=================
-There are two Ceph daemons that store data on disk:
+There are two Ceph daemons that store data on devices:
* **Ceph OSDs** (or Object Storage Daemons) are where most of the
data is stored in Ceph. Generally speaking, each OSD is backed by
There are two ways that OSDs can manage the data they store. Starting
with the Luminous 12.2.z release, the new default (and recommended) backend is
*BlueStore*. Prior to Luminous, the default (and only option) was
-*FileStore*.
+*Filestore*.
BlueStore
---------
Configuration variables set under the ``[client.radosgw.{instance-name}]``
section will not apply to rgw or radosgw-admin commands without an instance-name
specified in the command. Thus variables meant to be applied to all RGW
-instances or all radosgw-admin commands can be put into the ``[global]`` or the
-``[client]`` section to avoid specifying instance-name.
+instances or all radosgw-admin options can be put into the ``[global]`` or the
+``[client]`` section to avoid specifying ``instance-name``.
-``rgw frontends``
+``rgw_frontends``
:Description: Configures the HTTP frontend(s). The configuration for multiple
frontends can be provided in a comma-delimited list. Each frontend
:Type: String
:Default: ``beast port=7480``
-``rgw data``
+``rgw_data``
-:Description: Sets the location of the data files for Ceph Object Gateway.
+:Description: Sets the location of the data files for Ceph RADOS Gateway.
:Type: String
:Default: ``/var/lib/ceph/radosgw/$cluster-$id``
-``rgw enable apis``
+``rgw_enable_apis``
:Description: Enables the specified APIs.
.. note:: Enabling the ``s3`` API is a requirement for
- any radosgw instance that is meant to
+ any ``radosgw`` instance that is meant to
participate in a `multi-site <../multisite>`_
configuration.
:Type: String
:Default: ``s3, swift, swift_auth, admin`` All APIs.
-``rgw cache enabled``
+``rgw_cache_enabled``
:Description: Whether the Ceph Object Gateway cache is enabled.
:Type: Boolean
:Default: ``true``
-``rgw cache lru size``
+``rgw_cache_lru_size``
:Description: The number of entries in the Ceph Object Gateway cache.
:Type: Integer
:Default: ``10000``
-``rgw socket path``
+``rgw_socket_path``
:Description: The socket path for the domain socket. ``FastCgiExternalServer``
uses this socket. If you do not specify a socket path, Ceph
:Type: String
:Default: N/A
-``rgw fcgi socket backlog``
+``rgw_fcgi_socket_backlog``
:Description: The socket backlog for fcgi.
:Type: Integer
:Default: ``1024``
-``rgw host``
+``rgw_host``
:Description: The host for the Ceph Object Gateway instance. Can be an IP
address or a hostname.
:Default: ``0.0.0.0``
-``rgw port``
+``rgw_port``
:Description: Port the instance listens for requests. If not specified,
Ceph Object Gateway runs external FastCGI.
:Default: None
-``rgw dns name``
+``rgw_dns_name``
:Description: The DNS name of the served domain. See also the ``hostnames`` setting within regions.
:Type: String
:Default: None
-``rgw script uri``
+``rgw_script_uri``
:Description: The alternative value for the ``SCRIPT_URI`` if not set
in the request.
:Default: None
-``rgw request uri``
+``rgw_request_uri``
:Description: The alternative value for the ``REQUEST_URI`` if not set
in the request.
:Default: None
-``rgw print continue``
+``rgw_print_continue``
:Description: Enable ``100-continue`` if it is operational.
:Type: Boolean
:Default: ``true``
-``rgw remote addr param``
+``rgw_remote_addr_param``
:Description: The remote address parameter. For example, the HTTP field
containing the remote address, or the ``X-Forwarded-For``
:Default: ``REMOTE_ADDR``
-``rgw op thread timeout``
+``rgw_op_thread_timeout``
:Description: The timeout in seconds for open threads.
:Type: Integer
:Default: 600
-``rgw op thread suicide timeout``
+``rgw_op_thread_suicide_timeout``
:Description: The time ``timeout`` in seconds before a Ceph Object Gateway
process dies. Disabled if set to ``0``.
:Default: ``0``
-``rgw thread pool size``
+``rgw_thread_pool_size``
:Description: The size of the thread pool.
:Type: Integer
:Default: 100 threads.
-``rgw num control oids``
+``rgw_num_control_oids``
:Description: The number of notification objects used for cache synchronization
between different ``rgw`` instances.
:Default: ``8``
-``rgw init timeout``
+``rgw_init_timeout``
:Description: The number of seconds before Ceph Object Gateway gives up on
initialization.
:Default: ``30``
-``rgw mime types file``
+``rgw_mime_types_file``
-:Description: The path and location of the MIME types. Used for Swift
+:Description: The path and location of the MIME-types file. Used for Swift
auto-detection of object types.
:Type: String
:Default: ``/etc/mime.types``
-``rgw s3 success create obj status``
+``rgw_s3_success_create_obj_status``
:Description: The alternate success status response for ``create-obj``.
:Type: Integer
:Default: ``0``
-``rgw resolve cname``
+``rgw_resolve_cname``
:Description: Whether ``rgw`` should use DNS CNAME record of the request
hostname field (if hostname is not equal to ``rgw dns name``).
:Default: ``false``
-``rgw obj stripe size``
+``rgw_obj_stripe_size``
:Description: The size of an object stripe for Ceph Object Gateway objects.
See `Architecture`_ for details on striping.
:Default: ``4 << 20``
-``rgw extended http attrs``
+``rgw_extended_http_attrs``
:Description: Add new set of attributes that could be set on an entity
(user, bucket or object). These extra attributes can be set
:Example: "content_foo, content_bar, x-foo-bar"
-``rgw exit timeout secs``
+``rgw_exit_timeout_secs``
:Description: Number of seconds to wait for a process before exiting
unconditionally.
:Default: ``120``
-``rgw get obj window size``
+``rgw_get_obj_window_size``
:Description: The window size in bytes for a single object request.
:Type: Integer
:Default: ``16 << 20``
-``rgw get obj max req size``
+``rgw_get_obj_max_req_size``
:Description: The maximum request size of a single get operation sent to the
Ceph Storage Cluster.
:Default: ``4 << 20``
-``rgw relaxed s3 bucket names``
+``rgw_relaxed_s3_bucket_names``
:Description: Enables relaxed S3 bucket names rules for US region buckets.
:Type: Boolean
:Default: ``false``
-``rgw list buckets max chunk``
+``rgw_list_buckets_max_chunk``
:Description: The maximum number of buckets to retrieve in a single operation
when listing user buckets.
:Default: ``1000``
-``rgw override bucket index max shards``
+``rgw_override_bucket_index_max_shards``
:Description: Represents the number of shards for the bucket index object,
a value of zero indicates there is no sharding. It is not
:Default: ``0``
-``rgw curl wait timeout ms``
+``rgw_curl_wait_timeout_ms``
:Description: The timeout in milliseconds for certain ``curl`` calls.
:Type: Integer
:Default: ``1000``
-``rgw copy obj progress``
+``rgw_copy_obj_progress``
:Description: Enables output of object progress during long copy operations.
:Type: Boolean
:Default: ``true``
-``rgw copy obj progress every bytes``
+``rgw_copy_obj_progress_every_bytes``
:Description: The minimum bytes between copy progress output.
:Type: Integer
:Default: ``1024 * 1024``
-``rgw admin entry``
+``rgw_admin_entry``
:Description: The entry point for an admin request URL.
:Type: String
:Default: ``admin``
-``rgw content length compat``
+``rgw_content_length_compat``
-:Description: Enable compatibility handling of FCGI requests with both CONTENT_LENGTH AND HTTP_CONTENT_LENGTH set.
+:Description: Enable compatibility handling of FCGI requests with both ``CONTENT_LENGTH`` and ``HTTP_CONTENT_LENGTH`` set.
:Type: Boolean
:Default: ``false``
-``rgw bucket quota ttl``
+``rgw_bucket_quota_ttl``
:Description: The amount of time in seconds cached quota information is
trusted. After this timeout, the quota information will be
:Default: ``600``
-``rgw user quota bucket sync interval``
+``rgw_user_quota_bucket_sync_interval``
:Description: The amount of time in seconds bucket quota information is
accumulated before syncing to the cluster. During this time,
:Default: ``180``
-``rgw user quota sync interval``
+``rgw_user_quota_sync_interval``
:Description: The amount of time in seconds user quota information is
accumulated before syncing to the cluster. During this time,
:Default: ``180``
-``rgw bucket default quota max objects``
+``rgw_bucket_default_quota_max_objects``
:Description: Default max number of objects per bucket. Set on new users,
if no other quota is specified. Has no effect on existing users.
:Default: ``-1``
-``rgw bucket default quota max size``
+``rgw_bucket_default_quota_max_size``
:Description: Default max capacity per bucket, in bytes. Set on new users,
if no other quota is specified. Has no effect on existing users.
:Default: ``-1``
-``rgw user default quota max objects``
+``rgw_user_default_quota_max_objects``
:Description: Default max number of objects for a user. This includes all
objects in all buckets owned by the user. Set on new users,
:Default: ``-1``
-``rgw user default quota max size``
+``rgw_user_default_quota_max_size``
:Description: The value for user max size quota in bytes set on new users,
if no other quota is specified. Has no effect on existing users.
:Default: ``-1``
-``rgw verify ssl``
+``rgw_verify_ssl``
:Description: Verify SSL certificates while making requests.
:Type: Boolean
There are two options in particular to look at when looking to increase the
aggressiveness of lifecycle processing:
-``rgw lc max worker``
+``rgw_lc_max_worker``
:Description: This option specifies the number of lifecycle worker threads
to run in parallel, thereby processing bucket and index
:Type: Integer
:Default: ``3``
-``rgw lc max wp worker``
+``rgw_lc_max_wp_worker``
:Description: This option specifies the number of threads in each lifecycle
workers work pool. This option can help accelerate processing each bucket.
These values can be tuned based upon your specific workload to further increase the
aggressiveness of lifecycle processing. For a workload with a larger number of buckets (thousands)
-you would look at increasing the ``rgw lc max worker`` value from the default value of 3 whereas a
+you would look at increasing the ``rgw_lc_max_worker`` value from the default value of 3 whereas for a
workload with a smaller number of buckets but higher number of objects (hundreds of thousands)
-per bucket you would look at tuning ``rgw lc max wp worker`` from the default value of 3.
+per bucket you would consider decreasing ``rgw_lc_max_wp_worker`` from the default value of 3.
:NOTE: When looking to tune either of these specific values please validate the
current Cluster performance and Ceph Object Gateway utilization before increasing.
$ radosgw-admin gc list
- Note: specify --include-all to list all entries, including unexpired
+ Note: specify ``--include-all`` to list all entries, including unexpired
Garbage collection is a background activity that may
execute continuously or during times of low loads, depending upon how the
configuration parameters.
-``rgw gc max objs``
+``rgw_gc_max_objs``
:Description: The maximum number of objects that may be handled by
garbage collection in one garbage collection processing cycle.
:Default: ``32``
-``rgw gc obj min wait``
+``rgw_gc_obj_min_wait``
:Description: The minimum wait time before a deleted object may be removed
and handled by garbage collection processing.
:Default: ``2 * 3600``
-``rgw gc processor max time``
+``rgw_gc_processor_max_time``
:Description: The maximum time between the beginning of two consecutive garbage
collection processing cycles.
:Default: ``3600``
-``rgw gc processor period``
+``rgw_gc_processor_period``
:Description: The cycle time for garbage collection processing.
:Type: Integer
:Default: ``3600``
-``rgw gc max concurrent io``
+``rgw_gc_max_concurrent_io``
:Description: The maximum number of concurrent IO operations that the RGW garbage
collection thread will use when purging old data.
As an initial step towards tuning Ceph Garbage Collection to be more aggressive the following options are suggested to be increased from their default configuration values:
-``rgw gc max concurrent io = 20``
-``rgw gc max trim chunk = 64``
+``rgw_gc_max_concurrent_io = 20``
+``rgw_gc_max_trim_chunk = 64``
:NOTE: Modifying these values requires a restart of the RGW service.
file under each ``[client.radosgw.{instance-name}]`` instance.
-``rgw zone``
+``rgw_zone``
:Description: The name of the zone for the gateway instance. If no zone is
set, a cluster-wide default can be configured with the command
:Default: None
-``rgw zonegroup``
+``rgw_zonegroup``
:Description: The name of the zonegroup for the gateway instance. If no
zonegroup is set, a cluster-wide default can be configured with
:Default: None
-``rgw realm``
+``rgw_realm``
:Description: The name of the realm for the gateway instance. If no realm is
set, a cluster-wide default can be configured with the command
:Default: None
-``rgw run sync thread``
+``rgw_run_sync_thread``
:Description: If there are other zones in the realm to sync from, spawn threads
to handle the sync of data and metadata.
:Default: ``true``
-``rgw data log window``
+``rgw_data_log_window``
:Description: The data log entries window in seconds.
:Type: Integer
:Default: ``30``
-``rgw data log changes size``
+``rgw_data_log_changes_size``
:Description: The number of in-memory entries to hold for the data changes log.
:Type: Integer
:Default: ``1000``
-``rgw data log obj prefix``
+``rgw_data_log_obj_prefix``
:Description: The object name prefix for the data log.
:Type: String
:Default: ``data_log``
-``rgw data log num shards``
+``rgw_data_log_num_shards``
:Description: The number of shards (objects) on which to keep the
data changes log.
:Default: ``128``
-``rgw md log max shards``
+``rgw_md_log_max_shards``
:Description: The maximum number of shards for the metadata log.
:Type: Integer
:Default: ``64``
-.. important:: The values of ``rgw data log num shards`` and
- ``rgw md log max shards`` should not be changed after sync has
+.. important:: The values of ``rgw_data_log_num_shards`` and
+ ``rgw_md_log_max_shards`` should not be changed after sync has
started.
S3 Settings
===========
-``rgw s3 auth use ldap``
+``rgw_s3_auth_use_ldap``
-:Description: Should S3 authentication use LDAP.
+:Description: Should S3 authentication use LDAP?
:Type: Boolean
:Default: ``false``
Swift Settings
==============
-``rgw enforce swift acls``
+``rgw_enforce_swift_acls``
:Description: Enforces the Swift Access Control List (ACL) settings.
:Type: Boolean
:Default: ``true``
-``rgw swift token expiration``
+``rgw_swift_token_expiration``
:Description: The time in seconds for expiring a Swift token.
:Type: Integer
:Default: ``24 * 3600``
-``rgw swift url``
+``rgw_swift_url``
:Description: The URL for the Ceph Object Gateway Swift API.
:Type: String
:Default: None
-``rgw swift url prefix``
+``rgw_swift_url_prefix``
:Description: The URL prefix for the Swift API, to distinguish it from
the S3 API endpoint. The default is ``swift``, which
:Example: "/swift-testing"
-``rgw swift auth url``
+``rgw_swift_auth_url``
:Description: Default URL for verifying v1 auth tokens (if not using internal
Swift auth).
:Default: None
-``rgw swift auth entry``
+``rgw_swift_auth_entry``
:Description: The entry point for a Swift auth URL.
:Type: String
:Default: ``auth``
-``rgw swift account in url``
+``rgw_swift_account_in_url``
:Description: Whether or not the Swift account name should be included
in the Swift API URL.
:Default: ``false``
-``rgw swift versioning enabled``
+``rgw_swift_versioning_enabled``
:Description: Enables the Object Versioning of OpenStack Object Storage API.
This allows clients to put the ``X-Versions-Location`` attribute
:Default: ``false``
-``rgw trust forwarded https``
+``rgw_trust_forwarded_https``
:Description: When a proxy in front of radosgw is used for ssl termination, radosgw
does not know whether incoming http connections are secure. Enable
================
-``rgw log nonexistent bucket``
+``rgw_log_nonexistent_bucket``
:Description: Enables Ceph Object Gateway to log a request for a non-existent
bucket.
:Default: ``false``
-``rgw log object name``
+``rgw_log_object_name``
-:Description: The logging format for an object name. See manpage
+:Description: The logging format for an object name. See ma npage
:manpage:`date` for details about format specifiers.
:Type: Date
:Default: ``%Y-%m-%d-%H-%i-%n``
-``rgw log object name utc``
+``rgw_log_object_name_utc``
:Description: Whether a logged object name includes a UTC time.
If ``false``, it uses the local time.
:Default: ``false``
-``rgw usage max shards``
+``rgw_usage_max_shards``
:Description: The maximum number of shards for usage logging.
:Type: Integer
:Default: ``32``
-``rgw usage max user shards``
+``rgw_usage_max_user_shards``
:Description: The maximum number of shards used for a single user's
usage logging.
:Default: ``1``
-``rgw enable ops log``
+``rgw_enable_ops_log``
:Description: Enable logging for each successful Ceph Object Gateway operation.
:Type: Boolean
:Default: ``false``
-``rgw enable usage log``
+``rgw_enable_usage_log``
:Description: Enable the usage log.
:Type: Boolean
:Default: ``false``
-``rgw ops log rados``
+``rgw_ops_log_rados``
:Description: Whether the operations log should be written to the
Ceph Storage Cluster backend.
:Default: ``true``
-``rgw ops log socket path``
+``rgw_ops_log_socket_path``
:Description: The Unix domain socket for writing operations logs.
:Type: String
:Default: None
-``rgw ops log data backlog``
+``rgw_ops_log_data_backlog``
:Description: The maximum data backlog data size for operations logs written
to a Unix domain socket.
:Default: ``5 << 20``
-``rgw usage log flush threshold``
+``rgw_usage_log_flush_threshold``
:Description: The number of dirty merged entries in the usage log before
flushing synchronously.
:Default: 1024
-``rgw usage log tick interval``
+``rgw_usage_log_tick_interval``
:Description: Flush pending usage log data every ``n`` seconds.
:Type: Integer
:Default: ``30``
-``rgw log http headers``
+``rgw_log_http_headers``
:Description: Comma-delimited list of HTTP headers to include with ops
log entries. Header names are case insensitive, and use
:Example: "http_x_forwarded_for, http_x_special_k"
-``rgw intent log object name``
+``rgw_intent_log_object_name``
-:Description: The logging format for the intent log object name. See manpage
+:Description: The logging format for the intent log object name. See the manpage
:manpage:`date` for details about format specifiers.
:Type: Date
:Default: ``%Y-%m-%d-%i-%n``
-``rgw intent log object name utc``
+``rgw_intent_log_object_name_utc``
-:Description: Whether the intent log object name includes a UTC time.
+:Description: Whether the intent log object name uses UTC time.
If ``false``, it uses the local time.
:Type: Boolean
=================
-``rgw keystone url``
+``rgw_keystone_url``
:Description: The URL for the Keystone server.
:Type: String
:Default: None
-``rgw keystone api version``
+``rgw_keystone_api_version``
:Description: The version (2 or 3) of OpenStack Identity API that should be
used for communication with the Keystone server.
:Default: ``2``
-``rgw keystone admin domain``
+``rgw_keystone_admin_domain``
:Description: The name of OpenStack domain with admin privilege when using
OpenStack Identity API v3.
:Default: None
-``rgw keystone admin project``
+``rgw_keystone_admin_project``
:Description: The name of OpenStack project with admin privilege when using
OpenStack Identity API v3. If left unspecified, value of
:Default: None
-``rgw keystone admin token``
+``rgw_keystone_admin_token``
-:Description: The Keystone admin token (shared secret). In Ceph RadosGW
+:Description: The Keystone admin token (shared secret). In Ceph RGW
authentication with the admin token has priority over
authentication with the admin credentials
- (``rgw keystone admin user``, ``rgw keystone admin password``,
- ``rgw keystone admin tenant``, ``rgw keystone admin project``,
- ``rgw keystone admin domain``). The Keystone admin token
+ (``rgw_keystone_admin_user``, ``rgw_keystone_admin_password``,
+ ``rgw_keystone_admin_tenant``, ``rgw_keystone_admin_project``,
+ ``rgw_keystone_admin_domain``). The Keystone admin token
has been deprecated, but can be used to integrate with
- older environments. Prefer ``rgw keystone admin token path``
- to avoid exposing the token.
+ older environments. It is preferred to instead configure
+ ``rgw_keystone_admin_token_path`` to avoid exposing the token.
:Type: String
:Default: None
-``rgw keystone admin token path``
+``rgw_keystone_admin_token_path``
:Description: Path to a file containing the Keystone admin token
(shared secret). In Ceph RadosGW authentication with
the admin token has priority over authentication with
the admin credentials
- (``rgw keystone admin user``, ``rgw keystone admin password``,
- ``rgw keystone admin tenant``, ``rgw keystone admin project``,
- ``rgw keystone admin domain``).
+ (``rgw_keystone_admin_user``, ``rgw_keystone_admin_password``,
+ ``rgw_keystone_admin_tenant``, ``rgw_keystone_admin_project``,
+ ``rgw_keystone_admin_domain``).
The Keystone admin token has been deprecated, but can be
used to integrate with older environments.
:Type: String
:Default: None
-``rgw keystone admin tenant``
+``rgw_keystone_admin_tenant``
:Description: The name of OpenStack tenant with admin privilege (Service Tenant) when
using OpenStack Identity API v2
:Default: None
-``rgw keystone admin user``
+``rgw_keystone_admin_user``
:Description: The name of OpenStack user with admin privilege for Keystone
- authentication (Service User) when OpenStack Identity API v2
+ authentication (Service User) when using OpenStack Identity API v2
:Type: String
:Default: None
-``rgw keystone admin password``
+``rgw_keystone_admin_password``
:Description: The password for OpenStack admin user when using OpenStack
- Identity API v2. Prefer ``rgw keystone admin password path``
- to avoid exposing the token.
+ Identity API v2. It is preferred to instead configure
+ ``rgw_keystone_admin_password_path`` to avoid exposing the token.
:Type: String
:Default: None
-``rgw keystone admin password path``
+``rgw_keystone_admin_password_path``
:Description: Path to a file containing the password for OpenStack
admin user when using OpenStack Identity API v2.
:Default: None
-``rgw keystone accepted roles``
+``rgw_keystone_accepted_roles``
-:Description: The roles requires to serve requests.
+:Description: The roles required to serve requests.
:Type: String
:Default: ``Member, admin``
-``rgw keystone token cache size``
+``rgw_keystone_token_cache_size``
:Description: The maximum number of entries in each Keystone token cache.
:Type: Integer
:Default: ``10000``
-``rgw keystone revocation interval``
+``rgw_keystone_revocation_interval``
:Description: The number of seconds between token revocation checks.
:Type: Integer
:Default: ``15 * 60``
-``rgw keystone verify ssl``
+``rgw_keystone_verify_ssl``
:Description: Verify SSL certificates while making token requests to keystone.
:Type: Boolean
Server-side encryption Settings
===============================
-``rgw crypt s3 kms backend``
+``rgw_crypt_s3_kms_backend``
:Description: Where the SSE-KMS encryption keys are stored. Supported KMS
systems are OpenStack Barbican (``barbican``, the default) and
Barbican Settings
=================
-``rgw barbican url``
+``rgw_barbican_url``
:Description: The URL for the Barbican server.
:Type: String
:Default: None
-``rgw keystone barbican user``
+``rgw_keystone_barbican_user``
:Description: The name of the OpenStack user with access to the `Barbican`_
secrets used for `Encryption`_.
:Type: String
:Default: None
-``rgw keystone barbican password``
+``rgw_keystone_barbican_password``
:Description: The password associated with the `Barbican`_ user.
:Type: String
:Default: None
-``rgw keystone barbican tenant``
+``rgw_keystone_barbican_tenant``
:Description: The name of the OpenStack tenant associated with the `Barbican`_
user when using OpenStack Identity API v2.
:Type: String
:Default: None
-``rgw keystone barbican project``
+``rgw_keystone_barbican_project``
:Description: The name of the OpenStack project associated with the `Barbican`_
user when using OpenStack Identity API v3.
:Type: String
:Default: None
-``rgw keystone barbican domain``
+``rgw_keystone_barbican_domain``
:Description: The name of the OpenStack domain associated with the `Barbican`_
user when using OpenStack Identity API v3.
HashiCorp Vault Settings
========================
-``rgw crypt vault auth``
+``rgw_crypt_vault_auth``
:Description: Type of authentication method to be used. The only method
currently supported is ``token``.
:Type: String
:Default: ``token``
-``rgw crypt vault token file``
+``rgw_crypt_vault_token_file``
:Description: If authentication method is ``token``, provide a path to the token
file, which should be readable only by Rados Gateway.
:Type: String
:Default: None
-``rgw crypt vault addr``
+``rgw_crypt_vault_addr``
:Description: Vault server base address, e.g. ``http://vaultserver:8200``.
:Type: String
:Default: None
-``rgw crypt vault prefix``
+``rgw_crypt_vault_prefix``
:Description: The Vault secret URL prefix, which can be used to restrict access
to a particular subset of the secret space, e.g. ``/v1/secret/data``.
:Type: String
:Default: None
-``rgw crypt vault secret engine``
+``rgw_crypt_vault_secret_engine``
:Description: Vault Secret Engine to be used to retrieve encryption keys: choose
between kv-v2, transit.
:Type: String
:Default: None
-``rgw crypt vault namespace``
+``rgw_crypt_vault_namespace``
:Description: If set, Vault Namespace provides tenant isolation for teams and individuals
on the same Vault Enterprise instance, e.g. ``acme/tenant1``
.. versionadded:: Nautilus
The ``civetweb`` frontend has a threading model that uses a thread per
-connection and hence automatically throttled by ``rgw thread pool size``
-configurable when it comes to accepting connections. The ``beast`` frontend is
+connection and hence is automatically throttled by ``rgw_thread_pool_size``
+configurable when it comes to accepting connections. The newer ``beast`` frontend is
not restricted by the thread pool size when it comes to accepting new
-connections, so a scheduler abstraction is introduced in Nautilus release which
-for supporting ways for scheduling requests in the future.
+connections, so a scheduler abstraction is introduced in the Nautilus release
+to support future methods of scheduling requests.
Currently the scheduler defaults to a throttler which throttles the active
connections to a configured limit. QoS based on mClock is currently in an
(swift auth, sts) metadata & data requests.
-``rgw max concurrent requests``
+``rgw_max_concurrent_requests``
-:Description: Maximum number of concurrent HTTP requests that the beast frontend
+:Description: Maximum number of concurrent HTTP requests that the Beast front end
will process. Tuning this can help to limit memory usage under
heavy load.
:Type: Integer
:Default: 1024
-``rgw scheduler type``
+``rgw_scheduler_type``
-:Description: The type of RGW Scheduler to use. Valid values are throttler,
- dmclock. Currently defaults to throttler which throttles beast
- frontend requests. dmclock is *experimental* and will need the
- experimental flag set
+:Description: The RGW scheduler to use. Valid values are ``throttler` and
+ ``dmclock``. Currently defaults to ``throttler`` which throttles Beast
+ frontend requests. ``dmclock` is *experimental* and requires the
+ ``dmclock`` to be included in the ``experimental_feature_enabled``
+ configuration option.
-The options below are to tune the experimental dmclock scheduler. For some
-further reading on dmclock, see :ref:`dmclock-qos`. `op_class` for the flags below is
-one of admin, auth, metadata or data.
+The options below tune the experimental dmclock scheduler. For
+additional reading on dmclock, see :ref:`dmclock-qos`. `op_class` for the flags below is
+one of ``admin``, ``auth``, ``metadata``, or ``data``.
``rgw_dmclock_<op_class>_res``
Generic IO Settings
===================
-``rbd compression hint``
+``rbd_compression_hint``
-:Description: Hint to send to the OSDs on write operations. If set to `compressible` and the OSD `bluestore compression mode` setting is `passive`, the OSD will attempt to compress the data. If set to `incompressible` and the OSD compression setting is `aggressive`, the OSD will not attempt to compress the data.
+:Description: Hint to send to the OSDs on write operations. If set to
+ ``compressible`` and the OSD ``bluestore_compression_mode``
+ setting is ``passive``, the OSD will attempt to compress data
+ If set to ``incompressible`` and the OSD compression setting
+ is ``aggressive``, the OSD will not attempt to compress data.
:Type: Enum
:Required: No
:Default: ``none``
:Values: ``none``, ``compressible``, ``incompressible``
-``rbd read from replica policy``
-:Description: policy for determining which OSD will receive read operations. If set to `default`, the primary OSD will always be used for read operations. If set to `balance`, read operations will be sent to a randomly selected OSD within the replica set. If set to `localize`, read operations will be sent to the closest OSD as determined by the CRUSH map. Note: this feature requires the cluster to be configured with a minimum compatible OSD release of Octopus.
+``rbd_read_from_replica_policy``
+
+:Description: Policy for determining which OSD will receive read operations.
+ If set to ``default``, each PG's primary OSD will always be used
+ for read operations. If set to ``balance``, read operations will
+ be sent to a randomly selected OSD within the replica set. If set
+ to ``localize``, read operations will be sent to the closest OSD
+ as determined by the CRUSH map. Note: this feature requires the
+ cluster to be configured with a minimum compatible OSD release of
+ Octopus.
:Type: Enum
:Required: No
:Default: ``default``
The librbd cache is enabled by default and supports three different cache
policies: write-around, write-back, and write-through. Writes return
immediately under both the write-around and write-back policies, unless there
-are more than ``rbd cache max dirty`` unwritten bytes to the storage cluster.
+are more than ``rbd_cache_max_dirty`` unwritten bytes to the storage cluster.
The write-around policy differs from the write-back policy in that it does
not attempt to service read requests from the cache, unlike the write-back
policy, and is therefore faster for high performance write workloads. Under the
RBD will not work with caching enabled.
-The ``ceph.conf`` file settings for RBD should be set in the ``[client]``
-section of your configuration file. The settings include:
-
+Option settings for RBD should be set in the ``[client]``
+section of your configuration file or the central config store. These settings
+include:
-``rbd cache``
+``rbd_cache``
:Description: Enable caching for RADOS Block Device (RBD).
:Type: Boolean
:Default: ``true``
-``rbd cache policy``
+``rbd_cache_policy``
:Description: Select the caching policy for librbd.
:Type: Enum
:Values: ``writearound``, ``writeback``, ``writethrough``
-``rbd cache writethrough until flush``
+``rbd_cache_writethrough_until_flush``
-:Description: Start out in write-through mode, and switch to write-back after the first flush request is received. Enabling this is a conservative but safe setting in case VMs running on rbd are too old to send flushes, like the virtio driver in Linux before 2.6.32.
+:Description: Start out in ``writethrough`` mode, and switch to ``writeback``
+ after the first flush request is received. Enabling is a
+ conservative but safe strategy in case VMs running on RBD volumes
+ are too old to send flushes, like the ``virtio`` driver in Linux
+ kernels older than 2.6.32.
:Type: Boolean
:Required: No
:Default: ``true``
-``rbd cache size``
+``rbd_cache_size``
-:Description: The RBD cache size in bytes.
+:Description: The per-volume RBD client cache size in bytes.
:Type: 64-bit Integer
:Required: No
:Default: ``32 MiB``
:Policies: write-back and write-through
-``rbd cache max dirty``
+``rbd_cache_max_dirty``
:Description: The ``dirty`` limit in bytes at which the cache triggers write-back. If ``0``, uses write-through caching.
:Type: 64-bit Integer
:Required: No
-:Constraint: Must be less than ``rbd cache size``.
+:Constraint: Must be less than ``rbd_cache_size``.
:Default: ``24 MiB``
:Policies: write-around and write-back
-``rbd cache target dirty``
+``rbd_cache_target_dirty``
:Description: The ``dirty target`` before the cache begins writing data to the data storage. Does not block writes to the cache.
:Type: 64-bit Integer
:Required: No
-:Constraint: Must be less than ``rbd cache max dirty``.
+:Constraint: Must be less than ``rbd_cache_max_dirty``.
:Default: ``16 MiB``
:Policies: write-back
-``rbd cache max dirty age``
+``rbd_cache_max_dirty_age``
:Description: The number of seconds dirty data is in the cache before writeback starts.
:Type: Float
disabled if caching is disabled or if the policy is write-around.
-``rbd readahead trigger requests``
+``rbd_readahead_trigger_requests``
:Description: Number of sequential read requests necessary to trigger read-ahead.
:Type: Integer
:Default: ``10``
-``rbd readahead max bytes``
+``rbd_readahead_max_bytes``
:Description: Maximum size of a read-ahead request. If zero, read-ahead is disabled.
:Type: 64-bit Integer
:Default: ``512 KiB``
-``rbd readahead disable after bytes``
+``rbd_readahead_disable_after_bytes``
-:Description: After this many bytes have been read from an RBD image, read-ahead is disabled for that image until it is closed. This allows the guest OS to take over read-ahead once it is booted. If zero, read-ahead stays enabled.
+:Description: After this many bytes have been read from an RBD image, read-ahead
+ is disabled for that image until it is closed. This allows the
+ guest OS to take over read-ahead once it is booted. If zero,
+ read-ahead stays enabled.
:Type: 64-bit Integer
:Required: No
:Default: ``50 MiB``
Image Features
==============
-RBD supports advanced features which can be specified via the command line when creating images or the default features can be specified via Ceph config file via 'rbd_default_features = <sum of feature numeric values>' or 'rbd_default_features = <comma-delimited list of CLI values>'
+RBD supports advanced features which can be specified via the command line when
+creating images or the default features can be configured via
+``rbd_default_features = <sum of feature numeric values>`` or
+``rbd_default_features = <comma-delimited list of CLI values>``.
``Layering``
-:Description: Layering enables you to use cloning.
+:Description: Layering enables cloning.
:Internal value: 1
:CLI value: layering
:Added in: v0.52 (Bobtail)
``Striping v2``
-:Description: Striping spreads data across multiple objects. Striping helps with parallelism for sequential read/write workloads.
+:Description: Striping spreads data across multiple objects. Striping helps with
+ parallelism for sequential read/write workloads.
:Internal value: 2
:CLI value: striping
:Added in: v0.55 (Bobtail)
``Exclusive locking``
-:Description: When enabled, it requires a client to get a lock on an object before making a write. Exclusive lock should only be enabled when a single client is accessing an image at the same time.
+:Description: When enabled, it requires a client to acquire a lock on an object
+ before making a write. Exclusive lock should only be enabled when
+ a single client is accessing an image at any given time.
:Internal value: 4
:CLI value: exclusive-lock
:Added in: v0.92 (Hammer)
``Object map``
-:Description: Object map support depends on exclusive lock support. Block devices are thin provisioned—meaning, they only store data that actually exists. Object map support helps track which objects actually exist (have data stored on a drive). Enabling object map support speeds up I/O operations for cloning; importing and exporting a sparsely populated image; and deleting.
+:Description: Object map support depends on exclusive lock support. Block
+ devices are thin provisioned, which means that they only store
+ data that actually has been written, ie. they are *sparse*. Object
+ map support helps track which objects actually exist (have data
+ stored on a device). Enabling object map support speeds up I/O
+ operations for cloning, importing and exporting a sparsely
+ populated image, and deleting.
:Internal value: 8
:CLI value: object-map
:Added in: v0.93 (Hammer)
``Fast-diff``
-:Description: Fast-diff support depends on object map support and exclusive lock support. It adds another property to the object map, which makes it much faster to generate diffs between snapshots of an image, and the actual data usage of a snapshot much faster.
+:Description: Fast-diff support depends on object map support and exclusive lock
+ support. It adds another property to the object map, which makes
+ it much faster to generate diffs between snapshots of an image.
+ It is also much faster to calculate the actual data usage of a
+ snapshot or volume (``rbd du``).
:Internal value: 16
:CLI value: fast-diff
:Added in: v9.0.1 (Infernalis)
``Deep-flatten``
-:Description: Deep-flatten makes rbd flatten work on all the snapshots of an image, in addition to the image itself. Without it, snapshots of an image will still rely on the parent, so the parent will not be delete-able until the snapshots are deleted. Deep-flatten makes a parent independent of its clones, even if they have snapshots.
+:Description: Deep-flatten enables ``rbd flatten`` to work on all snapshots of
+ an image, in addition to the image itself. Without it, snapshots
+ of an image will still rely on the parent, so the parent cannot be
+ deleted until the snapshots are first deleted. Deep-flatten makes
+ a parent independent of its clones, even if they have snapshots,
+ at the expense of using additional OSD device space.
:Internal value: 32
:CLI value: deep-flatten
:Added in: v9.0.2 (Infernalis)
``Journaling``
-:Description: Journaling support depends on exclusive lock support. Journaling records all modifications to an image in the order they occur. RBD mirroring utilizes the journal to replicate a crash consistent image to a remote cluster.
+:Description: Journaling support depends on exclusive lock support. Journaling
+ records all modifications to an image in the order they occur. RBD
+ mirroring can utilize the journal to replicate a crash-consistent
+ image to a remote cluster. It is best to let ``rbd-mirror``
+ manage this feature only as needed, as enabling it long term may
+ result in substantial additional OSD space consumption.
:Internal value: 64
:CLI value: journaling
:Added in: v10.0.1 (Jewel)
QOS Settings
============
-librbd supports limiting per image IO, controlled by the following
+librbd supports limiting per-image IO, controlled by the following
settings.
-``rbd qos iops limit``
+``rbd_qos_iops_limit``
:Description: The desired limit of IO operations per second.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos bps limit``
+``rbd_qos_bps_limit``
:Description: The desired limit of IO bytes per second.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos read iops limit``
+``rbd_qos_read_iops_limit``
:Description: The desired limit of read operations per second.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos write iops limit``
+``rbd_qos_write_iops_limit``
:Description: The desired limit of write operations per second.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos read bps limit``
+``rbd_qos_read_bps_limit``
:Description: The desired limit of read bytes per second.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos write bps limit``
+``rbd_qos_writ_bps_limit``
:Description: The desired limit of write bytes per second.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos iops burst``
+``rbd_qos_iops_burst``
:Description: The desired burst limit of IO operations.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos bps burst``
+``rbd_qos_bps_burst``
:Description: The desired burst limit of IO bytes.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos read iops burst``
+``rbd_qos_read_iops_burst``
:Description: The desired burst limit of read operations.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos write iops burst``
+``rbd_qos_write_iops_burst``
:Description: The desired burst limit of write operations.
:Type: Unsigned Integer
:Default: ``0``
-``rbd qos read bps burst``
+``rbd_qos_read_bps_burst``
-:Description: The desired burst limit of read bytes.
+:Description: The desired burst limit of read bytes per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``
-``rbd qos write bps burst``
+``rbd_qos_write_bps_burst``
-:Description: The desired burst limit of write bytes.
+:Description: The desired burst limit of write bytes per second.
:Type: Unsigned Integer
:Required: No
:Default: ``0``
-``rbd qos iops burst seconds``
+``rbd_qos_iops_burst_seconds``
:Description: The desired burst duration in seconds of IO operations.
:Type: Unsigned Integer
:Default: ``1``
-``rbd qos bps burst seconds``
+``rbd_qos_bps_burst_seconds``
-:Description: The desired burst duration in seconds of IO bytes.
+:Description: The desired burst duration in seconds.
:Type: Unsigned Integer
:Required: No
:Default: ``1``
-``rbd qos read iops burst seconds``
+``rbd_qos_read_iops_burst_seconds``
:Description: The desired burst duration in seconds of read operations.
:Type: Unsigned Integer
:Default: ``1``
-``rbd qos write iops burst seconds``
+``rbd_qos_write_iops_burst_seconds``
:Description: The desired burst duration in seconds of write operations.
:Type: Unsigned Integer
:Default: ``1``
-``rbd qos read bps burst seconds``
+``rbd_qos_read_bps_burst_seconds``
:Description: The desired burst duration in seconds of read bytes.
:Type: Unsigned Integer
:Default: ``1``
-``rbd qos write bps burst seconds``
+``rbd_qos_write_bps_burst_seconds``
:Description: The desired burst duration in seconds of write bytes.
:Type: Unsigned Integer
:Default: ``1``
-``rbd qos schedule tick min``
+``rbd_qos_schedule_tick_min``
:Description: The minimum schedule tick (in milliseconds) for QoS.
:Type: Unsigned Integer
projects / ceph.git / commitdiff