From: Sage Weil Date: Tue, 12 Apr 2016 17:24:56 +0000 (-0400) Subject: doc/configuration/filesystem-recommendations: recommend against ext4 X-Git-Tag: ses3-milestone4~27^2 X-Git-Url: http://git-server-git.apps.pok.os.sepia.ceph.com/?a=commitdiff_plain;h=refs%2Fpull%2F8556%2Fhead;p=ceph.git doc/configuration/filesystem-recommendations: recommend against ext4 Signed-off-by: Sage Weil --- diff --git a/doc/rados/configuration/filesystem-recommendations.rst b/doc/rados/configuration/filesystem-recommendations.rst index 7cf255575ed3..e0d558a37a07 100644 --- a/doc/rados/configuration/filesystem-recommendations.rst +++ b/doc/rados/configuration/filesystem-recommendations.rst @@ -29,28 +29,49 @@ Filesystems Ceph OSD Daemons rely heavily upon the stability and performance of the underlying filesystem. -.. note:: We currently recommend ``XFS`` for production deployments. - We recommend ``btrfs`` for testing, development, and any - non-critical deployments. We believe that ``btrfs`` has the correct - feature set and roadmap to serve Ceph in the long-term, but ``XFS`` - and ``ext4`` provide the necessary stability for today's deployments. - ``btrfs`` development is proceeding rapidly: users should be - comfortable installing the latest released upstream kernels and be - able to track development activity for critical bug fixes. - -Ceph OSD Daemons depend on the Extended Attributes (XATTRs) of the underlying -file system for various forms of internal object state and metadata. The -underlying filesystem must provide sufficient capacity for XATTRs. ``btrfs`` -does not bound the total xattr metadata stored with a file. ``XFS`` has a -relatively large limit (64 KB) that most deployments won't encounter, but the -``ext4`` is too small to be usable. +Recommended +----------- + +We currently recommend ``XFS`` for production deployments. + +We used to recommend ``btrfs`` for testing, development, and any non-critical +deployments becuase it has the most promising set of features. However, we +now plan to avoid using a kernel file system entirely with the new BlueStore +backend. ``btrfs`` is still supported and has a comparatively compelling +set of features, but be mindful of its stability and support status in your +Linux distribution. + +Not recommended +--------------- + +We recommend *against* using ``ext4`` due to limitations in the size +of xattrs it can store, and the problems this causes with the way Ceph +handles long RADOS object names. Although these issues will generally +not surface with Ceph clusters using only short object names (e.g., an +RBD workload that does not include long RBD image names), other users +like RGW make extensive use of long object names and can break. + +Starting with the Jewel release, the ``ceph-osd`` daemon will refuse +to start if the configured max object name cannot be safely stored on +``ext4``. If the cluster is only being used with short object names +(e.g., RBD only), you can continue using ``ext4`` by setting the +following configuration option:: + + osd max object name len = 256 + osd max object namespace len = 64 + +.. note:: This may result in difficult-to-diagnose errors if you try + to use RGW or other librados clients that do not properly + handle or politely surface any resulting ENAMETOOLONG + errors. Filesystem Background Info ========================== -The ``XFS``, ``btrfs`` and ``ext4`` file systems provide numerous advantages in highly -scaled data storage environments when `compared`_ to ``ext3``. +The ``XFS``, ``btrfs`` and ``ext4`` file systems provide numerous +advantages in highly scaled data storage environments when `compared`_ +to ``ext3``. ``XFS``, ``btrfs`` and ``ext4`` are `journaling file systems`_, which means that they are more robust when recovering from crashes, power outages, etc. These diff --git a/doc/rados/troubleshooting/troubleshooting-osd.rst b/doc/rados/troubleshooting/troubleshooting-osd.rst index df95c83dfb5f..d771d7d03e2c 100644 --- a/doc/rados/troubleshooting/troubleshooting-osd.rst +++ b/doc/rados/troubleshooting/troubleshooting-osd.rst @@ -365,9 +365,14 @@ might not have a recent enough version of ``glibc`` to support ``syncfs(2)``. Filesystem Issues ----------------- -Currently, we recommend deploying clusters with XFS or ext4. The btrfs +Currently, we recommend deploying clusters with XFS. The btrfs filesystem has many attractive features, but bugs in the filesystem may -lead to performance issues. +lead to performance issues. We do not recommend ext4 because xattr size +limitations break our support for long object names (needed for RGW). + +For more information, see `Filesystem Recommendations`_. + +.. _Filesystem Recommendations: ../configuration/filesystem-recommendations Insufficient RAM