#. Create the OSD. If no UUID is given, it will be set automatically when the
OSD starts up. The following command will output the OSD number, which you
- will need for subsequent steps. ::
+ will need for subsequent steps:
- ceph osd create [{uuid} [{id}]]
+ .. prompt:: bash $
+
+ ceph osd create [{uuid} [{id}]]
If the optional parameter {id} is given it will be used as the OSD id.
Note, in this case the command may fail if the number is already in use.
clusters are large. If {id} is not specified, the smallest available is
used.
-#. Create the default directory on your new OSD. ::
+#. Create the default directory on your new OSD:
- ssh {new-osd-host}
- sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
+ .. prompt:: bash $
+ ssh {new-osd-host}
+ sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
#. If the OSD is for a drive other than the OS drive, prepare it
- for use with Ceph, and mount it to the directory you just created::
+ for use with Ceph, and mount it to the directory you just created:
+
+ .. prompt:: bash $
- ssh {new-osd-host}
- sudo mkfs -t {fstype} /dev/{drive}
- sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}
+ ssh {new-osd-host}
+ sudo mkfs -t {fstype} /dev/{drive}
+ sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}
+#. Initialize the OSD data directory:
-#. Initialize the OSD data directory. ::
+ .. prompt:: bash $
- ssh {new-osd-host}
- ceph-osd -i {osd-num} --mkfs --mkkey
+ ssh {new-osd-host}
+ ceph-osd -i {osd-num} --mkfs --mkkey
The directory must be empty before you can run ``ceph-osd``.
#. Register the OSD authentication key. The value of ``ceph`` for
``ceph-{osd-num}`` in the path is the ``$cluster-$id``. If your
- cluster name differs from ``ceph``, use your cluster name instead.::
+ cluster name differs from ``ceph``, use your cluster name instead:
- ceph auth add osd.{osd-num} osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-{osd-num}/keyring
+ .. prompt:: bash $
+ ceph auth add osd.{osd-num} osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-{osd-num}/keyring
#. Add the OSD to the CRUSH map so that the OSD can begin receiving data. The
``ceph osd crush add`` command allows you to add OSDs to the CRUSH hierarchy
you specify only the root bucket, the command will attach the OSD directly
to the root, but CRUSH rules expect OSDs to be inside of hosts.
- Execute the following::
+ Execute the following:
+
+ .. prompt:: bash $
- ceph osd crush add {id-or-name} {weight} [{bucket-type}={bucket-name} ...]
+ ceph osd crush add {id-or-name} {weight} [{bucket-type}={bucket-name} ...]
You may also decompile the CRUSH map, add the OSD to the device list, add the
host as a bucket (if it's not already in the CRUSH map), add the device as an
be replaced. Unlike `Removing the OSD`_, replaced OSD's id and CRUSH map entry
need to be keep intact after the OSD is destroyed for replacement.
-#. Make sure it is safe to destroy the OSD::
+#. Make sure it is safe to destroy the OSD:
- while ! ceph osd safe-to-destroy osd.{id} ; do sleep 10 ; done
+ .. prompt:: bash $
-#. Destroy the OSD first::
+ while ! ceph osd safe-to-destroy osd.{id} ; do sleep 10 ; done
- ceph osd destroy {id} --yes-i-really-mean-it
+#. Destroy the OSD first:
+
+ .. prompt:: bash $
+
+ ceph osd destroy {id} --yes-i-really-mean-it
#. Zap a disk for the new OSD, if the disk was used before for other purposes.
- It's not necessary for a new disk::
+ It's not necessary for a new disk:
+
+ .. prompt:: bash $
- ceph-volume lvm zap /dev/sdX
+ ceph-volume lvm zap /dev/sdX
-#. Prepare the disk for replacement by using the previously destroyed OSD id::
+#. Prepare the disk for replacement by using the previously destroyed OSD id:
- ceph-volume lvm prepare --osd-id {id} --data /dev/sdX
+ .. prompt:: bash $
-#. And activate the OSD::
+ ceph-volume lvm prepare --osd-id {id} --data /dev/sdX
- ceph-volume lvm activate {id} {fsid}
+#. And activate the OSD:
+
+ .. prompt:: bash $
+
+ ceph-volume lvm activate {id} {fsid}
Alternatively, instead of preparing and activating, the device can be recreated
-in one call, like::
+in one call, like:
+
+ .. prompt:: bash $
- ceph-volume lvm create --osd-id {id} --data /dev/sdX
+ ceph-volume lvm create --osd-id {id} --data /dev/sdX
Starting the OSD
it is not yet running. The OSD is ``down`` and ``in``. You must start
your new OSD before it can begin receiving data. You may use
``service ceph`` from your admin host or start the OSD from its host
-machine::
+machine:
- sudo systemctl start ceph-osd@{osd-num}
+ .. prompt:: bash $
+
+ sudo systemctl start ceph-osd@{osd-num}
Once you start your OSD, it is ``up`` and ``in``.
Once you have added your new OSD to the CRUSH map, Ceph will begin rebalancing
the server by migrating placement groups to your new OSD. You can observe this
-process with the `ceph`_ tool. ::
+process with the `ceph`_ tool. :
+
+ .. prompt:: bash $
- ceph -w
+ ceph -w
You should see the placement group states change from ``active+clean`` to
``active, some degraded objects``, and finally ``active+clean`` when migration
completes. (Control-c to exit.)
-
.. _Add/Move an OSD: ../crush-map#addosd
.. _ceph: ../monitoring
Before you remove an OSD, it is usually ``up`` and ``in``. You need to take it
out of the cluster so that Ceph can begin rebalancing and copying its data to
-other OSDs. ::
+other OSDs. :
- ceph osd out {osd-num}
+ .. prompt:: bash $
+
+ ceph osd out {osd-num}
Observe the Data Migration
Once you have taken your OSD ``out`` of the cluster, Ceph will begin
rebalancing the cluster by migrating placement groups out of the OSD you
-removed. You can observe this process with the `ceph`_ tool. ::
+removed. You can observe this process with the `ceph`_ tool. :
+
+ .. prompt:: bash $
- ceph -w
+ ceph -w
You should see the placement group states change from ``active+clean`` to
``active, some degraded objects``, and finally ``active+clean`` when migration
``active+remapped`` state. If you are in this case, you should mark
the OSD ``in`` with:
- ``ceph osd in {osd-num}``
+ .. prompt:: bash $
+
+ ceph osd in {osd-num}
to come back to the initial state and then, instead of marking ``out``
the OSD, set its weight to 0 with:
- ``ceph osd crush reweight osd.{osd-num} 0``
+ .. prompt:: bash $
+
+ ceph osd crush reweight osd.{osd-num} 0
After that, you can observe the data migration which should come to its
end. The difference between marking ``out`` the OSD and reweighting it
After you take an OSD out of the cluster, it may still be running.
That is, the OSD may be ``up`` and ``out``. You must stop
-your OSD before you remove it from the configuration. ::
+your OSD before you remove it from the configuration:
+
+ .. prompt:: bash $
- ssh {osd-host}
- sudo systemctl stop ceph-osd@{osd-num}
+ ssh {osd-host}
+ sudo systemctl stop ceph-osd@{osd-num}
Once you stop your OSD, it is ``down``.
#. Let the cluster forget the OSD first. This step removes the OSD from the CRUSH
map, removes its authentication key. And it is removed from the OSD map as
well. Please note the :ref:`purge subcommand <ceph-admin-osd>` is introduced in Luminous, for older
- versions, please see below ::
+ versions, please see below:
+
+ .. prompt:: bash $
- ceph osd purge {id} --yes-i-really-mean-it
+ ceph osd purge {id} --yes-i-really-mean-it
#. Navigate to the host where you keep the master copy of the cluster's
- ``ceph.conf`` file. ::
+ ``ceph.conf`` file:
- ssh {admin-host}
- cd /etc/ceph
- vim ceph.conf
+ .. prompt:: bash $
-#. Remove the OSD entry from your ``ceph.conf`` file (if it exists). ::
+ ssh {admin-host}
+ cd /etc/ceph
+ vim ceph.conf
+
+#. Remove the OSD entry from your ``ceph.conf`` file (if it exists)::
[osd.1]
host = {hostname}
-#. From the host where you keep the master copy of the cluster's ``ceph.conf`` file,
- copy the updated ``ceph.conf`` file to the ``/etc/ceph`` directory of other
- hosts in your cluster.
+#. From the host where you keep the master copy of the cluster's ``ceph.conf``
+ file, copy the updated ``ceph.conf`` file to the ``/etc/ceph`` directory of
+ other hosts in your cluster.
-If your Ceph cluster is older than Luminous, instead of using ``ceph osd purge``,
-you need to perform this step manually:
+If your Ceph cluster is older than Luminous, instead of using ``ceph osd
+purge``, you need to perform this step manually:
#. Remove the OSD from the CRUSH map so that it no longer receives data. You may
also decompile the CRUSH map, remove the OSD from the device list, remove the
device as an item in the host bucket or remove the host bucket (if it's in the
CRUSH map and you intend to remove the host), recompile the map and set it.
- See `Remove an OSD`_ for details. ::
+ See `Remove an OSD`_ for details:
+
+ .. prompt:: bash $
+
+ ceph osd crush remove {name}
+
+#. Remove the OSD authentication key:
+
+ .. prompt:: bash $
+
+ ceph auth del osd.{osd-num}
- ceph osd crush remove {name}
+ The value of ``ceph`` for ``ceph-{osd-num}`` in the path is the
+ ``$cluster-$id``. If your cluster name differs from ``ceph``, use your
+ cluster name instead.
-#. Remove the OSD authentication key. ::
+#. Remove the OSD:
- ceph auth del osd.{osd-num}
+ .. prompt:: bash $
- The value of ``ceph`` for ``ceph-{osd-num}`` in the path is the ``$cluster-$id``.
- If your cluster name differs from ``ceph``, use your cluster name instead.
+ ceph osd rm {osd-num}
-#. Remove the OSD. ::
+ for example:
- ceph osd rm {osd-num}
- #for example
- ceph osd rm 1
+ .. prompt:: bash $
+ ceph osd rm 1
.. _Remove an OSD: ../crush-map#removeosd
projects / ceph.git / commitdiff