ceph orch host ls [--format yaml] [--host-pattern <name>] [--label <label>] [--host-status <status>] [--detail]
-In commands of this form, the arguments "host-pattern", "label", and
-"host-status" are optional and are used for filtering.
+In commands of this form, the arguments ``host-pattern``, ``label``, and
+``host-status`` are optional and are used for filtering.
-- "host-pattern" is a regex that matches against hostnames and returns only
+- ``host-pattern`` is a regular expression that matches against hostnames and returns only
matching hosts.
-- "label" returns only hosts with the specified label.
-- "host-status" returns only hosts with the specified status (currently
- "offline" or "maintenance").
+- ``label`` returns only hosts with the specified label.
+- ``host-status`` returns only hosts with the specified status (currently
+ ``offline`` or ``maintenance``).
- Any combination of these filtering flags is valid. It is possible to filter
against name, label and status simultaneously, or to filter against any
proper subset of name, label and status.
-The "detail" parameter provides more host related information for cephadm based
+The ``detail`` parameter provides more host related information for cephadm-based
clusters. For example:
.. prompt:: bash #
ceph orch host ls --detail
-::
+.. code-block:: console
HOSTNAME ADDRESS LABELS STATUS VENDOR/MODEL CPU HDD SSD NIC
ceph-master 192.168.122.73 _admin QEMU (Standard PC (Q35 + ICH9, 2009)) 4C/4T 4/1.6TB - 1
1 hosts in cluster
-.. _cephadm-adding-hosts:
-
+
+.. _cephadm-adding-hosts:
+
Adding Hosts
============
.. prompt:: bash #
- ssh-copy-id -f -i /etc/ceph/ceph.pub root@*<new-host>*
+ ssh-copy-id -f -i /etc/ceph/ceph.pub root@<new-host>
For example:
.. prompt:: bash #
- ceph orch host add *<newhost>* [*<ip>*] [*<label1> ...*]
+ ceph orch host add <newhost> [<ip>] [<label1> ...]
For example:
ceph orch host add host4 10.10.0.104 --labels _admin
+
.. _cephadm-removing-hosts:
Removing Hosts
.. prompt:: bash #
- ceph orch host drain *<host>*
+ ceph orch host drain <host>
The ``_no_schedule`` and ``_no_conf_keyring`` labels will be applied to the
host. See :ref:`cephadm-special-host-labels`.
-If you want to drain daemons but leave managed `ceph.conf` and keyring
+If you want to drain daemons but leave the managed ``ceph.conf`` and keyring
files on the host, you may pass the ``--keep-conf-keyring`` flag to the
-drain command.
+drain command:
.. prompt:: bash #
- ceph orch host drain *<host>* --keep-conf-keyring
+ ceph orch host drain <host> --keep-conf-keyring
This will apply the ``_no_schedule`` label to the host but not the
``_no_conf_keyring`` label.
The ``orch host drain`` command also supports a ``--zap-osd-devices``
flag. Setting this flag while draining a host will cause cephadm to zap
-the devices of the OSDs it is removing as part of the drain process
+the devices of the OSDs it is removing as part of the drain process.
.. prompt:: bash #
- ceph orch host drain *<host>* --zap-osd-devices
+ ceph orch host drain <host> --zap-osd-devices
-Use the following command to determine whether any daemons are still on the
-host:
+Run a command of the following form to determine whether any daemons are
+still on the host:
.. prompt:: bash #
- ceph orch ps <host>
+ ceph orch ps <host>
After all daemons have been removed from the host, remove the host from the
-cluster by running the following command:
+cluster by running a command of the following form:
.. prompt:: bash #
ceph orch host rm <host>
-Offline host removal
+
+Offline Host Removal
--------------------
If a host is offline and can not be recovered, it can be removed from the
.. warning:: This can potentially cause data loss. This command forcefully
purges OSDs from the cluster by calling ``osd purge-actual`` for each OSD.
Any service specs that still contain this host should be manually updated.
+ For more information, see :ref:`orchestrator-cli-service-spec`.
+
.. _orchestrator-host-labels:
-Host labels
+Host Labels
===========
The orchestrator supports assigning labels to hosts. Labels
-are free form and have no particular meaning by itself and each host
-can have multiple labels. They can be used to specify placement
-of daemons. See :ref:`orch-placement-by-labels`
+are free-form and have no particular meaning by itself and each host
+can have multiple labels. They can be used to specify the placement
+of daemons. For more information, see :ref:`orch-placement-by-labels`.
Labels can be added when adding a host with the ``--labels`` flag:
.. _cephadm-special-host-labels:
-Special host labels
+Special Host Labels
-------------------
The following host labels have a special meaning to cephadm. All start with ``_``.
* ``_no_conf_keyring``: *Do not deploy config files or keyrings on this host*.
This label is effectively the same as ``_no_schedule`` but instead of working for
- daemons it works for client keyrings and ceph conf files that are being managed
- by cephadm
+ daemons it works for client keyrings and ``ceph.conf`` files that are being managed
+ by cephadm.
* ``_no_autotune_memory``: *Do not autotune memory on this host*.
This label will prevent daemon memory from being tuned even when the
- ``osd_memory_target_autotune`` or similar option is enabled for one or more daemons
+ :confval:`osd_memory_target_autotune` or similar option is enabled for one or more daemons
on that host.
-* ``_admin``: *Distribute client.admin and ceph.conf to this host*.
+* ``_admin``: *Distribute config files and keyrings to this host*.
By default, an ``_admin`` label is applied to the first host in the cluster (where
bootstrap was originally run), and the ``client.admin`` key is set to be distributed
to that host via the ``ceph orch client-keyring ...`` function. Adding this label
- to additional hosts will normally cause cephadm to deploy config and keyring files
+ to additional hosts will normally cause cephadm to deploy configuration and keyring files
in ``/etc/ceph``. Starting from versions 16.2.10 (Pacific) and 17.2.1 (Quincy) in
addition to the default location ``/etc/ceph/`` cephadm also stores config and keyring
files in the ``/var/lib/ceph/<fsid>/config`` directory.
+
Maintenance Mode
================
Putting a host into *maintenance mode* stops all Ceph daemons on the host.
Run a command of the following form to put a host into maintenance mode or
-to take a host out of maintenance mode:
+to take a host out of maintenance mode:
.. prompt:: bash #
ceph orch host maintenance exit <hostname> [--force] [--offline]
* Adding the ``--force`` flag to the ``enter`` command allows the user to
- bypass warnings (but not alerts).
+ bypass warnings (but not alerts).
* Adding the ``--yes-i-really-mean-it`` flag to the ``enter`` command
bypasses all safety checks and makes an attempt to force the host into
maintenance mode.
.. warning:: Using the ``--yes-i-really-mean-it`` flag to force the host to
enter maintenance mode can cause loss of data availability, breakdown of
- the mon quorum due to too few running monitors, unresponsive Manager
+ the Monitor quorum due to too few running Monitors, unresponsive Manager
module commands (such as ``ceph orch . . .`` commands), and other issues.
Use this flag only if you're absolutely certain that you know what you're
doing.
-See also :ref:`cephadm-fqdn`
+See also :ref:`cephadm-fqdn`.
+
Rescanning Host Devices
=======================
The ``with-summary`` flag provides a breakdown of the number of HBAs found and scanned, together
with any that failed:
-.. prompt:: bash [ceph:root@rh9-ceph1/]#
+.. prompt:: bash #
ceph orch host rescan rh9-ceph1 --with-summary
-
-::
+
+.. code-block:: console
Ok. 2 adapters detected: 2 rescanned, 0 skipped, 0 failed (0.32s)
-Creating many hosts at once
+
+Creating many Hosts at Once
===========================
-Many hosts can be added at once using
-``ceph orch apply -i`` by submitting a multi-document YAML file:
+Many hosts can be added at once using ``ceph orch apply -i`` by submitting
+a multi-document YAML file:
.. code-block:: yaml
addr: 192.168.0.12
This can be combined with :ref:`service specifications<orchestrator-cli-service-spec>`
-to create a cluster spec file to deploy a whole cluster in one command. see
-``cephadm bootstrap --apply-spec`` also to do this during bootstrap. Cluster
-SSH Keys must be copied to hosts prior to adding them.
+to create a cluster spec file to deploy a whole cluster in one command.
+Command of the form ``cephadm bootstrap --apply-spec`` can be used to also do
+this during bootstrap.
+
+Cluster SSH Keys must be copied to hosts prior to adding them,
+see :ref:`cephadm-ssh-configuration` below.
+
-Setting the initial CRUSH location of host
-==========================================
+Setting the Initial CRUSH Location of a Host
+============================================
-Hosts can contain a ``location`` identifier which will instruct cephadm to
+Hosts can contain a ``location`` identifier which will instruct cephadm to
create a new CRUSH host bucket located in the specified hierarchy.
You can specify more than one element of the tree when doing so (for
instance if you want to ensure that the rack that a host is being
root: default
rack: rack1
-.. note::
+.. note::
The ``location`` attribute will affect only the initial CRUSH location.
- Subsequent changes to the ``location`` property will be ignored.
+ Subsequent changes to the ``location`` property will be ignored.
Removing a host will not remove an associated CRUSH bucket unless the
``--rm-crush-entry`` flag is provided to the ``orch host rm`` command.
See also :ref:`crush_map_default_types`.
-Removing a host from the CRUSH map
+
+Removing a Host from the CRUSH Map
==================================
The ``ceph orch host rm`` command has support for removing the associated host bucket
from the CRUSH map. This is done by providing the ``--rm-crush-entry`` flag.
-.. prompt:: bash [ceph:root@host1/]#
+.. prompt:: bash #
ceph orch host rm host1 --rm-crush-entry
it fails to do so, cephadm will report the failure and the host will remain under
cephadm control.
-.. note::
+.. note::
Removal from the CRUSH map will fail if there are OSDs deployed on the
host. If you would like to remove all the host's OSDs as well, please start
by using the ``ceph orch host drain`` command to do so. Once the OSDs
- have been removed, then you may direct cephadm remove the CRUSH bucket
+ have been removed, then you may direct cephadm to remove the CRUSH bucket
along with the host using the ``--rm-crush-entry`` flag.
+
OS Tuning Profiles
==================
Cephadm can be used to manage operating system tuning profiles that apply
-``sysctl`` settings to sets of hosts.
+``sysctl`` settings to sets of hosts.
To do so, create a YAML spec file in the following format:
fs.file-max: 1000000
vm.swappiness: '13'
-Apply the tuning profile with the following command:
+Apply the tuning profile by running a command of the following form:
.. prompt:: bash #
ceph orch tuned-profile apply -i <tuned-profile-file-name>
This profile is written to a file under ``/etc/sysctl.d/`` on each host
-specified in the ``placement`` block, then ``sysctl --system`` is
+specified in the ``placement`` block, and then ``sysctl --system`` is
run on the host.
.. note::
sysctl settings are applied in lexicographical order (sorted by the filename
in which the setting is specified), you may want to carefully choose
the ``profile_name`` in your spec so that it is applied before or after other
- conf files. Careful selection ensures that values supplied here override or
+ configuration files. Careful selection ensures that values supplied here override or
do not override those in other ``sysctl.d`` files as desired.
.. note::
ceph orch tuned-profile ls
-.. note::
+.. note::
To make modifications and re-apply a profile, pass ``--format yaml`` to the
``tuned-profile ls`` command. The ``tuned-profile ls --format yaml`` command
Removing Profiles
-----------------
-To remove a previously applied profile, run this command:
+To remove a previously applied profile, run a command of the following form:
.. prompt:: bash #
ceph orch tuned-profile rm <profile-name>
-When a profile is removed, cephadm cleans up the file previously written to ``/etc/sysctl.d``.
+When a profile is removed, cephadm cleans up the file previously written
+to ``/etc/sysctl.d``, and then ``sysctl --system`` is run on the host.
Modifying Profiles
ceph orch tuned-profile rm-setting <profile-name> <setting-name>
-.. note::
+.. note::
Modifying the placement requires re-applying a profile with the same name.
Remember that profiles are tracked by their names, so when a profile with the
same name as an existing profile is applied, it overwrites the old profile
unless the ``--no-overwrite`` flag is passed.
+
+.. _cephadm-ssh-configuration:
+
SSH Configuration
=================
with those hosts in a secure way.
-Default behavior
+Default Behavior
----------------
-Cephadm stores an SSH key in the monitor that is used to
+Cephadm stores an SSH key in the :ref:`Monitor configuration database <ceph-conf-database>` that is used to
connect to remote hosts. When the cluster is bootstrapped, this SSH
key is generated automatically and no additional configuration
is necessary.
ceph config-key set mgr/cephadm/ssh_identity_key -i <key>
ceph config-key set mgr/cephadm/ssh_identity_pub -i <pub>
-You will then need to restart the mgr daemon to reload the configuration with:
+You will then need to restart the Manager daemon to reload the
+configuration with:
.. prompt:: bash #
ceph mgr fail
+
.. _cephadm-ssh-user:
-Configuring a different SSH user
-----------------------------------
+Configuring a Different SSH User
+--------------------------------
-Cephadm must be able to log into all the Ceph cluster nodes as an user
+Cephadm must be able to log into all the Ceph cluster nodes as a user
that has enough privileges to download container images, start containers
and execute commands without prompting for a password. If you do not want
to use the "root" user (default option in cephadm), you must provide
cephadm the name of the user that is going to be used to perform all the
-cephadm operations. Use the command:
+cephadm operations. Run a command of the following form:
.. prompt:: bash #
ceph cephadm set-user <user>
-Prior to running this the cluster SSH key needs to be added to this users
-authorized_keys file and non-root users must have passwordless sudo access.
+Prior to running this, the cluster SSH key needs to be added to this user's
+``authorized_keys`` file and non-root users must have passwordless sudo access.
-Customizing the SSH configuration
+Customizing the SSH Configuration
---------------------------------
Cephadm generates an appropriate ``ssh_config`` file that is
-used for connecting to remote hosts. This configuration looks
+used for connecting to remote hosts. The configuration looks
something like this::
Host *
There are two ways to customize this configuration for your environment:
#. Import a customized configuration file that will be stored
- by the monitor with:
+ by the Monitor with:
.. prompt:: bash #
ceph config set mgr mgr/cephadm/ssh_config_file <path>
We do *not recommend* this approach. The path name must be
- visible to *any* mgr daemon, and cephadm runs all daemons as
+ visible to *any* Manager daemon, and cephadm runs all daemons as
containers. That means that the file must either be placed
inside a customized container image for your deployment, or
- manually distributed to the mgr data directory
+ manually distributed to the Manager data directory
(``/var/lib/ceph/<cluster-fsid>/mgr.<id>`` on the host, visible at
``/var/lib/ceph/mgr/ceph-<id>`` from inside the container).
-Setting up CA signed keys for the cluster
+
+Setting up CA-signed Keys for the Cluster
-----------------------------------------
Cephadm also supports using CA signed keys for SSH authentication
across cluster nodes. In this setup, instead of needing a private
-key and public key, we instead need a private key and certificate
-created by signing that private key with a CA key. For more info
-on setting up nodes for authentication using a CA signed key, see
+key and a public key, we instead need a private key and a certificate
+created by signing the public key with a CA key. For more information
+on setting up nodes for authentication using a CA-signed key, see
:ref:`cephadm-bootstrap-ca-signed-keys`. Once you have your private
-key and signed cert, they can be set up for cephadm to use by running:
+key and signed certificate, they can be set up for cephadm to use by running
+commands of the following form:
.. prompt:: bash #
ceph config-key set mgr/cephadm/ssh_identity_key -i <private-key-file>
ceph config-key set mgr/cephadm/ssh_identity_cert -i <signed-cert-file>
+
.. _cephadm-fqdn:
-Fully qualified domain names vs bare host names
+Fully Qualified Domain Names vs Bare Host Names
===============================================
.. note::
- cephadm demands that the name of the host given via ``ceph orch host add``
+ Cephadm demands that the name of the host given via ``ceph orch host add``
equals the output of ``hostname`` on remote hosts.
Otherwise cephadm can't be sure that names returned by
``ceph * metadata`` match the hosts known to cephadm. This might result
in a :ref:`cephadm-stray-host` warning.
-When configuring new hosts, there are two **valid** ways to set the
+When configuring new hosts, there are two **valid** ways to set the
``hostname`` of a host:
-1. Using the bare host name. In this case:
+#. Using the bare host name. In this case:
-- ``hostname`` returns the bare host name.
-- ``hostname -f`` returns the FQDN.
+ - ``hostname`` returns the bare host name.
+ - ``hostname -f`` returns the FQDN.
-2. Using the fully qualified domain name as the host name. In this case:
+#. Using the fully qualified domain name as the host name. In this case:
-- ``hostname`` returns the FQDN
-- ``hostname -s`` return the bare host name
+ - ``hostname`` returns the FQDN.
+ - ``hostname -s`` return the bare host name.
Note that ``man hostname`` recommends ``hostname`` to return the bare
host name:
Which means, ``man hostname`` recommends ``hostname`` to return the bare
host name. This in turn means that Ceph will return the bare host names
when executing ``ceph * metadata``. This in turn means cephadm also
-requires the bare host name when adding a host to the cluster:
+requires the bare host name when adding a host to the cluster:
``ceph orch host add <bare-name>``.
..
projects / ceph.git / commitdiff