From 80517806eb9939ddef3d2eadbdb2d7e4b31d4beb Mon Sep 17 00:00:00 2001 From: Ville Ojamo <14869000+bluikko@users.noreply.github.com> Date: Thu, 7 Aug 2025 11:16:39 +0700 Subject: [PATCH] doc/install: Small improvements in manual-freebsd-deployment.rst Update text and commands for the cluster vanity name deprecation: - Explain that vanity names are deprecated (taken from ceph-conf.rst). - Remove cluster name parameters from commands. - Change cluster name variable in commands to simply "ceph". Linkify some terms in the introduction text. Use double backticks for data like paths, files consistently. Don't capitalize in the middle of a sentence. Don't capitalize "daemons". Capitalize "MDS", the concept, consistently. Missing space in "disk format", "disk layout", "discussion list". Fix typoed "comingling", "resolved". Fix copypasta zpool command that is supposed to add cache and not log. Try to improve grammar and add missing full stops. Use admonitions for note paragraphs. Remove trailing '/' from directory path to be consistent with other paths. Fix missing empty line causing RST syntax rendered verbatim and with wrong formatting. Replace all tabs with four spaces. Wrap very long text line. Don't render colon in "Some text.:" preceding preformatted blocks. Try to fix a preformatted table with columns all over the place. Checked how it should look on Quincy but the output is different. Promptify CLI commands but don't touch the long list of installation commands. They might need reviewing anyways. Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com> --- doc/install/manual-freebsd-deployment.rst | 319 +++++++++++----------- 1 file changed, 161 insertions(+), 158 deletions(-) diff --git a/doc/install/manual-freebsd-deployment.rst b/doc/install/manual-freebsd-deployment.rst index d32ad96a5e136..44ffb63b44747 100644 --- a/doc/install/manual-freebsd-deployment.rst +++ b/doc/install/manual-freebsd-deployment.rst @@ -3,17 +3,17 @@ ============================== This a largely a copy of the regular Manual Deployment with FreeBSD specifics. -The difference lies in two parts: The underlying diskformat, and the way to use +The difference lies in two parts: the underlying disk format, and the way to use the tools. All Ceph clusters require at least one monitor, and at least as many OSDs as copies of an object stored on the cluster. Bootstrapping the initial monitor(s) is the first step in deploying a Ceph Storage Cluster. Monitor deployment also sets important criteria for the entire cluster, such as the number of replicas -for pools, the number of placement groups per OSD, the heartbeat intervals, -whether authentication is required, etc. Most of these values are set by -default, so it's useful to know about them when setting up your cluster for -production. +for pools, the number of :ref:`placement groups` per OSD, the heartbeat intervals, +whether :ref:`authentication ` is required, etc. Most of these +values are set by default, so it's useful to know about them when setting up +your cluster for production. We will set up a cluster with ``node1`` as the monitor node, and ``node2`` and ``node3`` for OSD nodes. @@ -42,15 +42,17 @@ We will set up a cluster with ``node1`` as the monitor node, and ``node2`` and -Disklayout on FreeBSD -===================== +Disk Layout on FreeBSD +====================== + +Current implementation works on ZFS pools. -Current implementation works on ZFS pools +* All Ceph data is created in ``/var/lib/ceph``. +* Log files go into ``/var/log/ceph``. +* PID files go into ``/var/log/run``. +* One ZFS pool is allocated per OSD, like: -* All Ceph data is created in /var/lib/ceph -* Log files go into /var/log/ceph -* PID files go into /var/log/run -* One ZFS pool is allocated per OSD, like:: + .. prompt:: bash # gpart create -s GPT ada1 gpart add -t freebsd-zfs -l osd.1 ada1 @@ -60,31 +62,35 @@ Current implementation works on ZFS pools Please note that this is different from the Ceph journals. Cache and log are totally transparent for Ceph, and help the file system to keep the system consistent and help performance. - Assuming that ada2 is an SSD:: + Assuming that ``ada2`` is an SSD: + + .. prompt:: bash # gpart create -s GPT ada2 gpart add -t freebsd-zfs -l osd.1-log -s 1G ada2 zpool add osd.1 log gpt/osd.1-log gpart add -t freebsd-zfs -l osd.1-cache -s 10G ada2 - zpool add osd.1 log gpt/osd.1-cache + zpool add osd.1 cache gpt/osd.1-cache -* Note: *UFS2 does not allow large xattribs* +.. note:: UFS2 does not allow large xattribs. Configuration ------------- -As per FreeBSD default parts of extra software go into ``/usr/local/``. Which +As per FreeBSD default, parts of extra software go into ``/usr/local``. This means that for ``/etc/ceph.conf`` the default location is ``/usr/local/etc/ceph/ceph.conf``. Smartest thing to do is to create a softlink -from ``/etc/ceph`` to ``/usr/local/etc/ceph``:: +from ``/etc/ceph`` to ``/usr/local/etc/ceph``: + +.. prompt:: bash # ln -s /usr/local/etc/ceph /etc/ceph -A sample file is provided in ``/usr/local/share/doc/ceph/sample.ceph.conf`` -Note that ``/usr/local/etc/ceph/ceph.conf`` will be found by most tools, +A sample file is provided in ``/usr/local/share/doc/ceph/sample.ceph.conf``. +Note that ``/usr/local/etc/ceph/ceph.conf`` will be found by most tools, and linking it to ``/etc/ceph/ceph.conf`` will help with any scripts that are found -in extra tools, scripts, and/or discussionlists. +in extra tools, scripts, and/or discussion lists. Monitor Bootstrapping ===================== @@ -98,27 +104,18 @@ a number of things: block devices, and object storage gateway interfaces too, so ``fsid`` is a bit of a misnomer. -- **Cluster Name:** Ceph clusters have a cluster name, which is a simple string - without spaces. The default cluster name is ``ceph``, but you may specify - a different cluster name. Overriding the default cluster name is - especially useful when you are working with multiple clusters and you need to - clearly understand which cluster your are working with. - - For example, when you run multiple clusters in a :ref:`multisite configuration `, - the cluster name (e.g., ``us-west``, ``us-east``) identifies the cluster for - the current CLI session. **Note:** To identify the cluster name on the - command line interface, specify a :ref:`Ceph configuration file ` with the - cluster name (e.g., ``ceph.conf``, ``us-west.conf``, ``us-east.conf``, etc.). - Also see CLI usage (``ceph --cluster {cluster-name}``). +- **Cluster Name:** Note that cluster *vanity names* are deprecated and may be + removed entirely from future releases. We *strongly urge* that new clusters be + provisioned only with the default name ``ceph``. - **Monitor Name:** Each monitor instance within a cluster has a unique name. In common practice, the Ceph Monitor name is the host name (we recommend one - Ceph Monitor per host, and no commingling of Ceph OSD Daemons with + Ceph Monitor per host, and no comingling of Ceph OSD daemons with Ceph Monitors). You may retrieve the short hostname with ``hostname -s``. - **Monitor Map:** Bootstrapping the initial monitor(s) requires you to - generate a monitor map. The monitor map requires the ``fsid``, the cluster - name (or uses the default), and at least one host name and its IP address. + generate a monitor map. The monitor map requires the ``fsid`` and at least + one host name and its IP address. - **Monitor Keyring**: Monitors communicate with each other via a secret key. You must generate a keyring with a monitor secret and provide @@ -144,56 +141,56 @@ The procedure is as follows: #. Log in to the initial monitor node(s):: - ssh {hostname} + ssh {hostname} For example:: - ssh node1 + ssh node1 #. Ensure you have a directory for the Ceph configuration file. By default, Ceph uses ``/etc/ceph``. When you install ``ceph``, the installer will create the ``/etc/ceph`` directory automatically. :: - ls /etc/ceph + ls /etc/ceph #. Create a Ceph configuration file. By default, Ceph uses - ``ceph.conf``, where ``ceph`` reflects the cluster name. :: + ``ceph.conf``. :: - sudo vim /etc/ceph/ceph.conf + sudo vim /etc/ceph/ceph.conf #. Generate a unique ID (i.e., ``fsid``) for your cluster. :: - uuidgen + uuidgen #. Add the unique ID to your Ceph configuration file. :: - fsid = {UUID} + fsid = {UUID} For example:: - fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993 + fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993 #. Add the initial monitor(s) to your Ceph configuration file. :: - mon initial members = {hostname}[,{hostname}] + mon initial members = {hostname}[,{hostname}] For example:: - mon initial members = node1 + mon initial members = node1 #. Add the IP address(es) of the initial monitor(s) to your Ceph configuration file and save the file. :: - mon host = {ip-address}[,{ip-address}] + mon host = {ip-address}[,{ip-address}] For example:: - mon host = 192.168.0.1 + mon host = 192.168.0.1 **Note:** You may use IPv6 addresses instead of IPv4 addresses, but you must set ``ms bind ipv6`` to ``true``. See `Network Configuration @@ -201,140 +198,142 @@ The procedure is as follows: #. Create a keyring for your cluster and generate a monitor secret key. :: - ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' + ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' #. Generate an administrator keyring, generate a ``client.admin`` user and add the user to the keyring. :: - sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' + sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' #. Add the ``client.admin`` key to the ``ceph.mon.keyring``. :: - ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring + ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring #. Generate a monitor map using the hostname(s), host IP address(es) and the FSID. Save it as ``/tmp/monmap``:: - monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap + monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap For example:: - monmaptool --create --add node1 192.168.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap + monmaptool --create --add node1 192.168.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap #. Create a default data directory (or directories) on the monitor host(s). :: - sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname} + sudo mkdir /var/lib/ceph/mon/ceph-{hostname} For example:: - sudo mkdir /var/lib/ceph/mon/ceph-node1 + sudo mkdir /var/lib/ceph/mon/ceph-node1 See `Monitor Config Reference - Data`_ for details. #. Populate the monitor daemon(s) with the monitor map and keyring. :: - sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring + sudo -u ceph ceph-mon --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring For example:: - sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring + sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring #. Consider settings for a Ceph configuration file. Common settings include the following:: - [global] - fsid = {cluster-id} - mon initial members = {hostname}[, {hostname}] - mon host = {ip-address}[, {ip-address}] - public network = {network}[, {network}] - cluster network = {network}[, {network}] - auth cluster required = cephx - auth service required = cephx - auth client required = cephx - osd journal size = {n} - osd pool default size = {n} # Write an object n times. - osd pool default min size = {n} # Allow writing n copy in a degraded state. - osd pool default pg num = {n} - osd pool default pgp num = {n} - osd crush chooseleaf type = {n} + [global] + fsid = {cluster-id} + mon initial members = {hostname}[, {hostname}] + mon host = {ip-address}[, {ip-address}] + public network = {network}[, {network}] + cluster network = {network}[, {network}] + auth cluster required = cephx + auth service required = cephx + auth client required = cephx + osd journal size = {n} + osd pool default size = {n} # Write an object n times. + osd pool default min size = {n} # Allow writing n copy in a degraded state. + osd pool default pg num = {n} + osd pool default pgp num = {n} + osd crush chooseleaf type = {n} In the foregoing example, the ``[global]`` section of the configuration might look like this:: - [global] - fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993 - mon initial members = node1 - mon host = 192.168.0.1 - public network = 192.168.0.0/24 - auth cluster required = cephx - auth service required = cephx - auth client required = cephx - osd journal size = 1024 - osd pool default size = 3 - osd pool default min size = 2 - osd pool default pg num = 333 - osd pool default pgp num = 333 - osd crush chooseleaf type = 1 + [global] + fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993 + mon initial members = node1 + mon host = 192.168.0.1 + public network = 192.168.0.0/24 + auth cluster required = cephx + auth service required = cephx + auth client required = cephx + osd journal size = 1024 + osd pool default size = 3 + osd pool default min size = 2 + osd pool default pg num = 333 + osd pool default pgp num = 333 + osd crush chooseleaf type = 1 #. Touch the ``done`` file. Mark that the monitor is created and ready to be started:: - sudo touch /var/lib/ceph/mon/ceph-node1/done + sudo touch /var/lib/ceph/mon/ceph-node1/done #. And for FreeBSD an entry for every monitor needs to be added to the config - file. (The requirement will be removed in future releases). + file. (The requirement will be removed in future releases.) The entry should look like:: [mon] [mon.node1] - host = node1 # this name can be resolve + host = node1 # this name can be resolved #. Start the monitor(s). For FreeBSD we use the rc.d init scripts (called bsdrc in Ceph):: - sudo service ceph start start mon.node1 + sudo service ceph start start mon.node1 + + For this to work ``/etc/rc.conf`` also needs the entry to enable the + ``ceph`` services:: - For this to work /etc/rc.conf also needs the entry to enable ceph:: - cat 'ceph_enable="YES"' >> /etc/rc.conf + cat 'ceph_enable="YES"' >> /etc/rc.conf #. Verify that Ceph created the default pools. :: - ceph osd lspools + ceph osd lspools You should see output like this:: - 0 data - 1 metadata - 2 rbd + 0 data + 1 metadata + 2 rbd #. Verify that the monitor is running. :: - ceph -s + ceph -s You should see output that the monitor you started is up and running, and - you should see a health error indicating that placement groups are stuck - inactive. It should look something like this:: + you should see a health error indicating that placement groups are ``stuck + inactive``. It should look something like this:: - cluster a7f64266-0894-4f1e-a635-d0aeaca0e993 - health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds - monmap e1: 1 mons at {node1=192.168.0.1:6789/0}, election epoch 1, quorum 0 node1 - osdmap e1: 0 osds: 0 up, 0 in - pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects - 0 kB used, 0 kB / 0 kB avail - 192 creating + cluster a7f64266-0894-4f1e-a635-d0aeaca0e993 + health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds + monmap e1: 1 mons at {node1=192.168.0.1:6789/0}, election epoch 1, quorum 0 node1 + osdmap e1: 0 osds: 0 up, 0 in + pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects + 0 kB used, 0 kB / 0 kB avail + 192 creating - **Note:** Once you add OSDs and start them, the placement group health errors - should disappear. See the next section for details. + .. note:: Once you add OSDs and start them, the placement group health errors + should disappear. See the next section for details. .. _freebsd_adding_osds: @@ -345,7 +344,7 @@ Once you have your initial monitor(s) running, you should add OSDs. Your cluster cannot reach an ``active + clean`` state until you have enough OSDs to handle the number of copies of an object (e.g., ``osd pool default size = 2`` requires at least two OSDs). After bootstrapping your monitor, your cluster has a default -CRUSH map; however, the CRUSH map doesn't have any Ceph OSD Daemons mapped to +CRUSH map; however, the CRUSH map doesn't have any Ceph OSD daemons mapped to a Ceph Node. @@ -359,24 +358,24 @@ OSDs with the long form procedure, execute the following on ``node2`` and #. Connect to the OSD host. :: - ssh {node-name} + ssh {node-name} #. Generate a UUID for the OSD. :: - uuidgen + uuidgen #. Create the OSD. If no UUID is given, it will be set automatically when the OSD starts up. The following command will output the OSD number, which you will need for subsequent steps. :: - ceph osd create [{uuid} [{id}]] + ceph osd create [{uuid} [{id}]] #. Create the default directory on your new OSD. :: - ssh {new-osd-host} - sudo mkdir /var/lib/ceph/osd/{cluster-name}-{osd-number} + ssh {new-osd-host} + sudo mkdir /var/lib/ceph/osd/ceph-{osd-number} Above are the ZFS instructions to do this for FreeBSD. @@ -387,33 +386,30 @@ OSDs with the long form procedure, execute the following on ``node2`` and #. Initialize the OSD data directory. :: - ssh {new-osd-host} - sudo ceph-osd -i {osd-num} --mkfs --mkkey --osd-uuid [{uuid}] + ssh {new-osd-host} + sudo ceph-osd -i {osd-num} --mkfs --mkkey --osd-uuid [{uuid}] The directory must be empty before you can run ``ceph-osd`` with the - ``--mkkey`` option. In addition, the ceph-osd tool requires specification - of custom cluster names with the ``--cluster`` option. + ``--mkkey`` option. -#. Register the OSD authentication key. The value of ``ceph`` for - ``ceph-{osd-num}`` in the path is the ``$cluster-$id``. If your - cluster name differs from ``ceph``, use your cluster name instead.:: +#. Register the OSD authentication key. :: - sudo ceph auth add osd.{osd-num} osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/{cluster-name}-{osd-num}/keyring + sudo ceph auth add osd.{osd-num} osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-{osd-num}/keyring #. Add your Ceph Node to the CRUSH map. :: - ceph [--cluster {cluster-name}] osd crush add-bucket {hostname} host + ceph osd crush add-bucket {hostname} host For example:: - ceph osd crush add-bucket node1 host + ceph osd crush add-bucket node1 host #. Place the Ceph Node under the root ``default``. :: - ceph osd crush move node1 root=default + ceph osd crush move node1 root=default #. Add the OSD to the CRUSH map so that it can begin receiving data. You may @@ -421,11 +417,11 @@ OSDs with the long form procedure, execute the following on ``node2`` and bucket (if it's not already in the CRUSH map), add the device as an item in the host, assign it a weight, recompile it and set it. :: - ceph [--cluster {cluster-name}] osd crush add {id-or-name} {weight} [{bucket-type}={bucket-name} ...] + ceph osd crush add {id-or-name} {weight} [{bucket-type}={bucket-name} ...] For example:: - ceph osd crush add osd.0 1.0 host=node1 + ceph osd crush add osd.0 1.0 host=node1 #. After you add an OSD to Ceph, the OSD is in your configuration. However, @@ -436,22 +432,22 @@ OSDs with the long form procedure, execute the following on ``node2`` and After adding the OSD to ``ceph.conf``:: - sudo service ceph start osd.{osd-num} + sudo service ceph start osd.{osd-num} For example:: - sudo service ceph start osd.0 - sudo service ceph start osd.1 + sudo service ceph start osd.0 + sudo service ceph start osd.1 In this case, to allow the start of the daemon at each reboot you must create an empty file like this:: - sudo touch /var/lib/ceph/osd/{cluster-name}-{osd-num}/bsdrc + sudo touch /var/lib/ceph/osd/ceph-{osd-num}/bsdrc For example:: - sudo touch /var/lib/ceph/osd/ceph-0/bsdrc - sudo touch /var/lib/ceph/osd/ceph-1/bsdrc + sudo touch /var/lib/ceph/osd/ceph-0/bsdrc + sudo touch /var/lib/ceph/osd/ceph-1/bsdrc Once you start your OSD, it is ``up`` and ``in``. @@ -462,36 +458,39 @@ Adding MDS In the below instructions, ``{id}`` is an arbitrary name, such as the hostname of the machine. -#. Create the mds data directory.:: +#. Create the MDS data directory. :: - mkdir -p /var/lib/ceph/mds/{cluster-name}-{id} + mkdir -p /var/lib/ceph/mds/ceph-{id} -#. Create a keyring.:: +#. Create a keyring. :: - ceph-authtool --create-keyring /var/lib/ceph/mds/{cluster-name}-{id}/keyring --gen-key -n mds.{id} + ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-{id}/keyring --gen-key -n mds.{id} -#. Import the keyring and set caps.:: +#. Import the keyring and set caps. :: - ceph auth add mds.{id} osd "allow rwx" mds "allow *" mon "allow profile mds" -i /var/lib/ceph/mds/{cluster}-{id}/keyring + ceph auth add mds.{id} osd "allow rwx" mds "allow *" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-{id}/keyring -#. Add to ceph.conf.:: +#. Add to ``ceph.conf``. :: - [mds.{id}] - host = {id} + [mds.{id}] + host = {id} -#. Start the daemon the manual way.:: +#. Start the :ref:`daemon ` the manual way. :: - ceph-mds --cluster {cluster-name} -i {id} -m {mon-hostname}:{mon-port} [-f] + ceph-mds -i {id} -m {mon-hostname}:{mon-port} [-f] -#. Start the daemon the right way (using ceph.conf entry).:: +#. Start the daemon the right way (using ``ceph.conf`` entry). :: - service ceph start + service ceph start #. If starting the daemon fails with this error:: - mds.-1.0 ERROR: failed to authenticate: (22) Invalid argument + mds.-1.0 ERROR: failed to authenticate: (22) Invalid argument - Then make sure you do not have a keyring set in ceph.conf in the global section; move it to the client section; or add a keyring setting specific to this mds daemon. And verify that you see the same key in the mds data directory and ``ceph auth get mds.{id}`` output. + Then make sure you do not have a keyring set in ``ceph.conf`` in the ``global`` + section; move it to the ``client`` section; or add a keyring setting specific + to this MDS daemon. And verify that you see the same key in the MDS data + directory and ``ceph auth get mds.{id}`` output. #. Now you are ready to `create a Ceph file system`_. @@ -500,25 +499,29 @@ Summary ======= Once you have your monitor and two OSDs up and running, you can watch the -placement groups peer by executing the following:: +placement groups peer by executing the following: + +.. prompt:: bash # + + ceph -w - ceph -w +To view the tree, execute the following: -To view the tree, execute the following:: +.. prompt:: bash # - ceph osd tree + ceph osd tree You should see output that looks something like this:: - # id weight type name up/down reweight - -1 2 root default - -2 2 host node1 - 0 1 osd.0 up 1 - -3 1 host node2 - 1 1 osd.1 up 1 + # id weight type name up/down reweight + -1 2 root default + -2 2 host node1 + 0 1 osd.0 up 1 + -3 1 host node2 + 1 1 osd.1 up 1 To add (or remove) additional monitors, see :ref:`adding-and-removing-monitors`. -To add (or remove) additional Ceph OSD Daemons, see `Add/Remove OSDs`_. +To add (or remove) additional Ceph OSD daemons, see `Add/Remove OSDs`_. .. _Add/Remove OSDs: ../../rados/operations/add-or-rm-osds -- 2.39.5