From 11fef2275d9810cfc279fba0d4c789fb89ac9c3b Mon Sep 17 00:00:00 2001 From: Ken Dreyer Date: Thu, 21 May 2015 12:54:30 -0600 Subject: [PATCH] doc: recommend opening entire 6800-7300 port range Prior to this commit, the Network Configuration Reference guide and Troubleshooting guide recommended opening a number of ports that were unique to the number of daemons that we ran. This doesn't really cover all use cases. Users can easily restart daemons in ways that cause the daemons to bind to higher ports. This leads to OSDs or MDSs binding to ports that are firewalled. Update the Network Configuration Reference guide and Troubleshooting guides to simply recommend that users open all the ports between 6800 and 7300 on their OSDs and MDSs. http://tracker.ceph.com/issues/11688 Refs: #11688 Signed-off-by: Ken Dreyer --- .../configuration/network-config-ref.rst | 30 ++++++++----------- .../troubleshooting/troubleshooting-mon.rst | 5 ++-- 2 files changed, 16 insertions(+), 19 deletions(-) diff --git a/doc/rados/configuration/network-config-ref.rst b/doc/rados/configuration/network-config-ref.rst index 440e476f42025..3e0aa9e165c51 100644 --- a/doc/rados/configuration/network-config-ref.rst +++ b/doc/rados/configuration/network-config-ref.rst @@ -103,8 +103,10 @@ MDS IP Tables ------------- A :term:`Ceph Metadata Server` listens on the first available port on the public -network beginning at port 6800. Ensure that you open one port beginning at port -6800 for each Ceph Metadata Server that runs on the Ceph Node. When you add the +network beginning at port 6800. Note that this behavior is not deterministic, so +if you are running more than one OSD or MDS on the same host, or if you restart +the daemons within a short window of time, the daemons will bind to higher +ports. You should open the entire 6800-7300 range by default. When you add the rule using the example below, make sure you replace ``{iface}`` with the public network interface (e.g., ``eth0``, ``eth1``, etc.), ``{ip-address}`` with the IP address of the public network and ``{netmask}`` with the netmask of the public @@ -112,16 +114,17 @@ network. For example:: - sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:6810 -j ACCEPT + sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT OSD IP Tables ------------- By default, Ceph OSD Daemons `bind`_ to the first available ports on a Ceph Node -beginning at port 6800. Ensure that you open at least three ports beginning at -port 6800 for each OSD that runs on the host. Each Ceph OSD Daemon on a Ceph -Node may use up to three ports: +beginning at port 6800. Note that this behavior is not deterministic, so if you +are running more than one OSD or MDS on the same host, or if you restart the +daemons within a short window of time, the daemons will bind to higher ports. +Each Ceph OSD Daemon on a Ceph Node may use up to three ports: #. One for talking to clients and monitors. #. One for sending data to other OSDs. @@ -144,11 +147,9 @@ Node may use up to three ports: | cCCC | \---------------/ -Ports are node-specific, so you don't need to open any more ports than the -number of ports needed by Ceph daemons running on that Ceph Node. You may -consider opening a few additional ports in case a daemon fails and restarts -without letting go of the port such that the restarted daemon binds to a new -port. +When a daemon fails and restarts without letting go of the port, the restarted +daemon will bind to a new port. You should open the entire 6800-7300 port range +to handle this possibility. If you set up separate public and cluster networks, you must add rules for both the public network and the cluster network, because clients will connect using @@ -158,15 +159,10 @@ network. When you add the rule using the example below, make sure you replace ``{ip-address}`` with the IP address and ``{netmask}`` with the netmask of the public or cluster network. For example:: - sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:6810 -j ACCEPT - -Be sure to replace the "6810" in the above example with an upper bound that -reflects the number of daemons you will be running on this host. + sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT .. tip:: If you run Ceph Metadata Servers on the same Ceph Node as the Ceph OSD Daemons, you can consolidate the public network configuration step. - Ensure that you open the number of ports required for each daemon per host. - Ceph Networks diff --git a/doc/rados/troubleshooting/troubleshooting-mon.rst b/doc/rados/troubleshooting/troubleshooting-mon.rst index b1ac0619538dd..5d51774ca1386 100644 --- a/doc/rados/troubleshooting/troubleshooting-mon.rst +++ b/doc/rados/troubleshooting/troubleshooting-mon.rst @@ -378,9 +378,10 @@ like this appropriately:: You may also need to add rules to IP tables on your Ceph hosts to ensure that clients can access the ports associated with your Ceph monitors (i.e., port -6789 by default) and Ceph OSDs (i.e., 6800 et. seq. by default). For example:: +6789 by default) and Ceph OSDs (i.e., 6800 through 7300 by default). For +example:: - iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:6810 -j ACCEPT + iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:7300 -j ACCEPT Everything Failed! Now What? -- 2.39.5