TCP Hole Punching, how to establish TCP connection across NAT

Actually this post doesn't focus on how to establish a TCP connection across, however its concern is whether JAVA network API, especially TCP, supports using a single local port to listen for incoming TCP connections, and to initiate multiple outgoing TCP connections concurrently. Java really does it by SO_RESUEADDR, which is a socket option supports by all major operation systems.

This post is heavily inspired by the paper "Peer-to-Peer Communication Across Network Address Translator" which published by Byran Ford, Pyda Srisuresh and Dan Kegel.

Background
When talking about P2P, we will easily associate it with another question, how does P2P traverse across NAT? Many people know STUN, and STUN on UDP is easier to implement, also easier to understand. But UDP has its cons, for example, many NATs may disable UDP protocal, and UDP is not so reliable as TCP.

TCP hole punching(STUN on TCP) is more complicated to implement and understand, due to those we have mentioned at the beginning. Anyway we can use SO_RESUEADDR to achieve it.

Solution
I will give a java implementation which will listen and initiate connection on a single local port. Code will explain my idea.

  1 package org.clinic4j.net;
  2 
  3 import java.io.BufferedReader;
  4 import java.io.IOException;
  5 import java.io.InputStream;
  6 import java.io.InputStreamReader;
  7 import java.io.OutputStream;
  8 import java.io.PrintWriter;
  9 import java.net.InetSocketAddress;
 10 import java.net.ServerSocket;
 11 import java.net.Socket;
 12 
 13 /**
 14  * Just for testing socket SO_RESUEADDR. If set SO_RESUEADDR to true, we can use
 15  * a single local port to listen for incoming TCP connections, and to initiate
 16  * multiple outgoing TCP connections concurrently. By this way we can implement
 17  * TCP hole punching(establish P2P connection traversal through NAT over TCP).
 18  */
 19 public class TcpPeer {
 20   // TCP port is a different source from UDP port, it means you can listen on
 21   // same port for both TCP and UDP at the same time.
 22   private int localport = 7890;
 23   private ServerSocket peerSock;
 24   private Socket serverSocket;
 25 
 26   public TcpPeer(final String serverHost, final int serverPort, final int localPort)
 27           throws Exception {
 28     this.localport = localPort;
 29 
 30     Thread server = new Thread(new Runnable() {
 31 
 32       @Override
 33       public void run() {
 34         try {
 35           peerSock = new ServerSocket();
 36           peerSock.setReuseAddress(true);
 37           peerSock.bind(new InetSocketAddress("localhost", localport));
 38           System.out.println("[Server]The server is listening on " + localport + ".");
 39 
 40           while (true) {
 41             try {
 42               serverSocket = peerSock.accept();
 43               // just means finishing handshaking, and connection
 44               // established.
 45               System.out.println("[Server]New connection accepted"
 46                       + serverSocket.getInetAddress() + ":" + serverSocket.getPort());
 47 
 48               BufferedReader br = getReader(serverSocket);
 49               PrintWriter pw = getWriter(serverSocket);
 50               String req = br.readLine();
 51               System.out.println("[Server][REQ]" + req);
 52               pw.println(req);
 53 
 54               pw.close();
 55               br.close();
 56             } catch (IOException e) {
 57               e.printStackTrace();
 58             } finally {
 59               try {
 60                 if (serverSocket != null)
 61                   serverSocket.close();
 62               } catch (IOException e) {
 63                 e.printStackTrace();
 64               }
 65             }
 66           }
 67         } catch (Exception e) {
 68           e.printStackTrace();
 69         }
 70       }
 71 
 72     });
 73     //  server.setDaemon(true);
 74     server.start();
 75 
 76     // sleep several seconds before launch of client
 77     Thread.currentThread().sleep(5 * 1000);
 78 
 79     final int retry = 5;
 80     Thread client = new Thread(new Runnable() {
 81 
 82       @Override
 83       public void run() {
 84         Socket socket = new Socket();
 85         try {
 86           socket.setReuseAddress(true);
 87           System.out.println("[Client]socket.isBound():" + socket.isBound());
 88           socket.bind(new InetSocketAddress("localhost", localport));
 89           for (int i = 1; i < retry; i++) {
 90             try {
 91               socket.connect(new InetSocketAddress(serverHost, serverPort));
 92               System.out.println("[Client]connect to " + serverHost + ":"
 93                       + serverPort + " successfully.");
 94               break;
 95             } catch (Exception e) {
 96               System.out.println("[Client]fail to connect " + serverHost + ":"
 97                       + serverPort + ", try again.");
 98               Thread.currentThread().sleep(i * 2 * 1000);
100             }
101           }
102 
103           PrintWriter pw = getWriter(socket);
104           String msg = "hello world!";
105           pw.println(msg);
106 
107           BufferedReader br = getReader(socket);
108           String resp = br.readLine();
109           System.out.println("[Client][RESP-1]" + resp);
113           pw.close();
114           br.close();
115         } catch (Exception e) {
116           e.printStackTrace();
117         } finally {
118           try {
119             socket.close();
120           } catch (Exception e) {
121             e.printStackTrace();
122           }
123         }
124       }
125 
126     });
127     client.start();
128   }
129 
130   private PrintWriter getWriter(Socket socket) throws IOException {
131     OutputStream socketOut = socket.getOutputStream();
132     return new PrintWriter(socketOut, true);
133   }
134 
135   private BufferedReader getReader(Socket socket) throws IOException {
136     InputStream socketIn = socket.getInputStream();
137     return new BufferedReader(new InputStreamReader(socketIn));
138   }
139 
140   public static void main(String[] args) throws Exception {
141     if (args.length != 3) {
142       System.out.println("[Usage] java " + TcpPeer.class.getCanonicalName()
143               + " [serverHost] [serverPort] [localPort]");
144       System.exit(0);
145     }
146 
147     new TcpPeer(args[0], Integer.parseInt(args[1]), Integer.parseInt(args[2]));
148   }
149 }
150 

Now we launch 2 jvm processes:
ps#1> java org.clinic4j.net.TcpPeer locahost 2000 4000
ps#2> java org.clinic4j.net.TcpPeer locahost 4000 2000

Finally when 2 processes got stable, they will give below outputs:
ps#1>

[Server]The server is listening on 2000.
[Client]socket.isBound():false
[Client]connect to localhost:4000 successfully.
[Client][RESP-1]hello world!

ps#2>

[Server]The server is listening on 4000.
[Server]New connection accepted/127.0.0.1:2000
[Server][REQ]hello world!
[Client]socket.isBound():false
[Client]fail to connect localhost:2000, try again.
[Client]fail to connect localhost:2000, try again.
[Client]fail to connect localhost:2000, try again.
[Client]fail to connect localhost:2000, try again.
java.net.SocketException: Socket is not connected
at java.net.Socket.getOutputStream(Socket.java:816)
at org.clinic4j.net.TcpPeer.getWriter(TcpPeer.java:136)
at org.clinic4j.net.TcpPeer.access$6(TcpPeer.java:135)
at org.clinic4j.net.TcpPeer$2.run(TcpPeer.java:108)
at java.lang.Thread.run(Thread.java:619)

From the output, we can figure out that interaction flow as below:

1. ps#1 listen on 2000.
2. ps#2 listen on 4000.
3. ps#1 connect to ps#2, from localhost:2000 -> localhost:4000.
4. ps#2 close the connection which established at step#3. 
5. ps#2 try to connect to ps#1 at 2000, failed!

Why ps#2 cannot connect to ps#1 at step#4?  I cannot make a simple conclusion, actually i just suspect it is caused by the underlying TCP stack mechanism.

Lets check what is netstat right after step#3.

When ps#1 connects to ps#2, there are 2 established TCP connections, "192.168.2.197:2000 -> 192.168.2.107:4000" is used for ps#1 to send request to ps#2, and "192.168.2.107:4000 -> 192.168.2.107:2000" is used for ps#2 to response to ps#1.

Then what is the net status right after step#4?

* 192.168.2.107 is localhost
Now we can see TCP connection(127.0.0.1:4000->127.0.0.1:2000) is TIME_WAIT, and the other one(127.0.0.1:2000->127.0.0.1:4000) is already closed and re-collected by operation system.

My suspicion is when ps#2 try to connect to ps#1, the connection(127.0.0.1:4000->127.0.0.1:2000) cannot be established, as it is still TIME_WAIT, that is why I got failure at step#5(pls give your comments). Does it means if we keep retrying step#5 till (127.0.0.1:4000->127.0.0.1:2000) is closed, the connection will be established??(seem not such case)


Fix
As at step#4 I closed the connection, it will caused a underlying TCP connection to TIME_WAIT, in the fixed revision, I will just keep the connection up.

  1 package org.clinic4j.net;
  2 
  3 import java.io.BufferedReader;
  4 import java.io.IOException;
  5 import java.io.InputStream;
  6 import java.io.InputStreamReader;
  7 import java.io.OutputStream;
  8 import java.io.PrintWriter;
  9 import java.net.InetSocketAddress;
 10 import java.net.ServerSocket;
 11 import java.net.Socket;
 12 
 13 /**
 14  * Just for testing socket SO_RESUEADDR. If set SO_RESUEADDR to true, we can use
 15  * a single local port to listen for incoming TCP connections, and to initiate
 16  * multiple outgoing TCP connections concurrently. By this way we can implement
 17  * TCP hole punching(establish P2P connection traversal through NAT over TCP).
 18  */
 19 public class TcpPeer {
 20   // TCP port is a different source from UDP port, it means you can listen on
 21   // same port for both TCP and UDP at the same time.
 22   private int localport = 7890;
 23   private ServerSocket peerSock;
 24   private Socket serverSocket;
 25 
 26   public TcpPeer(final String serverHost, final int serverPort, final int localPort)
 27           throws Exception {
 28     this.localport = localPort;
 29 
 30     Thread server = new Thread(new Runnable() {
 31 
 32       @Override
 33       public void run() {
 34         try {
 35           peerSock = new ServerSocket();
 36           peerSock.setReuseAddress(true);
 37           peerSock.bind(new InetSocketAddress("localhost", localport));
 38           System.out.println("[Server]The server is listening on " + localport + ".");
 39 
 40           while (true) {
 41             try {
 42               serverSocket = peerSock.accept();
 43               // just means finishing handshaking, and connection
 44               // established.
 45               System.out.println("[Server]New connection accepted"
 46                       + serverSocket.getInetAddress() + ":" + serverSocket.getPort());
 47 
 48               BufferedReader br = getReader(serverSocket);
 49               PrintWriter pw = getWriter(serverSocket);
 50               String req = br.readLine();
 51               System.out.println("[Server][REQ]" + req);
 52               pw.println(req);
 53 
 54 //              pw.close();
 55 //              br.close();
 56             } catch (IOException e) {
 57               e.printStackTrace();
 58             } finally {
 59 //              try {
 60 //                if (serverSocket != null)
 61 //                  serverSocket.close();
 62 //              } catch (IOException e) {
 63 //                e.printStackTrace();
 64 //              }
 65             }
 66           }
 67         } catch (Exception e) {
 68           e.printStackTrace();
 69         }
 70       }
 71 
 72     });
 73     // server.setDaemon(true);
 74     server.start();
 75 
 76     Thread.currentThread();
 77     // sleep several seconds before launch of client
 78     Thread.sleep(5 * 1000);
 79 
 80     final int retry = 5;
 81     Thread client = new Thread(new Runnable() {
 82 
 83       @Override
 84       public void run() {
 85         Socket socket = new Socket();
 86         try {
 87           socket.setReuseAddress(true);
 88           System.out.println("[Client]socket.isBound():" + socket.isBound());
 89           socket.bind(new InetSocketAddress("localhost", localport));
 90           for (int i = 1; i < retry; i++) {
 91             try {
 92               socket.connect(new InetSocketAddress(serverHost, serverPort));
 93               System.out.println("[Client]connect to " + serverHost + ":"
 94                       + serverPort + " successfully.");
 95               break;
 96             } catch (Exception e) {
 97               System.out.println("[Client]fail to connect " + serverHost + ":"
 98                       + serverPort + ", try again.");
 99               Thread.currentThread().sleep(i * 2 * 1000);
100               /**
101                * PeerA and PeerB
102                * <p>
103                * Alternatively, A's TCP implementation might
104                * instead notice that A has an active listen socket
105                * on that port waiting for incoming connection
106                * attempts. Since B's SYN looks like an incoming
107                * connection attempt, A's TCP creates a new stream
108                * socket with which to associate the new TCP
109                * session, and hands this new socket to the
110                * application via the application's next accept()
111                * call on its listen socket. A's TCP then responds
112                * to B with a SYN-ACK as above, and TCP connection
113                * setup proceeds as usual for client/server-style
114                * connections.
115                * <p>
116                * Since A's prior outbound connect() attempt to B
117                * used a combination of source and destination
118                * endpoints that is now in use by another socket,
119                * namely the one just returned to the application
120                * via accept(), A's asynchronous connect() attempt
121                * must fail at some point, typically with an
122                * “address in use” error. The application
123                * nevertheless has the working peer-to- peer stream
124                * socket it needs to communicate with B, so it
125                * ignores this failure.
126                */
127               if (i == retry - 1) {
128                 System.out
129                         .println("[Client]Use the socket returned by ServerSocket.");
130 
131                 socket = serverSocket;
132               }
133             }
134           }
135 
136           PrintWriter pw = getWriter(socket);
137           String msg = "hello world!";
138           pw.println(msg);
139 
140           /**
141            * Got response from the server socket.
142            */
143           BufferedReader br = getReader(socket);
144           String resp = br.readLine();
145           System.out.println("[Client][RESP-1]" + resp);
146 
147           /**
148            * The client thread of other process will send request. If
149            * fail to establish connection with other peer, the Socket
150            * return by the ServerSocket will be used.
151            */
152           resp = br.readLine();
153           System.out.println("[Client][RESP-2]" + resp);
154 //          pw.close();
155 //          br.close();
156         } catch (Exception e) {
157           e.printStackTrace();
158         } finally {
159 //          try {
160 //            socket.close();
161 //          } catch (Exception e) {
162 //            e.printStackTrace();
163 //          }
164         }
165       }
166 
167     });
168     client.start();
169   }
170 
171   private PrintWriter getWriter(Socket socket) throws IOException {
172     OutputStream socketOut = socket.getOutputStream();
173     return new PrintWriter(socketOut, true);
174   }
175 
176   private BufferedReader getReader(Socket socket) throws IOException {
177     InputStream socketIn = socket.getInputStream();
178     return new BufferedReader(new InputStreamReader(socketIn));
179   }
180 
181   public static void main(String[] args) throws Exception {
182     if (args.length != 3) {
183       System.out.println("[Usage] java " + TcpPeer.class.getCanonicalName()
184               + " [serverHost] [serverPort] [localPort]");
185       System.exit(0);
186     }
187 
188     new TcpPeer(args[0], Integer.parseInt(args[1]), Integer.parseInt(args[2]));
189   }
190 }
191 

You can see, I have commented those codes which will close connection, and also reuse the Socket returned by ServerSocket if cannot connect with the other peer. And at this time, it works as my expectation.
ps#1>

[Server]The server is listening on 2000.
[Client]socket.isBound():false
[Client]connect to localhost:4000 successfully.
[Client][RESP-1]hello world!
[Client][RESP-2]hello world!

ps#2>

[Server]The server is listening on 4000.
[Server]New connection accepted/127.0.0.1:2000
[Server][REQ]hello world!
[Client]socket.isBound():false
[Client]fail to connect localhost:2000, try again.
[Client]fail to connect localhost:2000, try again.
[Client]fail to connect localhost:2000, try again.
[Client]fail to connect localhost:2000, try again.
[Client]Use the socket returned by ServerSocket.

Please do keep in mind that the socket must be eventually closed by some way.