How to emulate network timeout?

Before march into how to emulate network timeout, we have to understand the details of network timeout first.

Socket Timeout

We already knew that there are 3 times of handshake before establish a tcp connection.
- client send 'SYN'.(client: SYN_SENT)
- server response 'SYN'+'ACK'(client: SYN_SENT, server:SYN_SENT)
- client response 'ACK'(client:ESTABLISHED, server(got ACK):ESTABLISHED)

Client Connects to a Nonexistent Port

In this case, host's IP is valid, however the tcp port is nonexistent, and in general server will response 'connection refused'.

Client Connects to a Nonexistent IP

In this case, client will get 'host unreachable'.

Packet Lost During Establishing Connection

As there are 3 times of handshake, each packet maybe lost in network.

SYN of client lost

As no 'ACK' of 'SYN', client will keep resending 'SYN'. If the process of resending fail finally, connection timeout occurs.

SYN+ACK of server lost

As no 'ACK' of server's 'SYN', server will keep resending 'SYN'. If the process of resending fail finally, connection timeout occurs at server side. Also client can't get 'ACK', it behaves like last case.

ACK of client lost

In this case, client will regard that connection has been established successfully, however the state of server is still SYN_SENT, so if client try to send data, it won't reach the server, and as no ACK of sending, client will retransmit, and finally get connection timeout.

Refer to 深入理解socket网络异常

Emulate Socket Timeout

On *nix OS, we can achieve this by command 'iptables'. A full iptables reference can be found here and here

By means of 'iptables', we will emulate 2 kinds of timeout. Let's IGPE is deployed on a remote server, and IGPE port is 9090.

Emulate Connection Timeout

Connection timeout means no connection established at all, in general thay says the handshakes of TCP fails. The client can be very sure that no any request data reaches the backend if connection timeout.

Use below command to drop the response SYN packet(flag SYN set).

iptables -A OUTPUT -p tcp -m tcp --tcp-flags SYN SYN --sport 9090 -j DROP

As any pakcet lost during handshake will trigger connection timeout, we can approach this by many means.

The arguments of --tcp-flags is a little confused, i am not completedly understood, you can refer to the manual page. In above rule, I am trying to drop all outgoing packets whose 'SYN' flag has been set, and source tcp port is 9090.

Use below command to view your rules:

iptables -L -v

And then remember to use below command to clear all your rules:

iptables -F

Emulate Read Timeout

Read timeout means the connection has established successfully, however the client fail to get response(wail to timeout). In this case, the client can't be sure whether the request data has reached the backend or not(or say whether the request has been handled by the backend or not).

Use below command to drop the data response packet(flag PSH set).

iptables -A OUTPUT -p tcp -m tcp --tcp-flags PSH PSH --sport 9090 -j DROP

- PSH flag means there is data transfer in packet, not only a control packet.(oh NO, PSH aims to inform TCP buffer to send data immediately, no need to wait buffer is fulll, refer to http://packetlife.net/blog/2011/mar/2/tcp-flags-psh-and-urg/)
- I only wanna drop the response data packet, and emulate read timeout by this way.

The Retionale

In a client/server deployment, the server port will be fixed however the client tcp port is randomly picked. To a INPUT chain(the client request), the source port is client tcp port, however to a OUTPUT chain(server response), the source port is server tcp port. This must be clarified.

In my first try, I try to emulate connection timeout by below rule:

iptables -A INPUT -p tcp -m tcp --dport 9090 -j DROP

It won't result in a connction timeout, as all incoming packets will be dropped, that says no 'ACK' packet of request 'SYN' will be returned, in this case client will regard that the host is unreachable(not connection timeout). The key of emulating connection timeout is let client get the 'ACK' pakcet, however no 'SYN' packet from server, in this case, client will keep issuing 'SYN' packets and at final regard it as timeout.

If try to emulate read timeout, we must affect the OUTPUT chain.

Also seem there are build-in tool in linux can emulate these 2 situations, refer to netem

Referenced Documents

• iptable examples - http://www.cyberciti.biz/tips/linux-iptables-examples.html
• iptable manual - http://www.cyberciti.biz/tips/linux-iptables-examples.html
• understanding socket exception - http://blog.csdn.net/wy5761/article/details/17232495
• netem - http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
• Simulate delayed and dropped packets on Linux - http://stackoverflow.com/questions/614795/simulate-delayed-and-dropped-packets-on-linux