Solaris: Handling idle socket connections

(based on the writings of Gregory Bedigian, Joonbo Yoon and Neil Putnam, Sun Microsystems)

It is not uncommon to see lots of idle TCP connections (run netstat -a on Solaris to check the state of TCP connections on Solaris) in real world production systems. Here is a little background on the IDLE state for TCP socket connections:

TCP endpoints are in IDLE state when first created. After the socket creation (listening socket or the client socket), a call is normally made to the bind() system call in order to make the server port to listen or connect() call from client to communicate with the server.

In the instance where bind() is not used in the server program; or the equivalent connect() in the client program, the state of the TCP connection will be IDLE. Check the tcp.h for relevant comments.

% grep IDLE /usr/include/inet/tcp.h
#define TCPS_IDLE               -5      /* idle (opened, but not bound) */

Typically on Solaris, IDLE TCP connections are maintained indefinitely once created, even if no communication occurs between host systems. Note that keeping the connection open may consume host and/or application resources. However there are no TCP/IP tunable parameters to tweak idle, unbound socket connections.

Other possible scenarios where we may see lots of IDLE connections

• The client has aborted the connection without notifying the server.
  • eg., Client crash, client OS crash.. etc. In such cases, the server will be waiting on a non-existing client.
  
  
• The active communication between the client and the server has been interrupted. In this case the client and the server are waiting on each other.
  • eg., Network outage or a broken network path between the two hosts.

In any case, server treats them as valid connections; and the resources allocated to those connections will be kept intact. Next few paragraphs show how to clean up such idle connections.

TCP keepalive option on Solaris

TCP keepalive is a feature provided by many TCP implementations, including Solaris, as a way to clean up idle connections in situations like the ones mentioned above. Applications must enable this feature with the SO_KEEPALIVE socket option via the setsockopt(3SOCKET) socket call. Once enabled, a keepalive probe packet is sent to the other end of the socket provided the connection has remained in the ESTABLISHED state and has been idle for the specified time frame. This time frame is the value specified by the TCP tunable tcp_keepalive_interval.

A keepalive probe packet is handled just like any other TCP packet which requires an acknowledgment (ACK) from the other end of the socket connection. It will be retransmitted per the standard retransmission backoff algorithm. If no response is received by the time specified for the other TCP tunable, tcp_ip_abort_interval, the connection is terminated, as would be the case for any other unacknowledged packet. Hence the actual maximum idle time of a connection utilizing TCP keepalive, which has no responding peer will therefore be:

           tcp_keepalive_interval + tcp_ip_abort_interval

.
To set the relevant TCP tunables on Solaris, run:

/usr/sbin/ndd -set /dev/tcp tcp_keepalive_interval <value in milliseconds>
/usr/sbin/ndd -set /dev/tcp tcp_ip_abort_interval <value in milliseconds>

The above parameters are global and will affect the entire system. Keep in mind that TCP keepalive probes have no effect on inactive connections as long as the remote host is still responding to probes. However care should be taken to ensure the above parameters remain at a high enough value to avoid unnecessary traffic and other issues such as prematurely closing active connections in situations where a few packets have gone missing.
____________________
Technorati tags:
Solaris | OpenSolaris | TCP | Networking | Troubleshooting