Apache HTTP Server Version 1.3

Connections in the FIN_WAIT_2 state and Apache

1. What is the FIN_WAIT_2 state?

   Starting with the Apache 1.2 betas, people are reporting many more connections in the FIN_WAIT_2 state (as reported by netstat) than they saw using older versions. When the server closes a TCP connection, it sends a packet with the FIN bit sent to the client, which then responds with a packet with the ACK bit set. The client then sends a packet with the FIN bit set to the server, which responds with an ACK and the connection is closed. The state that the connection is in during the period between when the server gets the ACK from the client and the server gets the FIN from the client is known as FIN_WAIT_2. See the TCP RFC for the technical details of the state transitions.

   The FIN_WAIT_2 state is somewhat unusual in that there is no timeout defined in the standard for it. This means that on many operating systems, a connection in the FIN_WAIT_2 state will stay around until the system is rebooted. If the system does not have a timeout and too many FIN_WAIT_2 connections build up, it can fill up the space allocated for storing information about the connections and crash the kernel. The connections in FIN_WAIT_2 do not tie up an httpd process.

2. But why does it happen?

   There are numerous reasons for it happening, some of them may not yet be fully clear. What is known follows.

   Buggy clients and persistent connections

   Several clients have a bug which pops up when dealing with persistent connections (aka keepalives). When the connection is idle and the server closes the connection (based on the KeepAliveTimeout), the client is programmed so that the client does not send back a FIN and ACK to the server. This means that the connection stays in the FIN_WAIT_2 state until one of the following happens:
   • The client opens a new connection to the same or a different site, which causes it to fully close the older connection on that socket.
   • The user exits the client, which on some (most?) clients causes the OS to fully shutdown the connection.
   • The FIN_WAIT_2 times out, on servers that have a timeout for this state.

   If you are lucky, this means that the buggy client will fully close the connection and release the resources on your server. However, there are some cases where the socket is never fully closed, such as a dialup client disconnecting from their provider before closing the client. In addition, a client might sit idle for days without making another connection, and thus may hold its end of the socket open for days even though it has no further use for it. This is a bug in the browser or in its operating system's TCP implementation.

   The clients on which this problem has been verified to exist:

   • Mozilla/3.01 (X11; I; FreeBSD 2.1.5-RELEASE i386)
   • Mozilla/2.02 (X11; I; FreeBSD 2.1.5-RELEASE i386)
   • Mozilla/3.01Gold (X11; I; SunOS 5.5 sun4m)
   • MSIE 3.01 on the Macintosh
   • MSIE 3.01 on Windows 95

   This does not appear to be a problem on:

   • Mozilla/3.01 (Win95; I)

   It is expected that many other clients have the same problem. What a client should do is periodically check its open socket(s) to see if they have been closed by the server, and close their side of the connection if the server has closed. This check need only occur once every few seconds, and may even be detected by a OS signal on some systems (e.g., Win95 and NT clients have this capability, but they seem to be ignoring it).

   Apache cannot avoid these FIN_WAIT_2 states unless it disables persistent connections for the buggy clients, just like we recommend doing for Navigator 2.x clients due to other bugs. However, non-persistent connections increase the total number of connections needed per client and slow retrieval of an image-laden web page. Since non-persistent connections have their own resource consumptions and a short waiting period after each closure, a busy server may need persistence in order to best serve its clients.

   As far as we know, the client-caused FIN_WAIT_2 problem is present for all servers that support persistent connections, including Apache 1.1.x and 1.2.

   A necessary bit of code introduced in 1.2

   While the above bug is a problem, it is not the whole problem. Some users have observed no FIN_WAIT_2 problems with Apache 1.1.x, but with 1.2b enough connections build up in the FIN_WAIT_2 state to crash their server. The most likely source for additional FIN_WAIT_2 states is a function called lingering_close() which was added between 1.1 and 1.2. This function is necessary for the proper handling of persistent connections and any request which includes content in the message body (e.g., PUTs and POSTs). What it does is read any data sent by the client for a certain time after the server closes the connection. The exact reasons for doing this are somewhat complicated, but involve what happens if the client is making a request at the same time the server sends a response and closes the connection. Without lingering, the client might be forced to reset its TCP input buffer before it has a chance to read the server's response, and thus understand why the connection has closed. See the appendix for more details.

   The code in lingering_close() appears to cause problems for a number of factors, including the change in traffic patterns that it causes. The code has been thoroughly reviewed and we are not aware of any bugs in it. It is possible that there is some problem in the BSD TCP stack, aside from the lack of a timeout for the FIN_WAIT_2 state, exposed by the lingering_close code that causes the observed problems.

   What can I do about it?

   There are several possible workarounds to the problem, some of which work better than others.

   Add a timeout for FIN_WAIT_2

   The obvious workaround is to simply have a timeout for the FIN_WAIT_2 state. This is not specified by the RFC, and could be claimed to be a violation of the RFC, but it is widely recognized as being necessary. The following systems are known to have a timeout:
   • FreeBSD versions starting at 2.0 or possibly earlier.
   • NetBSD version 1.2(?)
   • OpenBSD all versions(?)
   • BSD/OS 2.1, with the K210-027 patch installed.
   • Solaris as of around version 2.2. The timeout can be tuned by using ndd to modify tcp_fin_wait_2_flush_interval, but the default should be appropriate for most servers and improper tuning can have negative impacts.
   • Linux 2.0.x and earlier(?)
   • HP-UX 10.x defaults to terminating connections in the FIN_WAIT_2 state after the normal keepalive timeouts. This does not refer to the persistent connection or HTTP keepalive timeouts, but the SO_LINGER socket option which is enabled by Apache. This parameter can be adjusted by using nettune to modify parameters such as tcp_keepstart and tcp_keepstop. In later revisions, there is an explicit timer for connections in FIN_WAIT_2 that can be modified; contact HP support for details.
   • SGI IRIX can