Accedian is now part of Cisco  |

Avatar photo
By Guillaume Savary

TCP Series #2: How to close TCP sessions and diagnose disconnections?

Things that can go wrong when you close TCP sessions

This is the second in a series of articles covering everything that you need to know to troubleshoot performance issues impacting applications that rely on the TCP protocol. After studying how TCP sessions are established in our first article, we will now see what can go wrong when you close TCP sessions.

Paradoxically, is it more complex to close a TCP connection than it is to create one! This is due to the fact that resources must be correctly released on both sides of the connection, host A and host B.

Figure 1 – Simplified TCP closing with FIN.
Figure 1 – Simplified TCP closing with FIN.

The standard way to close TCP sessions is to send a FIN packet, then wait for a FIN response from the other party.

  1. A sends a FIN packet and waits for a response; it can release some resources but awaits the response of the other part (Fin Wait)
  2. receives the FIN packet and must release resources; it waits for a closing application level (Close Wait)
  3. B can now send a FIN to and then await its acknowledgement (Last Ack wait).
  4. A can now fully close its job, but it must wait for network collision (?) (Time Wait); it may have to send the final ACK another time.
  5. B eventually receives the final ACK and destroys (kills) the connection.

This works fine in a perfect world. However, what happens when one part of the conversation is broken? That’s why the Reset (RST) packet exists.

Figure 2 – RST sent to force the end of a TCP session.
Figure 2 – RST sent to force the end of a TCP session.

Those abnormal terminations (i.e., either an aborted setup or a disconnection) could appear due to:

  • A lack of resources or network interruption
  • A crash/bug during the session
  • While one pair has already closed its part of the connexion, the other part continues to send data
  • The server refuses to open a connection to the client

Skylight™ provides metrics to see if you close TCP connections properly or not. By selecting the theme “TCP Events“, you get the count of FIN and RST packets in both directions.

Figure 3 – SkyLIGHT PVX showing client IPs sorted by server RST.
Figure 3 – Skylight showing client IPs sorted by server RST.

If the SYN rate per connection and the server RST are both high (Figure 3 – 1st row), this means that the server is refusing the client connection demands. With a drill down to the conversations (see Figure 4), you will have the precise server ports and applications that cause this issue. Hereunder, we see some attempts to connect to a VPN from an IP address using port “1194”.

Figure 4 – SkyLIGHT PVX showing the cause of RST without any connections.
Figure 4 – Skylight showing the cause of RST without any connections.

For Advanced Users of Skylight: If you want to filter on data with or without RST or FIN packets, Skylight provides some custom filters. In the previous article, we already saw how to filter on connections.

  • FIN: fin.count, fin.count.srv, fin.count.clt
  • RST: rst.count, rst.count.srv, rst.count.clt

Examples:

  • Flows with only RST and no FIN:
    rst.count > 0 and fin.count = 0
  • Flows with RST but without connections:
    rst.count > 0 and ct.count = 0

Sometimes, RST packets are quite “normal”. For example when the user manually interrupts a huge data transfer. The TCP session is sending packets as fast as possible, so when the client sends the FIN and closes its part, the server is still sending lots of data for a moment. In this case, the client sends RST packets until the server stops sending data. In this case, the is as client FIN (than server FIN), but in addition, you will see some RST packets.

It is also important to note that some applications do not close sessions properly and simply use an RST to close every session. While this is not a good practice, you must be aware that some applications are developed this way.

It could also be relevant to graph some RST or FIN metrics over time. Skylight provides a metric to graph the rate of RST per connexion over time using Custom Graphs (Figure 5).

Figure 5 – SkyLIGHT PVX custom chart showing TCP RST over time in both directions (from client and server).
Figure 5 – Skylight custom chart showing TCP RST over time in both directions (from client and server).

CONCLUSION

We have seen how closing a TCP connection can be more complex than opening one. The session can be closed by a double FIN, by a mix of FIN + RST, or only by RST packets. However, RST packets can also be sent without any connection.

Skylighthelps to diagnose session issues by reporting statistics about FINRSTSYN, and connections. It is also able to graph all the metrics over time, especially the RST per connection.

As we saw in the first article, we can use pages like Top Client Zone and Top Servers toanalyzee where the issue is from.

In a next article, we will have a look at how TCP retransmission works, and when it can be an issue or not.

In the meantime; if you would like to give it a try, just download our evaluation virtual appliance or download our troubleshooting guide

Related Posts