Profiling TCP: An In-depth Analysis of Processing Costs

There are currently several proposals for large, disruptive changes to the Linux TCP implementation to improve network performance and scalability. These proposals, including relocating part or all of the TCP stack, header splitting, and dedicated network caches, require potentially sizeable changes to the Linux TCP implementation. Before considering these modifications, we believe the behavior of the Linux TCP stack should be revisited to be better understood, studied, and optimized.

We provide an in-depth analysis of CPU profiles for both the Linux 2.4 and the 2.6 TCP stacks. For Linux 2.4.16, we vary several aspects of networking and profile the TCP stack for a detailed look at how the code path changes under load. We show that after years of development and enhancements the limitations of high performance networking remain the same even across different kernel versions and hardware.

For the Linux 2.6.14 TCP stack, CPU profiling is performed using callgraph support from OProfile. The callgraphs and newly developed callgraph examination tools are used to determine where high level TCP functions, such as TCP receive and TCP send message, are spending CPU cycles. These results are then examined to determine bottlenecks and potential areas for improvement with respect to CPU usage. Under load, we show why TCP receive dominates costs even for large files.

For each kernel, we propose several possible nondisruptive enhancements. Furthermore, we present correlations and trends from the 2.4 to the 2.6 TCP stack and trends over generations of machine hardware. Our results validate and expand upon existing TCP profiling work.

By: Jason LaVoie, Erich Nahum, Robert Flynn

Published in: RC24245 in 2007


This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.


Questions about this service can be mailed to .