Profiling TCP: An In-depth Analysis of Processing Costs

There are currently several proposals for large, disruptive changes to the Linux TCP implementation to improve network performance and scalability. These proposals, including relocating part or all of the TCP stack, header splitting, and dedicated network caches, require potentially sizeable changes to the Linux TCP implementation. Before considering these modifications, we believe the behavior of the Linux TCP stack should be revisited to be better understood, studied, and optimized.

We provide an in-depth analysis of CPU profiles for both the Linux 2.4 and the 2.6 TCP stacks. For Linux 2.4.16, we vary several aspects of networking and profile the TCP stack for a detailed look at how the code path changes under load. We show that after years of development and enhancements the limitations of high performance networking remain the same even across different kernel versions and hardware.

For the Linux 2.6.14 TCP stack, CPU profiling is performed using callgraph support from OProfile. The callgraphs and newly developed callgraph examination tools are used to determine where high level TCP functions, such as TCP receive and TCP send message, are spending CPU cycles. These results are then examined to determine bottlenecks and potential areas for improvement with respect to CPU usage. Under load, we show why TCP receive dominates costs even for large files.

For each kernel, we propose several possible nondisruptive enhancements. Furthermore, we present correlations and trends from the 2.4 to the 2.6 TCP stack and trends over generations of machine hardware. Our results validate and expand upon existing TCP profiling work.