Storage over IP: A Performance Study

In the past, the storage model assumed the presence of storage attached to every host server. This type of host server-attached storage relied primarily on the Small Computer System Interface (SCSI) command protocol. The preferred transport for the SCSI command protocol in this server-attached storage model was Parallel SCSI where the storage devices were connected to the host server via a cable-based parallel bus. However, as the need for storage and servers grew, the limitations of this technology became obvious. The physical characteristics of the cable limit the number of storage devices as well as the distance of the storage devices from the host server. Also, the concept of attaching storage to every host server means that the storage had to be managed on a per-host server basis.

The lack of scalability and manageability of the host server-attached storage model led to the evolution of the concept of a storage area network. Storage devices are assumed to be independent machines that provide storage service via a network to a multitude of host servers. The advent of networking infrastructure capable of gigabit speeds as well as the development of transport protocols capable of sustaining such speeds further facilitates the service of storage over large-distance networks. Most storage area networks use Fibre Channel [3]; other storage area network technologies are Infiniband [6], VaxClusters [19], HIPPI [27] and IP [4,15,16].

This paper focuses on storage area networks based on the pervasive IP networking technology. The advantages of IP networks are obvious. First, using the same IP technology for both regular (non-storage) networks as well as storage networks removes the need to have two different types of networks in any infrastructure. Also, the use of a single popular networking infrastructure can leverage widely available network management skills. Second, the presence of well tested and established protocols allow IP networks both wide-area connectivity, scalable routing and addressing as well as proven bandwidth sharing capabilities. Third, the emergence of Gigabit Ethernet and the future arrival of 10 Gigabit Ethernet seems to indicate that the bandwidth requirements of serving storage over a network should not be an issue [1]. Finally, the commodity availability of IP networking infrastructure indicates the cost of building a storage area network will not be prohibitive.

The key contribution of this paper is to compare these three approaches for IP storage area networks with the help of both micro-benchmarks as well as macro-benchmarks. In the microbenchmarks, the three approaches were compared with respect to latency and throughput by measuring their sensitivity to varying block sizes as well as CPU, I/O bus and memory speeds. The micro-benchmark analysis was projected onto the real world by running macro-benchmarks on each of the three approaches.

The results show that contrary to intuition, the hardware approaches are not inherently superior in terms of performance, which is surprising because of the cost of hardware offload. The results indicate that while the hardware support decreases the CPU utilization-to-throughput ratio for large block sizes, the hardware support can itself be a performance bottleneck that hurts the rate of I/O operations. Consequently, not only is the software approach superior to the hardware approaches in terms of latency and throughput, but also outperforms the hardware approaches in
macro-benchmarks. This projects for the need for an intelligent partitioning of storage transport functionality between hardware and software that can take advantage of the rate of increase in the computing power of general-purpose processors.