Revisiting Link-Layer Storage Networking

This paper revisits the historical concept of link-layer, block-oriented storage networking in light of commodity-based, high-bandwidth local area networks. It examines how advances in data communication technology such as virtual LANs and switched gigabit Ethernet make it relatively simple and inexpensive to re-centralize the storage resources used by clusters of workstations and servers. In particular, the work is motivated by an interest in how to support scalable and efficient access to read-only and private read-write data such as root file systems, swap partitions, log files and static web pages. These techniques complement, but do not replace, higher level distributed file systems whose primary goal is to provide coherent access to shared read/write data. This paper describes the design and implementation of a very simple, link-layer protocol, the Ethernet Block Device (EBD), for accessing remote block devices. It compares the EBD prototype to a locally attached disk and to similar, network-based techniques that use TCP/IP such as the Linux Network Block Device (NBD), as well as higher level distributed file systems such as NFS. Functionally, the implementation is compared with a local disk to determine what restrictions and visible differences exist. The performance evaluation is based on a series of standard disk and file access benchmarks run on commodity servers and standard networking equipment. The performance results show that for large sequential reads and random reads and writes EBD generally outperforms comparative network storage technologies by 15% to 30%, and performance is best when the data set fits into the server’s memory. On a benchmark that is very metadata-intensive and does small sequential operations, EBD and NBD perform similarly. On certain tests, EBD shows a maximum request latency is that is about four (4) times that of NFS, but an average latency that is approximately the same. This suggests that work toward eliminating the occasional very slow response to EBD requests should yield a significant improvement in overall performance.