Benchmarking for Power and Performance

There has been a tremendous increase in focus on power consumption and cooling of computer systems from both the design and management perspectives. Managing power has significant implications for system performance, and has drawn the attention of the computer architecture and systems research communities. Researchers rely on benchmarks to develop models of system behavior and experimentally evaluate new ideas. But benchmarking for combined power and performance analysis has unique features distinct from traditional performance benchmarking.

In this extended abstract, we present our experiences with adapting performance benchmarks for use in power/performance research. We focus on two areas: the problem of variability and its effect on system power management and that of collecting correlated power and performance data. For the first, we have learned that benchmarks need to expose at least three sources of power/performance variability – workload intensity, workload behavior and component-level differences. Benchmarks not only have to test across all of these forms of variability, but they also must capture the dynamic nature of real workloads and real systems. The workload and the system’s response to it change across time, and how fast and well the system responds to change is an important consideration in evaluating its power management capabilities. In the second area, we have developed tools for collecting correlated power and performance data, and we briefly discuss our experience with them.