Configuration is the process whereby components are assembled or adjusted to produce a functional system that operates at a specified level of performance. Today, the complexity of configuration is a major impediment to deploying and managing computer systems. We describe an approach to quantifying configuration complexity, with the ultimate goal of producing a configuration complexity benchmark. Our belief is that such a benchmark can drive progress towards self-configuring systems. Unlike traditional workload-based performance benchmarks, our approach is process-based. It generates metrics that reflect the level of human involvement in the configuration process, quantified by interaction time and probability of successful configuration. It computes the metrics using a model of a standardized human operator, calibrated in advance by a user study that measures operator behavior on a set of parameterized canonical configuration actions. The model captures the human component of configuration complexity at low cost and provides representativeness and reproducibility.
By: Aaron B. Brown, Joseph L. Hellerstein
Published in: RC23146 in 2004
LIMITED DISTRIBUTION NOTICE:
This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.
Questions about this service can be mailed to reports@us.ibm.com .