Measurements of Failure Rate in Commercial Software

In the history of empirical failure rate measurement, one problem that continues to plague researchers and practitioners is that of measuring the customer perceived failure rate of commercial software. There is extensive experience and data on the measurement of hardware failure rates, but reports on software failure rate are scarce. The few that exist are from some very specialized and unique applications, or indirect measures using hardware logs or models based on faults. Unfortunately, even order of magnitude measures of failure rate for commercial software which is widely distributed, is not truly available. Given repeated reports on the criticality of software, and its significance, the industry flounders for some real baselines. This paper reports the failure rate of a several million line of code commercial software product distributed to hundreds of thousands of customers. To first order of approximation, the MTBF plateaus at around 4 years and 2 years for successive releases of the software. The changes in the failure rate as a function of severity, release and time are also provided.