MDL Estimation for Small Sample Sizes and Its Application to Linear Regression

In this paper we examine the problem of performing Minimum Description Length (MDL) based estimation for small sample sizes. By this we mean cases where asymptotic forms are not appropriate. The path to our solution is embedded in a careful examination of the issue of completeness in MDL-based estimation. We develop specific MDL-based code-length formulas for use as objective functions in linear regression (fitting functions linear in the adjustable parameters). The problem we address is regression where we not only want to find the best fit for a given model order/complexity (e.g. polynomial degree), but also the best order. The accuracy of this fitting process will be assessed by how well the fitted function matches the ``true'' underlying function in simulation experiments. We succeed in obtaining several non-asymptotic expressions (code-length formulas) for the problem we consider and these results are compared with each other and with the ``classic'' asymptotic (k/2) log n MDL formula in fairly extensive simulation curve-fitting experiments. All the non-asymptotic formulas do better than (k/2) log n. Index Terms: Minimum Description Length principle, Stochastic Complexity, order estimation, model selection, statistical estimation, linear regression.