Pattern Classification using the Principle of Parsimony: A Least square Kernel Machine with Box Constraints

Among various principles in the theory of pattern classification to improve generalization, one of the most widely used principle is Occam's razor or the principle of parsimony. Structural risk minimization (SRM) and minimum description length (MDL) principle and their variants are essentially two different forms of Occam's razor used in pattern classification. In this article, we present a modified view of Occam's razor and use this principle to design a kernel-based classifier. The proposed classifier has close relationships with a widely different variety of “least-square” kernel-machines such as adaptive ridge regression, least square support vector machine, regularized least square classifier, LASSO (least absolute shrinkage and selection operator), and generalized LASSO. However, unlike the existing ``least-square'' kernel machines, the proposed classifier uses box constraints on the priors, and the box constraints are derived from the modified principle of parsimony. Experimental results demonstrate that the proposed classifier is capable of outperforming SVM in terms of cross-validation scores on several datasets. In addition, we also prescribe some kernel functions other than the Gaussian for obtaining better performance on real-life datasets.