Pattern Classification using the Principle of Parsimony: A Least square Kernel Machine with Box Constraints

Among various principles in the theory of pattern classification to improve generalization, one of the most widely used principle is Occam's razor or the principle of parsimony. Structural risk minimization (SRM) and minimum description length (MDL) principle and their variants are essentially two different forms of Occam's razor used in pattern classification. In this article, we present a modified view of Occam's razor and use this principle to design a kernel-based classifier. The proposed classifier has close relationships with a widely different variety of “least-square” kernel-machines such as adaptive ridge regression, least square support vector machine, regularized least square classifier, LASSO (least absolute shrinkage and selection operator), and generalized LASSO. However, unlike the existing ``least-square'' kernel machines, the proposed classifier uses box constraints on the priors, and the box constraints are derived from the modified principle of parsimony. Experimental results demonstrate that the proposed classifier is capable of outperforming SVM in terms of cross-validation scores on several datasets. In addition, we also prescribe some kernel functions other than the Gaussian for obtaining better performance on real-life datasets.

By: Jayanta Basak

Published in: RI07011 in 2008


This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.


Questions about this service can be mailed to .