Starting from the Naïve Bayes model, we develop a new concept for aggregating items of evidence in classification problems. We show that in Naïve Bayes, each feature variable contributes a multiplicative adjustment factor to the estimated class probability. We next introduce a way of controlling the importance of the feature variables by raising each adjustment factor to a different power. The powers are chosen so as to maximize the accuracy of estimated class probabilities on the training data, and their optimal values are obtained by fitting a logistic regression model whose explanatory variables are constructed from the feature variables of the classification problem. This optimization accomplishes more than what feature selection does for Naïve Bayes. We call this new model family the Adjusted Probability Model (APM). We also define a regularized version, APMR. Experiments demonstrate that APMR is surprisingly effective. Assigning different degrees of importance to the feature variables seems to remove much of the naïveté from Naïve Bayes.
By: Se June Hong, Jonathan Hosking, Ramesh Natarajan
Published in: RC22393 in 2002
LIMITED DISTRIBUTION NOTICE:
This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.
Questions about this service can be mailed to reports@us.ibm.com .