We explore how to improve the performance of our Question Answering system by using redundancy and reinforcement. We have deployed in our system a variety of agents, each of which is tuned to a different class of question types, but with considerable overlap. One source of redundancy and reinforcement is from the multiple agents: many questions give rise to two or more sets of candidate answers, which can be merged to provide better performance than any single agent. We note relative improvement of up to 16.3% using the Mean Reciprocal Rank metric, and 11.9% using the Confidence Weight Score metric. We also investigate new approaches we call QA-by-Dossier and QA-by-Dossier-with-Constraints, in which additional questions are asked to generate constraints on the answers to the original question. This can reduce the confidence of many wrong answers and reinforce many good ones, resulting in one experiment in an increase in precision from .43 to .95.
By: John Prager, Jennifer Chu-Carroll, Krzysztof Czuba
Published in: RC22936 in 2003
LIMITED DISTRIBUTION NOTICE:
This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.
Questions about this service can be mailed to reports@us.ibm.com .