Towards the Open Advancement of Question Answering Systems

On February 27-28, 2008, a group of researchers from industry and academia met to discuss the state of the Question Answering (QA) field. The discussion focused on recent experiences from funded research programs (e.g. AQUAINT, HALO) and open evaluations (e.g. TREC, NTCIR). The group acknowledged that funded research programs and evaluations have been instrumental in establishing fundamental QA research. However, major advances in the field of QA are yet to be realized. Advances that can openly accelerate progress and greatly generalize QA technologies to produce scalable, more adaptable methodologies and business applications are within our reach. Although there are deep technical challenges, we believe these challenges can be effectively addressed by open collaboration in the open-source development of integrated QA technologies aimed at well-chosen sets of QA applications. This document captures the thoughts and proposals from the workshop regarding open advancement: the use of shared system models, open-source components, collections of challenge problems and common evaluation metrics, so that the contribution of each technology to end to end performance can be accurately measured and the community as a whole can uniformly advance system performance on an ever broadening range of QA problems.