Tools and Methods for Building Watson

The DeepQA team built the Watson QA system for Jeopardy! in under four years by adopting a metrics-driven research and development methodology. This methodology relies heavily on disciplined integration of new and improved components, extensive experimentation at the component and end-to-end system level, and informative error analysis. To support this methodology, we adopted a formal protocol for integrating components into the overall system and running end-to-end integration tests, assembled a powerful computing environment for running a large volume of high-throughput experiments, and built several tools for deploying experiments and evaluating results. We describe our software development and integration protocol, the DeepQA computing environment for development, and the tools we built and used to create Watson for Jeopardy!. We also briefly allude to some of the more recent enhancements to these tools and methods as we extend Watson to operate in commercial applications.