Evaluating Knowledge Base Relevancy

Whenever possible, it is preferable to reuse knowledge base content rather than redevelop it. The problem is how to determine the relevancy of an existing knowledge base to a new problem domain, or to determine which existing knowledge base is most relevant. In this paper, we describe an implementation of an algorithm and methodology for comparing the relevancy of different knowledge bases to a set of problem tickets. Using distance metrics that have been applied in the field of Text Clustering, we systematically compare the text of the problem domain to the text of each knowledge base. We then allow an expert in the domain area to check the validity of the problem/solution matches and determine where to draw the line between the sets of solved and unsolved problem tickets. This approach makes it possible to objectively compare thousands of problems against thousands of solutions in a matter of hours, to determine an approximate percent coverage of the solutions when applied to the problem domain. We have implemented our approach, and we present the results of performing knowledge base relevancy evaluation on a computer helpdesk problem ticket set compared to different knowledge base solution sets.