We describe a system for rapidly determining document similarity among a set of documents obtained from an information retrieval (IR) system. We obtain a ranked list of the most important terms in each document using a rapid phrase recognizer system. We store these in a database and compute document similarity using a simple database query. If the number of terms found to not be contained in both documents is less than some predetermined threshold compared to the total number of terms in the document, these documents are determined to be very similar.
By: James W. Cooper, Anni R. Coden, Eric W. Brown
Published in: Proceedings of the 35th Annual Hawaii International Conference on System Sciences. Los Alamitos, CA, , IEEE Computer Society Press, p.1153-9 in 2002
Please obtain a copy of this paper from your local library. IBM cannot distribute this paper externally.
Questions about this service can be mailed to reports@us.ibm.com .