Creating a Corpus of Clinical Notes Manually Tagged for Part-of-Speech Information

This paper presents a pilot project aimed at generating a corpus of linguistically annotated clinical data to be used for training and testing of NLP and other text processing applications and techniques in the medical domain. We describe and discuss the process of training three medical index experts to perform linguistic annotation. We list some of the challenges as well as encouraging results pertaining to inter-rater agreement and consistency of annotation for part-of-speech information and lay out future directions for this initiative. We also present preliminary experimental results indicating the necessity for adapting state-of-the-art POS taggers to the sublanguage domain of clinical notes.

By: Serguei V. Pakhomov, Anni Coden, Christopher G. Chute

Published in: RC23178 in 2004

LIMITED DISTRIBUTION NOTICE:

This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.

rc23178.pdf

Questions about this service can be mailed to reports@us.ibm.com .