September 2010 brings the JAMIA architecture paper for cTAKES (clinical Text Analysis and Knowledge Extraction System), an open-source system that extracts information from the free text of medical records, developed at Mayo Clinic on top of the Apache UIMA framework and the OpenNLP toolkit (Savova et al., JAMIA 17(5):507-513). The paper measures each component separately on an annotated corpus of Mayo notes, and is the first peer-reviewed write-up of a general clinical NLP pipeline distributed as reusable software.

Context

The reference tool for extracting concepts from biomedical text is MetaMap, from the National Library of Medicine, tuned on the literature indexed in MEDLINE. Clinical text — ward notes, discharge letters, reports — has quite different properties: telegraphic phrasing, dosage abbreviations, numeric values (120/80 mmHg, 500 mg BID), negation on every line. A finding written as “no evidence of pneumonia” contains the string pneumonia while asserting the absence of the condition. A system that maps strings to concepts without modelling negation and status ends up reversing the meaning of the note.

cTAKES started inside Mayo around 2006 to grind through large volumes of notes from the enterprise record. The 2010 paper fixes its architecture and measures its components.

Architecture

cTAKES is a pipeline of Apache UIMA annotators. UIMA (Unstructured Information Management Architecture) is a Java framework that originated at IBM Research and was later donated to the Apache Software Foundation, where it entered incubation in 2006; the UIMA 1.0 specification became an OASIS Standard in March 2009 (docs.oasis-open.org/uima/v1.0). The data model at its centre is the CAS (Common Analysis Structure): a shared structure that holds the document together with all the annotations added along the way. Each annotator reads the annotations already present in the CAS and deposits new ones; a Type System declared in XML defines the classes (Token, Sentence, NamedEntity and the like). Chaining the annotators yields an Aggregate Analysis Engine.

Building on UIMA carries a concrete consequence: the components are independent modules, replaceable and reorderable, and each one is evaluated on its own. The paper relies on exactly this and reports a separate accuracy for every stage.

The annotator sequence described in the paper is this:

  • sentence boundary detector and tokenizer, derived from OpenNLP and adapted to clinical text (abbreviations such as Dr. or q.d. do not end a sentence);
  • normalizer, which brings inflected forms back to a canonical form;
  • POS tagger (part-of-speech) and shallow parser (chunker), which assigns first-level syntactic units — noun, verb and prepositional phrases — without attempting a full parse;
  • dictionary lookup, which maps phrases to UMLS concepts;
  • named entity recognizer, which structures mentions into clinical types;
  • negation and status annotator, which assign the assertion attributes.

The critical point

The dictionary lookup does not query the whole UMLS Metathesaurus. cTAKES uses a subset of UMLS version 2008AB, restricted to the concepts that come from SNOMED CT and RxNorm (the former for disorders, procedures and anatomy, the latter for drugs). It is the filter by terminology and by semantic type that brings the system down onto the clinical domain: rather than fishing out every biomedical concept, the lookup only looks at the entity families a chart note actually needs.

Two assertion annotators work over the mentions fished out this way. Negation uses the NegEx method of Chapman et al. (J Biomed Inform 34(5):301-310, 2001): a pattern-based algorithm that searches for negation terms within a window of seven tokens to the left and right of a mention, with a list of pseudo-negations (for instance “not ruled out”) to exclude because they mislead. Status assigns each mention one value among current, history of, family history of, possible. It is these two stages, after recognition, that separate “the patient has diagnosis X” from “diagnosis X is absent, past, in the family or merely hypothesised”.

What follows

The figures on the Mayo corpus read one component at a time, and that is the most useful part of the documentation for anyone deciding whether to adopt the system. The reference linguistic annotations come from 273 notes (about 100,650 tokens, 7,299 sentences); the named entity annotation covers 160 notes (47,975 tokens, 1,466 disorder mentions).

On the base components the paper reports: sentence boundary detector and tokenizer at accuracy 0.949, POS tagger 0.936, shallow parser at F-score 0.924. On entity recognition: F-score 0.715 on exact match (precise spans) and 0.824 on overlapping spans. Negation reaches accuracy 0.943, status sits at 0.859. The gap between exact and overlapping on entities — about ten points — tells you how often the system finds the right concept but gets the span boundaries wrong: a distinction that weighs downstream, because a correct concept with an imprecise span is recoverable, a missed concept is not.

Every stage is a measurable, replaceable UIMA annotator: that makes the pipeline a base to adapt from, not a closed result. Anyone working on text other than Mayo’s inherits the architecture and the Type System, then retrains or replaces the stages that depend on the data.

Limits

The components are tuned for English and for Mayo-style notes: telegraphic, US-American. The figures above hold on that corpus; using it on different text — paediatric, specialist, or in another language — means re-measuring each stage, without taking for granted that the figures hold elsewhere. For Italian the language-dependent stages (tokenizer, POS tagger, the NegEx list itself) would have to be redone, and a useful dictionary lookup presupposes Italian clinical terminology coverage that today is fragmentary. cTAKES also does not de-identify text: removing personal information is a separate problem, to be solved before sharing data. Status is limited to four discrete values and does not model fine temporal reasoning (when a condition began, whether it has resolved, whether it is planned), which stays outside the pipeline described.

It remains to be seen whether the output of a lookup restricted to SNOMED CT and RxNorm is enough for downstream uses (cohort identification, assisted coding), or whether the 2008AB subset needs widening to other UMLS sources as the domain expands.

Cover image: The Mayo Clinic complex in Rochester, Minnesota: the low Mayo Medical School building in the foreground with the tall main Mayo… — photo by Jonathunder, CC BY-SA 3.0 — https://commons.wikimedia.org/wiki/File:MayoClinic&MedSchool2006-05-07.JPG