Using Natural Language Processing to Screen for Adverse Postoperative Events

The Agency for Healthcare Research and Quality (AHRQ) has developed a set of 20 patient safety indicators to use as measures to screen for potential adverse events that may occur during hospitalization. The measures utilize administrative data to screen for specific International Classification of Diseases, Ninth Revision, Clinical Modification codes that might indicate a medical injury. The Centers for Medicare & Medicaid Services as well as private organizations use the AHRQ method to create ratings on individual healthcare institutions. However, according to researchers, administrative data have some limitations in this context. There are concerns about the validity of the codes and the difficulty of determining from discharge codes whether a disease entity existed prior to hospital admission or occurred during the hospitalization. The emergence of electronic medical records (EMRs) coupled with support from the federal government as part of the Affordable Care Act for healthcare information technology has created an alternative source of clinical information about hospital-related safety events. One example is natural language processing, an automated approach that extracts specific medical concepts from textual medical documents that do not rely on discharge codes. There have been few studies examining the use of natural language processing tools for the detection of adverse events. The researchers recently conducted a study to evaluate an approach based on language processing to identify postoperative complications in a multihospital healthcare network using the same EMR. They reported study results in the Journal of the American Medical Association [2011;306(8):848-855]. The researchers hypothesized that the language processing searches would be more accurate in predicting surgical complications compared with patient safety indicators identified using administrative discharge data. The cross-sectional study included 2974 patients undergoing surgical procedures at 6 Veterans Health Administration (VA) medical centers from 1999 to 2006. The primary outcome measures were postoperative occurrences of acute renal failure requiring dialysis, deep vein thrombosis, pulmonary embolism, sepsis, pneumonia, or myocardial infarction. The adverse events were identified using data gathered as part of the VA Surgical Quality Improvement Program; the events used for this study are also included as patient safety indicator events. The researchers determined the sensitivity and specificity of the natural language processing approach to identify the complications and compared the approach with patient safety indicators that use discharge coding data. Median age of the study participants was 64.5 years and 95% were male; 82% had an American Society of Anesthesiologists preoperative score of ≥3. Of the surgeries included, 38% were classified as general surgical procedures, 21% were orthopedic surgeries, and 14% were vascular procedures. The postoperative rates for the events included in the study were acute renal failure requiring dialysis, 2%; pulmonary embolism, 0.7%; deep vein thrombosis, 1%; sepsis, 7%; pneumonia, 16%; and myocardial infarction, 2%. The analysis found that, in general, using a natural language processing–based approach had higher sensitivities and lower specificities compared with the patient safety indicator. Natural language processing correctly identified 82% of cases of acute renal failure requiring dialysis, versus 38% of cases correctly identified using patient safety indicators (P<.001). Likewise, natural language processing correctly identified 59% of cases of pulmonary embolism/deep vein thrombosis versus 46% using patient safety indicators (P=.30). Comparisons of sensitivity differences of natural language processing compared with patient safety indicators for the other events were, respectively, sepsis, 89% versus 34% (P<.001); pneumonia, 64% versus 5% (P<.001); and myocardial infarction, 91% versus 89% (P=.67). Results of analyses for comparisons of natural language processing versus patient safety indicators to determine rates of specificity were, respectively, acute renal failure, 94% versus 100% (P<.001); pulmonary embolism/deep vein thrombosis, 91% versus 98% (P<.001); sepsis, 94% versus 99% (P<.001); pneumonia, 95% versus 99% (P<.001); and myocardial infarction, 95% versus 99% (P=.67). In conclusion, the researchers said, “using natural language processing with an electronic medical record greatly improves postoperative complication identification compared with patient safety indicators.…Different query strategies produced varying sensitivity and specificity, which in many cases could be improved through combining individual queries to optimize test characteristics.”