External Validation of a Widely Implemented Sepsis Prediction Model in Hospitalized Patients | Critical Care Medicine | JN Learning | AMA Ed Hub [Skip to Content]
[Skip to Content Landing]

External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients

Educational Objective
To externally validate the Epic Sepsis Model (ESM) in the prediction of sepsis and evaluate its potential clinical value compared with usual care.
1 Credit CME
Key Points

Question  How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis?

Findings  In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.

Meaning  This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level.

Abstract

Importance  The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM’s ability to identify patients with sepsis has not been adequately evaluated despite widespread use.

Objective  To externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care.

Design, Setting, and Participants  This retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019.

Exposure  The ESM score, calculated every 15 minutes.

Main Outcomes and Measures  Sepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours. Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics). Alert fatigue was evaluated by comparing the clinical value of different alerting strategies.

Results  We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.

Conclusions and Relevance  This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.

Sign in to take quiz and track your certificates

Buy This Activity

JN Learning™ is the home for CME and MOC from the JAMA Network. Search by specialty or US state and earn AMA PRA Category 1 CME Credit™ from articles, audio, Clinical Challenges and more. Learn more about CME/MOC

Article Information

Accepted for Publication: April 18, 2021.

Published Online: June 21, 2021. doi:10.1001/jamainternmed.2021.2626

Correction: This article was corrected on August 2, 2021, to fix an error in the number needed to evaluate presented in Table 2 and the Results.

Corresponding Author: Karandeep Singh, MD, MMSc, Department of Learning Health Sciences, University of Michigan Medical School, 1161H NIB, 300 N Ingalls St, Ann Arbor, MI 48109 (kdpsingh@umich.edu).

Author Contributions: Dr Singh had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Wong, Otles, Pestrue, Phillips, Penoza, Singh.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Wong, Otles, Ghous, Singh.

Critical revision of the manuscript for important intellectual content: Wong, Otles, Donnelly, Krumm, McCullough, DeTroyer-Cooley, Pestrue, Phillips, Konye, Penoza, Singh.

Statistical analysis: Wong, Otles, Donnelly, Krumm, McCullough, Singh.

Administrative, technical, or material support: Wong, Otles, Pestrue, Phillips, Konye, Penoza.

Supervision: Singh.

Conflict of Interest Disclosures: Dr Donnelly reported receiving grants from the National Institutes of Health, National Heart, Lung, and Blood Institute K12 Scholar during the conduct of the study; and personal fees from the American College of Emergency Physicians as an editor of Annals of Emergency Medicine outside the submitted work. No other disclosures were reported.

Funding/Support: Mr Otles was supported by grant T32GM007863 from the National Institutes of Health. Dr Donnelly was supported by grant K12HL138039 from the National Heart, Lung, and Blood Institute.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Rivers  E , Nguyen  B , Havstad  S ,  et al; Early Goal-Directed Therapy Collaborative Group.  Early goal-directed therapy in the treatment of severe sepsis and septic shock.   N Engl J Med. 2001;345(19):1368-1377. doi:10.1056/NEJMoa010307 PubMedGoogle ScholarCrossref
2.
Yealy  DM , Kellum  JA , Huang  DT ,  et al; ProCESS Investigators.  A randomized trial of protocol-based care for early septic shock.   N Engl J Med. 2014;370(18):1683-1693. doi:10.1056/NEJMoa1401602 PubMedGoogle Scholar
3.
Gao  F , Melody  T , Daniels  DF , Giles  S , Fox  S .  The impact of compliance with 6-hour and 24-hour sepsis bundles on hospital mortality in patients with severe sepsis: a prospective observational study.   Crit Care. 2005;9(6):R764-R770. doi:10.1186/cc3909 PubMedGoogle ScholarCrossref
4.
Sawyer  AM , Deal  EN , Labelle  AJ ,  et al.  Implementation of a real-time computerized sepsis alert in nonintensive care unit patients.   Crit Care Med. 2011;39(3):469-473. doi:10.1097/CCM.0b013e318205df85 PubMedGoogle ScholarCrossref
5.
Semler  MW , Weavind  L , Hooper  MH ,  et al.  An electronic tool for the evaluation and treatment of sepsis in the ICU: a randomized controlled trial.   Crit Care Med. 2015;43(8):1595-1602. doi:10.1097/CCM.0000000000001020 PubMedGoogle ScholarCrossref
6.
Giannini  HM , Ginestra  JC , Chivers  C ,  et al.  A machine learning algorithm to predict severe sepsis and septic shock: development, implementation, and impact on clinical practice.   Crit Care Med. 2019;47(11):1485-1492. doi:10.1097/CCM.0000000000003891 PubMedGoogle ScholarCrossref
7.
Downing  NL , Rolnick  J , Poole  SF ,  et al.  Electronic health record–based clinical decision support alert for severe sepsis: a randomised evaluation.   BMJ Qual Saf. 2019;28(9):762-768. doi:10.1136/bmjqs-2018-008765 PubMedGoogle ScholarCrossref
8.
Delahanty  RJ , Alvarez  J , Flynn  LM , Sherwin  RL , Jones  SS .  Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis.   Ann Emerg Med. 2019;73(4):334-344. doi:10.1016/j.annemergmed.2018.11.036 PubMedGoogle ScholarCrossref
9.
Afshar  M , Arain  E , Ye  C ,  et al.  Patient outcomes and cost-effectiveness of a sepsis care quality improvement program in a health system.   Crit Care Med. 2019;47(10):1371-1379. doi:10.1097/CCM.0000000000003919 PubMedGoogle ScholarCrossref
10.
Guidi  JL , Clark  K , Upton  MT ,  et al.  Clinician perception of the effectiveness of an automated early warning and response system for sepsis in an academic medical center.   Ann Am Thorac Soc. 2015;12(10):1514-1519. doi:10.1513/AnnalsATS.201503-129OC PubMedGoogle ScholarCrossref
11.
Ginestra  JC , Giannini  HM , Schweickert  WD ,  et al.  Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock.   Crit Care Med. 2019;47(11):1477-1484. doi:10.1097/CCM.0000000000003803 PubMedGoogle ScholarCrossref
12.
Rolnick  JA , Weissman  GE .  Early warning systems: the neglected importance of timing.   J Hosp Med. 2019;14(7):445-447. doi:10.12788/jhm.3229 PubMedGoogle ScholarCrossref
13.
Makam  AN , Nguyen  OK , Auerbach  AD .  Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review.   J Hosp Med. 2015;10(6):396-402. doi:10.1002/jhm.2347 PubMedGoogle ScholarCrossref
14.
Benthin  C , Pannu  S , Khan  A , Gong  M ; NHLBI Prevention and Early Treatment of Acute Lung Injury (PETAL) Network.  The nature and variability of automated practice alerts derived from electronic health records in a U.S. nationwide critical care research network.   Ann Am Thorac Soc. 2016;13(10):1784-1788. doi:10.1513/AnnalsATS.201603-172BC PubMedGoogle Scholar
15.
Van Calster  B , Wynants  L , Timmerman  D , Steyerberg  EW , Collins  GS .  Predictive analytics in health care: how can we know it works?   J Am Med Inform Assoc. 2019;26(12):1651-1654. doi:10.1093/jamia/ocz130 PubMedGoogle ScholarCrossref
16.
Davis  SE , Lasko  TA , Chen  G , Siew  ED , Matheny  ME .  Calibration drift in regression and machine learning models for acute kidney injury.   J Am Med Inform Assoc. 2017;24(6):1052-1061. doi:10.1093/jamia/ocx030 PubMedGoogle ScholarCrossref
17.
Caldwell  P . We’ve spent billions to fix our medical records, and they’re still a mess: here’s why. Mother Jones. Published October 21, 2015. Accessed April 24, 2020. https://www.motherjones.com/politics/2015/10/epic-systems-judith-faulkner-hitech-ehr-interoperability/
18.
Rhee  C , Dantes  RB , Epstein  L , Klompas  M .  Using objective clinical data to track progress on preventing and treating sepsis: CDC’s new “Adult Sepsis Event” surveillance strategy.   BMJ Qual Saf. 2019;28(4):305-309. doi:10.1136/bmjqs-2018-008331 PubMedGoogle ScholarCrossref
19.
Rhee  C , Dantes  R , Epstein  L ,  et al; CDC Prevention Epicenter Program.  Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014.   JAMA. 2017;318(13):1241-1249. doi:10.1001/jama.2017.13836 PubMedGoogle ScholarCrossref
20.
Centers for Disease Control and Prevention. Hospital toolkit for adult sepsis surveillance. Published March 2018. Accessed February 11, 2021. https://www.cdc.gov/sepsis/pdfs/Sepsis-Surveillance-Toolkit-Mar-2018_508.pdf
21.
Henry  KE , Hager  DN , Pronovost  PJ , Saria  S .  A targeted real-time early warning score (TREWScore) for septic shock.   Sci Transl Med. 2015;7(299):299ra122. doi:10.1126/scitranslmed.aab3719 PubMedGoogle Scholar
22.
Oh  J , Makar  M , Fusco  C ,  et al.  A generalizable, data-driven approach to predict daily risk of Clostridium difficile infection at two large academic health centers.   Infect Control Hosp Epidemiol. 2018;39(4):425-433. doi:10.1017/ice.2018.16 PubMedGoogle ScholarCrossref
23.
Singh  K , Valley  TS , Tang  S ,  et al.  Evaluating a widely implemented proprietary deterioration index model among hospitalized COVID-19 patients.   Ann Am Thorac Soc. 2020. Published online December 24, 2020. doi:10.1513/AnnalsATS.202006-698OC PubMedGoogle Scholar
24.
R Core Team. R: a language and environment for statistical computing. Published online 2020. Accessed May 4, 2021. http://www.r-project.org/
25.
pROC: Display and analyze ROC curves [R package pROC version 1.16.2]. Accessed April 23, 2020. https://CRAN.R-project.org/package=pROC
26.
Singh  K. The runway package for R. Accessed October 21, 2020. https://github.com/ML4LHS/runway
27.
Bennett  T , Russell  S , King  J ,  et al  Accuracy of the Epic Sepsis Prediction Model in a regional health system.   arXiv. Preprint posted online February 19, 2019. https://arxiv.org/abs/1902.07276 Google Scholar
28.
Healthcare Cost and Utilization Project. HCUP weighted summary statistics report: NIS 2018 core file means of continuous data elements. Accessed March 8, 2021. https://www.hcup-us.ahrq.gov/db/nation/nis/tools/stats/MaskedStats_NIS_2018_Core_Weighted.PDF
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
jn-learning_Modal_Multimedia_LoginSubscribe_Purchase
Close
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
jn-learning_Modal_Multimedia_LoginSubscribe_Purchase
Close
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right
Close

Name Your Search

Save Search
Close
With a personal account, you can:
  • Track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
jn-learning_Modal_SaveSearch_NoAccess_Purchase
Close

Lookup An Activity

or

Close

My Saved Searches

You currently have no searches saved.

Close
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right
Close