[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
[Skip to Content Landing]

Development and Validation of a Tool to Identify Patients With Type 2 Diabetes at High Risk of Hypoglycemia-Related Emergency Department or Hospital Use

Educational Objective
To describe the development and validation of a tool to categorize risk of hypoglycemic-related emergency department (ED) or hospital use in patients with type 2 diabetes.
1 Credit CME
Key Points

Question  Can electronic medical records be used to reliably categorize risk of future hypoglycemia-related emergency department or hospital use in patients with type 2 diabetes?

Findings  We developed and validated a risk stratification tool that categorized patients’ 12-month risk of hypoglycemia-related utilization using only 6 electronic medical record–based inputs (patient history of hypoglycemia-related utilization, insulin use, sulfonylurea use, emergency department use, chronic kidney disease, and age). Tool performance was validated in 2 fully independent populations.

Meaning  This hypoglycemia risk stratification tool could facilitate efficient targeting of population management interventions to reduce hypoglycemia risk and improve patient safety.


Importance  Hypoglycemia-related emergency department (ED) or hospital use among patients with type 2 diabetes (T2D) is clinically significant and possibly preventable.

Objective  To develop and validate a tool to categorize risk of hypoglycemic-related utilization in patients with T2D.

Design, Setting, and Participants  Using recursive partitioning with a split-sample design, we created a classification tree based on potential predictors of hypoglycemia-related ED or hospital use. The resulting model was transcribed into a tool for practical application and tested in 1 internal and 2 fully independent, external samples. Development and internal testing was conducted in a split sample of 206 435 patients with T2D from Kaiser Permanente Northern California (KPNC), an integrated health care system. The tool was externally tested in 1 335 966 Veterans Health Administration and 14 972 Group Health Cooperative patients with T2D.

Exposures  Based on a literature review, we identified 156 candidate predictor variables (prebaseline exposures) using data collected from electronic medical records.

Main Outcomes and Measures  Hypoglycemia-related ED or hospital use during 12 months of follow-up.

Results  The derivation sample (n = 165 148) had a mean (SD) age of 63.9 (13.0) years and included 78 576 (47.6%) women. The crude annual rate of at least 1 hypoglycemia-related ED or hospital encounter in the KPNC derivation sample was 0.49%. The resulting hypoglycemia risk stratification tool required 6 patient-specific inputs: number of prior episodes of hypoglycemia-related utilization, insulin use, sulfonylurea use, prior year ED use, chronic kidney disease stage, and age. We categorized the predicted 12-month risk of any hypoglycemia-related utilization as high (>5%), intermediate (1%-5%), or low (<1%). In the internal validation sample, 2.0%, 10.7%, and 87.3% were categorized as high, intermediate, and low risk, respectively, with observed 12-month hypoglycemia-related utilization rates of 6.7%, 1.4%, and 0.2%, respectively. There was good discrimination in the internal validation KPNC sample (C statistic = 0.83) and both external validation samples (Veterans Health Administration: C statistic = 0.81; Group Health Cooperative: C statistic = 0.79).

Conclusions and Relevance  This hypoglycemia risk stratification tool categorizes the 12-month risk of hypoglycemia-related utilization in patients with T2D using only 6 inputs. This tool could facilitate targeted population management interventions, potentially reducing hypoglycemia risk and improving patient safety and quality of life.

Sign in to take quiz and track your certificates

Buy This Activity

JN Learning™ is the home for CME and MOC from the JAMA Network. Search by specialty or US state and earn AMA PRA Category 1 CME Credit™ from articles, audio, Clinical Challenges and more. Learn more about CME/MOC

Article Information

Corresponding Author: Andrew J. Karter, PhD, Division of Research, Kaiser Permanente Northern California, 2000 Broadway, Oakland, CA 94612 (andy.j.karter@kp.org).

Accepted for Publication: June 20, 2017.

Published Online: August 21, 2017. doi:10.1001/jamainternmed.2017.3844

Author Contributions: Dr Karter had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Karter, Warton, Lipska, Moffet, Huang, Miller.

Acquisition, analysis, or interpretation of data: Karter, Warton, Lipska, Ralston, Moffet, Jackson, Miller.

Drafting of the manuscript: Karter, Warton, Jackson.

Critical revision of the manuscript for important intellectual content: Karter, Warton, Lipska, Ralston, Moffet, Huang, Miller.

Statistical analysis: Karter, Warton, Miller.

Obtained funding: Karter, Moffet.

Administrative, technical, or material support: Karter, Ralston, Moffet, Jackson, Miller.

Supervision: Karter, Ralston, Huang.

Conflict of Interest Disclosures: The National Institutes of Health supplied additional funding for our hypoglycemia-related research (NIDDK R01DK103721, R01DK081796). Drs Karter and Huang are also supported by the NIDDK Centers for Diabetes Translational Research (P30 DK092924 and P30 DK092949, respectively). Dr Huang was supported by K24 DK105340. Dr Lipska receives support from the Centers for Medicare & Medicaid Services to develop and maintain publicly reported quality measures, and from the Yale Claude D. Pepper Older Americans Independence Center (P30AG021342) and the National Institute on Aging through the Paul Beeson Career Development Award (K23AG048359). No other disclosures are reported.

Funding/Support: This project was funded by the US Food and Drug Administration (FDA BAA-13 00119).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Gregg  EW, Li  Y, Wang  J,  et al.  Changes in diabetes-related complications in the United States, 1990-2010.  N Engl J Med. 2014;370(16):1514-1523.PubMedGoogle ScholarCrossref
Pogach  L, Aron  D.  Balancing hypoglycemia and glycemic control: a public health approach for insulin safety.  JAMA. 2010;303(20):2076-2077.PubMedGoogle ScholarCrossref
Lee  SJ.  So much insulin, so much hypoglycemia.  JAMA Intern Med. 2014;174(5):686-688.PubMedGoogle ScholarCrossref
Lipska  KJ.  Improving safety of diabetes mellitus management.  JAMA Intern Med. 2014;174(10):1612-1613.PubMedGoogle ScholarCrossref
Workgroup on Hypoglycemia, American Diabetes Association.  Defining and reporting hypoglycemia in diabetes: a report from the American Diabetes Association Workgroup on Hypoglycemia.  Diabetes Care. 2005;28(5):1245-1249.PubMedGoogle ScholarCrossref
Majumdar  SR, Hemmelgarn  BR, Lin  M, McBrien  K, Manns  BJ, Tonelli  M.  Hypoglycemia associated with hospitalization and adverse events in older people: population-based cohort study.  Diabetes Care. 2013;36(11):3585-3590.PubMedGoogle ScholarCrossref
Lipska  KJ, Ross  JS, Wang  Y,  et al.  National trends in US hospital admissions for hyperglycemia and hypoglycemia among Medicare beneficiaries, 1999 to 2011.  JAMA Intern Med. 2014;174(7):1116-1124.PubMedGoogle ScholarCrossref
Huang  ES, Laiteerapong  N, Liu  JY, John  PM, Moffet  HH, Karter  AJ.  Rates of complications and mortality in older patients with diabetes mellitus: the diabetes and aging study.  JAMA Intern Med. 2014;174(2):251-258.PubMedGoogle ScholarCrossref
Budnitz  DS, Lovegrove  MC, Shehab  N, Richards  CL.  Emergency hospitalizations for adverse drug events in older Americans.  N Engl J Med. 2011;365(21):2002-2012.PubMedGoogle ScholarCrossref
Signorovitch  JE, Macaulay  D, Diener  M,  et al.  Hypoglycaemia and accident risk in people with type 2 diabetes mellitus treated with non-insulin antidiabetes drugs.  Diabetes Obes Metab. 2013;15(4):335-341.PubMedGoogle ScholarCrossref
Stahn  A, Pistrosch  F, Ganz  X,  et al.  Relationship between hypoglycemic episodes and ventricular arrhythmias in patients with type 2 diabetes and cardiovascular diseases: silent hypoglycemias and silent arrhythmias.  Diabetes Care. 2014;37(2):516-520.PubMedGoogle ScholarCrossref
Whitmer  RA, Karter  AJ, Yaffe  K, Quesenberry  CP  Jr, Selby  JV.  Hypoglycemic episodes and risk of dementia in older patients with type 2 diabetes mellitus.  JAMA. 2009;301(15):1565-1572.PubMedGoogle ScholarCrossref
McCoy  RG, Van Houten  HK, Ziegenfuss  JY, Shah  ND, Wermers  RA, Smith  SA.  Increased mortality of patients with diabetes reporting severe hypoglycemia.  Diabetes Care. 2012;35(9):1897-1901.PubMedGoogle ScholarCrossref
Zoungas  S, Patel  A, Chalmers  J,  et al; ADVANCE Collaborative Group.  Severe hypoglycemia and risks of vascular events and death.  N Engl J Med. 2010;363(15):1410-1418.PubMedGoogle ScholarCrossref
Karter  AJ, Subramanian  U, Saha  C,  et al.  Barriers to insulin initiation: the translating research into action for diabetes insulin starts project.  Diabetes Care. 2010;33(4):733-735.PubMedGoogle ScholarCrossref
Laiteerapong  N, Karter  AJ, Liu  JY,  et al.  Correlates of quality of life in older adults with diabetes: the diabetes & aging study.  Diabetes Care. 2011;34(8):1749-1753.PubMedGoogle ScholarCrossref
Nicolucci  A, Pintaudi  B, Rossi  MC,  et al.  The social burden of hypoglycemia in the elderly.  Acta Diabetol. 2015;52(4):677-685.PubMedGoogle ScholarCrossref
Peusens  G, De Jonghe  K, De Rood  I, ,  et al.  Phytoplasmas in pome fruit trees: update of their presence and their vectors in Belgium.  Commun Agric Appl Biol Sci. 2015;80(2):143-148.PubMedGoogle Scholar
Amiel  SA, Dixon  T, Mann  R, Jameson  K.  Hypoglycaemia in type 2 diabetes.  Diabet Med. 2008;25(3):245-254.PubMedGoogle ScholarCrossref
Meulstee  M, Whittemore  R, Watts  SA.  Development of an educational program on prevention of hypoglycemic events among elderly veterans with type 2 diabetes.  Diabetes Educ. 2015;41(6):690-697.PubMedGoogle ScholarCrossref
Rondags  SM, de Wit  M, Snoek  FJ.  HypoAware: development and pilot study of a brief and partly web-based psychoeducational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia.  Diabet Med. 2016;33(2):184-191.PubMedGoogle ScholarCrossref
Elliott  J, Rankin  D, Jacques  RM,  et al; NIHR DAFNE Research Study Group.  A cluster randomized controlled non-inferiority trial of 5-day dose adjustment for normal eating (DAFNE) training delivered over 1 week versus 5-day DAFNE training delivered over 5 weeks: the DAFNE 5 × 1-day trial.  Diabet Med. 2015;32(3):391-398.PubMedGoogle ScholarCrossref
Seaquist  ER, Anderson  J, Childs  B,  et al; American Diabetes Association; Endocrine Society.  Hypoglycemia and diabetes: a report of a workgroup of the American Diabetes Association and the Endocrine Society.  J Clin Endocrinol Metab. 2013;98(5):1845-1859.PubMedGoogle ScholarCrossref
Heinemann  L, Devries  JH.  Evidence for continuous glucose monitoring: sufficient for reimbursement?  Diabet Med. 2014;31(2):122-125.PubMedGoogle ScholarCrossref
Miller  DR, Pogach  L.  Longitudinal approaches to evaluate health care quality and outcomes: the Veterans Health Administration diabetes epidemiology cohorts.  J Diabetes Sci Technol. 2008;2(1):24-32.PubMedGoogle ScholarCrossref
Huang  ES, Liu  JY, Moffet  HH, John  PM, Karter  AJ.  Glycemic control, complications, and death in older diabetic patients: the diabetes and aging study.  Diabetes Care. 2011;34(6):1329-1336.PubMedGoogle ScholarCrossref
Ginde  AA, Blanc  PG, Lieberman  RM, Camargo  CA  Jr.  Validation of ICD-9-CM coding algorithm for improved identification of hypoglycemia visits.  BMC Endocr Disord. 2008;8:4.PubMedGoogle ScholarCrossref
Miller  SI, Wallace  RJ  Jr, Musher  DM, Septimus  EJ, Kohl  S, Baughn  RE.  Hypoglycemia as a manifestation of sepsis.  Am J Med. 1980;68(5):649-654.PubMedGoogle ScholarCrossref
Moons  KG, Altman  DG, Vergouwe  Y, Royston  P.  Prognosis and prognostic research: application and impact of prognostic models in clinical practice.  BMJ. 2009;338:b606.PubMedGoogle ScholarCrossref
Royston  P, Moons  KG, Altman  DG, Vergouwe  Y.  Prognosis and prognostic research: developing a prognostic model.  BMJ. 2009;338:b604.PubMedGoogle ScholarCrossref
Altman  DG, Vergouwe  Y, Royston  P, Moons  KG.  Prognosis and prognostic research: validating a prognostic model.  BMJ. 2009;338:b605.PubMedGoogle ScholarCrossref
Steyerberg  EW.  Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009.
Justice  AC, Covinsky  KE, Berlin  JA.  Assessing the generalizability of prognostic information.  Ann Intern Med. 1999;130(6):515-524.PubMedGoogle ScholarCrossref
Breiman  L, Friedman  JH, Olshen  RA, Stone  CJ.  Classification and Regression Trees. Monterey, CA: Wadsworth & Brooks/Cole Advanced Books & Software; 1984.
Fonarow  GC, Adams  KF  Jr, Abraham  WT, Yancy  CW, Boscardin  WJ; ADHERE Scientific Advisory Committee, Study Group, and Investigators.  Risk stratification for in-hospital mortality in acutely decompensated heart failure: classification and regression tree analysis.  JAMA. 2005;293(5):572-580.PubMedGoogle ScholarCrossref
Lewis  RJ. An Introduction to Classification and Regression Tree (CART) Analysis. 2000. http://citeseerx.ist.psu.edu/viewdoc/download?doi= Accessed July 6, 2017.
Hosmer  DW, Lemeshow  S.  Applied Logistic Regression. New York, NY: John Wiley & Sons; 1989.
Chambless  LE, Cummiskey  CP, Cui  G.  Several methods to assess improvement in risk prediction models: extension to survival analysis.  Stat Med. 2011;30(1):22-38.PubMedGoogle ScholarCrossref
Moons  KG, de Groot  JA, Bouwmeester  W,  et al.  Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.  PLoS Med. 2014;11(10):e1001744.PubMedGoogle ScholarCrossref
Meurer  WJ, Tolles  J.  Logistic regression diagnostics: understanding how well a model predicts outcomes.  JAMA. 2017;317(10):1068-1069.PubMedGoogle ScholarCrossref
Walter  LC, Brand  RJ, Counsell  SR,  et al.  Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization.  JAMA. 2001;285(23):2987-2994.PubMedGoogle ScholarCrossref
Moffet  HH, Adler  N, Schillinger  D,  et al.  Cohort profile: the Diabetes Study of Northern California (DISTANCE)—objectives and design of a survey follow-up study of social health disparities in a managed care population.  Int J Epidemiol. 2009;38(1):38-47.PubMedGoogle ScholarCrossref
Levey  AS, Bosch  JP, Lewis  JB, Greene  T, Rogers  N, Roth  D; Modification of Diet in Renal Disease Study Group.  A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation.  Ann Intern Med. 1999;130(6):461-470.PubMedGoogle ScholarCrossref
Senior  PA, Bellin  MD, Alejandro  R,  et al; Clinical Islet Transplantation Consortium.  Consistency of quantitative scores of hypoglycemia severity and glycemic lability and comparison with continuous glucose monitoring system measures in long-standing type 1 diabetes.  Diabetes Technol Ther. 2015;17(4):235-242.PubMedGoogle ScholarCrossref
Olsen  SE, Bjørgaas  MR, Åsvold  BO,  et al.  Impaired awareness of hypoglycemia in adults with type 1 diabetes is not associated with autonomic dysfunction or peripheral neuropathy.  Diabetes Care. 2016;39(3):426-433.PubMedGoogle ScholarCrossref
Rondags  SM, de Wit  M, van Tulder  MW, Diamant  M, Snoek  FJ.  HypoAware—a brief and partly web-based psycho-educational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia: design of a cost-effectiveness randomised controlled trial.  BMC Endocr Disord. 2015;15:43.PubMedGoogle ScholarCrossref
Shepard  JA, Vajda  K, Nyer  M, Clarke  W, Gonder-Frederick  L.  Understanding the construct of fear of hypoglycemia in pediatric type 1 diabetes.  J Pediatr Psychol. 2014;39(10):1115-1125.PubMedGoogle ScholarCrossref
Rondags  SM, de Wit  M, Twisk  JW, Snoek  FJ.  Effectiveness of HypoAware, a brief partly web-based psychoeducational intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycemia: a cluster randomized controlled trial.  Diabetes Care. 2016;39(12):2190-2196.PubMedGoogle ScholarCrossref
Pogach  L, Aron  D.  The other side of quality improvement in diabetes for seniors: a proposal for an overtreatment glycemic measure.  Arch Intern Med. 2012;172(19):1510-1512.PubMedGoogle ScholarCrossref
Tseng  CL, Soroka  O, Maney  M, Aron  DC, Pogach  LM.  Assessing potential glycemic overtreatment in persons at hypoglycemic risk.  JAMA Intern Med. 2014;174(2):259-268.PubMedGoogle ScholarCrossref
McCoy  RG, Lipska  KJ, Yao  X, Ross  JS, Montori  VM, Shah  ND.  Intensive treatment and severe hypoglycemia among adults with type 2 diabetes.  JAMA Intern Med. 2016;176(7):969-978.PubMedGoogle ScholarCrossref
Lipska  KJ, Kosiborod  M.  Hypoglycemia and adverse outcomes: marker or mediator?  Rev Cardiovasc Med. 2011;12(3):132-135.PubMedGoogle Scholar
Rodriguez-Gutierrez  R, Lipska  KJ, McCoy  RG, Ospina  NS, Ting  HH, Montori  VM; Hypoglycemia as a Quality Measure in Diabetes Study Group.  Hypoglycemia as an indicator of good diabetes care.  BMJ. 2016;352:i1084.PubMedGoogle ScholarCrossref
Rodriguez-Gutierrez  R, Ospina  NS, McCoy  RG, Lipska  KJ, Shah  ND, Montori  VM; Hypoglycemia as a Quality Measure in Diabetes Study Group.  Inclusion of hypoglycemia in clinical practice guidelines and performance measures in the care of patients with diabetes.  JAMA Intern Med. 2016;176(11):1714-1716.PubMedGoogle ScholarCrossref
Bonds  DE, Miller  ME, Dudl  J,  et al.  Severe hypoglycemia symptoms, antecedent behaviors, immediate consequences and association with glycemia medication usage: secondary analysis of the ACCORD clinical trial data.  BMC Endocr Disord. 2012;12:5.PubMedGoogle ScholarCrossref
Seligman  HK, Bolger  AF, Guzman  D, López  A, Bibbins-Domingo  K.  Exhaustion of food budgets at month’s end and hospital admissions for hypoglycemia.  Health Aff (Millwood). 2014;33(1):116-123.PubMedGoogle ScholarCrossref
Basu  S, Berkowitz  SA, Seligman  H.  The monthly cycle of hypoglycemia: an observational claims-based study of emergency room visits, hospital admissions, and costs in a commercially insured population.  Med Care. 2017;55(7):639-645.PubMedGoogle ScholarCrossref
Berkowitz  SA, Karter  AJ, Lyles  CR,  et al.  Low socioeconomic status is associated with increased risk for hypoglycemia in diabetes patients: the Diabetes Study of Northern California (DISTANCE).  J Health Care Poor Underserved. 2014;25(2):478-490.PubMedGoogle ScholarCrossref
Frier  BM, Jensen  MM, Chubb  BD.  Hypoglycaemia in adults with insulin-treated diabetes in the UK: self-reported frequency and effects.  Diabet Med. 2016;33(8):1125-1132.PubMedGoogle ScholarCrossref
Östenson  CG, Geelhoed-Duijvestijn  P, Lahtela  J, Weitgasser  R, Markert Jensen  M, Pedersen-Bjergaard  U.  Self-reported non-severe hypoglycaemic events in Europe.  Diabet Med. 2014;31(1):92-101.PubMedGoogle ScholarCrossref
Sarkar  U, Karter  AJ, Liu  JY, Moffet  HH, Adler  NE, Schillinger  D.  Hypoglycemia is more common among type 2 diabetes patients with limited health literacy: the Diabetes Study of Northern California (DISTANCE).  J Gen Intern Med. 2010;25(9):962-968.PubMedGoogle ScholarCrossref
Lipska  KJ, Warton  EM, Huang  ES,  et al.  HbA1c and risk of severe hypoglycemia in type 2 diabetes: the Diabetes and Aging Study.  Diabetes Care. 2013;36(11):3535-3542.PubMedGoogle ScholarCrossref
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right

Name Your Search

Save Search
With a personal account, you can:
  • Track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience

Lookup An Activity



My Saved Searches

You currently have no searches saved.

With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right