[Skip to Content]
Access to paid content on this site is currently suspended due to excessive activity being detected from your IP address Please contact the publisher to request reinstatement.
[Skip to Content Landing]

Evaluation of an Algorithm for Identifying Ocular Conditions in Electronic Health Record Data

Educational Objective To develop a method to accurately identify the presence or absence of ocular conditions of interest using electronic health record data.
1 Credit CME
Key Points

Question  What method other than assessing administrative billing codes can researchers, using big data, apply to accurately identify patients with ocular diseases of interest?

Findings  In this study of the electronic health records of 122 339 eye care recipients, a newly developed and validated algorithm that searches structured and unstructured data in electronic health records successfully detected most patients with and without exfoliation syndrome.

Meaning  Algorithms may enhance the ability of researchers to make use of big data to study patients with ocular diseases.


Importance  For research involving big data, researchers must accurately identify patients with ocular diseases or phenotypes of interest. Reliance on administrative billing codes alone for this purpose is limiting.

Objective  To develop a method to accurately identify the presence or absence of ocular conditions of interest using electronic health record (EHR) data.

Design, Setting, and Participants  This study is a retrospective analysis of the EHR data of patients (n = 122 339) in the Sight Outcomes Research Collaborative Ophthalmology Data Repository who received eye care at participating academic medical centers between August 1, 2012, and August 31, 2017. An algorithm that searches structured and unstructured (free-text) EHR data for conditions of interest was developed and then tested to determine how well it could detect the presence or absence of exfoliation syndrome (XFS). The algorithm was trained to search for evidence of XFS among a sample of patients with and without XFS (n = 200) by reviewing International Classification of Diseases, Ninth Revision or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-9 or ICD-10) billing codes, the patient’s problem list, and text within the ocular examination section and unstructured (free-text) data in the EHR. The likelihood that each patient had XFS was estimated using logistic least absolute shrinkage and selection operator (LASSO) regression. The EHR data of all patients were run through the algorithm to generate an XFS probability score for each patient. The algorithm was validated with review of EHRs by glaucoma specialists.

Main Outcomes and Measures  Positive predictive value (PPV) and negative predictive value (NPV) of the algorithm were computed as the proportion of patients correctly classified with XFS or without XFS.

Results  This study included 122 339 patients, with a mean (SD) age of 52.4 (25.1) years. Of these patients, 69 002 (56.4%) were female and 99 579 (81.4%) were white. The algorithm assigned a less than 10% probability of XFS for 121 085 patients (99.0%) as well as an XFS probability score of more than 75% for 543 patients (0.4%), more than 90% for 353 patients (0.3%), and more than 99% for 83 patients (0.07%). Validated by glaucoma specialists, the algorithm had a PPV of 95.0% (95% CI, 89.5%-97.7%) and an NPV of 100% (95% CI, 91.2%-100%). When there was ICD-9 or ICD-10 billing code documentation of XFS, in 86% or 96% of the records, respectively, evidence of XFS was also recorded elsewhere in the EHR. Conversely, when there was clinical examination or free-text evidence of XFS, it was documented with ICD-9 codes only approximately 40% of the time and even less often with ICD-10 codes.

Conclusions and Relevance  The algorithm developed, tested, and validated in this study appears to be better at identifying the presence or absence of XFS in EHR data than the conventional approach of assessing only billing codes; such an algorithm may enhance the ability of investigators to use EHR data to study patients with ocular diseases.

Sign in to take quiz and track your certificates

Buy This Activity

JN Learning™ is the home for CME and MOC from the JAMA Network. Search by specialty or US state and earn AMA PRA Category 1 CME Credit™ from articles, audio, Clinical Challenges and more. Learn more about CME/MOC

Article Information

Accepted for Publication: December 5, 2018.

Corresponding Author: Joshua D. Stein, MD, MS, W. K. Kellogg Eye Center, Department of Ophthalmology and Visual Sciences, University of Michigan Medical School, 1000 Wall St, Ann Arbor, MI 48105 (jdstein@med.umich.edu).

Published Online: February 21, 2019. doi:10.1001/jamaophthalmol.2018.7051

Author Contributions: Dr Stein had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Stein.

Acquisition, analysis, or interpretation of data: All authors.

Drafting of the manuscript: Stein, Kamat, Woodward, Cowall.

Critical revision of the manuscript for important intellectual content: Rahman, Andrews, Ehrlich, Shah, Boese, Woodward, Trager, Narayanaswamy, Hanauer.

Statistical analysis: Rahman, Andrews, Narayanaswamy.

Obtained funding: Ehrlich.

Administrative, technical, or material support: Stein, Cowall, Hanauer.

Supervision: Andrews, Woodward.

Conflict of Interest Disclosures: Dr Stein reported grants from NIH, grants from Lighthouse Guild, and grants from Research to Prevent Blindness during the conduct of the study. Dr Ehrlich reported grants from National Institutes of Health during the conduct of the study; grants from Blue Cross Blue Shield of Michigan Foundation and grants from American Society of Cataract and Refractive Surgery Foundation outside the submitted work. Dr Shah reported personal fees from Allergan and personal fees from Glaukos outside the submitted work. Dr Woodward reported grants from National Eye Institute during the conduct of the study. No other disclosures were reported.

Funding/Support: This study was supported by grants R01 EY026641 (Dr Stein), K23EY027848 (Dr Ehrlich), and K23EY023596 (Dr Woodward) from the National Institutes of Health; a grant from the Lighthouse Guild; and an unrestricted grant to the University of Michigan Department of Ophthalmology and Visual Sciences from Research to Prevent Blindness.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Meeting Presentation: The results of this study were presented at the 2018 American Ophthalmological Society Annual Meeting; May 18, 2018; Dana Point, California.

Musch  DC, Niziol  LM, Stein  JD, Kamyar  RM, Sugar  A.  Prevalence of corneal dystrophies in the United States: estimates from claims data.  Invest Ophthalmol Vis Sci. 2011;52(9):6959-6963. doi:10.1167/iovs.11-7771PubMedGoogle ScholarCrossref
Wang  SY, Andrews  CA, Herman  WH, Gardner  TW, Stein  JD.  Incidence and risk factors for developing diabetic retinopathy among youths with type 1 or type 2 diabetes throughout the United States.  Ophthalmology. 2017;124(4):424-430. doi:10.1016/j.ophtha.2016.10.031PubMedGoogle ScholarCrossref
Wu  AM, Wu  CM, Tseng  VL,  et al.  Characteristics associated with receiving cataract surgery in the US Medicare and Veterans Health Administration populations.  JAMA Ophthalmol. 2018;136(7):738-745. doi:10.1001/jamaophthalmol.2018.1361PubMedGoogle ScholarCrossref
Coleman  AL, Yu  F, Evans  SJ.  Use of gonioscopy in Medicare beneficiaries before glaucoma surgery.  J Glaucoma. 2006;15(6):486-493. doi:10.1097/01.ijg.0000212287.62798.8fPubMedGoogle ScholarCrossref
Coleman  AL, Yu  F, Rowe  S.  Visual field testing in glaucoma Medicare beneficiaries before surgery.  Ophthalmology. 2005;112(3):401-406. doi:10.1016/j.ophtha.2004.09.034PubMedGoogle ScholarCrossref
Stein  JD, Andrews  C, Musch  DC, Green  C, Lee  PP.  Sight-threatening ocular diseases remain underdiagnosed among children of less affluent families.  Health Aff (Millwood). 2016;35(8):1359-1366. doi:10.1377/hlthaff.2015.1007PubMedGoogle ScholarCrossref
Elam  AR, Andrews  C, Musch  DC, Lee  PP, Stein  JD.  Large disparities in receipt of glaucoma care between enrollees in Medicaid and those with commercial health insurance.  Ophthalmology. 2017;124(10):1442-1448. doi:10.1016/j.ophtha.2017.05.003PubMedGoogle ScholarCrossref
Stein  JD, Grossman  DS, Mundy  KM, Sugar  A, Sloan  FA.  Severe adverse events after cataract surgery among Medicare beneficiaries.  Ophthalmology. 2011;118(9):1716-1723. doi:10.1016/j.ophtha.2011.02.024PubMedGoogle ScholarCrossref
Atchison  EA, Wood  KM, Mattox  CG, Barry  CN, Lum  F, MacCumber  MW.  The real-world effect of intravitreous anti-vascular endothelial growth factor drugs on intraocular pressure: an analysis using the IRIS registry.  Ophthalmology. 2018;125(5):676-682. doi:10.1016/j.ophtha.2017.11.027PubMedGoogle ScholarCrossref
VanderBeek  BL, Bonaffini  SG, Ma  L.  Association of compounded bevacizumab with postinjection endophthalmitis.  JAMA Ophthalmol. 2015;133(10):1159-1164. doi:10.1001/jamaophthalmol.2015.2556PubMedGoogle ScholarCrossref
Gower  EW, Keay  LJ, Stare  DE,  et al.  Characteristics of endophthalmitis after cataract surgery in the United States Medicare population.  Ophthalmology. 2015;122(8):1625-1632. doi:10.1016/j.ophtha.2015.04.036PubMedGoogle ScholarCrossref
Stein  JD, Lum  F, Lee  PP, Rich  WL  III, Coleman  AL.  Use of health care claims data to study patients with ophthalmologic conditions.  Ophthalmology. 2014;121(5):1134-1141. doi:10.1016/j.ophtha.2013.11.038PubMedGoogle ScholarCrossref
Muir  KW, Gupta  C, Gill  P, Stein  JD.  Accuracy of International Classification of Diseases, Ninth Revision, Clinical Modification billing codes for common ophthalmic conditions.  JAMA Ophthalmol. 2013;131(1):119-120. doi:10.1001/jamaophthalmol.2013.577PubMedGoogle ScholarCrossref
Lau  M, Prenner  JL, Brucker  AJ, VanderBeek  BL.  Accuracy of billing codes used in the therapeutic care of diabetic retinopathy.  JAMA Ophthalmol. 2017;135(7):791-794. doi:10.1001/jamaophthalmol.2017.1595PubMedGoogle ScholarCrossref
Adler-Milstein  J, Jha  AK.  HITECH Act drove large gains in hospital electronic health record adoption.  Health Aff (Millwood). 2017;36(8):1416-1422. doi:10.1377/hlthaff.2016.1651PubMedGoogle ScholarCrossref
Lim  MC, Boland  MV, McCannel  CA,  et al.  Adoption of electronic health records and perceptions of financial and clinical outcomes among ophthalmologists in the United States.  JAMA Ophthalmol. 2018;136(2):164-170. doi:10.1001/jamaophthalmol.2017.5978PubMedGoogle ScholarCrossref
Liao  KP, Cai  T, Savova  GK,  et al.  Development of phenotype algorithms using electronic medical records and incorporating natural language processing.  BMJ. 2015;350:h1885. doi:10.1136/bmj.h1885PubMedGoogle ScholarCrossref
American Medical Association.  Physician ICD-9-CM 2006: International Classification of Diseases, 9th Revision, Clinical Modification. Vol 1. Chicago, IL: AMA Press; 2006.
American Medical Association. ICD-10-CM 2017: The Complete Official Code Book. Chicago, IL: AMA Press; 2016.
Anastasopoulos  E, Topouzis  F, Wilson  MR,  et al.  Characteristics of pseudoexfoliation in the Thessaloniki Eye Study.  J Glaucoma. 2011;20(3):160-166. doi:10.1097/IJG.0b013e3181d9d8bdPubMedGoogle ScholarCrossref
Ariga  M, Nivean  M, Utkarsha  P.  Pseudoexfoliation syndrome.  J Curr Glaucoma Pract. 2013;7(3):118-120. doi:10.5005/jp-journals-10008-1148PubMedGoogle ScholarCrossref
Hanauer  DA.  EMERSE: the electronic medical record search engine.  AMIA Annu Symp Proc. 2006;941.PubMedGoogle Scholar
Kim  SM, Kim  Y, Jeong  K, Jeong  H, Kim  J.  Logistic LASSO regression for the diagnosis of breast cancer using clinical demographic data and the BI-RADS lexicon for ultrasonography.  Ultrasonography. 2018;37(1):36-42. doi:10.14366/usg.16045PubMedGoogle ScholarCrossref
Naumann  GO, Schlötzer-Schrehardt  U, Küchle  M.  Pseudoexfoliation syndrome for the comprehensive ophthalmologist: intraocular and systemic manifestations.  Ophthalmology. 1998;105(6):951-968. doi:10.1016/S0161-6420(98)96020-1PubMedGoogle ScholarCrossref
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right

Name Your Search

Save Search
With a personal account, you can:
  • Track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience

Lookup An Activity



My Saved Searches

You currently have no searches saved.

With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right