[Skip to Content]
[Skip to Content Landing]

A Quantitative Severity Scale for Retinopathy of Prematurity Using Deep Learning to Monitor Disease Regression After Treatment

Educational Objective To apply an automated retinopathy of prematurity (ROP) vascular severity score obtained using a deep learning algorithm and to assess its utility for objectively monitoring ROP regression after treatment.
1 Credit CME
Key Points

Question  Can a quantitative measurement of retinopathy of prematurity severity be used over time to monitor disease regression after treatment?

Findings  In this cohort study of at-risk infants using data collected for the Imaging and Informatics in Retinopathy of Prematurity study, the quantitative retinopathy of prematurity vascular severity score developed using an automated deep learning–based plus disease classifier consistently reflected clinical disease posttreatment regression in 46 included eyes with laser or bevacizumab treatment.

Meaning  Tracking quantitative measurements of retinopathy of prematurity severity may be an effective method of following disease regression and identifying patients at risk for recurrence after retinopathy of prematurity treatment.

Abstract

Importance  Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide, but treatment failure and disease recurrence are important causes of adverse outcomes in patients with treatment-requiring ROP (TR-ROP).

Objectives  To apply an automated ROP vascular severity score obtained using a deep learning algorithm and to assess its utility for objectively monitoring ROP regression after treatment.

Design, Setting, and Participants  This retrospective cohort study used data from the Imaging and Informatics in ROP consortium, which comprises 9 tertiary referral centers in North America that screen high volumes of at-risk infants for ROP. Images of 5255 clinical eye examinations from 871 infants performed between July 2011 and December 2016 were assessed for eligibility in the present study. The disease course was assessed with time across the numerous examinations for patients with TR-ROP. Infants born prematurely meeting screening criteria for ROP who developed TR-ROP and who had images captured within 4 weeks before and after treatment as well as at the time of treatment were included.

Main Outcomes and Measures  The primary outcome was mean (SD) ROP vascular severity score before, at time of, and after treatment. A deep learning classifier was used to assign a continuous ROP vascular severity score, which ranged from 1 (normal) to 9 (most severe), at each examination. A secondary outcome was the difference in ROP vascular severity score among eyes treated with laser or the vascular endothelial growth factor antagonist bevacizumab. Differences between groups for both outcomes were assessed using unpaired 2-tailed t tests with Bonferroni correction.

Results  Of 5255 examined eyes, 91 developed TR-ROP, of which 46 eyes met the inclusion criteria based on the available images. The mean (SD) birth weight of those patients was 653 (185) g, with a mean (SD) gestational age of 24.9 (1.3) weeks. The mean (SD) ROP vascular severity scores significantly increased 2 weeks prior to treatment (4.19 [1.75]), peaked at treatment (7.43 [1.89]), and decreased for at least 2 weeks after treatment (4.00 [1.88]) (all P < .001). Eyes requiring retreatment with laser had higher ROP vascular severity scores at the time of initial treatment compared with eyes receiving a single treatment (P < .001).

Conclusions and Relevance  This quantitative ROP vascular severity score appears to consistently reflect clinical disease progression and posttreatment regression in eyes with TR-ROP. These study results may have implications for the monitoring of patients with ROP for treatment failure and disease recurrence and for determining the appropriate level of disease severity for primary treatment in eyes with aggressive disease.

Sign in to take quiz and track your certificates

Buy This Activity

JN Learning™ is the home for CME and MOC from the JAMA Network. Search by specialty or US state and earn AMA PRA Category 1 CME Credit™ from articles, audio, Clinical Challenges and more. Learn more about CME/MOC

Article Information

Accepted for Publication: April 14, 2019.

Corresponding Author: Michael F. Chiang, MD, Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, 3375 SW Terwilliger Blvd, Portland, OR 97239 (chiangm@ohsu.edu).

Published Online: July 3, 2019. doi:10.1001/jamaophthalmol.2019.2442

Author Contributions: Drs Gupta and Campbell contributed equally to this work and are considered co–first authors. Dr Chiang had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Campbell, Taylor, Erdogmus, Ioannidis, Kalpathy-Cramer, Kim, Chiang.

Acquisition, analysis, or interpretation of data: Gupta, Campbell, Taylor, Brown, Ostmo, Chan, Dy, Kalpathy-Cramer, Kim, Chiang.

Drafting of the manuscript: Gupta, Campbell, Brown.

Critical revision of the manuscript for important intellectual content: Campbell, Taylor, Ostmo, Chan, Dy, Erdogmus, Ioannidis, Kalpathy-Cramer, Kim, Chiang.

Statistical analysis: Gupta, Campbell, Chiang.

Obtained funding: Erdogmus, Ioannidis, Kalpathy-Cramer, Chiang.

Administrative, technical, or material support: Gupta, Campbell, Brown, Ostmo, Kalpathy-Cramer, Chiang.

Supervision: Campbell, Chan, Dy, Erdogmus, Kalpathy-Cramer, Chiang.

Conflict of Interest Disclosures: Dr Campbell reported receiving grants from Genentech and personal fees from Allergan outside the submitted work. Dr Chan reported receiving personal fees from Alcon, Allergan, Visunex Medical Systems, Beyeonics, and Genentech outside the submitted work. Dr Dy reported receiving grants from the National Science Foundation. Dr Ioannidis reported receiving grants from National Science Foundation and from the National Institutes of Health during the conduct of the study. Dr Kalpathy-Cramer reported receiving grants from the National Eye Institute and the National Science Foundation during the conduct of the study and receiving personal fees from INFOTECHSoft outside the submitted work. Dr Chiang reported receiving grants from the National Institutes of Health, the National Science Foundation, and Genentech; receiving nonfinancial support from Clarity Medical Systems; receiving personal fees from Novartis during the conduct of the study; and being an equity owner in Initeleretina outside the submitted work. Drs Campbell and Brown, Ms Ostmo, and Drs Chan, Dy, Erdogmus, Ioannidis, Kalpathy-Cramer, and Chiang have a patent application pending on the described technology.

Funding/Support: This project was supported by grants R01EY19474, K12EY027720, and P30EY10572 from the National Institutes of Health; by grants SCH-1622679, SCH-1622542, and SCH-1622536 from the National Science Foundation; and by Research to Prevent Blindness in the form of unrestricted departmental funding to the Casey Institute and a Career Development Award (Dr Campbell).

Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Group Information: The members of the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Consortium follow. Oregon Health & Science University: Michael F. Chiang, MD, Susan Ostmo, MS, Sang Jin Kim, MD, PhD, Kemal Sonmez, PhD, and J. Peter Campbell, MD, MPH. University of Illinois at Chicago: R. V. Paul Chan, MD and Karyn Jonas, RN. Columbia University: Jason Horowitz, MD, Osode Coki, RN, Cheryl-Ann Eccles, RN, and Leora Sarna, RN. Weill Cornell Medical College: Anton Orlin, MD. Bascom Palmer Eye Institute: Audina Berrocal, MD, and Catherin Negron, BA. William Beaumont Hospital: Kimberly Denser, MD, Kristi Cumming, RN, Tammy Osentoski, RN, Tammy Check, RN, and Mary Zajechowski, RN. Children’s Hospital Los Angeles: Thomas Lee, MD, Evan Kruger, BA, and Kathryn McGovern, MPH. Cedars Sinai Hospital: Charles Simmons, MD, Raghu Murthy, MD, and Sharon Galvis, NNP. LA Biomedical Research Institute: Jerome Rotter, MD, Ida Chen, PhD, Xiaohui Li, MD, Kent Taylor, PhD, and Kaye Roll, RN. Massachusetts General Hospital: Jayashree Kalpathy-Cramer, PhD. Northeastern University: Deniz Erdogmus, PhD, and Stratis Ioannidis, PhD. Asociacion para Evitar la Ceguera en Mexico: Maria Ana Martinez-Castellanos, MD, Samantha Salinas-Longoria, MD, Rafael Romero, MD, Andrea Arriola, MD, Francisco Olguin-Manriquez, MD, Miroslava Meraz-Gutierrez, MD, Carlos M. Dulanto-Reinoso, MD, and Cristina Montero-Mendoza, MD.

Meeting Presentation: This paper was presented at the Annual Meeting of the Association for Research in Vision and Ophthalmology; May 1, 2018; Honolulu, Hawaii.

References
1.
Cryotherapy for Retinopathy of Prematurity Cooperative Group.  Multicenter trial of cryotherapy for retinopathy of prematurity: preliminary results.  Arch Ophthalmol. 1988;106(4):471-479. doi:10.1001/archopht.1988.01060130517027PubMedGoogle ScholarCrossref
2.
Gilbert  C.  Retinopathy of prematurity: a global perspective of the epidemics, population of babies at risk and implications for control.  Early Hum Dev. 2008;84(2):77-82. doi:10.1016/j.earlhumdev.2007.11.009PubMedGoogle ScholarCrossref
3.
The Committee for the Classification of Retinopathy of Prematurity.  An international classification of retinopathy of prematurity.  Arch Ophthalmol. 1984;102(8):1130-1134. doi:10.1001/archopht.1984.01040030908011PubMedGoogle ScholarCrossref
4.
International Committee for the Classification of Retinopathy of Prematurity.  The international classification of retinopathy of prematurity revisited.  Arch Ophthalmol. 2005;123(7):991-999. doi:10.1001/archopht.123.7.991PubMedGoogle ScholarCrossref
5.
Early Treatment For Retinopathy Of Prematurity Cooperative Group.  Revised indications for the treatment of retinopathy of prematurity: results of the early treatment for retinopathy of prematurity randomized trial.  Arch Ophthalmol. 2003;121(12):1684-1694. doi:10.1001/archopht.121.12.1684PubMedGoogle ScholarCrossref
6.
Mintz-Hittner  HA, Kennedy  KA, Chuang  AZ; BEAT-ROP Cooperative Group.  Efficacy of intravitreal bevacizumab for stage 3+ retinopathy of prematurity.  N Engl J Med. 2011;364(7):603-615. doi:10.1056/NEJMoa1007374PubMedGoogle ScholarCrossref
7.
Fierson  WM; American Academy of Pediatrics Section on Ophthalmology; American Academy of Ophthalmology; American Association for Pediatric Ophthalmology and Strabismus; American Association of Certified Orthoptists.  Screening examination of premature infants for retinopathy of prematurity.  Pediatrics. 2018;142(6):e20183061. doi:10.1542/peds.2018-3061PubMedGoogle ScholarCrossref
8.
Reynolds  JD, Dobson  V, Quinn  GE,  et al; CRYO-ROP and LIGHT-ROP Cooperative Study Groups.  Evidence-based screening criteria for retinopathy of prematurity: natural history data from the CRYO-ROP and LIGHT-ROP studies.  Arch Ophthalmol. 2002;120(11):1470-1476. doi:10.1001/archopht.120.11.1470PubMedGoogle ScholarCrossref
9.
Mintz-Hittner  HA.  Retinopathy of prematurity: intravitreal injections of bevacizumab: timing, technique, and outcomes.  J AAPOS. 2016;20(6):478-480. doi:10.1016/j.jaapos.2016.10.002PubMedGoogle ScholarCrossref
10.
Darwish  D, Chee  R-I, Patel  SN,  et al.  Anti-Vascular endothelial growth factor and the evolving management paradigm for retinopathy of prematurity.  Asia Pac J Ophthalmol (Phila). 2018;7(3):136-144. doi:10.22608/APO.201850PubMedGoogle Scholar
11.
Fleck  BW, Williams  C, Juszczak  E,  et al; BOOST II Retinal Image Digital Analysis (RIDA) Group.  An international comparison of retinopathy of prematurity grading performance within the Benefits of Oxygen Saturation Targeting II trials.  Eye (Lond). 2018;32(1):74-80. doi:10.1038/eye.2017.150PubMedGoogle ScholarCrossref
12.
Campbell  JP, Kalpathy-Cramer  J, Erdogmus  D,  et al; Imaging and Informatics in Retinopathy of Prematurity Research Consortium.  Plus disease in retinopathy of prematurity: a continuous spectrum of vascular abnormality as a basis of diagnostic variability.  Ophthalmology. 2016;123(11):2338-2344. doi:10.1016/j.ophtha.2016.07.026PubMedGoogle ScholarCrossref
13.
Kalpathy-Cramer  J, Campbell  JP, Erdogmus  D,  et al; Imaging and Informatics in Retinopathy of Prematurity Research Consortium.  Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis.  Ophthalmology. 2016;123(11):2345-2351. doi:10.1016/j.ophtha.2016.07.020PubMedGoogle ScholarCrossref
14.
Campbell  JP, Ataer-Cansizoglu  E, Bolon-Canedo  V,  et al; Imaging and Informatics in ROP (i-ROP) Research Consortium.  Expert diagnosis of plus disease in retinopathy of prematurity from computer-based image analysis.  JAMA Ophthalmol. 2016;134(6):651-657. doi:10.1001/jamaophthalmol.2016.0611PubMedGoogle ScholarCrossref
15.
Chiang  MF, Thyparampil  PJ, Rabinowitz  D.  Interexpert agreement in the identification of macular location in infants at risk for retinopathy of prematurity.  Arch Ophthalmol. 2010;128(9):1153-1159. doi:10.1001/archophthalmol.2010.199PubMedGoogle ScholarCrossref
16.
Chiang  MF, Jiang  L, Gelman  R, Du  YE, Flynn  JT.  Interexpert agreement of plus disease diagnosis in retinopathy of prematurity.  Arch Ophthalmol. 2007;125(7):875-880. doi:10.1001/archopht.125.7.875PubMedGoogle ScholarCrossref
17.
Campbell  JP, Ryan  MC, Lore  E,  et al; Imaging & Informatics in Retinopathy of Prematurity Research Consortium.  Diagnostic discrepancies in retinopathy of prematurity classification.  Ophthalmology. 2016;123(8):1795-1801. doi:10.1016/j.ophtha.2016.04.035PubMedGoogle ScholarCrossref
18.
Quinn  GE, Ells  A, Capone  A  Jr,  et al; e-ROP (Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity) Cooperative Group.  Analysis of discrepancy between diagnostic clinical examination findings and corresponding evaluation of digital images in the Telemedicine Approaches to Evaluating Acute-Phase Retinopathy of Prematurity study.  JAMA Ophthalmol. 2016;134(11):1263-1270. doi:10.1001/jamaophthalmol.2016.3502PubMedGoogle ScholarCrossref
19.
Scott  KE, Kim  DY, Wang  L,  et al.  Telemedical diagnosis of retinopathy of prematurity intraphysician agreement between ophthalmoscopic examination and image-based interpretation.  Ophthalmology. 2008;115(7):1222-1228. doi:10.1016/j.ophtha.2007.09.006PubMedGoogle ScholarCrossref
20.
Mintz-Hittner  HA, Geloneck  MM, Chuang  AZ.  Clinical management of recurrent retinopathy of prematurity after intravitreal bevacizumab monotherapy.  Ophthalmology. 2016;123(9):1845-1855. doi:10.1016/j.ophtha.2016.04.028PubMedGoogle ScholarCrossref
21.
Mueller  B, Salchow  DJ, Waffenschmidt  E,  et al.  Treatment of type I ROP with intravitreal bevacizumab or laser photocoagulation according to retinal zone.  Br J Ophthalmol. 2017;101(3):365-370. doi:10.1136/bjophthalmol-2016-308375PubMedGoogle Scholar
22.
Wittenberg  LA, Jonsson  NJ, Chan  RVP, Chiang  MF.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity.  J Pediatr Ophthalmol Strabismus. 2012;49(1):11-19. doi:10.3928/01913913-20110222-01PubMedGoogle ScholarCrossref
23.
Wallace  DK.  Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis).  Trans Am Ophthalmol Soc. 2007;105:594-615.PubMedGoogle Scholar
24.
Ataer-Cansizoglu  E, Bolon-Canedo  V, Campbell  JP,  et al; i-ROP Research Consortium.  Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity: performance of the “i-ROP” system and image features associated with expert diagnosis.  Transl Vis Sci Technol. 2015;4(6):5. doi:10.1167/tvst.4.6.5PubMedGoogle ScholarCrossref
25.
Brown  JM, Campbell  JP, Beers  A,  et al; Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium.  Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks.  JAMA Ophthalmol. 2018;136(7):803-810. doi:10.1001/jamaophthalmol.2018.1934PubMedGoogle ScholarCrossref
26.
Taylor  S, Brown  JM, Gupta  K,  et al; Imaging and Informatics in Retinopathy of Prematurity Consortium.  Monitoring disease progression with a quantitative severity scale for retinopathy of prematurity using deep learning  [published online July 3, 2019].  JAMA Ophthalmol. doi:10.1001/jamaophthalmol.2019.2433Google Scholar
27.
World Medical Association.  World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects.  JAMA. 2013;310(20):2191-2194. doi:10.1001/jama.2013.281053PubMedGoogle ScholarCrossref
28.
Ryan  MC, Ostmo  S, Jonas  K,  et al.  Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology.  AMIA Annu Symp Proc. 2014;2014:1902-1910.PubMedGoogle Scholar
29.
Redd  TK, Campbell  JP, Brown  JM,  et al; Imaging and Informatics in Retinopathy of Prematurity (i-ROP) Research Consortium.  Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity.  [published online November 23, 2018].  Br J Ophthalmol. 2018;bjophthalmol-2018-313156. doi:10.1136/bjophthalmol-2018-313156PubMedGoogle Scholar
30.
Gupta  MP, Chan  RVP, Anzures  R, Ostmo  S, Jonas  K, Chiang  MF; Imaging & Informatics in ROP Research Consortium.  Practice patterns in retinopathy of prematurity treatment for disease milder than recommended by guidelines.  Am J Ophthalmol. 2016;163:1-10. doi:10.1016/j.ajo.2015.12.005PubMedGoogle ScholarCrossref
31.
Walz  JM, Bemme  S, Reichl  S,  et al; Retina.net ROP-Register-Studiengruppe.  Treated cases of retinopathy of prematurity in Germany: 5-year data from the Retina.net ROP registry  [in German].  Ophthalmologe. 2018;115(6):476-488. doi:10.1007/s00347-018-0701-5PubMedGoogle ScholarCrossref
32.
VanderVeen  DK, Melia  M, Yang  MB, Hutchinson  AK, Wilson  LB, Lambert  SR.  Anti-vascular endothelial growth factor therapy for primary treatment of type 1 retinopathy of prematurity: a report by the American Academy of Ophthalmology.  Ophthalmology. 2017;124(5):619-633. doi:10.1016/j.ophtha.2016.12.025PubMedGoogle ScholarCrossref
33.
Sankar  MJ, Sankar  J, Chandra  P.  Anti-vascular endothelial growth factor (VEGF) drugs for treatment of retinopathy of prematurity.  Cochrane Database Syst Rev. 2018;1(1):CD009734. doi:10.1002/14651858.CD009734.pub3PubMedGoogle Scholar
34.
Hewing  NJ, Kaufman  DR, Chan  RVP, Chiang  MF.  Plus disease in retinopathy of prematurity: qualitative analysis of diagnostic process by experts.  JAMA Ophthalmol. 2013;131(8):1026-1032. doi:10.1001/jamaophthalmol.2013.135PubMedGoogle ScholarCrossref
35.
Brown  JM, Campbell  JP, Beers  A,  et al. Fully automated disease severity assessment and treatment monitoring in retinopathy of prematurity using deep learning. In: Zhang  J, Chen  P-H, eds.  Proceedings of SPIE 10579, Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications. Bellingham, WA: SPIE; 2018, doi:10.1117/12.2295942.
36.
Hartnett  ME.  Pathophysiology and mechanisms of severe retinopathy of prematurity.  Ophthalmology. 2015;122(1):200-210. doi:10.1016/j.ophtha.2014.07.050PubMedGoogle ScholarCrossref
37.
Hajrasouliha  AR, Garcia-Gonzales  JM, Shapiro  MJ, Yoon  H, Blair  MP.  Reactivation of retinopathy of prematurity three years after treatment with bevacizumab.  Ophthalmic Surg Lasers Imaging Retina. 2017;48(3):255-259. doi:10.3928/23258160-20170301-10PubMedGoogle ScholarCrossref
38.
Snyder  LL, Garcia-Gonzalez  JM, Shapiro  MJ, Blair  MP.  Very late reactivation of retinopathy of prematurity after monotherapy with intravitreal bevacizumab.  Ophthalmic Surg Lasers Imaging Retina. 2016;47(3):280-283. doi:10.3928/23258160-20160229-12PubMedGoogle ScholarCrossref
39.
Kang  KB, Orlin  A, Lee  TC, Chiang  MF, Chan  RVP.  The use of digital imaging in the identification of skip areas after laser treatment for retinopathy of prematurity and its implications for education and patient care.  Retina. 2013;33(10):2162-2169. doi:10.1097/IAE.0b013e31828e6969PubMedGoogle ScholarCrossref
40.
Ting  DSW, Wu  W-C, Toth  C.  Deep learning for retinopathy of prematurity screening.  [published online November 23, 2018].  Br J Ophthalmol. 2018;bjophthalmol-2018-313290. doi:10.1136/bjophthalmol-2018-313290PubMedGoogle Scholar
41.
Ting  DSW, Pasquale  LR, Peng  L,  et al.  Artificial intelligence and deep learning in ophthalmology.  Br J Ophthalmol. 2019;103(2):167-175. doi:10.1136/bjophthalmol-2018-313173PubMedGoogle ScholarCrossref
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
jn-learning_Modal_LoginSubscribe_Purchase
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
jn-learning_Modal_LoginSubscribe_Purchase
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right

Name Your Search

Save Search
With a personal account, you can:
  • Track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
jn-learning_Modal_SaveSearch_NoAccess_Purchase

Lookup An Activity

or

My Saved Searches

You currently have no searches saved.

With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right
Topics
State Requirements