Development and Validation of a Deep Learning System to Detect Glaucomatous Optic Neuropathy | Glaucoma | JN Learning | AMA Ed Hub [Skip to Content]
[Skip to Content Landing]

Development and Validation of a Deep Learning System to Detect Glaucomatous Optic Neuropathy Using Fundus Photographs

Educational Objective
To establish a deep learning system for detection of glaucomatous optic neuropathy using retinal fundus images and glaucoma diagnosis with convoluted neural networks that has the ability to be generalized across populations.
1 Credit CME
Key Points

Question  How does a deep learning system compare with professional human graders in detecting glaucomatous optic neuropathy?

Findings  In this cross-sectional study, the deep learning system showed a sensitivity and specificity of greater than 90% for detecting glaucomatous optic neuropathy in a local validation data set, in 3 clinical-based data sets, and in a real-world distribution data set. The deep learning system showed lower sensitivity when tested in multiethnic and website-based data sets.

Meaning  This assessment of fundus images suggests that deep learning systems can provide a tool with high sensitivity and specificity that might expedite screening for glaucomatous optic neuropathy.


Importance  A deep learning system (DLS) that could automatically detect glaucomatous optic neuropathy (GON) with high sensitivity and specificity could expedite screening for GON.

Objective  To establish a DLS for detection of GON using retinal fundus images and glaucoma diagnosis with convoluted neural networks (GD-CNN) that has the ability to be generalized across populations.

Design, Setting, and Participants  In this cross-sectional study, a DLS for the classification of GON was developed for automated classification of GON using retinal fundus images obtained from the Chinese Glaucoma Study Alliance, the Handan Eye Study, and online databases. The researchers selected 241 032 images were selected as the training data set. The images were entered into the databases on June 9, 2009, obtained on July 11, 2018, and analyses were performed on December 15, 2018. The generalization of the DLS was tested in several validation data sets, which allowed assessment of the DLS in a clinical setting without exclusions, testing against variable image quality based on fundus photographs obtained from websites, evaluation in a population-based study that reflects a natural distribution of patients with glaucoma within the cohort and an additive data set that has a diverse ethnic distribution. An online learning system was established to transfer the trained and validated DLS to generalize the results with fundus images from new sources. To better understand the DLS decision-making process, a prediction visualization test was performed that identified regions of the fundus images utilized by the DLS for diagnosis.

Exposures  Use of a deep learning system.

Main Outcomes and Measures  Area under the receiver operating characteristics curve (AUC), sensitivity and specificity for DLS with reference to professional graders.

Results  From a total of 274 413 fundus images initially obtained from CGSA, 269 601 images passed initial image quality review and were graded for GON. A total of 241 032 images (definite GON 29 865 [12.4%], probable GON 11 046 [4.6%], unlikely GON 200 121 [83%]) from 68 013 patients were selected using random sampling to train the GD-CNN model. Validation and evaluation of the GD-CNN model was assessed using the remaining 28 569 images from CGSA. The AUC of the GD-CNN model in primary local validation data sets was 0.996 (95% CI, 0.995-0.998), with sensitivity of 96.2% and specificity of 97.7%. The most common reason for both false-negative and false-positive grading by GD-CNN (51 of 119 [46.3%] and 191 of 588 [32.3%]) and manual grading (50 of 113 [44.2%] and 183 of 538 [34.0%]) was pathologic or high myopia.

Conclusions and Relevance  Application of GD-CNN to fundus images from different settings and varying image quality demonstrated a high sensitivity, specificity, and generalizability for detecting GON. These findings suggest that automated DLS could enhance current screening programs in a cost-effective and time-efficient manner.

Sign in to take quiz and track your certificates

Buy This Activity

JN Learning™ is the home for CME and MOC from the JAMA Network. Search by specialty or US state and earn AMA PRA Category 1 CME Credit™ from articles, audio, Clinical Challenges and more. Learn more about CME/MOC

Article Information

Accepted for Publication: July 14, 2019.

Published Online: September 12, 2019. doi:10.1001/jamaophthalmol.2019.3501

Correction: This article was corrected on December 1, 2019, to fix an error in the byline.

Corresponding Authors: Ningli Wang, MD, PhD, Beijing Tongren Hospital, Capital Medical University; Beijing Institute of Ophthalmology, No.1 Dongjiaominxiang Street, Dongcheng District, Beijing 100730, China (; Mai Xu, PhD, School of Electronic and Information Engineering, Beihang University, Beijing 100191, China (

Author Contributions: Drs H. Liu, M. Xu, and N. Wang had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs M. Xu and N. Wang contributed equally to this work.

Concept and design: H. Liu, Qiao, Zhang, H. Wang, Yang, Moghimi, Lai, Hu, Y. Xu, Kang, Ji, Tham, Ting, Wong, Z. Wang, M. Xu, N. Wang.

Acquisition, analysis, or interpretation of data: H. Liu, L. Liu, Wormstone, Qiao, P. Liu, Li, H. Wang, Mou, Pang, Yang, Zangwill, Hou, Bowd, Chen, Chang, Tham, Cheung, Wong, Weinreb, M. Xu.

Drafting of the manuscript: H. Liu, L. Liu, Qiao, Zhang, Li, Pang, Chen, Tham, M. Xu.

Critical revision of the manuscript for important intellectual content: H. Liu, Wormstone, Qiao, P. Liu, H. Wang, Mou, Yang, Zangwill, Moghimi, Hou, Bowd, Lai, Hu, Y. Xu, Kang, Ji, Chang, Tham, Cheung, Ting, Wong, Z. Wang, Weinreb, M. Xu, N. Wang.

Statistical analysis: H. Liu, L. Liu, Pang, Lai, Chen, Tham, Wong, M. Xu.

Obtained funding: H. Liu, Mou, Pang, Zangwill, Chen, Weinreb.

Administrative, technical, or material support: L. Liu, Qiao, Zhang, P. Liu, Li, H. Wang, Yang, Zangwill, Moghimi, Hou, Bowd, Chen, Hu, Y. Xu, Kang, Ji, Tham, Cheung, Z. Wang, Weinreb, M. Xu.

Supervision: P. Liu, Chen, Tham, Ting, Wong, Z. Wang, M. Xu, N. Wang.

Conflict of Interest Disclosures: Dr Zangwill reports grants from the National Eye Institute during the conduct of the study and research and equipment support from Heidelberg Engineering, Optovue, Carl Zeiss Meditec, and Topcon. Dr Ting reported having a patent pending for a deep learning system for retinal diseases, not related to this work. Dr Wong reported receiving personal fees from Allergan, personal fees from Bayer, personal fees from Boehringer Ingelheim, personal fees from Genentech, personal fees from Merck, personal fees from Novartis, personal fees from Oxurion, and personal fees from Roche outside the submitted work and he is a shareholder in Plano and EyRIS. No other disclosures were reported.

Funding/Support: The research has received funding from the National Natural Science Fund Projects of China (81271005), Beijing Municipal Administration of Hospitals Qingmiao Projects (QMS20180210), the Priming Scientific Research Foundation for the Junior Researcher in Beijing Tongren Hospital (Dr H. Liu; 2016-YJJ-ZZL-021), Beijing Tongren Hospital Top Talent Training Program, and Medical Synergy Science and Technology Innovation Research (Z181100001918035).

Role of the Funder/Sponsor: The funding organizations had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Tham  YC, Li  X, Wong  TY, Quigley  HA, Aung  T, Cheng  CY.  Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis.  Ophthalmology. 2014;121(11):2081-2090. doi:10.1016/j.ophtha.2014.05.013PubMedGoogle ScholarCrossref
Quigley  HA, Broman  AT.  The number of people with glaucoma worldwide in 2010 and 2020.  Br J Ophthalmol. 2006;90(3):262-267. doi:10.1136/bjo.2005.081224PubMedGoogle ScholarCrossref
Hood  DC, Raza  AS, de Moraes  CG, Liebmann  JM, Ritch  R.  Glaucomatous damage of the macula.  Prog Retin Eye Res. 2013;32:1-21. doi:10.1016/j.preteyeres.2012.08.003PubMedGoogle ScholarCrossref
Tatham  AJ, Weinreb  RN, Medeiros  FA.  Strategies for improving early detection of glaucoma: the combined structure-function index.  Clin Ophthalmol. 2014;8:611-621. doi:10.2147/OPTH.S44586PubMedGoogle Scholar
Weinreb  RN, Aung  T, Medeiros  FA.  The pathophysiology and treatment of glaucoma: a review.  JAMA. 2014;311(18):1901-1911. doi:10.1001/jama.2014.3192PubMedGoogle ScholarCrossref
Zhao  D, Guallar  E, Gajwani  P,  et al; SToP Glaucoma Study Group.  Optimizing glaucoma screening in high-risk population: design and 1-year findings of the screening to prevent (SToP) glaucoma study.  Am J Ophthalmol. 2017;180:18-28. doi:10.1016/j.ajo.2017.05.017PubMedGoogle ScholarCrossref
Fleming  C, Whitlock  EP, Beil  T, Smit  B, Harris  RP.  Screening for primary open-angle glaucoma in the primary care setting: an update for the US preventive services task force.  Ann Fam Med. 2005;3(2):167-170. doi:10.1370/afm.293PubMedGoogle ScholarCrossref
Esteva  A, Kuprel  B, Novoa  RA,  et al.  Dermatologist-level classification of skin cancer with deep neural networks.  Nature. 2017;542(7639):115-118. doi:10.1038/nature21056PubMedGoogle ScholarCrossref
Ehteshami Bejnordi  B, Veta  M, Johannes van Diest  P,  et al; the CAMELYON16 Consortium.  Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.  JAMA. 2017;318(22):2199-2210. doi:10.1001/jama.2017.14585PubMedGoogle ScholarCrossref
Korfiatis  P, Kline  TL, Coufalova  L,  et al.  MRI texture features as biomarkers to predict MGMT methylation status in glioblastomas.  Med Phys. 2016;43(6):2835-2844. doi:10.1118/1.4948668PubMedGoogle ScholarCrossref
Gulshan  V, Peng  L, Coram  M,  et al.  Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.  JAMA. 2016;316(22):2402-2410. doi:10.1001/jama.2016.17216PubMedGoogle ScholarCrossref
Ting  DSW, Cheung  CY, Lim  G,  et al.  Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes.  JAMA. 2017;318(22):2211-2223. doi:10.1001/jama.2017.18152PubMedGoogle ScholarCrossref
Li  Z, He  Y, Keel  S, Meng  W, Chang  RT, He  M.  Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs.  Ophthalmology. 2018;125(8):1199-1206. doi:10.1016/j.ophtha.2018.01.023PubMedGoogle ScholarCrossref
Wong  TY, Bressler  NM.  Artificial intelligence with deep learning technology looks into diabetic retinopathy screening.  JAMA. 2016;316(22):2366-2367. doi:10.1001/jama.2016.17563PubMedGoogle ScholarCrossref
Castelvecchi  D.  Can we open the black box of AI?  Nature. 2016;538(7623):20-23. doi:10.1038/538020aPubMedGoogle ScholarCrossref
Verghese  A, Shah  NH, Harrington  RA.  What this computer needs is a physician: humanism and artificial intelligence.  JAMA. 2018;319(1):19-20. doi:10.1001/jama.2017.19198PubMedGoogle ScholarCrossref
He  K, Zhang  X, Ren  S,  et al. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE; 2016:770-778. doi:10.1109/CVPR.2016.90
Haleem  MS, Han  L, van Hemert  J, Li  B.  Automatic extraction of retinal features from colour retinal images for glaucoma diagnosis: a review.  Comput Med Imaging Graph. 2013;37(7-8):581-596. doi:10.1016/j.compmedimag.2013.09.005PubMedGoogle ScholarCrossref
Szegedy  C, Liu  W, Jia  Y,  et al. Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE; 2015:1-9. doi:10.1109/CVPR.2015.7298594
Zeiler  MD, Fergus  R. Visualizing and understanding convolutional networks. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T, eds. Computer Vision—ECCV 2014. Cham, Switzerland: Springer; 2014:818-833. doi:10.1007/978-3-319-10590-1_53
Singh  A, Dutta  MK, ParthaSarathi  M, Uher  V, Burget  R.  Image processing based automatic diagnosis of glaucoma using wavelet features of segmented optic disc from fundus image.  Comput Methods Programs Biomed. 2016;124:108-120. doi:10.1016/j.cmpb.2015.10.010PubMedGoogle ScholarCrossref
Issac  A, Partha Sarathi  M, Dutta  MK.  An adaptive threshold based image processing technique for improved glaucoma detection and classification.  Comput Methods Programs Biomed. 2015;122(2):229-244. doi:10.1016/j.cmpb.2015.08.002PubMedGoogle ScholarCrossref
Chakrabarty  L, Joshi  GD, Chakravarty  A, Raman  GV, Krishnadas  SR, Sivaswamy  J.  Automated detection of glaucoma from topographic features of the optic nerve head in color fundus photographs.  J Glaucoma. 2016;25(7):590-597. doi:10.1097/IJG.0000000000000354PubMedGoogle ScholarCrossref
Xiangyu Chen, Yanwu Xu, Jiang Liu, Damon Wing Kee Wong, Tien Yin Wong.  Glaucoma detection based on deep convolutional neural network.  Conf Proc IEEE Eng Med Biol Soc. 2015;2015:715-718. doi:10.1109/EMBC.2015.7318462PubMedGoogle Scholar
Annan Li, Jun Cheng, Jiang Liu, Damon Wing Kee Wong.  Integrating holistic and local deep features for glaucoma classification.  Conf Proc IEEE Eng Med Biol Soc. 2016;2016:1328-1331. doi:10.1109/EMBC.2016.7590952PubMedGoogle Scholar
Christopher  M, Belghith  A, Bowd  C,  et al.  Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs.  Sci Rep. 2018;8(1):16685. doi:10.1038/s41598-018-35044-9PubMedGoogle ScholarCrossref
Shibata  N, Tanito  M, Mitsuhashi  K,  et al.  Development of a deep residual learning algorithm to screen for glaucoma from fundus photography.  Sci Rep. 2018;8(1):14665. doi:10.1038/s41598-018-33013-wPubMedGoogle ScholarCrossref
Meier  J, Bock  R, Michelson  G,  et al. Effects of preprocessing eye fundus images on appearance based glaucoma classification. Proceedings of the International Conference on Computer Analysis of Images and Patterns. Berlin, Germany: Springer; 2007:165-172. doi:10.1007/978-3-540-74272-2_21
Bock  R, Meier  J, Michelson  G,  et al. Classifying glaucoma with image-based features from fundus photographs. In: Hamprecht FA, Schnörr C, Jähne B, eds. Pattern Recognition. DAGM 2007. Heidelberg, Germany: Springer; 2007. doi:10.1007/978-3-540-74936-3_36
Bock  R, Meier  J, Nyúl  LG, Hornegger  J, Michelson  G.  Glaucoma risk index: automated glaucoma detection from color fundus images.  Med Image Anal. 2010;14(3):471-481. doi:10.1016/ ScholarCrossref
Keerthi  SS, Shevade  SK, Bhattacharyya  C,  et al.  Improvements to Platt’s SMO algorithm for SVM classifier design.  Neural Comput. 2014;13(3):637-649. doi:10.1162/089976601300014493Google ScholarCrossref
Wang  NL, Hao  J, Zhen  Y,  et al.  A population-based investigation of circadian rhythm of intraocular pressure in habitual position among healthy subjects: the Handan Eye Study.  J Glaucoma. 2016;25(7):584-589. doi:10.1097/IJG.0000000000000351PubMedGoogle ScholarCrossref
Zhang  Y, Li  SZ, Li  L, Thomas  R, Wang  NL.  The Handan Eye Study: comparison of screening methods for primary angle closure suspects in a rural Chinese population.  Ophthalmic Epidemiol. 2014;21(4):268-275. doi:10.3109/09286586.2014.929707PubMedGoogle ScholarCrossref
Cho  HK, Kee  C.  Population-based glaucoma prevalence studies in Asians.  Surv Ophthalmol. 2014;59(4):434-447. doi:10.1016/j.survophthal.2013.09.003PubMedGoogle ScholarCrossref
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right

Name Your Search

Save Search
With a personal account, you can:
  • Track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience

Lookup An Activity



My Saved Searches

You currently have no searches saved.

With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right