[Skip to Content]

Quiz

Surgery

Association Between Surgeon Technical Skills and Patient Outcomes

Educational Objective
To examine the variation in technical skills of practicing surgeons, the association between skills and patient outcomes, and the amount of variation in patient outcomes explained by technical skills.

1 Credit CME

JAMA Surgery

Published Online: August 19, 2020

Original Investigation

Article Information and Disclosures

Jonah J. Stulberg, MD, PhD, MPH^1,2;

Reiping Huang, PhD^1,2;

Lindsey Kreutzer, MPH^1,2;

Kristen Ban, MD^1,3;

Bradley J. Champagne, MD⁴;

Scott R. Steele, MD⁵;

Julie K. Johnson, PhD^1,2;

Jane L. Holl, MD, MPH⁶;

Caprice C. Greenberg, MD, MPH⁷;

Karl Y. Bilimoria, MD, MS^1,2,3

Author Affiliations

¹Surgical Outcomes and Quality Improvement Center, Feinberg School of Medicine, Department of Surgery, Northwestern University, Chicago, Illinois
²Center for Health Services and Outcomes Research, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
³American College of Surgeons, Chicago, Illinois
⁴Department of Colorectal Surgery, Cleveland Clinic, Fairview, Ohio
⁵Department of Colorectal Surgery, Cleveland Clinic, Cleveland, Ohio
⁶Biological Sciences Division, The University of Chicago, Chicago, Illinois
⁷Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison

Read article on JAMA Network

Read Article Take Quiz

Key Points

Question Is the technical skill of a surgeon associated with patient outcomes?

Findings In this quality improvement study including 17 practicing surgeons and outcome data from the American College of Surgeons National Surgical Quality Improvement Program as a reference group, better technical skill scores, assessed via intraoperative video, were statistically significantly associated with lower rates of any complication, unplanned reoperation, and death or serious morbidity. Overall, technical skill scores were associated with approximately 26% of the variation in risk-adjusted complication rates.

Meaning Quality improvement efforts aimed at improving patient outcomes should include a focus on improving surgeon-level technical skill.

Abstract

Importance Postoperative complications remain common after surgery, but little is known about the extent of variation in operative technical skill and whether variation is associated with patient outcomes.

Objectives To examine the (1) variation in technical skill scores of practicing surgeons, (2) association between technical skills and patient outcomes, and (3) amount of variation in patient outcomes explained by a surgeon’s technical skill.

Design, Setting, and Participants In this quality improvement study, 17 practicing surgeons submitted a video of a laparoscopic right hemicolectomy that was then rated by at least 10 blinded peer surgeons and 2 expert raters. The association between surgeon technical skill scores and risk-adjusted outcomes was examined using data from the American College of Surgeons National Surgical Quality Improvement Program. The association between technical skill scores and outcomes was examined for colorectal procedures and noncolorectal procedures (ie, assessed on whether technical skills demonstrated during colectomy were associated with patient outcomes across other cases). In addition, the proportion of patient outcomes explained by technical skill scores was examined using robust regression techniques. The study was conducted from September 23, 2016, to February 10, 2018; data analysis was performed from November 2018 to January 2019.

Exposures Colorectal and noncolorectal procedures.

Main Outcomes and Measures Any complication, mortality, unplanned hospital readmission, unplanned reoperation related to principal procedure, surgical site infection, and death or serious morbidity.

Results Of the 17 surgeons included in the study, 13 were men (76%). The participants had a range from 1 to 28 years in surgical practice (median, 11 years). Based on 10 or more reviewers per video and with a maximum quality score of 5, overall technical skill scores ranged from 2.8 to 4.6. From 2014 to 2016, study participants performed a total of 3063 procedures (1120 colectomies). Higher technical skill scores were significantly associated with lower rates of any complication (15.5% vs 20.6%, P = .03; Spearman rank-order correlation coefficient r = −0.54, P = .03), unplanned reoperation (4.7% vs 7.2%, P = .02; r = −0.60, P = .01), and a composite measure of death or serious morbidity (15.9% vs 21.4%, P = .02; r = −0.60, P = .01) following colectomy. Similar associations were found between colectomy technical skill scores and patient outcomes for all types of procedures performed by a surgeon. Overall, technical skill scores appeared to account for 25.8% of the variation in postcolectomy complication rates and 27.5% of the variation when including noncolectomy complication rates.

Conclusions and Relevance The findings of this study suggest that there is wide variation in technical skill among practicing surgeons, accounting for more than 25% of the variation in patient outcomes. Higher colectomy technical skill scores appear to be associated with lower complication rates for colectomy and for all other procedures performed by a surgeon. Efforts to improve surgeon technical skills may result in better patient outcomes.

Take Quiz

Sign in to take quiz and track your certificates

Buy This Activity

Article Information

Accepted for Publication: April 28, 2020.

Published Online: August 19, 2020. doi:10.1001/jamasurg.2020.3007

Corrections: This article was corrected on October 21, 2020, to fix an error in the author byline and on May 12, 2021, to fix an error in the Methods and Table 2.

Corresponding Author: Jonah J. Stulberg, MD, PhD, MPH, Surgical Outcomes and Quality Improvement Center, Feinberg School of Medicine, Department of Surgery, Northwestern University, 633 N St Clair St, 20th Floor, Chicago, IL 60610 (jonah.stulberg@northwestern.edu).

Author Contributions: Dr Stulberg had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Stulberg, Kreutzer, Ban, Champagne, Holl, Greenberg, Bilimoria.

Acquisition, analysis, or interpretation of data: Stulberg, Huang, Ban, Steele, Johnson, Holl, Greenberg, Bilimoria.

Drafting of the manuscript: Stulberg, Huang, Kreutzer, Holl, Bilimoria.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Stulberg, Huang, Bilimoria.

Obtained funding: Bilimoria.

Administrative, technical, or material support: Stulberg, Kreutzer, Ban, Champagne.

Supervision: Stulberg, Champagne, Steele, Bilimoria.

Conflict of Interest Disclosures: Dr Stulberg reported paid consultant services for Intuitive Surgical and receiving an Intuitive Foundation grant for unrelated work. Dr Greenberg reported receiving nonfinancial support from Johnson & Johnson outside the submitted work and is president and founder of the Academy for Surgical Coaching. No other disclosures were reported.

Meeting Presentation: This work was presented in part at the American College of Surgeons Clinical Congress; October 22, 2018; Boston, Massachusetts.

Additional Contributions: We thank the members of the Video-Based Coaching Collaborative for their efforts throughout this project. We also acknowledge the work of the Wisconsin Coaching Collaborative from the University of Wisconsin (Janet Dombrowski, MHSA, Caprice C. Greenberg, MD, MPH, and Sudha R. Pavuluri Quamme, MD, MS), who worked with us throughout this project on efforts to improve surgical care through coaching; they were paid as consultants to conduct the coach training.

Additional Information: Information about the Illinois Surgical Quality Improvement Collaborative can be found at http://www.ISQIC.org.

References

Archampong D , Borowski D , Wille-Jørgensen P , Iversen LH . Workload and surgeon’s specialty for outcome after colorectal cancer surgery. Cochrane Database Syst Rev. 2012;(3):CD005391. doi:10.1002/14651858.CD005391.pub3 PubMed Google Scholar

Hall BL , Huffman KM , Hamilton BH , et al. Profiling individual surgeon performance using information from a high-quality clinical registry: opportunities and limitations. J Am Coll Surg. 2015;221(5):901-913. doi:10.1016/j.jamcollsurg.2015.07.454 PubMed Google Scholar Crossref

Quinn CM , Bilimoria KY , Chung JW , Ko CY , Cohen ME , Stulberg JJ . Creating individual surgeon performance assessments in a statewide hospital surgical quality improvement collaborative. J Am Coll Surg. 2018;227(3):303-312.e3. doi:10.1016/j.jamcollsurg.2018.06.002 PubMed Google Scholar Crossref

Keller DS , Delaney CP , Hashemi L , Haas EM . A national evaluation of clinical and economic outcomes in open versus laparoscopic colorectal surgery. Surg Endosc. 2016;30(10):4220-4228. doi:10.1007/s00464-015-4732-6 PubMed Google Scholar Crossref

Birkmeyer JD , Finks JF , O’Reilly A , et al; Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369(15):1434-1442. doi:10.1056/NEJMsa1300625 PubMed Google Scholar Crossref

de Montbrun SL , Roberts PL , Lowry AC , et al. A novel approach to assessing technical competence of colorectal surgery residents: the development and evaluation of the Colorectal Objective Structured Assessment of Technical Skill (COSATS). Ann Surg. 2013;258(6):1001-1006. doi:10.1097/SLA.0b013e31829b32b8 PubMed Google Scholar Crossref

Martin JA , Regehr G , Reznick R , et al. Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273-278. doi:10.1002/bjs.1800840237 PubMed Google Scholar

de Montbrun S , Roberts PL , Satterthwaite L , MacRae H . Implementing and evaluating a national certification technical skills examination: the Colorectal Objective Structured Assessment of Technical Skill. Ann Surg. 2016;264(1):1-6. doi:10.1097/SLA.0000000000001620 PubMed Google Scholar Crossref

Champagne BJ , Steele SR , Hendren SK , et al. The American Society of Colon and Rectal Surgeons Assessment tool for performance of laparoscopic colectomy. Dis Colon Rectum. 2017;60(7):738-744. doi:10.1097/DCR.0000000000000817 PubMed Google Scholar Crossref

Merkow RP , Ju MH , Chung JW , et al. Underlying reasons associated with hospital readmission following surgery in the United States. JAMA. 2015;313(5):483-495. doi:10.1001/jama.2014.18614 PubMed Google Scholar Crossref

Huffman KM , Cohen ME , Ko CY , Hall BL . A comprehensive evaluation of statistical reliability in ACS NSQIP profiling models. Ann Surg. 2015;261(6):1108-1113. doi:10.1097/SLA.0000000000000913 PubMed Google Scholar Crossref

Merkow RP , Hall BL , Cohen ME , et al. Validity and feasibility of the American College of Surgeons colectomy composite outcome quality measure. Ann Surg. 2013;257(3):483-489. doi:10.1097/SLA.0b013e318273bf17 PubMed Google Scholar Crossref

Cohen ME , Ko CY , Bilimoria KY , et al. Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus. J Am Coll Surg. 2013;217(2):336-346.e1. doi:10.1016/j.jamcollsurg.2013.02.027 PubMed Google Scholar Crossref

Varban OA , Greenberg CC , Schram J , et al; Michigan Bariatric Surgery Collaborative. Surgical skill in bariatric surgery: does skill in one procedure predict outcomes for another? Surgery. 2016;160(5):1172-1181. doi:10.1016/j.surg.2016.04.033 PubMed Google Scholar Crossref

Fecso AB , Szasz P , Kerezov G , Grantcharov TP . The effect of technical performance on patient outcomes in surgery: a systematic review. Ann Surg. 2017;265(3):492-501. doi:10.1097/SLA.0000000000001959 PubMed Google Scholar Crossref

Holst D , Kowalewski TM , White LW , et al. Crowd-sourced assessment of technical skills: differentiating animate surgical skill through the wisdom of crowds. J Endourol. 2015;29(10):1183-1188. doi:10.1089/end.2015.0104 PubMed Google Scholar Crossref

AMA CME Accreditation Information

Credit Designation Statement: The American Medical Association designates this Journal-based CME activity activity for a maximum of 1.00  AMA PRA Category 1 Credit(s)™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.

Successful completion of this CME activity, which includes participation in the evaluation component, enables the participant to earn up to:

1.00 Medical Knowledge MOC points in the American Board of Internal Medicine's (ABIM) Maintenance of Certification (MOC) program;;
1.00 Self-Assessment points in the American Board of Otolaryngology – Head and Neck Surgery’s (ABOHNS) Continuing Certification program;
1.00 MOC points in the American Board of Pediatrics’ (ABP) Maintenance of Certification (MOC) program;
1.00 Lifelong Learning points in the American Board of Pathology’s (ABPath) Continuing Certification program; and
1.00 CME points in the American Board of Surgery’s (ABS) Continuing Certification program

It is the CME activity provider's responsibility to submit participant completion information to ACCME for the purpose of granting MOC credit.