[Skip to Content]
[Skip to Content Landing]

Adjusted Analyses in Studies Addressing Therapy and HarmUsers’ Guides to the Medical Literature

Educational Objective
To understand fundamental concepts underlying adjusted analyses in observational studies.
1 Credit CME

Observational studies almost always have bias because prognostic factors are unequally distributed between patients exposed or not exposed to an intervention. The standard approach to dealing with this problem is adjusted or stratified analysis. Its principle is to use measurement of risk factors to create prognostically homogeneous groups and to combine effect estimates across groups.

The purpose of this Users’ Guide is to introduce readers to fundamental concepts underlying adjustment as a way of dealing with prognostic imbalance and to the basic principles and relative trustworthiness of various adjustment strategies.

One alternative to the standard approach is propensity analysis, in which groups are matched according to the likelihood of membership in exposed or unexposed groups. Propensity methods can deal with multiple prognostic factors, even if there are relatively few patients having outcome events. However, propensity methods do not address other limitations of traditional adjustment: investigators may not have measured all relevant prognostic factors (or not accurately), and unknown factors may bias the results.

A second approach, instrumental variable analysis, relies on identifying a variable associated with the likelihood of receiving the intervention but not associated with any prognostic factor or with the outcome (other than through the intervention); this could mimic randomization. However, as with assumptions of other adjustment approaches, it is never certain if an instrumental variable analysis eliminates bias.

Although all these approaches can reduce the risk of bias in observational studies, none replace the balance of both known and unknown prognostic factors offered by randomization.

Sign in to take quiz and track your certificates

Buy This Activity

JN Learning™ is the home for CME and MOC from the JAMA Network. Search by specialty or US state and earn AMA PRA Category 1 CME Credit™ from articles, audio, Clinical Challenges and more. Learn more about CME/MOC

Article Information

Corresponding Author: Thomas Agoritsas, MD, PhD, Divisions of Clinical Epidemiology and General Internal Medicine, Department of Internal Medicine, Rehabilitation and Geriatrics (DMIRG), University Hospitals of Geneva, Rue Gabrielle-Perret-Gentil 4, 1211 Genève 14, Switzerland (thomas.agoritsas@gmail.com).

Author Contributions: Drs Agoritsas and Guyatt had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: Dr Agoritsas was financially supported by the Fellowship for Prospective Researchers grant P3SMP3-155290/1 from the Swiss National Science Foundation.

Role of the Funder/Sponsor: The Swiss National Science Foundation had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Rathore  SS, Epstein  AJ, Volpp  KG, Krumholz  HM.  Regionalization of care for acute coronary syndromes.  JAMA. 2005;293(11):1383-1387.PubMedGoogle ScholarCrossref
Mehta  SR, Cannon  CP, Fox  KA,  et al.  Routine vs selective invasive strategies in patients with acute coronary syndromes.  JAMA. 2005;293(23):2908-2917.PubMedGoogle ScholarCrossref
Stukel  TA, Fisher  ES, Wennberg  DE,  et al.  Analysis of observational studies in the presence of treatment selection bias.  JAMA. 2007;297(3):278-285.PubMedGoogle ScholarCrossref
Del Fiol  G, Workman  TE, Gorman  PN.  Clinical questions raised by clinicians at the point of care.  JAMA Intern Med. 2014;174(5):710-718.PubMedGoogle ScholarCrossref
Walsh  M, Perkovic  V, Manns  B,  et al. Therapy (randomized trials). In: Guyatt  G, Rennie  D, Meade  MO, Cook  DJ, eds.  Users’ Guides to the Medical Literature. 3rd ed. New York, NY: McGraw-Hill; 2015.
Sullivan  P, Goldmann  D.  The promise of comparative effectiveness research.  JAMA. 2011;305(4):400-401.PubMedGoogle ScholarCrossref
Hochman  M, McCormick  D.  Characteristics of published comparative effectiveness studies of medications.  JAMA. 2010;303(10):951-958.PubMedGoogle ScholarCrossref
Demaria  AN.  Comparative effectiveness research.  J Am Coll Cardiol. 2009;53(11):973-975.PubMedGoogle ScholarCrossref
Dahabreh  IJ, Kent  DM.  Can the learning health care system be educated with observational data?  JAMA. 2014;312(2):129-130.PubMedGoogle ScholarCrossref
Hemkens  LG, Contopoulos-Ioannidis  DG, Ioannidis  JP.  Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials.  BMJ. 2016;352:i493.PubMedGoogle ScholarCrossref
Kennedy  CC, Jaeschke  R, Keitz  S,  et al.  Tips for teachers of evidence-based medicine: adjusting for prognostic imbalances (confounding variables) in studies on therapy or harm.  J Gen Intern Med. 2008;23(3):337-343.PubMedGoogle ScholarCrossref
Ioannidis  JP, Haidich  AB, Pappa  M,  et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies.  JAMA. 2001;286(7):821-830.PubMedGoogle ScholarCrossref
Saquib  N, Saquib  J, Ioannidis  JP.  Practices and impact of primary outcome adjustment in randomized controlled trials.  BMJ. 2013;347:f4313.PubMedGoogle ScholarCrossref
Levine  M, Walter  S, Lee  H, Haines  T, Holbrook  A, Moyer  V.  Users’ Guides to the Medical Literature, IV: how to use an article about harm.  JAMA. 1994;271(20):1615-1619.PubMedGoogle ScholarCrossref
Brignardello-Petersen  R, Ioannidis  JPA, Tomlinson  G, Guyatt  G. Surprising results of randomized trials. In: Guyatt  G, Rennie  D, Meade  MO, Cook  DJ, eds.  Users’ Guides to the Medical Literature. 3rd ed. New York, NY: McGraw-Hill; 2015.
Guyatt  GH, Oxman  AD, Vist  GE,  et al; GRADE Working Group.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.  BMJ. 2008;336(7650):924-926.PubMedGoogle ScholarCrossref
Stanner  SA, Hughes  J, Kelly  CN, Buttriss  J.  A review of the epidemiological evidence for the “antioxidant hypothesis”.  Public Health Nutr. 2004;7(3):407-422.PubMedGoogle ScholarCrossref
Willcox  JK, Ash  SL, Catignani  GL.  Antioxidants and prevention of chronic disease.  Crit Rev Food Sci Nutr. 2004;44(4):275-295.PubMedGoogle ScholarCrossref
Bjelakovic  G, Nikolova  D, Gluud  C.  Antioxidant supplements to prevent mortality.  JAMA. 2013;310(11):1178-1179.PubMedGoogle ScholarCrossref
Bjelakovic  G, Nikolova  D, Gluud  LL, Simonetti  RG, Gluud  C.  Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases.  Cochrane Database Syst Rev. 2012;3(3):CD007176.PubMedGoogle Scholar
Ebrahim  S, Walter  SD, Cook  DJ, Jaeschke  R, Guyatt  G. Correlation and regression. In: Guyatt  G, Rennie  D, Meade  MO, Cook  DJ, eds.  Users’ Guides to the Medical Literature. 3rd ed. New York, NY: McGraw-Hill; 2015.
Guyatt  GH, Haynes  RB, Sackett  DL. Analyzing data. In: Haynes  RB, Sackett  DL, Guyatt  GH, Tugwell  P, eds.  Clinical Epidemiology: How to Do Clinical Practice Research. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2006.
Kleinbaum  DG, Klein  M.  Survival Analysis: A Self-Learning Text. 3rd ed. New York, NY: Springer; 2012.
Kern  LM, Malhotra  S, Barrón  Y,  et al.  Accuracy of electronically reported “meaningful use” clinical quality measures.  Ann Intern Med. 2013;158(2):77-83.PubMedGoogle ScholarCrossref
Courvoisier  DS, Combescure  C, Agoritsas  T, Gayet-Ageron  A, Perneger  TV.  Performance of logistic regression modeling.  J Clin Epidemiol. 2011;64(9):993-1000.PubMedGoogle ScholarCrossref
Guyatt  GH. Determining prognosis and creating clinical decision rules. In: Haynes  RB, Sackett  DL, Guyatt  GH, Tugwell  P, eds.  Clinical Epidemiology: How to Do Clinical Practice Research. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2006.
Austin  PC, Steyerberg  EW.  The number of subjects per variable required in linear regression analyses.  J Clin Epidemiol. 2015;68(6):627-636.PubMedGoogle ScholarCrossref
Stukel  TA, Lucas  FL, Wennberg  DE.  Long-term outcomes of regional variations in intensity of invasive vs medical management of Medicare patients with acute myocardial infarction.  JAMA. 2005;293(11):1329-1337.PubMedGoogle ScholarCrossref
Rosenbaum  PR, Rubin  DB.  Reducing bias in observational studies using subclassification on the propensity score.  J Am Stat Assoc. 1984;79(387):516-524.Google ScholarCrossref
Austin  PC.  An introduction to propensity score methods for reducing the effects of confounding in observational studies.  Multivariate Behav Res. 2011;46(3):399-424.PubMedGoogle ScholarCrossref
Austin  PC.  A tutorial and case study in propensity score analysis.  Multivariate Behav Res. 2011;46(1):119-151.PubMedGoogle ScholarCrossref
Biondi-Zoccai  G, Romagnoli  E, Agostoni  P,  et al.  Are propensity scores really superior to standard multivariable analysis?  Contemp Clin Trials. 2011;32(5):731-740.PubMedGoogle ScholarCrossref
Kurth  T, Walker  AM, Glynn  RJ,  et al.  Results of multivariable logistic regression, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect.  Am J Epidemiol. 2006;163(3):262-270.PubMedGoogle ScholarCrossref
Austin  PC.  The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies.  Stat Med. 2010;29(20):2137-2148.PubMedGoogle ScholarCrossref
Austin  PC, Stuart  EA.  The performance of inverse probability of treatment weighting and full matching on the propensity score in the presence of model misspecification when estimating the effect of treatment on survival outcomes [published online April 30, 2015].  Stat Methods Med Res. doi:10.1177/0962280215584401Google Scholar
Austin  PC.  Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006.  J Thorac Cardiovasc Surg. 2007;134(5):1128-1135.PubMedGoogle ScholarCrossref
Austin  PC.  Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies.  Pharm Stat. 2011;10(2):150-161.PubMedGoogle ScholarCrossref
Crown  WH.  Propensity-score matching in economic analyses.  Appl Health Econ Health Policy. 2014;12(1):7-18.PubMedGoogle ScholarCrossref
Kuss  O.  The z-difference can be used to measure covariate balance in matched propensity score analyses.  J Clin Epidemiol. 2013;66(11):1302-1307.PubMedGoogle ScholarCrossref
Ali  MS, Groenwold  RH, Belitser  SV,  et al.  Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal.  J Clin Epidemiol. 2015;68(2):112-121.PubMedGoogle ScholarCrossref
Luo  Z, Gardiner  JC, Bradley  CJ.  Applying propensity score methods in medical research.  Med Care Res Rev. 2010;67(5):528-554.PubMedGoogle ScholarCrossref
Shah  BR, Laupacis  A, Hux  JE, Austin  PC.  Propensity score methods gave similar results to traditional regression modeling in observational studies.  J Clin Epidemiol. 2005;58(6):550-559.PubMedGoogle ScholarCrossref
D’Agostino  RB  Jr, D’Agostino  RB  Sr.  Estimating treatment effects using observational data.  JAMA. 2007;297(3):314-316.PubMedGoogle ScholarCrossref
Cepeda  MS, Boston  R, Farrar  JT, Strom  BL.  Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders.  Am J Epidemiol. 2003;158(3):280-287.PubMedGoogle ScholarCrossref
Rassen  JA, Glynn  RJ, Brookhart  MA, Schneeweiss  S.  Covariate selection in high-dimensional propensity score analyses of treatment effects in small samples.  Am J Epidemiol. 2011;173(12):1404-1413.PubMedGoogle ScholarCrossref
Schneeweiss  S, Rassen  JA, Glynn  RJ, Avorn  J, Mogun  H, Brookhart  MA.  High-dimensional propensity score adjustment in studies of treatment effects using health care claims data.  Epidemiology. 2009;20(4):512-522.PubMedGoogle ScholarCrossref
Martens  EP, Pestman  WR, de Boer  A,  et al.  Systematic differences in treatment effect estimates between propensity score methods and logistic regression.  Int J Epidemiol. 2008;37(5):1142-1147.PubMedGoogle ScholarCrossref
Dahabreh  IJ, Sheldrick  RC, Paulus  JK,  et al.  Do observational studies using propensity score methods agree with randomized trials?  Eur Heart J. 2012;33(15):1893-1901.PubMedGoogle ScholarCrossref
Zhang  Z, Ni  H, Xu  X.  Observational studies using propensity score analysis underestimated the effect sizes in critical care medicine.  J Clin Epidemiol. 2014;67(8):932-939.PubMedGoogle ScholarCrossref
Newhouse  JP, McClellan  M.  Econometrics in outcomes research: the use of instrumental variables.  Annu Rev Public Health. 1998;19:17-34.PubMedGoogle ScholarCrossref
Chen  Y, Briesacher  BA.  Use of instrumental variable in prescription drug research with observational data: a systematic review.  J Clin Epidemiol. 2011;64(6):687-700.PubMedGoogle ScholarCrossref
Davies  NM, Smith  GD, Windmeijer  F, Martin  RM.  Issues in the reporting and conduct of instrumental variable studies.  Epidemiology. 2013;24(3):363-369.PubMedGoogle ScholarCrossref
Brookhart  MA, Rassen  JA, Schneeweiss  S.  Instrumental variable methods in comparative safety and effectiveness research.  Pharmacoepidemiol Drug Saf. 2010;19(6):537-554.PubMedGoogle ScholarCrossref
Fang  G, Brooks  JM, Chrischilles  EA.  Apples and oranges? interpretations of risk adjustment and instrumental variable estimates of intended treatment effects using observational data.  Am J Epidemiol. 2012;175(1):60-65.PubMedGoogle ScholarCrossref
Concato  J, Lawler  EV, Lew  RA, Gaziano  JM, Aslan  M, Huang  GD.  Observational methods in comparative effectiveness research.  Am J Med. 2010;123(12)(suppl 1):e16-e23.PubMedGoogle ScholarCrossref
Fang  G, Brooks  JM, Chrischilles  EA.  Comparison of instrumental variable analysis using a new instrument with risk adjustment methods to reduce confounding by indication.  Am J Epidemiol. 2012;175(11):1142-1151.PubMedGoogle ScholarCrossref
Lee  CC, Ho  HC, Hsiao  SH,  et al.  Infectious complications in head and neck cancer patients treated with cetuximab: propensity score and instrumental variable analysis.  PLoS One. 2012;7(11):e50163.PubMedGoogle ScholarCrossref
Lalani  T, Cabell  CH, Benjamin  DK,  et al.  Analysis of the impact of early surgery on in-hospital mortality of native valve endocarditis: use of propensity score and instrumental variable methods to adjust for treatment-selection bias.  Circulation. 2010;121(8):1005-1013.PubMedGoogle ScholarCrossref
Swanson  SA, Hernán  MA.  Commentary: how to report instrumental variable analyses (suggestions welcome).  Epidemiology. 2013;24(3):370-374.PubMedGoogle ScholarCrossref
Brookhart  MA, Schneeweiss  S.  Preference-based instrumental variable methods for the estimation of treatment effects.  Int J Biostat. 2007;3(1):14.PubMedGoogle ScholarCrossref
Swanson  SA, Hernán  MA.  Think globally, act globally: an epidemiologist’s perspective on instrumental variable estimation.  Stat Sci. 2014;29(3):371-374.PubMedGoogle ScholarCrossref
Swanson  SA, Robins  JM, Miller  M, Hernán  MA.  Selecting on treatment: a pervasive form of bias in instrumental variable analyses.  Am J Epidemiol. 2015;181(3):191-197.PubMedGoogle ScholarCrossref
Martens  EP, Pestman  WR, de Boer  A,  et al.  Instrumental variables: application and limitations.  Epidemiology. 2006;17(3):260-267.PubMedGoogle ScholarCrossref
Tan  HJ, Norton  EC, Ye  Z,  et al.  Long-term survival following partial vs radical nephrectomy among older patients with early-stage kidney cancer.  JAMA. 2012;307(15):1629-1635.PubMedGoogle ScholarCrossref
Pirracchio  R, Sprung  C, Payen  D, Chevret  S.  Benefits of ICU admission in critically ill patients: whether instrumental variable methods or propensity scores should be used.  BMC Med Res Methodol. 2011;11:132.PubMedGoogle ScholarCrossref
Crown  WH, Henk  HJ, Vanness  DJ.  Some cautions on the use of instrumental variables estimators in outcomes research.  Value Health. 2011;14(8):1078-1084.PubMedGoogle ScholarCrossref
Garabedian  LF, Chu  P, Toh  S, Zaslavsky  AM, Soumerai  SB.  Potential bias of instrumental variable analyses for observational comparative effectiveness research.  Ann Intern Med. 2014;161(2):131-138.PubMedGoogle ScholarCrossref
Finucane  FF, Madans  JH, Bush  TL,  et al.  Decreased risk of stroke among postmenopausal hormone users.  Arch Intern Med. 1993;153(1):73-79.PubMedGoogle ScholarCrossref
Wassertheil-Smoller  S, Hendrix  SL, Limacher  M,  et al.  Effect of estrogen plus progestin on stroke in postmenopausal women: the Women’s Health Initiative.  JAMA. 2003;289(20):2673-2684.PubMedGoogle ScholarCrossref
Devereaux  PJ, Yang  H, Yusuf  S,  et al.  Effects of extended-release metoprolol succinate in patients undergoing non-cardiac surgery (POISE trial).  Lancet. 2008;371(9627):1839-1847.PubMedGoogle ScholarCrossref
Blais  L, Desgagné  A, LeLorier  J.  3-Hydroxy-3-methylglutaryl coenzyme A reductase inhibitors and the risk of cancer.  Arch Intern Med. 2000;160(15):2363-2368.PubMedGoogle ScholarCrossref
Dale  KM, Coleman  CI, Henyan  NN, Kluger  J, White  CM.  Statins and cancer risk: a meta-analysis.  JAMA. 2006;295(1):74-80.PubMedGoogle ScholarCrossref
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right

Name Your Search

Save Search
With a personal account, you can:
  • Track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience

Lookup An Activity


My Saved Searches

You currently have no searches saved.

With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right
State Requirements