[Skip to Content]
[Skip to Content Landing]

Adjusted Analyses in Studies Addressing Therapy and HarmUsers’ Guides to the Medical Literature

Educational Objective
To understand fundamental concepts underlying adjusted analyses in observational studies.
1 Credit CME
Abstract

Observational studies almost always have bias because prognostic factors are unequally distributed between patients exposed or not exposed to an intervention. The standard approach to dealing with this problem is adjusted or stratified analysis. Its principle is to use measurement of risk factors to create prognostically homogeneous groups and to combine effect estimates across groups.

The purpose of this Users’ Guide is to introduce readers to fundamental concepts underlying adjustment as a way of dealing with prognostic imbalance and to the basic principles and relative trustworthiness of various adjustment strategies.

One alternative to the standard approach is propensity analysis, in which groups are matched according to the likelihood of membership in exposed or unexposed groups. Propensity methods can deal with multiple prognostic factors, even if there are relatively few patients having outcome events. However, propensity methods do not address other limitations of traditional adjustment: investigators may not have measured all relevant prognostic factors (or not accurately), and unknown factors may bias the results.

A second approach, instrumental variable analysis, relies on identifying a variable associated with the likelihood of receiving the intervention but not associated with any prognostic factor or with the outcome (other than through the intervention); this could mimic randomization. However, as with assumptions of other adjustment approaches, it is never certain if an instrumental variable analysis eliminates bias.

Although all these approaches can reduce the risk of bias in observational studies, none replace the balance of both known and unknown prognostic factors offered by randomization.

Sign in to take quiz and track your certificates

Buy This Activity

JN Learning™ is the home for CME and MOC from the JAMA Network. Search by specialty or US state and earn AMA PRA Category 1 CME Credit™ from articles, audio, Clinical Challenges and more. Learn more about CME/MOC

Article Information

Corresponding Author: Thomas Agoritsas, MD, PhD, Divisions of Clinical Epidemiology and General Internal Medicine, Department of Internal Medicine, Rehabilitation and Geriatrics (DMIRG), University Hospitals of Geneva, Rue Gabrielle-Perret-Gentil 4, 1211 Genève 14, Switzerland (thomas.agoritsas@gmail.com).

Author Contributions: Drs Agoritsas and Guyatt had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.

Funding/Support: Dr Agoritsas was financially supported by the Fellowship for Prospective Researchers grant P3SMP3-155290/1 from the Swiss National Science Foundation.

Role of the Funder/Sponsor: The Swiss National Science Foundation had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

References
1.
Rathore  SS, Epstein  AJ, Volpp  KG, Krumholz  HM.  Regionalization of care for acute coronary syndromes.  JAMA. 2005;293(11):1383-1387.PubMedGoogle ScholarCrossref
2.
Mehta  SR, Cannon  CP, Fox  KA,  et al.  Routine vs selective invasive strategies in patients with acute coronary syndromes.  JAMA. 2005;293(23):2908-2917.PubMedGoogle ScholarCrossref
3.
Stukel  TA, Fisher  ES, Wennberg  DE,  et al.  Analysis of observational studies in the presence of treatment selection bias.  JAMA. 2007;297(3):278-285.PubMedGoogle ScholarCrossref
4.
Del Fiol  G, Workman  TE, Gorman  PN.  Clinical questions raised by clinicians at the point of care.  JAMA Intern Med. 2014;174(5):710-718.PubMedGoogle ScholarCrossref
5.
Walsh  M, Perkovic  V, Manns  B,  et al. Therapy (randomized trials). In: Guyatt  G, Rennie  D, Meade  MO, Cook  DJ, eds.  Users’ Guides to the Medical Literature. 3rd ed. New York, NY: McGraw-Hill; 2015.
6.
Sullivan  P, Goldmann  D.  The promise of comparative effectiveness research.  JAMA. 2011;305(4):400-401.PubMedGoogle ScholarCrossref
7.
Hochman  M, McCormick  D.  Characteristics of published comparative effectiveness studies of medications.  JAMA. 2010;303(10):951-958.PubMedGoogle ScholarCrossref
8.
Demaria  AN.  Comparative effectiveness research.  J Am Coll Cardiol. 2009;53(11):973-975.PubMedGoogle ScholarCrossref
9.
Dahabreh  IJ, Kent  DM.  Can the learning health care system be educated with observational data?  JAMA. 2014;312(2):129-130.PubMedGoogle ScholarCrossref
10.
Hemkens  LG, Contopoulos-Ioannidis  DG, Ioannidis  JP.  Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials.  BMJ. 2016;352:i493.PubMedGoogle ScholarCrossref
11.
Kennedy  CC, Jaeschke  R, Keitz  S,  et al.  Tips for teachers of evidence-based medicine: adjusting for prognostic imbalances (confounding variables) in studies on therapy or harm.  J Gen Intern Med. 2008;23(3):337-343.PubMedGoogle ScholarCrossref
12.
Ioannidis  JP, Haidich  AB, Pappa  M,  et al.  Comparison of evidence of treatment effects in randomized and nonrandomized studies.  JAMA. 2001;286(7):821-830.PubMedGoogle ScholarCrossref
13.
Saquib  N, Saquib  J, Ioannidis  JP.  Practices and impact of primary outcome adjustment in randomized controlled trials.  BMJ. 2013;347:f4313.PubMedGoogle ScholarCrossref
14.
Levine  M, Walter  S, Lee  H, Haines  T, Holbrook  A, Moyer  V.  Users’ Guides to the Medical Literature, IV: how to use an article about harm.  JAMA. 1994;271(20):1615-1619.PubMedGoogle ScholarCrossref
15.
Brignardello-Petersen  R, Ioannidis  JPA, Tomlinson  G, Guyatt  G. Surprising results of randomized trials. In: Guyatt  G, Rennie  D, Meade  MO, Cook  DJ, eds.  Users’ Guides to the Medical Literature. 3rd ed. New York, NY: McGraw-Hill; 2015.
16.
Guyatt  GH, Oxman  AD, Vist  GE,  et al; GRADE Working Group.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.  BMJ. 2008;336(7650):924-926.PubMedGoogle ScholarCrossref
17.
Stanner  SA, Hughes  J, Kelly  CN, Buttriss  J.  A review of the epidemiological evidence for the “antioxidant hypothesis”.  Public Health Nutr. 2004;7(3):407-422.PubMedGoogle ScholarCrossref
18.
Willcox  JK, Ash  SL, Catignani  GL.  Antioxidants and prevention of chronic disease.  Crit Rev Food Sci Nutr. 2004;44(4):275-295.PubMedGoogle ScholarCrossref
19.
Bjelakovic  G, Nikolova  D, Gluud  C.  Antioxidant supplements to prevent mortality.  JAMA. 2013;310(11):1178-1179.PubMedGoogle ScholarCrossref
20.
Bjelakovic  G, Nikolova  D, Gluud  LL, Simonetti  RG, Gluud  C.  Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases.  Cochrane Database Syst Rev. 2012;3(3):CD007176.PubMedGoogle Scholar
21.
Ebrahim  S, Walter  SD, Cook  DJ, Jaeschke  R, Guyatt  G. Correlation and regression. In: Guyatt  G, Rennie  D, Meade  MO, Cook  DJ, eds.  Users’ Guides to the Medical Literature. 3rd ed. New York, NY: McGraw-Hill; 2015.
22.
Guyatt  GH, Haynes  RB, Sackett  DL. Analyzing data. In: Haynes  RB, Sackett  DL, Guyatt  GH, Tugwell  P, eds.  Clinical Epidemiology: How to Do Clinical Practice Research. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2006.
23.
Kleinbaum  DG, Klein  M.  Survival Analysis: A Self-Learning Text. 3rd ed. New York, NY: Springer; 2012.
24.
Kern  LM, Malhotra  S, Barrón  Y,  et al.  Accuracy of electronically reported “meaningful use” clinical quality measures.  Ann Intern Med. 2013;158(2):77-83.PubMedGoogle ScholarCrossref
25.
Courvoisier  DS, Combescure  C, Agoritsas  T, Gayet-Ageron  A, Perneger  TV.  Performance of logistic regression modeling.  J Clin Epidemiol. 2011;64(9):993-1000.PubMedGoogle ScholarCrossref
26.
Guyatt  GH. Determining prognosis and creating clinical decision rules. In: Haynes  RB, Sackett  DL, Guyatt  GH, Tugwell  P, eds.  Clinical Epidemiology: How to Do Clinical Practice Research. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2006.
27.
Austin  PC, Steyerberg  EW.  The number of subjects per variable required in linear regression analyses.  J Clin Epidemiol. 2015;68(6):627-636.PubMedGoogle ScholarCrossref
28.
Stukel  TA, Lucas  FL, Wennberg  DE.  Long-term outcomes of regional variations in intensity of invasive vs medical management of Medicare patients with acute myocardial infarction.  JAMA. 2005;293(11):1329-1337.PubMedGoogle ScholarCrossref
29.
Rosenbaum  PR, Rubin  DB.  Reducing bias in observational studies using subclassification on the propensity score.  J Am Stat Assoc. 1984;79(387):516-524.Google ScholarCrossref
30.
Austin  PC.  An introduction to propensity score methods for reducing the effects of confounding in observational studies.  Multivariate Behav Res. 2011;46(3):399-424.PubMedGoogle ScholarCrossref
31.
Austin  PC.  A tutorial and case study in propensity score analysis.  Multivariate Behav Res. 2011;46(1):119-151.PubMedGoogle ScholarCrossref
32.
Biondi-Zoccai  G, Romagnoli  E, Agostoni  P,  et al.  Are propensity scores really superior to standard multivariable analysis?  Contemp Clin Trials. 2011;32(5):731-740.PubMedGoogle ScholarCrossref
33.
Kurth  T, Walker  AM, Glynn  RJ,  et al.  Results of multivariable logistic regression, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect.  Am J Epidemiol. 2006;163(3):262-270.PubMedGoogle ScholarCrossref
34.
Austin  PC.  The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies.  Stat Med. 2010;29(20):2137-2148.PubMedGoogle ScholarCrossref
35.
Austin  PC, Stuart  EA.  The performance of inverse probability of treatment weighting and full matching on the propensity score in the presence of model misspecification when estimating the effect of treatment on survival outcomes [published online April 30, 2015].  Stat Methods Med Res. doi:10.1177/0962280215584401Google Scholar
36.
Austin  PC.  Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006.  J Thorac Cardiovasc Surg. 2007;134(5):1128-1135.PubMedGoogle ScholarCrossref
37.
Austin  PC.  Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies.  Pharm Stat. 2011;10(2):150-161.PubMedGoogle ScholarCrossref
38.
Crown  WH.  Propensity-score matching in economic analyses.  Appl Health Econ Health Policy. 2014;12(1):7-18.PubMedGoogle ScholarCrossref
39.
Kuss  O.  The z-difference can be used to measure covariate balance in matched propensity score analyses.  J Clin Epidemiol. 2013;66(11):1302-1307.PubMedGoogle ScholarCrossref
40.
Ali  MS, Groenwold  RH, Belitser  SV,  et al.  Reporting of covariate selection and balance assessment in propensity score analysis is suboptimal.  J Clin Epidemiol. 2015;68(2):112-121.PubMedGoogle ScholarCrossref
41.
Luo  Z, Gardiner  JC, Bradley  CJ.  Applying propensity score methods in medical research.  Med Care Res Rev. 2010;67(5):528-554.PubMedGoogle ScholarCrossref
42.
Shah  BR, Laupacis  A, Hux  JE, Austin  PC.  Propensity score methods gave similar results to traditional regression modeling in observational studies.  J Clin Epidemiol. 2005;58(6):550-559.PubMedGoogle ScholarCrossref
43.
D’Agostino  RB  Jr, D’Agostino  RB  Sr.  Estimating treatment effects using observational data.  JAMA. 2007;297(3):314-316.PubMedGoogle ScholarCrossref
44.
Cepeda  MS, Boston  R, Farrar  JT, Strom  BL.  Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders.  Am J Epidemiol. 2003;158(3):280-287.PubMedGoogle ScholarCrossref
45.
Rassen  JA, Glynn  RJ, Brookhart  MA, Schneeweiss  S.  Covariate selection in high-dimensional propensity score analyses of treatment effects in small samples.  Am J Epidemiol. 2011;173(12):1404-1413.PubMedGoogle ScholarCrossref
46.
Schneeweiss  S, Rassen  JA, Glynn  RJ, Avorn  J, Mogun  H, Brookhart  MA.  High-dimensional propensity score adjustment in studies of treatment effects using health care claims data.  Epidemiology. 2009;20(4):512-522.PubMedGoogle ScholarCrossref
47.
Martens  EP, Pestman  WR, de Boer  A,  et al.  Systematic differences in treatment effect estimates between propensity score methods and logistic regression.  Int J Epidemiol. 2008;37(5):1142-1147.PubMedGoogle ScholarCrossref
48.
Dahabreh  IJ, Sheldrick  RC, Paulus  JK,  et al.  Do observational studies using propensity score methods agree with randomized trials?  Eur Heart J. 2012;33(15):1893-1901.PubMedGoogle ScholarCrossref
49.
Zhang  Z, Ni  H, Xu  X.  Observational studies using propensity score analysis underestimated the effect sizes in critical care medicine.  J Clin Epidemiol. 2014;67(8):932-939.PubMedGoogle ScholarCrossref
50.
Newhouse  JP, McClellan  M.  Econometrics in outcomes research: the use of instrumental variables.  Annu Rev Public Health. 1998;19:17-34.PubMedGoogle ScholarCrossref
51.
Chen  Y, Briesacher  BA.  Use of instrumental variable in prescription drug research with observational data: a systematic review.  J Clin Epidemiol. 2011;64(6):687-700.PubMedGoogle ScholarCrossref
52.
Davies  NM, Smith  GD, Windmeijer  F, Martin  RM.  Issues in the reporting and conduct of instrumental variable studies.  Epidemiology. 2013;24(3):363-369.PubMedGoogle ScholarCrossref
53.
Brookhart  MA, Rassen  JA, Schneeweiss  S.  Instrumental variable methods in comparative safety and effectiveness research.  Pharmacoepidemiol Drug Saf. 2010;19(6):537-554.PubMedGoogle ScholarCrossref
54.
Fang  G, Brooks  JM, Chrischilles  EA.  Apples and oranges? interpretations of risk adjustment and instrumental variable estimates of intended treatment effects using observational data.  Am J Epidemiol. 2012;175(1):60-65.PubMedGoogle ScholarCrossref
55.
Concato  J, Lawler  EV, Lew  RA, Gaziano  JM, Aslan  M, Huang  GD.  Observational methods in comparative effectiveness research.  Am J Med. 2010;123(12)(suppl 1):e16-e23.PubMedGoogle ScholarCrossref
56.
Fang  G, Brooks  JM, Chrischilles  EA.  Comparison of instrumental variable analysis using a new instrument with risk adjustment methods to reduce confounding by indication.  Am J Epidemiol. 2012;175(11):1142-1151.PubMedGoogle ScholarCrossref
57.
Lee  CC, Ho  HC, Hsiao  SH,  et al.  Infectious complications in head and neck cancer patients treated with cetuximab: propensity score and instrumental variable analysis.  PLoS One. 2012;7(11):e50163.PubMedGoogle ScholarCrossref
58.
Lalani  T, Cabell  CH, Benjamin  DK,  et al.  Analysis of the impact of early surgery on in-hospital mortality of native valve endocarditis: use of propensity score and instrumental variable methods to adjust for treatment-selection bias.  Circulation. 2010;121(8):1005-1013.PubMedGoogle ScholarCrossref
59.
Swanson  SA, Hernán  MA.  Commentary: how to report instrumental variable analyses (suggestions welcome).  Epidemiology. 2013;24(3):370-374.PubMedGoogle ScholarCrossref
60.
Brookhart  MA, Schneeweiss  S.  Preference-based instrumental variable methods for the estimation of treatment effects.  Int J Biostat. 2007;3(1):14.PubMedGoogle ScholarCrossref
61.
Swanson  SA, Hernán  MA.  Think globally, act globally: an epidemiologist’s perspective on instrumental variable estimation.  Stat Sci. 2014;29(3):371-374.PubMedGoogle ScholarCrossref
62.
Swanson  SA, Robins  JM, Miller  M, Hernán  MA.  Selecting on treatment: a pervasive form of bias in instrumental variable analyses.  Am J Epidemiol. 2015;181(3):191-197.PubMedGoogle ScholarCrossref
63.
Martens  EP, Pestman  WR, de Boer  A,  et al.  Instrumental variables: application and limitations.  Epidemiology. 2006;17(3):260-267.PubMedGoogle ScholarCrossref
64.
Tan  HJ, Norton  EC, Ye  Z,  et al.  Long-term survival following partial vs radical nephrectomy among older patients with early-stage kidney cancer.  JAMA. 2012;307(15):1629-1635.PubMedGoogle ScholarCrossref
65.
Pirracchio  R, Sprung  C, Payen  D, Chevret  S.  Benefits of ICU admission in critically ill patients: whether instrumental variable methods or propensity scores should be used.  BMC Med Res Methodol. 2011;11:132.PubMedGoogle ScholarCrossref
66.
Crown  WH, Henk  HJ, Vanness  DJ.  Some cautions on the use of instrumental variables estimators in outcomes research.  Value Health. 2011;14(8):1078-1084.PubMedGoogle ScholarCrossref
67.
Garabedian  LF, Chu  P, Toh  S, Zaslavsky  AM, Soumerai  SB.  Potential bias of instrumental variable analyses for observational comparative effectiveness research.  Ann Intern Med. 2014;161(2):131-138.PubMedGoogle ScholarCrossref
68.
Finucane  FF, Madans  JH, Bush  TL,  et al.  Decreased risk of stroke among postmenopausal hormone users.  Arch Intern Med. 1993;153(1):73-79.PubMedGoogle ScholarCrossref
69.
Wassertheil-Smoller  S, Hendrix  SL, Limacher  M,  et al.  Effect of estrogen plus progestin on stroke in postmenopausal women: the Women’s Health Initiative.  JAMA. 2003;289(20):2673-2684.PubMedGoogle ScholarCrossref
70.
Devereaux  PJ, Yang  H, Yusuf  S,  et al.  Effects of extended-release metoprolol succinate in patients undergoing non-cardiac surgery (POISE trial).  Lancet. 2008;371(9627):1839-1847.PubMedGoogle ScholarCrossref
71.
Blais  L, Desgagné  A, LeLorier  J.  3-Hydroxy-3-methylglutaryl coenzyme A reductase inhibitors and the risk of cancer.  Arch Intern Med. 2000;160(15):2363-2368.PubMedGoogle ScholarCrossref
72.
Dale  KM, Coleman  CI, Henyan  NN, Kluger  J, White  CM.  Statins and cancer risk: a meta-analysis.  JAMA. 2006;295(1):74-80.PubMedGoogle ScholarCrossref
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
jn-learning_Modal_LoginSubscribe_Purchase
If you are not a JN Learning subscriber, you can either:
Subscribe to JN Learning for one year
Buy this activity
jn-learning_Modal_LoginSubscribe_Purchase
With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right

Name Your Search

Save Search
With a personal account, you can:
  • Track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
jn-learning_Modal_SaveSearch_NoAccess_Purchase

Lookup An Activity

or

My Saved Searches

You currently have no searches saved.

With a personal account, you can:
  • Access free activities and track your credits
  • Personalize content alerts
  • Customize your interests
  • Fully personalize your learning experience
Education Center Collection Sign In Modal Right
Topics
State Requirements