Understanding systematic reviews and meta-analyses

Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine Roger F. Soll, MD H. Wallace Professor of Neonatology Larner College of Medicine, University of Vermont Coordinating Editor, Cochrane Neonatal President, Vermont Oxford Network Trusted evidence. Informed decisions. Better health. 1

Editorial Team Roger F. Soll Coordinating Editor Colleen Ovelman Managing Editor Caitlin O’Connell Assistant Managing Editor 2

Editorial Team Bill McGuire Hull York Medical School Co-coordinating Editor Cochrane Neonatal 3

Editorial Team New and Improved! The “Co-Co” 3

Editorial Team Michael Bracken Yale University Jeffrey Horbar Prakeshkumar Shah University of Vermont University of Toronto Gautham Suresh Baylor University 3

Associate Editors 4

Guest Discussants Danielle Ehret, MD, MPH Assistant Professor, University of Vermont Director, Global Health, Vermont Oxford Network Deirdre O'Reilly, MD, MPH Assistant Professor, University of Vermont Director, NPM Fellowship, University of Vermont

Sponsors Section on Neonatal-Perinatal Medicine

Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine Roger F. Soll, M.D. is the President of the Vermont Oxford Network and the Coordinating Editor of Cochrane Neonatal No other relevant financial issues to disclose

Understanding systematic reviews and meta-analyses in neonatal-perinatal medicine To develop an understanding of the strengths and weaknesses of evidence provided by systematic reviews and meta-analyses to inform our practice of neonatalperinatal medicine.

The Evidence Hierarchy What evidence should I use to inform my practice?

Understanding systematic reviews and meta-analysis Akobeng AK Archives of Disease in Childhood 2005;90:845-848.

“Narrative Reviews” vs. “Systematic Reviews” Review articles in the medical literature have traditionally been in the form of “narrative reviews” where experts in a particular field provide what is supposed to be a “summary of evidence” in that field. Narrative reviews, although still very common in the medical field, have been criticized because of the high risk of bias, and “systematic reviews” are preferred. Systematic reviews apply scientific strategies in ways that limit bias to the assembly, a critical appraisal, and synthesis of relevant studies that address a

The problem with traditional “Narrative Reviews” While traditional review articles or narrative reviews can be useful when conducted properly, there is evidence that they are usually of poor quality. Authors of narrative reviews often use informal, subjective methods to collect and interpret studies and tend to be selective in citing reports that reinforce their preconceived ideas or promote their own views on a topic. They are also rarely explicit about how they selected, assessed, and analyzed the primary studies, thereby not allowing readers to assess potential bias in the review process.

Search: “Neonate” Limit: Newborn: birth to 1 month; Randomized controlled trial (RCT); Past 10 years 5207 RCT identified

Systematic Overview Applies specific research strategies to identify appraise, and synthesize data from all relevant clinical studies “a study of studies”

Meta-analysis uantitative systematic reviews include meta-analyse statistical methods to combine the results of similar randomized controlled trials to produce a typical estimate of the effect size

Meta-analysis Meta-analysis demands the same methodological quality expected in a randomized controlled trial: prospectively designed protocol comprehensive and explicit search strategy strict criteria for inclusion of studies standard definitions of outcomes standard statistical techniques

Meta-analysis What’s the use of meta-analysis? increase statistical power increase precision of estimate explore differences between study results create structure for incorporating new evidence

Meta-Analysis: Methods Meta-analysis is a two-stage process 1. The first stage involves the calculation of a measure of treatment effect with its 95% confidence intervals (CI) for each individual study. The statistics that are usually used to measure treatment effect include odds ratios (OR), relative risks (RR), and risk differences.

Meta-Analysis Methods: Dichotomous Outcomes Intervention event rate (IER) rate at which an event occurs in the experimental group Control event rate (CER) rate at which an event occurs in the control group Risk ratio (Relative risk) the ratio of the risk in the experimental group to the risk in the control group Risk difference (absolute risk difference) the difference between the risks in the experimental and control groups Number needed to treat number of persons who must be treated for one person to benefit 1/absolute

Meta-Analysis Methods: Dichotomous Outcomes 95% Confidence Interval (CI) The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again (and again).

Clinically relevant measures of treatment effect Relative Risk and Relative Risk Reduction measure the strength of the association not the clinical importance Need to consider frequency of outcome in population Large RR: evidence of a causal relationship Large ARD: evidence of actual effect of treatment

Meta-Analysis Methods 2. In the second stage of meta-analysis, an overall treatment effect is calculated as a weighted average of the individual summary statistics. In meta-analysis, data from the individual studies are not simply combined as if they were from a single study. Greater weights are given to the results from studies that provide more information, because they are likely to be closer to the “true effect” we are trying to estimate. The weights are often the inverse of the variance (the square of the standard error) of the treatment effect, which relates closely to sample size. The typical graph for displaying the results of a meta-analysis is called a “forest plot”. We will look at “forest plots” later in the discussion today.

Appraising a systematic review with or without meta-analysis Although systematic reviews occupy the highest position in the hierarchy of evidence for articles on effectiveness of interventions, it should not be assumed that a study is valid merely because it is stated to be a systematic review. Just as in RCTs, the main issues to consider when appraising a systematic review can be condensed into three important areas: The validity of the trial methodology. The magnitude and precision of the treatment effect. The applicability of the results to your patient or

Have you ever metaanalysis you didn’t like? Steven Goodman MD, MHS, PhD Annals of Internal Medicine 1991; 114: 244-246

Problems with meta-analysis Only fair agreement between large clinical trials and meta-analyses LeLorier 1997

META-ANALYSIS OF MULTIPLE SMALL STUDIES COMPARED TO SINGLE LARGE STUDY ASPIRIN FOR PREVENTION OF PRE-ECLAMPSIA CHARACTERISTIC Odds Ratio (95% CI) Decreased 0.2 META-ANALYSIS 0.30 (0.20, 0.42) SINGLE LARGE RCT 0.82 (0.72, 1.05) 0.2 Risk Increased 0.5 1.0 2.0 4.0 0.5 1.0 2.0 4.0 Odds Ratio and 95% CI

Problems with meta-analysis Methodological flaws in meta-analyses Publication bias The tendency for investigators to preferentially submit studies with positive results, and the tendency for editors to choose positive studies for publication Heterogeneity Concerning variation in the direction or the degrees of results between individual studies

Language Bias PROPORTION OF CONTROLLED TRIALS PUBLISHED IN GERMAN AND ENGLISH % PUBLICATIONS 70% 60% 50% 40% 30% 20% 10% 0% GERMAN NON SI GNI FI CANT ENGLI SH SI GNI FI CANT EGGER 1997

Heterogeneity Inevitably, studies brought together in a systematic review will differ. Any kind of variability among studies in a systematic review may be termed heterogeneity. Clinical: Variability in the participants, interventions and outcomes studied Methodological: Variability in study design and risk of bias

RCTs Systematic Reviews “You cannot make a silk purse from a sow’s ear”

Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. BMJ 2009;339:b2535, doi: 10.1136/bmj.b2535

Checklist of Items to Include When Reporting a Systematic Review or Meta-Analysis Section/topic # Checklist item TITLE Title 1 Identify the report as a systematic review, meta-analysis, or both. 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. Rationale 3 Describe the rationale for the review in the context of what is already known. Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators. Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. Risk of bias in individual studies 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis. Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means). Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I2) for each metaanalysis. ABSTRACT Structured summary INTRODUCTION METHODS Reported on page #

Checklist of Items to Include When Reporting a Systematic Review or Meta-Analysis Section/topic # Checklist item Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. Risk of bias within studies 19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12). Results of individual studies 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot. Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency. Risk of bias across studies 22 Present results of any assessment of risk of bias across studies (see Item 15). Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers). Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias). Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research. 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review. RESULTS DISCUSSION FUNDING Funding Reported on page #

Cooling for newborns with hypoxic ischaemic encephalopathy Jacobs SE, Berg M, Hunt R, Tarnow-Mordi WO, Inder TE, Davis PG Cochrane Database Syst Rev. 2013 Jan 31;(1):CD003311. doi: 10.1002/14651858.CD003311.pub3. .

Cooling for newborns with hypoxic ischaemic encephalopathy Objectives: To determine the effect of therapeutic hypothermia on death and long-term neurodevelopmental disability, and to ascertain clinically important side effects in newborn infants with HIE. Jacobs and colleagues. Cochrane Database Syst Rev. 2013 Jan 31; (1):CD003311. doi: 10.1002/14651858.CD003311.pub3.

PICOT Question

Cooling for newborns with hypoxic ischaemic encephalopathy PICOT Question In newborn infants with HIE, does therapeutic hypothermia compared to conventional management decrease the risk of death or long-term neurodevelopmental disability at 18 to 24 months of age.

Searching databases Bank items Cast iron bank Mechanical bank items 14813 805 items 336

Search strategy We use the standard search strategy of Cochrane Neonatal to search Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library, OVID Medline; MEDLINE via PubMed for the previous year; and CINAHL. We also search clinical trials databases and the reference lists of retrieved articles for randomized controlled trials and quasi-randomized trials.

Searching bibliographic databases 2000 - 2019 Therapeutic hypothermia Newborn Clinical trials Randomized controlled trials 14423 items 1383 items 169 items 127 items

Data synthesis If we identify multiple studies that we consider to be sufficiently similar, we will perform metaanalysis using Review Manager 5 (Review Manager 2014). For categorical outcomes, we will calculate the typical estimates of RR and RD, each with its 95% CI; for continuous outcomes, we will calculate the mean difference or the standardized mean difference, each with its 95% CI. We will use a fixed-effect model to combine data where it is reasonable to assume that studies were estimating the same underlying treatment

The forest plot The plot shows, at a glance, information from the individual studies that went into the meta-analysis, and an estimate of the overall results. It also allows a visual assessment of the amount of variation between the results of the studies (heterogeneity).

Death or Major Disability in Survivors: Relative Risk

Death or Major Disability in Survivors: Risk Difference

Annotated Forest Plot Death or Major Disability in Survivors: Relative Risk Point estimate Confidence interval Individual study Typical estimate I squared Line of identity

Heterogeneity Chi2 assesses whether observed differences in results are compatible with chance alone. I2 quantifying inconsistency across studies A rough guide to interpretation is as follows: 0% to 40%: might not be important; 30% to 60%: may represent moderate heterogeneity 50% to 90%: may represent substantial heterogeneity

Page. Research designs in sports physical therapy. International journal of sports physical therapy. Oct 2012;7(5):482‐492.

Quality of evidence We use the GRADE approach to assess the quality of evidence for clinically relevant outcomes. We consider evidence from randomized controlled trials as high quality, downgrading the evidence one level for serious (or two levels for very serious) limitations based upon the following: design (risk of bias), consistency across studies, directness of the evidence, precision of estimates, and presence of publication bias.

Applicability/Generalizability of results to YOUR patients Health care professionals should always make judgements about whether the results of a study are applicable to their own patient or group of patients. These include similarity of study population to your population, benefit vs harm, patients' preferences, availability, and costs.

Meta-analysis Meta-analysis is useful In guiding evidence-based practice To formulate research questions To create a context in which to view new evidence

Guest Discussants Danielle Ehret, MD, MPH Assistant Professor, University of Vermont Director, Global Health, Vermont Oxford Network Deirdre O'Reilly, MD, MPH Assistant Professor, University of Vermont Director, NPM Fellowship, University of Vermont

Get your facts first, then you can distort them as you please Mark Twain