메타분석의 명과 암
Strengths and Limitations of Meta-Analysis
Article information
Trans Abstract
Meta-analysis is a statistical method that combines and synthesizes multiple studies and integrates their results. Meta-analysis increases the sample size, and in turn, the power to study the effects of interest by combining primary studies and providing a precise estimate of the effects. Data synthesized from meta-analyses are usually more beneficial than the results of narrative reviews. In a meta-analysis, the decisions are transparent, and statistical analysis yields an objective measure of the integrated quantitative evidence. The biases of narrative reviews can be limited or overcome by conducting a meta-analysis. The systematic approach and transparency in meta-analysis help to resolve conflicts and uncertainties between studies, while leading to significant conclusions. However, this method is controversial and may not always be the best tool. Moreover, meta-analysis has several shortcomings, and in some cases, it may not be appropriate. Although meta-analysis has been criticized due to its limitations, there are solutions to such problems. The aim of this review is to describe and discuss the strengths and weaknesses of meta-analysis.
INTRODUCTION
Meta-analysis is a statistical method that combines and synthesizes different studies and integrates their results into a single conclusion [1,2]. The word “meta” comes from Greek and means “after” or “beyond,” with meta-analysis meaning an “analysis of analyses” [3]. A meta-analysis increases the sample size, and consequently, the power to study the effects of interest by combining primary studies while considering the sizes of the studies included [2]. Thus, meta-analysis provides a more precise estimate of the effect. Meta-analysis is based on mathematical and statistical rules [4]; therefore, it is more objective than other methods, such as narrative reviews, and is less influenced by the author’s personal opinions. Thus, the use of meta-analysis is becoming increasingly popular [5-7]. Meta-analysis can be a powerful tool when it is well designed and performed appropriately. However, it may not always be the best tool, and is controversial [8]. Meta-analysis has several limitations, which can lead to misleading results; furthermore, it may not be appropriate in some cases. This review aims to describe and discuss the strengths and weaknesses of meta-analysis.
Meta-analysis vs. narrative review
Data synthesized from meta-analyses are usually more beneficial than the results of narrative reviews, because the latter have numerous disadvantages that can be overcome by meta-analysis [9]. Narrative reviews are based on subjective selection of publications; thus, study selection can be arbitrary [10]. In a narrative review, the lack of a specific search strategy increases the risk of failing to identify relevant studies on a given topic. In addition, the selected studies may not be critically appraised, the strength of the evidence may not be weighted, and no quantitative synthesis of the data may have been performed [10]. In narrative reviews, it may be difficult to compare the effects of studies that use varying metrics to measure a similar outcome. The reviewer in a narrative review qualitatively addresses a question by summarizing the findings of other studies and drawing a conclusion. Therefore, narrative reviews are prone to biases and errors, as different reviewers may draw different conclusions from the same evidence [11]. Therefore, narrative reviews should be considered opinion pieces with a low level of evidence. Hence, more transparent and reproducible systematic research syntheses may enable the resolution of any uncertainties when there are disagreements between the conclusions of studies. Meta-analysis minimizes bias by employing a methodological approach [12]. In meta-analysis, the decisions are transparent, and the statistical analysis results in an objective measure of the integrated quantitative evidence, which can then be repeated and verified. Moreover, meta-analysis converts the results of primary studies into a common metric as the effect size, thus different measures from primary studies can be compared against each other and yield conclusions that are more significant. By conducting a metaanalysis, the biases of narrative reviews can be limited or overcome. Therefore, meta-analysis is considered to provide evidence with the highest level of accuracy.
Strengths of meta-analysis
The methodological strengths of meta-analysis are listed in Table 1. It is more efficient to present a combined result than to describe the results of each individual study [8,13,14]. Meta-analysis can summarize and quantify the results from individual studies [15,16]. Additionally, it can clarify heterogeneity between the results of different studies and analyze differences in the results. An individual study may contain too few subjects in a particular subgroup. However, meta-analytic data from several individual studies may provide a clearer picture of the subgroup. Meta-analysis increases statistical power by increasing the sample size, and can determine small but clinically significant effects by combining data from numerous studies.
Strengths of meta-analysis
The precision of a study’s findings largely depends on the number of subjects. Statistically combining data from individual studies can provide a more precise estimate of the underlying effects than a single study. Thus, meta-analysis overcomes the limitation of small sample sizes of individual studies, detects the effects of interest, and reduces the risk of false-negative results. Moreover, meta-analysis can settle controversies resulting from studies with conflicting results. In addition, combining primary studies with varying sample sizes and patient populations can increase the generalizability of the results of individual studies; this allows the results of the meta-analysis to be generalized to a wider population. Appropriately examining the heterogeneity between individual studies allows the testing of novel hypotheses that have not been proposed in previous studies [17]. As meta-analysis summarizes currently existing knowledge, it may help in identifying areas that lack adequate evidence, thereby producing new research questions. Meta-analysis overcomes the problems and biases of traditional narrative reviews through a more transparent and subjective process that includes a systematic methodological approach.
Weaknesses of meta-analysis
The methodological weaknesses of meta-analysis are listed in Table 2. In addition, the limitations of meta-analysis as well as suggestions for addressing these have been described [8,13,14,18].
Weaknesses of meta-analysis
One number cannot summarize a research field
Summarizing large amounts of varying information using a single number is a controversial aspect of meta-analysis [19], as it ignores the fact that treatment effects may vary from study to study. However, a meta-analysis generalizes results despite differences in primary research and does not simply report a summary effect. If there is substantial heterogeneity, then the focus should shift from the summary effect to the heterogeneity itself. Meta-analysis provides a variety of tools to assess the pattern of heterogeneity, and to possibly explain it.
Mixing apples and oranges
Two main criticisms of meta-analysis are that it combines different types of studies (“mixing apples and oranges”) [8], and that the summary effect may ignore important differences between studies. Meta-analysis should be avoided if studies are too heterogeneous to be comparable, as the metaanalytical results may be meaningless and true effects may be obscured. However, meta-analyses, by their very nature, address broader questions than individual studies. Therefore, it can be said that a meta-analysis is similar to asking a question about fruits, for which both apples and oranges can contribute valuable information.
Garbage in, garbage out
The phrase “garbage in, garbage out” means that if a meta-analysis includes low-quality studies, its results will be biased and incorrect [20]. Meta-analysis includes a set of criteria for determining which studies to analyze. Hence, meta-analysis should be based on stricter criteria regarding the quality of studies to be included. When the available studies are flawed, a meta-analysis may employ sensitivity analyses to identify the influence of study biases.
Heterogeneity
In meta-analysis, heterogeneity refers to the degree of dissimilarity in the results of individual studies [2]. The main assumption for performing meta-analysis is that studies are homogenous in terms of populations, interventions, controls, and outcomes. Assessing the heterogeneity between primary studies is an important step in conducting a meta-analysis [21]. If there is substantial heterogeneity, the focus of the analysis should be on exploring and understanding the sources of the variation. Meta-analysis examines the existence of heterogeneity among primary studies and analyzes the variance in their results [2]. Subgroup analyses and meta-regression are used to explore the sources of heterogeneity. However, if there is a considerable amount of heterogeneity, it may not be appropriate to pool data in a meta-analysis.
Publication bias
Studies that report positive effects tend to be published more frequently than those that do not, and studies that report no significant results usually remain unpublished [22]. As meta-analysis includes only published studies, it may overestimate the actual magnitude of an effect [22]. This outcome is termed “publication bias.” To reduce the effect of publication bias on meta-analysis, serious efforts should be made to identify all relevant studies, because the outcome of the meta-analysis depends on the studies included. If the presence of publication bias is suspected, it can sometimes be detected by means of funnel plots and corresponding statistics, such as Egger’s linear regression test [23] for measuring funnel plot asymmetry or the “trim and fill” method to adjust summary estimates for the observed bias [24].
Not all variables are comparable
Some variables have no comparable measure for meta-analysis. Therefore, it may sometimes be necessary to construct new variables that present comparable concepts or restrict the analyses to common elements.
Meta-analysis can disagree with randomized trials
The main reason for discrepancies in meta-analysis is that it is based on heterogeneous and often small studies. The subjects in the individual studies may substantially differ with respect to diagnostic criteria, comorbidities, severity of disease, and geographic region. In contrast, in large randomized controlled trials, the target population is more limited. However, meta-analysis that is conducted appropriately may provide complementary valuable information.
Meta-analysis cannot overcome subjectivity
Meta-analysis relies on shared subjectivity, rather than objectivity. There is often a certain amount of subjectivity when deciding how similar studies should be before it is appropriate to combine them. Every form of analysis, including narrative reviews, requires certain subjective decisions. However, such decisions are always explicitly stated in a meta-analysis.
Meta-analysis deals only with the main effects
Meta-analysis deals with the main effects, and its results can be generalized to the target population. However, the effects of interactions may also be examined by moderator analysis.
CONCLUSION
Meta-analysis is a powerful tool when used judiciously to objectively summarize existing evidence regarding a specific issue. Moreover, meta-analysis provides a more objective appraisal of the evidence than narrative review, and attempts to minimize bias by utilizing a methodological approach. Meta-analysis provides a more precise estimate of the effect size and increases the generalizability of the results of individual studies. Therefore, it may enable the resolution of conflicts between studies, and yield conclusive results when individual studies are inconclusive. However, there are many caveats in the application of meta-analysis. Conclusions derived from meta-analysis are susceptible to the methodological quality of included studies, as well as to publication bias and the formulation of eligibility criteria. Although combining the data from independent studies using meta-analytical methods can improve statistical precision, it cannot altogether prevent bias. However, many criticisms of meta-analysis are true for narrative reviews as well [22]. Although meta-analysis is criticized for its limitations, solutions to these problems exist. A systematic approach and transparency in conducting meta-analysis help to resolve conflicts and uncertainties between studies and to derive meaningful conclusions. The use and value of metaanalysis is likely to increase in the future based on its power to reveal new findings.