| Home | E-Submission | Sitemap | Contact Us |  
logo
Korean J Med > Volume 94(5); 2019 > Article
메타분석의 명과 암

Abstract

Meta-analysis is a statistical method that combines and synthesizes multiple studies and integrates their results. Meta-analysis increases the sample size, and in turn, the power to study the effects of interest by combining primary studies and providing a precise estimate of the effects. Data synthesized from meta-analyses are usually more beneficial than the results of narrative reviews. In a meta-analysis, the decisions are transparent, and statistical analysis yields an objective measure of the integrated quantitative evidence. The biases of narrative reviews can be limited or overcome by conducting a meta-analysis. The systematic approach and transparency in meta-analysis help to resolve conflicts and uncertainties between studies, while leading to significant conclusions. However, this method is controversial and may not always be the best tool. Moreover, meta-analysis has several shortcomings, and in some cases, it may not be appropriate. Although meta-analysis has been criticized due to its limitations, there are solutions to such problems. The aim of this review is to describe and discuss the strengths and weaknesses of meta-analysis.

INTRODUCTION

Meta-analysis is a statistical method that combines and synthesizes different studies and integrates their results into a single conclusion [1,2]. The word “meta” comes from Greek and means “after” or “beyond,” with meta-analysis meaning an “analysis of analyses” [3]. A meta-analysis increases the sample size, and consequently, the power to study the effects of interest by combining primary studies while considering the sizes of the studies included [2]. Thus, meta-analysis provides a more precise estimate of the effect. Meta-analysis is based on mathematical and statistical rules [4]; therefore, it is more objective than other methods, such as narrative reviews, and is less influenced by the author’s personal opinions. Thus, the use of meta-analysis is becoming increasingly popular [5-7]. Meta-analysis can be a powerful tool when it is well designed and performed appropriately. However, it may not always be the best tool, and is controversial [8]. Meta-analysis has several limitations, which can lead to misleading results; furthermore, it may not be appropriate in some cases. This review aims to describe and discuss the strengths and weaknesses of meta-analysis.

Meta-analysis vs. narrative review

Data synthesized from meta-analyses are usually more beneficial than the results of narrative reviews, because the latter have numerous disadvantages that can be overcome by meta-analysis [9]. Narrative reviews are based on subjective selection of publications; thus, study selection can be arbitrary [10]. In a narrative review, the lack of a specific search strategy increases the risk of failing to identify relevant studies on a given topic. In addition, the selected studies may not be critically appraised, the strength of the evidence may not be weighted, and no quantitative synthesis of the data may have been performed [10]. In narrative reviews, it may be difficult to compare the effects of studies that use varying metrics to measure a similar outcome. The reviewer in a narrative review qualitatively addresses a question by summarizing the findings of other studies and drawing a conclusion. Therefore, narrative reviews are prone to biases and errors, as different reviewers may draw different conclusions from the same evidence [11]. Therefore, narrative reviews should be considered opinion pieces with a low level of evidence. Hence, more transparent and reproducible systematic research syntheses may enable the resolution of any uncertainties when there are disagreements between the conclusions of studies. Meta-analysis minimizes bias by employing a methodological approach [12]. In meta-analysis, the decisions are transparent, and the statistical analysis results in an objective measure of the integrated quantitative evidence, which can then be repeated and verified. Moreover, meta-analysis converts the results of primary studies into a common metric as the effect size, thus different measures from primary studies can be compared against each other and yield conclusions that are more significant. By conducting a metaanalysis, the biases of narrative reviews can be limited or overcome. Therefore, meta-analysis is considered to provide evidence with the highest level of accuracy.

Strengths of meta-analysis

The methodological strengths of meta-analysis are listed in Table 1. It is more efficient to present a combined result than to describe the results of each individual study [8,13,14]. Meta-analysis can summarize and quantify the results from individual studies [15,16]. Additionally, it can clarify heterogeneity between the results of different studies and analyze differences in the results. An individual study may contain too few subjects in a particular subgroup. However, meta-analytic data from several individual studies may provide a clearer picture of the subgroup. Meta-analysis increases statistical power by increasing the sample size, and can determine small but clinically significant effects by combining data from numerous studies.
The precision of a study’s findings largely depends on the number of subjects. Statistically combining data from individual studies can provide a more precise estimate of the underlying effects than a single study. Thus, meta-analysis overcomes the limitation of small sample sizes of individual studies, detects the effects of interest, and reduces the risk of false-negative results. Moreover, meta-analysis can settle controversies resulting from studies with conflicting results. In addition, combining primary studies with varying sample sizes and patient populations can increase the generalizability of the results of individual studies; this allows the results of the meta-analysis to be generalized to a wider population. Appropriately examining the heterogeneity between individual studies allows the testing of novel hypotheses that have not been proposed in previous studies [17]. As meta-analysis summarizes currently existing knowledge, it may help in identifying areas that lack adequate evidence, thereby producing new research questions. Meta-analysis overcomes the problems and biases of traditional narrative reviews through a more transparent and subjective process that includes a systematic methodological approach.

Weaknesses of meta-analysis

The methodological weaknesses of meta-analysis are listed in Table 2. In addition, the limitations of meta-analysis as well as suggestions for addressing these have been described [8,13,14,18].

One number cannot summarize a research field

Summarizing large amounts of varying information using a single number is a controversial aspect of meta-analysis [19], as it ignores the fact that treatment effects may vary from study to study. However, a meta-analysis generalizes results despite differences in primary research and does not simply report a summary effect. If there is substantial heterogeneity, then the focus should shift from the summary effect to the heterogeneity itself. Meta-analysis provides a variety of tools to assess the pattern of heterogeneity, and to possibly explain it.

Mixing apples and oranges

Two main criticisms of meta-analysis are that it combines different types of studies (“mixing apples and oranges”) [8], and that the summary effect may ignore important differences between studies. Meta-analysis should be avoided if studies are too heterogeneous to be comparable, as the metaanalytical results may be meaningless and true effects may be obscured. However, meta-analyses, by their very nature, address broader questions than individual studies. Therefore, it can be said that a meta-analysis is similar to asking a question about fruits, for which both apples and oranges can contribute valuable information.

Garbage in, garbage out

The phrase “garbage in, garbage out” means that if a meta-analysis includes low-quality studies, its results will be biased and incorrect [20]. Meta-analysis includes a set of criteria for determining which studies to analyze. Hence, meta-analysis should be based on stricter criteria regarding the quality of studies to be included. When the available studies are flawed, a meta-analysis may employ sensitivity analyses to identify the influence of study biases.

Heterogeneity

In meta-analysis, heterogeneity refers to the degree of dissimilarity in the results of individual studies [2]. The main assumption for performing meta-analysis is that studies are homogenous in terms of populations, interventions, controls, and outcomes. Assessing the heterogeneity between primary studies is an important step in conducting a meta-analysis [21]. If there is substantial heterogeneity, the focus of the analysis should be on exploring and understanding the sources of the variation. Meta-analysis examines the existence of heterogeneity among primary studies and analyzes the variance in their results [2]. Subgroup analyses and meta-regression are used to explore the sources of heterogeneity. However, if there is a considerable amount of heterogeneity, it may not be appropriate to pool data in a meta-analysis.

Publication bias

Studies that report positive effects tend to be published more frequently than those that do not, and studies that report no significant results usually remain unpublished [22]. As meta-analysis includes only published studies, it may overestimate the actual magnitude of an effect [22]. This outcome is termed “publication bias.” To reduce the effect of publication bias on meta-analysis, serious efforts should be made to identify all relevant studies, because the outcome of the meta-analysis depends on the studies included. If the presence of publication bias is suspected, it can sometimes be detected by means of funnel plots and corresponding statistics, such as Egger’s linear regression test [23] for measuring funnel plot asymmetry or the “trim and fill” method to adjust summary estimates for the observed bias [24].

Not all variables are comparable

Some variables have no comparable measure for meta-analysis. Therefore, it may sometimes be necessary to construct new variables that present comparable concepts or restrict the analyses to common elements.

Meta-analysis can disagree with randomized trials

The main reason for discrepancies in meta-analysis is that it is based on heterogeneous and often small studies. The subjects in the individual studies may substantially differ with respect to diagnostic criteria, comorbidities, severity of disease, and geographic region. In contrast, in large randomized controlled trials, the target population is more limited. However, meta-analysis that is conducted appropriately may provide complementary valuable information.

Meta-analysis cannot overcome subjectivity

Meta-analysis relies on shared subjectivity, rather than objectivity. There is often a certain amount of subjectivity when deciding how similar studies should be before it is appropriate to combine them. Every form of analysis, including narrative reviews, requires certain subjective decisions. However, such decisions are always explicitly stated in a meta-analysis.

Meta-analysis deals only with the main effects

Meta-analysis deals with the main effects, and its results can be generalized to the target population. However, the effects of interactions may also be examined by moderator analysis.

CONCLUSION

Meta-analysis is a powerful tool when used judiciously to objectively summarize existing evidence regarding a specific issue. Moreover, meta-analysis provides a more objective appraisal of the evidence than narrative review, and attempts to minimize bias by utilizing a methodological approach. Meta-analysis provides a more precise estimate of the effect size and increases the generalizability of the results of individual studies. Therefore, it may enable the resolution of conflicts between studies, and yield conclusive results when individual studies are inconclusive. However, there are many caveats in the application of meta-analysis. Conclusions derived from meta-analysis are susceptible to the methodological quality of included studies, as well as to publication bias and the formulation of eligibility criteria. Although combining the data from independent studies using meta-analytical methods can improve statistical precision, it cannot altogether prevent bias. However, many criticisms of meta-analysis are true for narrative reviews as well [22]. Although meta-analysis is criticized for its limitations, solutions to these problems exist. A systematic approach and transparency in conducting meta-analysis help to resolve conflicts and uncertainties between studies and to derive meaningful conclusions. The use and value of metaanalysis is likely to increase in the future based on its power to reveal new findings.

REFERENCES

1. Esterhuizen TM, Thabane L. Con: meta-analysis: some key limitations and potential solutions. Nephrol Dial Transplant 2016; 31:882–885.
crossref pmid

2. Walker E, Hernandez AV, Kattan MW. Meta-analysis: its strengths and limitations. Cleve Clin J Med 2008; 75:431–439.
crossref pmid

3. Glass GV. Primary, secondary, and meta-analysis of research. Educational Researcher 1976; 5:3–8.
crossref

4. Egger M, Smith GD, Phillips AN. Meta-analysis: principles and procedures. BMJ 1997; 315:1533–1537.
crossref pmid pmc

5. Lee YH. Meta-analysis of genetic association studies. Ann Lab Med 2015; 35:283–287.
crossref pmid pmc

6. Lee YH, Bae SC, Song GG. Hepatitis B virus (HBV) reactivation in rheumatic patients with hepatitis core antigen (HBV occult carriers) undergoing anti-tumor necrosis factor therapy. Clin Exp Rheumatol 2013; 31:118–121.


7. Lee YH, Choi SJ, Ji JD, Song GG. Overall and cause-specific mortality in systemic lupus erythematosus: an updated meta-analysis. Lupus 2016; 25:727–734.
crossref pmid

8. Finckh A, Tramèr MR. Primer: strengths and weaknesses of meta-analysis. Nat Clin Pract Rheumatol 2008; 4:146–152.
crossref pmid

9. Cooper HM, Rosenthal R. Statistical versus traditional procedures for summarizing research findings. Psychol Bull 1980; 87:442–449.
crossref pmid

10. Pae CU. Why systematic review rather than narrative review? Psychiatry Investig 2015; 12:417–419.
crossref pmid pmc

11. Cipriani A, Geddes J. Comparison of systematic and narrative reviews: the example of the atypical antipsychotics. Epidemiol Psichiatr Soc 2003; 12:146–153.
crossref pmid

12. Bailar JC 3rd. The promise and problems of meta-analysis. N Engl J Med 1997; 337:559–561.
crossref pmid

13. Ioannidis JP, Lau J. Pooling research results: benefits and limitations of meta-analysis. Jt Comm J Qual Improv 1999; 25:462–469.
crossref pmid

14. Noble JH Jr. Meta-analysis: methods, strengths, weaknesses, and political uses. J Lab Clin Med 2006; 147:7–20.
crossref pmid

15. Lee YH. Association between the neutrophil-to-lymphocyte ratio, and platelet-to-lymphocyte ratio and rheumatoid arthritis and their correlations with the disease activity: a meta-analysis. J Rheum Dis 2018; 25:169–178.
crossref

16. Lee YH, Song GG. Comparative efficacy and safety of secukinumab and adalimumab in patients with active ankylosing spondylitis: a Bayesian network meta-analysis of randomized controlled trials. J Rheum Dis 2017; 24:211–219.
crossref

17. Gotzsche PC. Why we need a broad perspective on metaanalysis. It may be crucially important for patients. BMJ 2000; 321:585–586.
crossref pmid pmc

18. Tranfield D, Denyer D, Smart P. Towards a methodology for developing evidence?informed management knowledge by means of systematic review. British Journal of Management 2003; 14:207–22.
crossref

19. Garg AX, Hackam D, Tonelli M. Systematic review and meta-analysis: when one study is just not enough. Clin J Am Soc Nephrol 2008; 3:253–260.
crossref pmid

20. Sharpe D. Of apples and oranges, file drawers and garbage: why validity issues in meta-analysis will not go away. Clin Psychol Rev 1997; 17:881–901.
crossref pmid

21. Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med 2002; 21:1539–1558.
crossref pmid

22. Dickersin K, Min YI. Publication bias: the problem that won't go away. Ann N Y Acad Sci 1993; 703:135–146; discussion 146-148.
crossref pmid

23. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315:629–634.
crossref pmid pmc

24. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000; 56:455–463.
crossref pmid

Table 1.
Strengths of meta-analysis
One number cannot summarize a research field
Mixing apples and oranges
Garbage in, garbage out
Heterogeneity
Publication bias
Not all variables are comparable
Potential disagreement with findings of randomized trials
Cannot overcome subjectivity
Only deals with main effects
Table 2.
Weaknesses of meta-analysis
One number cannot summarize a research field
Mixing apples and oranges
Garbage in, garbage out
Heterogeneity
Publication bias
Not all variables are comparable
Potential disagreement with findings of randomized trials
Cannot overcome subjectivity
Only deals with main effects
TOOLS
PDF Links  PDF Links
PubReader  PubReader
ePub Link  ePub Link
Full text via DOI  Full text via DOI
Download Citation  Download Citation
CrossRef TDM  CrossRef TDM
Supplementary Material  Supplementary Material
  E-Mail
  Print
Share:      
METRICS
0
Crossref
239
View
53
Download
Editorial Office
101-2501, Lotte Castle President, 109 Mapo-daero, Mapo-gu, Seoul 04146, Korea
Tel: +82-2-2271-6791   Fax: +82-2-790-0993    E-mail: kaim@kams.or.kr
About |  Browse Articles |  Current Issue |  For Authors and Reviewers
Copyright © The Korean Association of Internal Medicine. All rights reserved.                powerd by m2community