Why meta-analyses are important for complementary and alternative medicine research

Abstract

Complementary and alternative medicine (CAM) is defined as a group of interventions that are not generally considered part of conventional medicine. This definition already implies that CAM interventions are often not systematically studied; and the research evidence from single trials on CAM is often limited by small sample sizes, unclear methodology, and inadequate statistics. As a result, both, significant and insignificant results are often hard to interpret based on single trials. Summarizing the evidence from single CAM trials, qualitative systematic reviews still have to deal with the same problems as individual trials as they can only rely on the original reports. Thus, effects of CAM interventions are often underestimated or overestimated based on single trials or qualitative systematic reviews. While meta-analyses still are limited by the methodological shortcomings of the included studies, a well-conducted meta-analysis can deal with two common problems of CAM trials: inadequate statistics that rely on within-group comparisons and small underpowered sample sizes. Although large and high quality trials are urgently needed for most CAM interventions, funding often is limited. Until higher quality research is available, meta-analyses provide a useful tool to investigate the actual level of evidence of currently published CAM trials. This editorial presents examples of meta-analyses in the field of CAM and discusses how they contribute to the consolidation of evidence.

Key Words: Complementary therapies, Meta-analysis, Review, Randomized controlled trial, Bias

Core tip: The research evidence from single trials on complementary and alternative medicine (CAM) is often limited by small sample sizes, unclear methodology, and inadequate statistics. Qualitative systematic reviews still have to deal with the same problems as individual trials as they can only rely on the original reports. While meta-analyses still are limited by the methodological shortcomings of the included, they can deal with two common problems of CAM trials: inadequate statistics that rely on within-group comparisons and small underpowered sample sizes.

TEXT

The National Center for Complementary and Alternative Medicine of the National Institute of Health defines complementary and alternative medicine (CAM) as “a group of diverse medical and health care systems, practices, and products that are not generally considered part of conventional medicine”[1]. A growing number of randomized controlled trials (RCTs) aimed to investigate the effectiveness of CAM therapies in varied medical conditions. However, while these trials are urgently needed to consolidate evidence for interventions that have been - by definition - rarely studied systematically in the past, the research evidence from single trials on CAM is often limited by small sample sizes, unclear methodology, and inadequate statistics. Both, significant and insignificant results are often hard to interpret based on single trials. On the one hand, while randomized trials are generally conducted to compare effects of two or more different treatments on a specific condition, especially CAM trials often solely rely on within-group comparisons that do not take into account unspecific effects. Thereby, the value of having a control group is lost and it is often impossible to estimate the real specific effects of the intervention; overestimating its actual specific efficacy. On the other hand, small trials that often have to deal with marked baseline differences between groups are often underpowered to detect specific effects of the intervention. Based on those problem, it has even been encouraged to abandon RCT designs in CAM research altogether. However, most problems of CAM RCTs can be adequately addressed by proper methodology use. In recent years, a continuously growing number of systematic reviews have been conducted in order to summarize evidence from single CAM trials. While this tendency is definitely useful to consolidate evidence, qualitative reviews still have to deal with the same problems as individual trials as they can only rely on the original - often heavily biased - reports. This is where meta-analyses should come into play. While meta-analyses still are limited by the methodological shortcomings of the included trials and badly conducted meta-analyses can even worsen the situation, a well-conducted meta-analysis can deal with two common problems of CAM trials: inadequate statistics that rely on within-group comparisons and small underpowered sample sizes. By ignoring the original statistical method of the published analyses and by quantitatively pooling the results of several trials, a between-group analysis based on a larger sample can be created that compensates for at least some of the shortcomings of the original trials. Although a trained statistician would surely be able to assess the real effect sizes from the published data without relying on a published meta-analysis, CAM trials are often used to guide clinical decision making; and a clinician often will not be able to re-evaluate the statistics of a published trial. While it does not reduce the risk of bias of the original studies, a meta-analysis can address the problem of inadequate statistics and improve the power of the analysis. Take, e.g., yoga. Yoga has now become a popular means to improve health and well-being and several studies have investigated yoga’s effectiveness in varied medical conditions. In 2011, a systematic review on yoga for low back pain included a total of seven RCTs published until March 2011[2]. Five of the RCTs found significant effects of the yoga interventions while the other two did not. The systematic review concluded that yoga might be able to alleviate low back pain but that any definitive claims should be treated with caution[2]. A second systematic review on the same condition included trials that were published until January 2012[3]. While the first review refrained from meta-analyzing the data due to heterogeneity of the included trials, the second review was able to include a meta-analysis. Based on ten RCTs, this meta-analysis found strong evidence for short-term effectiveness and moderate evidence for long-term effectiveness of yoga for chronic low back pain in the most important patient-centered outcomes[3]. As a consequence, this review concluded that yoga can be recommended as an additional therapy to chronic low back pain patients. While the obvious divergences in conclusions might also be accounted to the increased number of included RCTs in the second review, they are likely to be at least partly based on the inclusion of a meta-analysis whose findings can go beyond just balancing the results of individual trials against each other. On the other hand, meta-analyses can also help to revise falsely overoptimistic conclusions of single trials. A 2012 systematic review on the effects of yoga for schizophrenia included three RCTS that were published until October 2011[4]. Despite the low number of eligible trials, the overall positive findings of those led to the conclusion that yoga could be helpful in reducing general psychopathology, positive, and negative symptoms in patients with schizophrenia. However, as these conclusions were based on the results that were reported in the original articles, and one out of three RCTs reported only within-group pre-post comparisons rather than between-group comparisons, these results were not robust against reporting bias. Accordingly, based on the same trials, a recent meta-analysis failed to find any effects of yoga on schizophrenia psychopathology[5]; resulting in the counterintuitive finding that a meta-analysis of three RCTs that all reported positive effects resulted in insignificant group differences.

A problem of meta-analyses - especially for but not limited to CAM research - is heterogeneity between trials; specifically clinical heterogeneity (differences in, e.g., interventions that might be labeled with the same term) which often results in statistical heterogeneity (differences in the interventions’ effects). While studies in a meta-analysis will inevitably differ from each other, substantial statistical heterogeneity can reduce the precision of effect estimates. Thus, authors of CAM-related meta-analyses should be aware of the heterogeneity of CAM interventions and define the focus of their meta-analysis as precisely as possible.

Small underpowered and poorly conducted trials are by no means only a problem of CAM research. However, as external funding for CAM trials is limited to non-existent in most countries, large well-conducted trials are especially difficult to conduct in this research area. While meta-analyses cannot compensate for low-quality original research, they provide a useful tool to investigate the actual level of evidence of currently published CAM trials until higher-quality research evidence is available.