Meta-analyses in the wonderland of neurology

Abstract

Meta-analyses are often misused and underused in neurology. This editorial provides some comments on the role of meta-analyses in neurological research. Recently, a huge increase in the number of meta-analyses and systematic reviews has been observed in neurological journals. The major strengths of meta-analyses are the increase of statistical power. However, as for any other investigative tool, meta-analytic research is a research method itself which can produce severe shortcomings. Specifically, the issues of search terms, time periods of published studies, databases used for searching, the definitions of inclusion and exclusion criteria for papers (which greatly affect clinical heterogeneity), publication bias; and the statistical methods used, dramatically influence the results of meta-analyses. The main problem of meta-analyses is that they cannot be expected to overcome the limitations of the studies they include (the so-called “garbage in, garbage out” phenomenon). Furthermore, most systematic reviews in the neurological literature lead to the unsatisfying and clinically frustrating statement “further studies are needed”. However it is much more frustrating to see how the gaps in scientific knowledge identified by meta-analyses have not been translated into serious efforts to fill them. Besides their role in evaluating efficacy and tolerability of drugs, meta-analyses may be used to assess diagnostic values of debatable clinical findings, as they represent powerful tools to try to answer questions not posed by individual studies and to settle controversies arising from conflicting claims.

Key Words: Clinical evaluation, Epilepsy, Meta-analysis, Migraine, Neurology

Core tip: Besides their role in evaluating efficacy and tolerability of drugs, meta-analyses may be used to assess diagnostic values of debatable clinical findings, as they represent powerful tools to try to answer questions not posed by individual studies and to settle controversies arising from conflicting claims.

META-ANALYSES AND THE NEUROLOGICAL UNANSWERED QUESTIONS

It is both astonishing and frustrating to consider how much meta-analyses are misused and underused in neurological research.

As a young neurology resident, I used to consider experienced neurologists as enlightened, trustworthy and truth holding people. After gaining some clinical experience in neurology and in evidence-based practice, I learnt to mistrust self assured people without worries, as I realized that truth does not exist in medicine, as it is an asymptotic process[1], not a divine revelation. Similarly, certainty in medicine does not exist, only probability does.

Neurology is probably the field of medicine most burdened with dilemmas on several crucial aspects of pathophysiology, diagnosis, and treatment of a number of diseases. Consider for instance the pathophysiological mechanisms involved in Alzheimer’s disease, the questionable therapeutical strategies against multiple sclerosis, and the endless discussions on cortical excitability in migraine or epilepsy. Questions in neurology seem to be much more than answers and as for the Holy Grail, the quest for definite conclusions is hard to be achieved but nevertheless remains an urgent need.

NEUROLOGICAL META-ANALYSES: REASONS FOR SUCCESS

Recently, a huge increase in the number of meta-analyses and systematic reviews has been observed in neurological journals. For instance, the number of articles published in four major neurological journals (Brain; Annals of Neurology; Neurology; and Journal of Neurology, Neurosurgery, and Psychiatry) increased from only 53 (1993-2002) to 187 (2003-2012)!

Such a great proliferation of meta-analyses may be easily understood: because of its inflated sample size, meta-analyses can detect treatment effects with greater statistical power, estimating these effects with greater precision than any single study.

SOME PITFALLS OF META-ANALYSES

However, improper use of meta-analyses may lead to erroneous conclusions regarding treatment efficacy. In fact, as for any other investigative tool, meta-analysis is a research method itself which can produce severe shortcomings.

Specifically, the issues of search terms, time periods of published studies, databases used for searching, the definitions of inclusion and exclusion criteria of papers (which greatly affect clinical heterogeneity), publication bias; and the statistical methods used, dramatically influence the results of meta-analyses.

Readers should be well aware of these pitfalls. After all, meta-analyses are human constructs, and as such they are fallible. All that glitters isn’t gold!

The main problem of meta-analyses is that they cannot be expected to overcome the limitations of the studies they include (the so-called “garbage in, garbage out” phenomenon). Furthermore, most systematic reviews in neurological literature lead to the unsatisfying and clinically frustrating statement “further studies are needed”. However, it is much more frustrating to see how the gaps in scientific knowledge identified by meta-analyses have not been translated into serious efforts to fill them.

ROLE OF META-ANALYSES IN NEUROLOGICAL RESEARCH: SOME PERSONAL EXAMPLES

Despite the above mentioned risks of pitfalls, how can meta-analyses help neurologists in their quest for answers

As a neurologist dealing with epilepsy and clinical neurophysiology I learnt to use meta-analyses as powerful tools to try to answer questions not posed by individual studies and to settle controversies arising from conflicting claims.

Meta-analyses allowed me not only to evaluate efficacy and tolerability of some neurological treatments[2,3], but also to better understand the diagnostic utility of some debatable clinical findings such as tongue biting, urinary incontinence or eye closure in the differential diagnosis of seizures[4-6]. Meta-analyses helped me to shed further light on the role of cortical excitability in the pathophysiology of migraine or idiopathic generalized epilepsies[7-10].

Finally, meta-analysis prompted me to consider the one single point of view of the view of one single point.

Dear neurologists, if there is no answer, just look for it! And may meta-analyses give you a hand!