Skip to Main Content

Editorial

Why do we need evidence-based methods in Cochrane?

Joanne E McKenzie, Mike J Clarke, Jackie Chandler
Editorial Article

If systematic reviews are to provide the information that people need when making decisions about health and social care, we need to be confident that the methods used to plan, conduct, and report these reviews result in valid and reliable evidence that fairly represents the sum of knowledge. This year’s Annual Cochrane Methods Symposium at the Cochrane Colloquium (colloquium.cochrane.org) focuses on why the methods used to produce reviews for evidence-based care should themselves be evidence-based.

As systematic reviewers, we devote long hours to scrutinising the evidence underpinning clinical practice. Yet how often do we question whether there is good evidence for the methods we use? Our confidence in the findings of more than 35,000 systematic reviews[1] rests on the evidence base underpinning the methods we use. Just as there are consequences arising from the choices we make about health and social care interventions, so too are there consequences when we choose the methods to use in systematic reviews. The DerSimonian and Laird random effects method offers an example of this.[2] Despite this being the most commonly used method, statistical simulation studies have shown that other methods perform better in certain circumstances.[3,4] However, further research is required to establish how often the use of an alternative method leads to different review conclusions and under what circumstances more than one method should be used. Methods research can allow us to make well-informed decisions about the most appropriate methods through the provision of information that allows us to understand trade-offs in terms of bias, efficiency, usability, and resource use.

Over recent decades, the evidence base for systematic review methods has been growing, with some areas having received considerable attention. Examples of methods research includes descriptive studies of reviews[5-7] and research using varied designs into the effects of alternative methods strategies, including search filters for eligible studies,[8,9] screening records retrieved from bibliographic databases,[10] meta-analysis methods,[11-13] prioritisation of updating of reviews,[14,15] and ways to summarise the findings of reviews.[16,17] However, many uncertainties and much debate remains, as illustrated by the findings of systematic reviews of methodology research, such as those conducted under the auspices of the Cochrane Methodology Review Group. These have included reviews relevant to searching for eligible studies,[18-20] obtaining data,[21] and investigating biases in the review process.[22]

The Methods Symposium will present a series of case studies to illustrate how methods have been developed, evaluated, and implemented for systematic reviews, within and beyond Cochrane. Discussion will cover different approaches to evaluating methods (assessing their strengths and weaknesses), the quality of evidence required to change methods, and how we bring about change in the methods used by people preparing, maintaining, and disseminating reviews. In addition, we will consider how best to determine where methods development is required and how to prioritise that development. There will be opportunity to discuss how research into review methods might be facilitated, using approaches such as the Studies Within A Review (SWAR) programme.[23]

Similar to research evaluating the effects of clinical interventions, there will be uncertainties and varied interpretations arising from research evaluating the effects of methods. What constitutes enough evidence for one person to implement a new method will not be enough for another. As systematic review methods proliferate, there will be an increasing impetus for reviewers, and review groups, to implement new methods, which they may deem have sufficient evidence. However, consistency of review methods has always been a desired and notable feature of Cochrane Reviews[24] and provides some guarantee to review users that the standard methods are appropriate and that the ensuing evidence is valid. This leaves Cochrane walking a tightrope: balancing consistency of methods with review authors' autonomy to use alternative methods. Cochrane is working toward developing processes to manage review developments. The Methods Symposium will allow for debate and discussion on these issues and we encourage participation of reviewers, editors, and methodologists.

The 2015 Cochrane Methods Symposium will take place on 2 October 2015 at the Palais Niederösterriech, Vienna, Austria. Find out more on the Cochrane Methods website.