Skip to Main Content

Editorial

In defence of reviews of small trials: underpinning the generation of evidence to inform practice

Helen HG Handoll, Peter Langhorne
Editorial Article

The value of systematic reviews of small trials has recently been questioned.[1] Contrary to the arguments of others who maintain that systematic reviews are crucial to avoiding waste[2], Roberts and Ker contend that systematic reviews of small trials "cause research waste" primarily because such reviews fail "to acknowledge the unreliability of small, single-centre trials".[1] We suggest that there is considerable awareness of the challenges of using small trials and that adherence to standard Cochrane methods helps counter the concerns surrounding the inclusion of small trials. This editorial illustrates why Cochrane Reviews of small trials are of value and how they can act as important grounds and platforms for trials that are large and robust enough to inform practice. We look at recently updated Cochrane Reviews on the treatment of an increasingly common fracture and the commonest impairment after stroke.[3,4]

Proximal humeral fractures, often termed shoulder fractures, are serious injuries that occur predominantly in older adults. Typically, the less disrupted or displaced fractures are treated by supporting the injured arm in a sling while the fracture mends. More severe fractures, in which two or more bony fragments are displaced, are often treated surgically. This usually involves putting the fracture fragments back in place and fixing them with a metal plate and screws. In more complex fractures, the humeral head is sometimes replaced by an artificial joint; more rarely, the whole joint is replaced. Rehabilitation, which often involves a combination of physiotherapy and home exercises, can be protracted. Treatment outcome is frequently unsatisfactory and poor shoulder function and pain can result in long term disability and, in frailer adults, increased dependence such as a move to a nursing home.

A Cochrane Review covering all interventions (including rehabilitation) for proximal humeral fractures was first published in 2001 and included nine small studies, with a further six listed as ongoing. This version and all six subsequent updates up to 2012 concluded that the evidence was insufficient to inform practice, noting in particular that "It remains unclear whether surgery, even for specific fracture types, will produce consistently better long term outcomes..."[5] The current version of the review, which includes 31 studies and lists a further 21 as ongoing, still notes the insufficiency of the evidence to most treatment decisions but now concludes: "There is high or moderate quality evidence that, compared with non-surgical treatment, surgery does not result in a better outcome at one and two years after injury for people with displaced proximal humeral fractures involving the humeral neck and is likely to result in a greater need for subsequent surgery".[3] This turnabout is solely because of the inclusion of evidence from a sufficiently powered, good-quality, multicentre randomised trial (the ProFHER trial) comparing surgery versus non-surgical treatment of displaced fractures for which surgery is increasingly being performed.[6]

The regularly updated Cochrane Review was instrumental in bringing about the delivery of this key primary research on the management of these injuries.

  • By highlighting the deficiency in the available evidence for surgical intervention, the review underpinned the need for the ProFHER trial, which was funded by the UK National Institute of Health Research.[7]
  • The review informed on the design of the ProFHER trial.[8] It endorsed the purposefully pragmatic design by highlighting the lack of robust evidence to inform treatment choices between different surgical or non-surgical interventions. Insights were gained from the critical appraisal of the included trials (e.g. noting the need for validated patient-reported outcome measures of function) and the contact with trialists of completed and abandoned trials (which highlighted frequent difficulties of recruitment in this area and reinforced the importance of a properly funded and professionally run multicentre trial).
  • Updates of the external evidence were regularly provided to the ProFHER Trial Steering Committee in order to ascertain that there was no new evidence that required a reappraisal of the trial's protocol. Two updates of the review were published during the conduct of the ProFHER trial.
  • The complete, open-access report of the trial set the results into context of the available evidence to illustrate the contribution made by the trial to the overall evidence for the trial comparison.[9] The current update of the Cochrane Review incorporates the ProFHER findings and sets them into the context of the currently and potentially available evidence for other comparisons.
  • The main results (two-year follow-up) of ProFHER are now available.[6] Five-year follow-up data are being collected for the ProFHER trial; the extension to follow-up is also supported by the findings of the Cochrane Review.

From being a neglected area served only by inadequate trials, a situation repeatedly highlighted by the Cochrane Review, the evidence for informing treatment decisions for these fractures is now accruing. Given that ProFHER is the first published of several multicentre prospectively registered trials testing treatment interventions for these fractures, the need to maintain this review is self-evident.

Loss of arm function is the most common impairment after stroke, and electromechanical (robot-assisted) arm training has been proposed as an approach to improving recovery.[4] The Cochrane Review of this intervention has been regularly updated and documents the development and publication of many small trials.[4] The 2012 version included 19 trials with 666 participants in total;[10] and the current version describes 34 trials with an average size of 34 participants (range 8 to 127). The review authors considered that the quality of the evidence was low. They concluded that despite promising results for arm recovery scores, and in view of the clinical heterogeneity of trials, "there is still a need for well-designed, large-scale, multicentre studies to evaluate benefits and harms of electromechanical-assisted arm training after stroke".[4] The UK National Institute of Health Research has commissioned such a trial, and researchers in Newcastle, UK, have successfully established the Robot Assisted Training for the Upper Limb after Stroke (RATULS) trial with a target recruitment of 720 participants.[11] Once again, the Cochrane Review was a key component in the development of the large, and hopefully definitive, trial. More broadly, a Cochrane overview of 31 reviews of different interventions for improving upper limb recovery after stroke has identified the shortage of high-quality evidence to support clinical decision-making in this clinical area and the need for larger definitive trials.[12]

These examples of how Cochrane Reviews of inadequate-sized trials have engendered and supported definitive studies are not atypical in the treatment of stroke or fractures. We are aware of many other examples in other areas, such as the direct influence of Cochrane Reviews, for example examining compression bandages[13], on a series of four definitive trials (VeNUS I, II, III and IV) on the management of venous ulcers.

Many Cochrane Review authors will know the frustration of having to report there is insufficient evidence to produce firm conclusions despite an enormous effort of identifying and, sometimes repeatedly, reviewing a medley of small trials. However, as shown here, such reviews still serve a crucial role by highlighting the evidence deficiency for key questions that merit the investment of substantive research.[14]

Related Cochrane Reviews