Abstract
BACKGROUND: Previous research has suggested that statistical power is suboptimal in many biomedical disciplines, but it is unclear whether power is better in trials for particular interventions, disorders, or outcome types. We therefore performed a detailed examination of power in trials of psychotherapy, pharmacotherapy, and complementary and alternative medicine (CAM) for mood, anxiety, and psychotic disorders. METHODS: We extracted data from the Cochrane Database of Systematic Reviews (Mental Health). We focused on continuous efficacy outcomes and estimated power to detect predetermined effect sizes (standardized mean difference [SMD] = 0.20-0.80, primary SMD = 0.40) and meta-analytic effect sizes (ES(MA)). We performed meta-regression to estimate the influence of including underpowered studies in meta-analyses. RESULTS: We included 256 reviews with 10 686 meta-analyses and 47 384 studies. Statistical power for continuous efficacy outcomes was very low across intervention and disorder types (overall median [IQR] power for SMD = 0.40: 0.32 [0.19-0.54]; for ES(MA): 0.23 [0.09-0.58]), only reaching conventionally acceptable levels (80%) for SMD = 0.80. Median power to detect the ES(MA) was higher in treatment-as-usual (TAU)/waitlist-controlled (0.49-0.63) or placebo-controlled (0.12-0.38) trials than in trials comparing active treatments (0.07-0.13). Adequately-powered studies produced smaller effect sizes than underpowered studies (B = -0.06, p ⩽ 0.001). CONCLUSIONS: Power to detect both predetermined and meta-analytic effect sizes in psychiatric trials was low across all interventions and disorders examined. Consistent with the presence of reporting bias, underpowered studies produced larger effect sizes than adequately-powered studies. These results emphasize the need to increase sample sizes and to reduce reporting bias against studies reporting null results to improve the reliability of the published literature.