Abstract
OBJECTIVE: To analyze distribution of "dramatic", large treatment effects. STUDY DESIGN & SETTING: Pareto distribution modeling of previously reported cohorts of 3,486 randomized trials (RCTs) that enrolled 1,532,459 patients and 730 non-randomized studies (NRS) enrolling 1,650,658 patients. RESULTS: We calculated the Pareto α parameter, which determines the tail of the distribution for various starting points of distribution [odds ratio(min) (OR(min))]. In default analysis using all data at OR(min) ≥1, Pareto distribution fit well to the treatment effects of RCTs favoring the new treatments (P = 0.21, Kolmogorov-Smirnov test) with best α = 2.32. For NRS, Pareto fit for OR(min) ≥2 with best α = 1.91. For RCTs, theoretical 99th percentile OR was 32.7. The actual 99th percentile OR was 25; which converted into relative risk (RR) = 7.1. The maximum observed effect size was OR = 121 (RR = 11.45). For NRS, theoretical 99th percentile was OR = 315. The actual 99th percentile OR was 294 (RR = 13). The maximum observed effect size was OR = 1473 (RR = 66). CONCLUSIONS: The effects sizes observed in RCTs and NRS considerably overlap. Large effects are rare and there is no clear threshold for dramatic effects that would obviate future RCTs.