Abstract
There is a persistent view of a significant minority in the medical informatics community that the randomized controlled trial (RCT) has a limited role to play in evaluating clinical information systems. A common reason voiced by skeptics is that these systems are fundamentally different from drug interventions, so the RCT is irrelevant. There is an urgent need to promote the use of RCTs, given the shift to evidence-based policy and the need to demonstrate cost-effectiveness of these systems. The authors suggest returning to first principles and argue that what is required is clarity about how to match methods to evaluation questions. The authors address common concerns about RCTs, and the extent to which they are fallacious, and also discuss the challenges of conducting RCTs in informatics and alternative study designs when randomized trials are infeasible. While neither a perfect nor universal evaluation method, RCTs form an important part of an evaluator's toolkit.