Abstract
Scientific inquiry involves observations and measurements, some of which are planned and some of which are not. The most interesting or unusual observations might be regarded as discoveries and therefore particularly worthy of publication. However, the observational process is fraught with inferential land mines, especially if the discoveries are serendipitous. Multiple observations increase the probability of false-positive conclusions and have led to many false and otherwise misleading publications. Statisticians recommend adjustments to final inferences with the goal of reducing the rate of false positives, a strategy that increases the rate of false negatives. Some scientists object to making such adjustments, arguing that it should not be more difficult to determine the validity of a discovery simply because other observations were made. Which tack is right? How does one decide that any particular scientific discovery is real? Unfortunately, there is no panacea, no one-size-fits-all approach. The goal of this commentary is to elucidate the issues and provide recommendations for conducting and reporting results of empirical studies, with emphasis on the problems of multiple comparisons and other types of multiplicities, including what I call "silent multiplicities." Because of the many observations, outcomes, subsets, treatments, etc, that are typically made or addressed in epidemiology and biomarker research, these recommendations may be particularly relevant for such studies. However, the lessons apply quite generally. I consider both frequentist and Bayesian statistical approaches.