Abstract
BACKGROUND: Item response theory (IRT) analyses of alcohol use disorder (AUD) and other psychological disorders are a predominant method for assessing overall and individual criterion severity for psychiatric diagnosis. However, no investigation has established the consistency of the relative criteria severities across different samples. METHOD: PubMed/Medline, PsycINFO, Web of Science and ProQuest databases were queried for entries relating to alcohol use and IRT. Study data were extracted using a standardized data entry sheet. Consistency of reported criteria severities across studies was analysed using generalizability theory to estimate generalized intraclass correlations (ICCs). RESULTS: A total of 451 citations were screened and 34 papers (30 unique samples) included in the research synthesis. The AUD criteria set exhibited low consistency in the ordering of criteria using both traditional [ICC = 0.16, 95% confidence interval (CI) 0.06-0.56] and generalized (ICC = 0.18, 95% CI 0.15-0.21) approaches. These results were partially accounted for by previously studied factors such as age and type of sample (e.g. clinical v. community), but the largest source of unreliability was the diagnostic instrument employed. CONCLUSIONS: Despite the robust finding of unidimensional structure of AUDs, inconsistency in the relative severities across studies suggests low replicability, challenging the generalizability of findings from any given study. Explicit modeling of well-studied factors like age and sample type is essential and increases the generalizability of findings. Moreover, while the development of structured diagnostic interviews is considered a landmark contribution toward improving psychiatric research, variability across instruments has not been fully appreciated and is substantial.