Abstract
This paper studies the behaviour of the Elo rating system when the underlying modelling assumptions are not met. The Elo rating system is a popular statistical technique used to analyse pairwise comparison data. It is perhaps best known for rating chess players. The crucial assumption behind the Elo rating system is that the probability of which players wins a chess game depends primarily on a real-valued parameter, that quantifies the player's "skill". This implicitly assumes that the binary relation "more likely to win against" is transitive. This paper studies how the Elo rating system behaves when this assumption is relaxed. First, we prove that once the assumption of transitivity is relaxed, the Elo ratings exhibits the undesirable property that estimated ratings are dependent on who plays who. Second, we prove that even when the assumption of transitivity is relaxed, for a given distribution with which players are selected, there is a unique point where the expected change of Elo ratings at that point is zero. This point represents the maximum likelihood estimator of the Elo ratings, given the observed data. Finally, we introduce a statistic that can be used to measure the intransitivity present in a game. We derive this measurement, and demonstrate on simulated data that it satisfies some useful sanity tests.