Abstract
Motivated by the need to model dose-response or dose-toxicity curves in clinical trials, we develop a new horseshoe-based prior for Bayesian isotonic regression modeling a binary outcome against an ordered categorical predictor, where the probability of the outcome is assumed to be monotonically non-decreasing with the predictor. The set of differences between outcome probabilities in consecutive categories of the predictor is equipped with a multivariate prior having support over simplex. The Dirichlet distribution, which can be derived from a normalized sum of independent gamma-distributed random variables, is a natural choice of prior, but using mathematical and simulation-based arguments, we show that the resulting posterior is prone to underflow and other numerical instabilities, even under simple data configurations. We propose an alternative prior based on horseshoe-type shrinkage that is numerically more stable. We show that this horseshoe-based prior is not subject to the numerical instability seen in the Dirichlet/gamma-based prior and that the horseshoe-based posterior can estimate the underlying true curve more efficiently than the Dirichlet-based one. We demonstrate the use of this prior in a model predicting the occurrence of radiation-induced lung toxicity in lung cancer patients as a function of dose delivered to normal lung tissue. Our methodology is implemented in the R package isotonicBayes and therefore suitable for use in the design of dose-finding studies or other dose-response modeling contexts.