We propose a time-varying generalization of the Bradley-Terry model that allows for nonparametric modeling of dynamic global rankings of distinct teams. We develop a novel estimator that relies on kernel smoothing to pre-process the pairwise comparisons over time and is applicable in sparse settings where the Bradley-Terry may not be fit. We obtain necessary and sufficient conditions for the existence and uniqueness of our estimator. We also derive time-varying oracle bounds for both the estimation error and the excess risk in the model-agnostic setting where the Bradley-Terry model is not necessarily the true data generating process. We thoroughly test the practical effectiveness of our model using both simulated and real world data and suggest an efficient data-driven approach for bandwidth tuning.
The Bradley-Terry model is widely used for pairwise comparison data analysis. In this paper, we analyze the asymptotic behavior of the maximum likelihood estimator of the Bradley-Terry model in its logistic parameterization, under a general class of linear identifiability constraints. We show that the constraint requiring the Bradley-Terry scores for all compared objects to sum to zero minimizes the sum of the variances of the estimated scores, and recommend using this constraint in practice.
The Bradley-Terry-Luce (BTL) model is a benchmark model for pairwise comparisons between individuals. Despite recent progress on the first-order asymptotics of several popular procedures, the understanding of uncertainty quantification in the BTL model remains largely incomplete, especially when the underlying comparison graph is sparse. In this paper, we fill this gap by focusing on two estimators that have received much recent attention: the maximum likelihood estimator (MLE) and the spectral estimator. Using a unified proof strategy, we derive sharp and uniform non-asymptotic expansions for…
Ranking problems based on pairwise comparisons, such as those arising in online gaming, often involve a large pool of items to order. In these situations, the gap in performance between any two items can be significant, and the smallest and largest winning probabilities can be very close to zero or one. Furthermore, each item may be compared only to a subset of all the items, so that not all pairwise comparisons are observed. In this paper, we study the performance of the Bradley-Terry-Luce model for ranking from pairwise comparison data under more realistic settings than those considered in t…
For nonbalanced paired comparisons, a wide variety of ranking methods have been proposed. One of the best popular methods is the Bradley-Terry model in which the ranking of a set of objects is decided by the maximum likelihood estimates (MLEs) of merits parameters. However, the existence of MLE for the Bradley-Terry model and its generalized models to allow for tied observation or home-field advantage or both to occur, crucially depends on the strong connection condition on the directed graph constructed by a win-loss matrix. When this condition fails, the MLE does not exist and hence there is…
In the Bradley-Terry model for paired comparisons, and its extensions to include order effects and ties, the maximum likelihood estimates of probabilities of certain outcomes can be 0 or 1 under certain data configurations. This poses problems for standard estimation methods. In this paper, we give algorithms for identifying the outcomes with estimated probability 0 or 1, and indicate how the remaining probabilities may be estimated and summarized.
The Bradley-Terry model assigns probabilities for the outcome of paired comparison experiments based on strength parameters associated with the objects being compared. We consider different proposed choices of prior parameter distributions for Bayesian inference of the strength parameters based on the paired comparison results. We evaluate them according to four desiderata motivated by the use of inferred Bradley-Terry parameters to rate teams on the basis of outcomes of a set of games: invariance under interchange of teams, invariance under interchange of winning and losing, normalizability a…
Optimal design theory for nonlinear regression studies local optimality on a given design space. We identify designs for the Bradley--Terry paired comparison model with small undirected graphs and prove that every saturated D-optimal design is represented by a path. We discuss the case of four alternatives in detail and derive explicit polynomial inequality descriptions for optimality regions in parameter space. Using these regions, for each point in parameter space we can prescribe a D-optimal design.
In a wide class of paired comparisons, especially in the sports games, in which all subjects are divided into several groups, the intragroup comparisons are dense and the intergroup comparisons are sparse. Typical examples include the NFL regular season. Motivated by these situations, we propose group sparsity for paired comparisons and show the consistency and asymptotical normality of the maximum likelihood estimate in the Bradley-Terry model when the number of parameters goes to infinity in this paper. Simulations are carried out to illustrate the group sparsity and asymptotical results.
This paper deals with the estimation of the unknown distribution of hidden random variables from the observation of pairwise comparisons between these variables. This problem is inspired by recent developments on Bradley-Terry models in random environment since this framework happens to be relevant to predict for instance the issue of a championship from the observation of a few contests per team. This paper provides three contributions on a Bayesian nonparametric approach to solve this problem. First, we establish contraction rates of the posterior distribution. We also propose a Markov Chain…
A number of applications (e.g., AI bot tournaments, sports, peer grading, crowdsourcing) use pairwise comparison data and the Bradley-Terry-Luce (BTL) model to evaluate a given collection of items (e.g., bots, teams, students, search results). Past work has shown that under the BTL model, the widely-used maximum-likelihood estimator (MLE) is minimax-optimal in estimating the item parameters, in terms of the mean squared error. However, another important desideratum for designing estimators is fairness. In this work, we consider fairness modeled by the notion of bias in statistics. We show that…
We consider the problem of estimating the probability matrix governing a tournament or linkage in graphs from incomplete observations, under the assumption that the probability matrix satisfies natural monotonicity constraints after being permuted in both rows and columns by some latent permutation. This condition is classical in the social sciences literature (see Fishburn(1973)) and has been studied in the statistics literature in recent work (see Chatterjee (2015) and Shah et al. (2016)). In this paper, we investigate in detail a natural estimator which bypasses the need to search over all …
The Bradley-Terry-Luce (BTL) model is a classic and very popular statistical approach for eliciting a global ranking among a collection of items using pairwise comparison data. In applications in which the comparison outcomes are observed as a time series, it is often the case that data are non-stationary, in the sense that the true underlying ranking changes over time. In this paper we are concerned with localizing the change points in a high-dimensional BTL model with piece-wise constant parameters. We propose novel and practicable algorithms based on dynamic programming that can consistentl…
