Positive predictive value and negative predictive value are two widely used parameters to assess the clinical usefulness of a medical diagnostic test. When there are two diagnostic tests, it is recommendable to make a comparative assessment of the values of these two parameters after applying the two tests to the same subjects (paired samples). The objective is then to make individual or global inferences about the difference or the ratio of the predictive value of the two diagnostic tests. These inferences are usually based on complex and not very intuitive expressions, some of which have subsequently been reformulated. We define the two properties of symmetry which any inference method must verify - symmetry in diagnoses and symmetry in the tests -, we propose new inference methods, and we define them with simple expressions. All of the methods are compared with each other, selecting the optimal method: (a) to obtain a confidence interval for the difference or ratio; (b) to perform an individual homogeneity test of the two predictive values; and (c) to carry out a global homogeneity test of the two predictive values.
Paired comparison data considered in this paper originate from the comparison of a large number N of individuals in couples. The dataset is a collection of results of contests between two individuals when each of them has faced n opponents, where n is much larger than N. Individual are represented by independent and identically distributed random parameters characterizing their abilities.The paper studies the maximum likelihood estimator of the parameters distribution. The analysis relies on the construction of a graphical model encoding conditional dependencies of the observations which are t…
For paired comparison experiments involving competing options described by two-level attributes several different methods of constructing designs having block paired observations under the main effects model are presented. These designs are compared to alternative designs available in the literature.
In paired comparison experiments respondents usually evaluate pairs of competing options. For this situation we introduce an appropriate model and derive optimal designs in the presence of second-order interactions when all attributes are dichotomous.
We present a Bayesian rating system based on the method of paired comparisons. Our system is a flexible generalization of the well-known Glicko, and in particular can better accommodate games with significant elements of luck. Our system is currently in use in the online game Duelyst II, and in that setting outperforms Glicko2.
A paired comparison analysis is the simplest way to make comparative judgments between objects where objects may be goods, services or skills. For a set of problems, this technique helps to choose the most important problem to solve first and/or provides the solution that will be the most effective. This paper presents the theory of paired comparisons method and contributes to the paired comparisons models by developing a new model based on t-distribution. The developed model is illustrated using a data set of citations among four famous journals of Statistics. Using Bayesian analysis, the jou…
In real life situations often paired comparisons involving alternatives of either full or partial profiles to mitigate cognitive burden are presented. For this situation the problem of finding optimal designs is considered in the presence of second-order interactions when all attributes have general common number of levels.
In psychological research often paired comparisons are used in which either full or partial profiles of the alternatives described by a common set of two-level attributes are presented. For this situation the problem of finding optimal designs is considered in the presence of third-order interactions.
The field of distribution-free predictive inference provides tools for provably valid prediction without any assumptions on the distribution of the data, which can be paired with any regression algorithm to provide accurate and reliable predictive intervals. The guarantees provided by these methods are typically marginal, meaning that predictive accuracy holds on average over both the training data set and the test point that is queried. However, it may be preferable to obtain a stronger guarantee of training-conditional coverage, which would ensure that most draws of the training data set res…
Thurstonian and Bradley-Terry models are the most commonly applied models in the analysis of paired comparison data. Since their introduction, numerous developments have been proposed in different areas. This paper provides an updated overview of these extensions, including how to account for object- and subject-specific covariates and how to deal with ordinal paired comparison data. Special emphasis is given to models for dependent comparisons. Although these models are more realistic, their use is complicated by numerical difficulties. We therefore concentrate on implementation issues. In pa…
It is shown how by not losing information on higher order interactions, optimal paired comparison designs involving alternatives of either full or partial profiles to reduce information overload as frequently encountered in applications can be constructed which enable identification of main effects up to third-order interactions when all attributes have general common number of levels.
Paired comparison models are used for analyzing data that involves pairwise comparisons among a set of objects. When the outcomes of the pairwise comparisons have no ties, the paired comparison models can be generalized as a class of binary response models. Receiver operating characteristic (ROC) curves and their corresponding areas under the curves are commonly used as performance metrics to evaluate the discriminating ability of binary response models. Despite their individual wide range of usage and their close connection to binary response models, ROC analysis to our knowledge has never be…
This paper proposes a new test for the comparison of conditional quantile curves when the outcome of interest, typically a duration, is subject to right censoring. The test can be applied both in the case of two independent samples and for paired data, and can be used for the comparison of quantiles at a fixed quantile level, a finite set of levels or a range of quantile levels. The asymptotic distribution of the proposed test statistics is obtained both under the null hypothesis and under local alternatives. We describe a bootstrap procedure in order to approximate the critical values, and pr…
This article introduces the bpcs R package (Bayesian Paired Comparison in Stan) and the statistical models implemented in the package. This package aims to facilitate the use of Bayesian models for paired comparison data in behavioral research. Bayesian analysis of paired comparison data allows parameter estimation even in conditions where the maximum likelihood does not exist, allows easy extension of paired comparison models, provide straightforward interpretation of the results with credible intervals, have better control of type I error, have more robust evidence towards the null hypothesi…
The predictive performance of any inferential model is critical to its practical success, but quantifying predictive performance is a subtle statistical problem. In this paper I show how the natural structure of any inferential problem defines a canonical measure of relative predictive performance and then demonstrate how approximations of this measure yield many of the model comparison techniques popular in statistics and machine learning.
In a wide class of paired comparisons, especially in the sports games, in which all subjects are divided into several groups, the intragroup comparisons are dense and the intergroup comparisons are sparse. Typical examples include the NFL regular season. Motivated by these situations, we propose group sparsity for paired comparisons and show the consistency and asymptotical normality of the maximum likelihood estimate in the Bradley-Terry model when the number of parameters goes to infinity in this paper. Simulations are carried out to illustrate the group sparsity and asymptotical results.
Missing data is a common issue in many biomedical studies. Under a paired design, some subjects may have missing values in either one or both of the conditions due to loss of follow-up, insufficient biological samples, etc. Such partially paired data complicate statistical comparison of the distribution of the variable of interest between the two conditions. In this paper, we propose a general class of test statistics based on the difference in weighted sample means without imposing any distributional or model assumption. An optimal weight is derived for this class of tests. Simulation studies…
Scientists often use a paired comparison of the areas under the receiver operating characteristic curves to decide which continuous cancer screening test has the best diagnostic accuracy. In the paired design, all participants are screened with both tests. Participants with unremarkable screening results enter a follow-up period. Participants with suspicious screening results and those who show evidence of disease during follow-up receive the gold standard test. The remaining participants are classified as non-cases, even though some may have occult disease. The standard analysis includes all …
Recently, new methods for model assessment, based on subsampling and posterior approximations, have been proposed for scaling leave-one-out cross-validation (LOO) to large datasets. Although these methods work well for estimating predictive performance for individual models, they are less powerful in model comparison. We propose an efficient method for estimating differences in predictive performance by combining fast approximate LOO surrogates with exact LOO subsampling using the difference estimator and supply proofs with regards to scaling characteristics. The resulting approach can be orde…
Method comparison studies are essential for development in medical and clinical fields. These studies often compare a cheaper, faster, or less invasive measuring method with a widely used one to see if they have sufficient agreement for interchangeable use. In the clinical and medical context, the response measurement is usually impacted not only by the measuring method but by the rater as well. This paper proposes a model-based approach to assess agreement of two measuring methods for paired repeated binary measurements under the scenario when the agreement between two measuring methods and t…
In this article, we review and compare a number of methods of spatial prediction. To demonstrate the breadth of available choices, we consider both traditional and more-recently-introduced spatial predictors. Specifically, in our exposition we review: traditional stationary kriging, smoothing splines, negative-exponential distance-weighting, Fixed Rank Kriging, modified predictive processes, a stochastic partial differential equation approach, and lattice kriging. This comparison is meant to provide a service to practitioners wishing to decide between spatial predictors. Hence, we provide tech…
Two procedures for checking Bayesian models are compared using a simple test problem based on the local Hubble expansion. Over four orders of magnitude, p-values derived from a global goodness-of-fit criterion for posterior probability density functions (Lucy 2017) agree closely with posterior predictive p-values. The former can therefore serve as an effective proxy for the difficult-to-calculate posterior predictive p-values.
The goal of this paper is to compare several widely used Bayesian model selection methods in practical model selection problems, highlight their differences and give recommendations about the preferred approaches. We focus on the variable subset selection for regression and classification and perform several numerical experiments using both simulated and real world data. The results show that the optimization of a utility estimate such as the cross-validation (CV) score is liable to finding overfitted models due to relatively high variance in the utility estimates when the data is scarce. This…
We investigate the forecasting ability of the most commonly used benchmarks in financial economics. We approach the usual caveats of probabilistic forecasts studies -small samples, limited models and non-holistic validations- by performing a comprehensive comparison of 15 predictive schemes during a time period of over 21 years. All densities are evaluated in terms of their statistical consistency, local accuracy and forecasting errors. Using a new composite indicator, the Integrated Forecast Score (IFS), we show that risk-neutral densities outperform historical-based predictions in terms of i…
