Active Bipartite Ranking with Smooth Posterior Distributions
Telecom Paris（电信巴黎学院）
James Cheshire, Stephan Clémençon
发表信息：
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, pages 2044--2052, year 2025, volume 258, series Proceedings of Machine Learning Research, publisher PMLR
In this article, bipartite ranking, a statistical learning problem involved in many applications and widely studied in the passive context, is approached in a much more general \textit{active setting} than the discrete one previously considered in the literature. While the latter assumes that the conditional distribution is piece wise constant, the framework we develop permits in contrast to deal with continuous conditional distributions, provided that they fulfill a Hölder smoothness constraint. We first show that a naive approach based on discretisation at a uniform level, fixed \textit{a priori} and consisting in applying next the active strategy designed for the discrete setting generally fails. Instead, we propose a novel algorithm, referred to as smooth-rank and designed for the continuous setting, which aims to minimise the distance between the ROC curve of the estimated ranking rule and the optimal one w.r.t. the $\sup$ norm. We show that, for a fixed confidence level $ε>0$ and probability $δ\in (0,1)$, smooth-rank is PAC$(ε,δ)$. In addition, we provide a problem dependent upper bound on the expected sampling time of smooth-rank and establish a problem dependent lower bound on the expected sampling time of any PAC$(ε,δ)$ algorithm. Beyond the theoretical analysis carried out, numerical results are presented, providing solid empirical evidence of the performance of the algorithm proposed, which compares favorably with alternative approaches.
This paper is devoted to the bipartite ranking problem, a classical statistical learning task, in a high dimensional setting. We propose a scoring and ranking strategy based on the PAC-Bayesian approach. We consider nonlinear additive scoring functions, and we derive non-asymptotic risk bounds under a sparsity assumption. In particular, oracle inequalities in probability holding under a margin condition assess the performance of our procedure, and prove its minimax optimality. An MCMC-flavored algorithm is proposed to implement our method, along with its behavior on synthetic and real-life dat…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；US Army Research Laboratory（美国陆军研究实验室）
In this paper, we present a general framework for distilling expectations with respect to the Bayesian posterior distribution of a deep neural network classifier, extending prior work on the Bayesian Dark Knowledge framework. The proposed framework takes as input "teacher" and student model architectures and a general posterior expectation of interest. The distillation method performs an online compression of the selected posterior expectation using iteratively generated Monte Carlo samples. We focus on the posterior predictive distribution and expected entropy as distillation targets. We inve…
Ranking sportsmen whose careers took place in different eras is often a contentious issue and the topic of much debate. In this paper we focus on cricket and examine what conclusions may be drawn about the ranking of Test batsmen using data on batting scores from the first Test in 1877 onwards. The overlapping nature of playing careers is exploited to form a bridge from past to present so that all players can be compared simultaneously, rather than just relative to their contemporaries. The natural variation in runs scored by a batsman is modelled by an additive log-linear model with year, age…
Université du Québec à Montréal（蒙特利尔魁北克大学）；Université de Montréal（蒙特利尔大学）；Université catholique de Louvain（天主教鲁汶大学）
A recommender system based on ranks is proposed, where an expert's ranking of a set of objects and a user's ranking of a subset of those objects are combined to make a prediction of the user's ranking of all objects. The rankings are assumed to be induced by latent continuous variables corresponding to the grades assigned by the expert and the user to the objects. The dependence between the expert and user grades is modelled by a copula in some parametric family. Given a prior distribution on the copula parameter, the user's complete ranking is predicted by the mode of the posterior predictive…
University of California, Berkeley（加州大学伯克利分校）；Microsoft Research（微软研究院）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Stony Brook University（石溪大学）；University of Washington（华盛顿大学）；Microsoft AI for Good（微软AI向善）
We propose a method for jointly inferring labels across a collection of data samples, where each sample consists of an observation and a prior belief about the label. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs. This formulation unifies various machine learning settings; the weak beliefs can come in the form of noisy or incomplete labels, likelihoods given by a different prediction mechanism on auxiliary input, or common-sense priors reflecting knowledge …
Surrogate Regret Bounds for Bipartite Ranking via Strongly Proper Losses
Indian Institute of Science（印度科学学院）
二分排名问题，其中实例被标记为正面或负面，目标是学习一个评分函数，该函数最大限度地减少对一对正面和负面实例的错误排名概率（或等效地，最大化ROC曲线下的面积），近年来已被广泛研究。一个占主导地位的理论和算法框架的问题已经减少二分排名成对分类，特别是，它是众所周知的，二分排名遗憾可以制定成对分类遗憾，这反过来又可以使用通常的遗憾界分类问题的上限。最近，Kotlowski et al.（2011）根据与标准（非成对）逻辑和指数损失的平衡版本相关的遗憾，显示了二分排序的遗憾界限。在本文中，我们表明，这种（非成对）代理遗憾界的二分排名可以得到一个广泛的类适当的（复合）损失，我们的术语作为强适当的。我们的证明技术比Kotlowski等人（2011）简单得多，并且依赖于Reid和Williamson（2010，2011）和其他人最近阐明的适当（复合）损失的性质。我们的结果产生明确的代理边界（没有隐藏的平衡条款）在各种强适当的损失，包括例如逻辑，指数，平方和平方铰链损失的特殊情况。我们还通过Clemencon和Robbiano（2011）的最新结果在某些低噪声条件下获得了更紧密的代理边界。
Bipartite ranking is an important supervised learning problem; however, unlike regression or classification, it has a quadratic dependence on the number of samples. To circumvent the prohibitive sample cost, many recent work focus on stochastic gradient-based methods. In this paper we consider an alternative approach, which leverages the structure of the widely-adopted pairwise squared loss, to obtain a stochastic and low cost algorithm that does not require stochastic gradients or learning rates. Using a novel uniform risk bound---based on matrix and vector concentration inequalities---we sho…
Many datasets give partial information about an ordering or ranking by indicating which team won a game, which item a user prefers, or who infected whom. We define a continuous spin system whose Gibbs distribution is the posterior distribution on permutations, given a probabilistic model of these interactions. Using the cavity method we derive a belief propagation algorithm that computes the marginal distribution of each node's position. In addition, the Bethe free energy lets us approximate the number of linear extensions of a partial order and perform model selection between competing probab…
Where machine-learned predictive risk scores inform high-stakes decisions, such as bail and sentencing in criminal justice, fairness has been a serious concern. Recent work has characterized the disparate impact that such risk scores can have when used for a binary classification task. This may not account, however, for the more diverse downstream uses of risk scores and their non-binary nature. To better account for this, in this paper, we investigate the fairness of predictive risk scores from the point of view of a bipartite ranking task, where one seeks to rank positive examples higher tha…
Ranked data arise in many areas of application ranging from the ranking of up-regulated genes for cancer to the ranking of academic statistics journals. Complications can arise when rankers do not report a full ranking of all entities; for example, they might only report their top--$M$ ranked entities after seeing some or all entities. It can also be useful to know whether rankers are equally informative, and whether some entities are effectively judged to be exchangeable. When there is important subgroup structure in the data, summaries such as aggregate (overall) rankings can be misleading. …
Recently, crowdsourcing has emerged as an effective paradigm for human-powered large scale problem solving in various domains. However, task requester usually has a limited amount of budget, thus it is desirable to have a policy to wisely allocate the budget to achieve better quality. In this paper, we study the principle of information maximization for active sampling strategies in the framework of HodgeRank, an approach based on Hodge Decomposition of pairwise ranking data with multiple workers. The principle exhibits two scenarios of active sampling: Fisher information maximization that lea…
Data in the form of ranking lists are frequently encountered, and combining ranking results from different sources can potentially generate a better ranking list and help understand behaviors of the rankers. Of interest here are the rank data under the following settings: (i) covariate information available for the ranked entities; (ii) rankers of varying qualities or having different opinions; and (iii) incomplete ranking lists for non-overlapping subgroups. We review some key ideas built around the Thurstone model family by researchers in the past few decades and provide a unifying approach …
BayesMallows is an R package for analyzing data in the form of rankings or preferences with the Mallows rank model, and its finite mixture extension, in a Bayesian probabilistic framework. The Mallows model is a well-known model, grounded on the idea that the probability density of an observed ranking decreases exponentially fast as its distance to the location parameter increases. Despite the model being quite popular, this is the first Bayesian implementation that allows a wide choice of distances, and that works well with a large amount of items to be ranked. BayesMallows supports footrule,…
LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI、巴黎电信学院、巴黎综合理工学院）；INRIA（法国国家信息与自动化研究所）
Many applications of AI involve scoring individuals using a learned function of their attributes. These predictive risk scores are then used to take decisions based on whether the score exceeds a certain threshold, which may vary depending on the context. The level of delegation granted to such systems in critical applications like credit lending and medical diagnosis will heavily depend on how questions of fairness can be answered. In this paper, we study fairness for the problem of learning scoring functions from binary labeled data, a classic learning task known as bipartite ranking. We arg…
University of Science and Technology of China（中国科学技术大学）；Baidu Talent Intelligence Center, Baidu Inc.（百度人才智能中心（百度公司））；Business Intelligence Lab, Baidu Research（百度研究院商业智能实验室）
The recent development of online recommender systems has a focus on collaborative ranking from implicit feedback, such as user clicks and purchases. Different from explicit ratings, which reflect graded user preferences, the implicit feedback only generates positive and unobserved labels. While considerable efforts have been made in this direction, the well-known pairwise and listwise approaches have still been limited by various challenges. Specifically, for the pairwise approaches, the assumption of independent pairwise preference is not always held in practice. Also, the listwise approaches…
Posterior Regimes and Latent Deception: Variational Bayesian Inference in Hidden Markov Models for Sequential Fraud Detection in Financial Transactions
University of Stirling（斯特灵大学）；University of Nottingham（诺丁汉大学）
Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
Department of Computer and Data Sciences, Case Western Reserve University（计算机与数据科学系，凯斯西储大学）；Department of Physics, Case Western Reserve University（物理系，凯斯西储大学）
London School of Economics（伦敦政治经济学院）；KAIST（韩国科学技术院）
In this paper, we study a popular method for inference of the Bradley-Terry model parameters, namely the MM algorithm, for maximum likelihood estimation and maximum a posteriori probability estimation. This class of models includes the Bradley-Terry model of paired comparisons, the Rao-Kupper model of paired comparisons allowing for tie outcomes, the Luce choice model, and the Plackett-Luce ranking model. We establish tight characterizations of the convergence rate for the MM algorithm, and show that it is essentially equivalent to that of a gradient descent algorithm. For the maximum likeliho…
Multistage ranking models, including the popular Plackett-Luce distribution (PL), rely on the assumption that the ranking process is performed sequentially, by assigning the positions from the top to the bottom one (forward order). A recent contribution to the ranking literature relaxed this assumption with the addition of the discrete-valued reference order parameter, yielding the novel Extended Plackett-Luce model (EPL). Inference on the EPL and its generalization into a finite mixture framework was originally addressed from the frequentist perspective. In this work, we propose the Bayesian …
Choice behavior and preferences typically involve numerous and subjective aspects that are difficult to be identified and quantified. For this reason, their exploration is frequently conducted through the collection of ordinal evidence in the form of ranking data. A ranking is an ordered sequence resulting from the comparative evaluation of a given set of items according to a specific criterion. Multistage ranking models, including the popular Plackett-Luce distribution (PL), rely on the assumption that the ranking process is performed sequentially, by assigning the positions from the top to t…
Bipartite ranking, which aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data, is widely adopted in various applications where sample prioritization is needed. Recently, there have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups defined by sensitive attributes. While there could be trade-off between fairness and performance, in this paper we propose a model agnostic post-processing framework for balancing them in the bipartite ranking scenario. Specifically, we …
Bayesian inverse reinforcement learning (IRL) methods are ideal for safe imitation learning, as they allow a learning agent to reason about reward uncertainty and the safety of a learned policy. However, Bayesian IRL is computationally intractable for high-dimensional problems because each sample from the posterior requires solving an entire Markov Decision Process (MDP). While there exist non-Bayesian deep IRL methods, these methods typically infer point estimates of reward functions, precluding rigorous safety and uncertainty analysis. We propose Bayesian Reward Extrapolation (B-REX), a high…
The ROC curve is the gold standard for measuring the performance of a test/scoring statistic regarding its capacity to discriminate between two statistical populations in a wide variety of applications, ranging from anomaly detection in signal processing to information retrieval, through medical diagnosis. Most practical performance measures used in scoring/ranking applications such as the AUC, the local AUC, the p-norm push, the DCG and others, can be viewed as summaries of the ROC curve. In this paper, the fact that most of these empirical criteria can be expressed as two-sample linear rank …
Bayesian genome-wide clustering and variable selection of transcriptomic data via rank-based mixtures
Oslo Centre for Biostatistics and Epidemiology, Department of Biostatistics, University of Oslo（奥斯陆大学生物统计与流行病学中心）；Department of Mathematics, University of Oslo（奥斯陆大学数学系）；Department of Economics and Statistics, University of Naples Federico II and CSEF（那不勒斯费德里科二世大学经济与统计系及CSEF）；The Norwegian Centre for Knowledge-driven Machine Learning, Integreat（挪威知识驱动机器学习中心）；Department of Cancer Genetics, Oslo University Hospital（奥斯陆大学医院癌症遗传学系）
Cluster-specific ranking and variable importance for Scottish regional deprivation via vine mixtures
Delft Institute of Applied Mathematics, Delft University of Technology（代尔夫特理工大学）；School of Mathematics and Maxwell Institute for Mathematical Sciences, University of Edinburgh（爱丁堡大学）；Department of Mathematics, School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学）
The aim of this work is to study the problem of prior elicitation for the Mallows model with Spearman's distance, a popular distance-based model for rankings or permutation data. Previous Bayesian inference for such model has been limited to the use of the uniform prior over the space of permutations. We present a novel strategy to elicit subjective prior beliefs on the location parameter of the model, discussing the interpretation of hyper-parameters and the implication of prior choices for the posterior analysis.
Area under ROC (AUC) is an important metric for binary classification and bipartite ranking problems. However, it is difficult to directly optimizing AUC as a learning objective, so most existing algorithms are based on optimizing a surrogate loss to AUC. One significant drawback of these surrogate losses is that they require pairwise comparisons among training data, which leads to slow running time and increasing local storage for online learning. In this work, we describe a new surrogate loss based on a reformulation of the AUC risk, which does not require pairwise comparison but rankings of…
