User preference learning is generally a hard problem. Individual preferences are typically unknown even to users themselves, while the space of choices is infinite. Here we study user preference learning from information-theoretic perspective. We model preference learning as a system with two interacting sub-systems, one representing a user with his/her preferences and another one representing an agent that has to learn these preferences. The user with his/her behaviour is modeled by a parametric preference function. To efficiently learn the preferences and reduce search space quickly, we propose the agent that interacts with the user to collect the most informative data for learning. The agent presents two proposals to the user for evaluation, and the user rates them based on his/her preference function. We show that the optimum agent strategy for data collection and preference learning is a result of maximin optimization of the normalized weighted Kullback-Leibler (KL) divergence between true and agent-assigned predictive user response distributions. The resulting value of KL-divergence, which we also call remaining system uncertainty (RSU), provides an efficient performance metric in the absence of the ground truth. This metric characterises how well the agent can predict user and, thus, the quality of the underlying learned user (preference) model. Our proposed agent comprises sequential mechanisms for user model inference and proposal generation. To infer the user model (preference function), Bayesian approximate inference is used in the agent. The data collection strategy is to generate proposals, responses to which help resolving uncertainty associated with prediction of the user responses the most. The efficiency of our approach is validated by numerical simulations. Also a real-life example of preference learning application is provided.
University of California Irvine（加州大学欧文分校）；Xerox Research Centre India（施乐印度研究中心）
We study the problem of ranking a set of items from nonactively chosen pairwise preferences where each item has feature information with it. We propose and characterize a very broad class of preference matrices giving rise to the Feature Low Rank (FLR) model, which subsumes several models ranging from the classic Bradley-Terry-Luce (BTL) (Bradley and Terry 1952) and Thurstone (Thurstone 1927) models to the recently proposed blade-chest (Chen and Joachims 2016) and generic low-rank preference (Rajkumar and Agarwal 2016) models. We use the technique of matrix completion in the presence of side i…
University of Nantes（南特大学）；University of Cambridge（剑桥大学）；Tencent Games（腾讯游戏）
In this paper we present a hybrid active sampling strategy for pairwise preference aggregation, which aims at recovering the underlying rating of the test candidates from sparse and noisy pairwise labelling. Our method employs Bayesian optimization framework and Bradley-Terry model to construct the utility function, then to obtain the Expected Information Gain (EIG) of each pair. For computational efficiency, Gaussian-Hermite quadrature is used for estimation of EIG. In this work, a hybrid active sampling strategy is proposed, either using Global Maximum (GM) EIG sampling or Minimum Spanning T…
In this paper we consider the collaborative ranking setting: a pool of users each provides a small number of pairwise preferences between $d$ possible items; from these we need to predict preferences of the users for items they have not yet seen. We do so by fitting a rank $r$ score matrix to the pairwise data, and provide two main contributions: (a) we show that an algorithm based on convex optimization provides good generalization guarantees once each user provides as few as $O(r\log^2 d)$ pairwise comparisons -- essentially matching the sample complexity required in the related matrix compl…
A number of applications require two-sample testing on ranked preference data. For instance, in crowdsourcing, there is a long-standing question of whether pairwise comparison data provided by people is distributed similar to ratings-converted-to-comparisons. Other examples include sports data analysis and peer grading. In this paper, we design two-sample tests for pairwise comparison data and ranking data. For our two-sample test for pairwise comparison data, we establish an upper bound on the sample complexity required to correctly distinguish between the distributions of the two sets of sam…
Many real-world engineering problems rely on human preferences to guide their design and optimization. We present PrefOpt, an open source package to simplify sequential optimization tasks that incorporate human preference feedback. Our approach extends an existing latent variable model for binary preferences to allow for observations of equivalent preference from users.
Conducting pairwise comparisons is a widely used approach in curating human perceptual preference data. Typically raters are instructed to make their choices according to a specific set of rules that address certain dimensions of image quality and aesthetics. The outcome of this process is a dataset of sampled image pairs with their associated empirical preference probabilities. Training a model on these pairwise preferences is a common deep learning approach. However, optimizing by gradient descent through mini-batch learning means that the "global" ranking of the images is not explicitly tak…
The literature on ranking from ordinal data is vast, and there are several ways to aggregate overall preferences from pairwise comparisons between objects. In particular, it is well known that any Nash equilibrium of the zero sum game induced by the preference matrix defines a natural solution concept (winning distribution over objects) known as a von Neumann winner. Many real-world problems, however, are inevitably multi-criteria, with different pairwise preferences governing the different criteria. In this work, we generalize the notion of a von Neumann winner to the multi-criteria setting b…
Stanford University（斯坦福大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）
This paper explores the preference-based top-$K$ rank aggregation problem. Suppose that a collection of items is repeatedly compared in pairs, and one wishes to recover a consistent ordering that emphasizes the top-$K$ ranked items, based on partially revealed preferences. We focus on the Bradley-Terry-Luce (BTL) model that postulates a set of latent preference scores underlying all items, where the odds of paired comparisons depend only on the relative scores of the items involved. We characterize the minimax limits on identifiability of top-$K$ ranked items, in the presence of random and non…
Active Preference Learning over Latent Preference Archetypes for Many-Objective Bayesian Optimization
University of Manchester（曼彻斯特大学）；University of Edinburgh（爱丁堡大学）；Aalto University（阿尔托大学）；University of Amsterdam（阿姆斯特丹大学）；ELLIS Institute Finland（芬兰ELLIS研究所）
Indian Institute of Science, Bangalore（印度科学学院，班加罗尔）；Google Inc., Mountain View（谷歌公司，山景城）
We consider the problem of optimal recovery of true ranking of $n$ items from a randomly chosen subset of their pairwise preferences. It is well known that without any further assumption, one requires a sample size of $Ω(n^2)$ for the purpose. We analyze the problem with an additional structure of relational graph $G([n],E)$ over the $n$ items added with an assumption of \emph{locality}: Neighboring items are similar in their rankings. Noting the preferential nature of the data, we choose to embed not the graph, but, its \emph{strong product} to capture the pairwise node relationships. Further…
We propose a novel parameterized family of Mixed Membership Mallows Models (M4) to account for variability in pairwise comparisons generated by a heterogeneous population of noisy and inconsistent users. M4 models individual preferences as a user-specific probabilistic mixture of shared latent Mallows components. Our key algorithmic insight for estimation is to establish a statistical connection between M4 and topic models by viewing pairwise comparisons as words, and users as documents. This key insight leads us to explore Mallows components with a separable structure and leverage recent adva…
We consider the problem of estimating a ranking on a set of items from noisy pairwise comparisons given item features. We address the fact that pairwise comparison data often reflects irrational choice, e.g. intransitivity. Our key observation is that two items compared in isolation from other items may be compared based on only a salient subset of features. Formalizing this framework, we propose the salient feature preference model and prove a finite sample complexity result for learning the parameters of our model and the underlying ranking with maximum likelihood estimation. We also provide…
University of Virginia（弗吉尼亚大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
We propose the Heterogeneous Thurstone Model (HTM) for aggregating ranked data, which can take the accuracy levels of different users into account. By allowing different noise distributions, the proposed HTM model maintains the generality of Thurstone's original framework, and as such, also extends the Bradley-Terry-Luce (BTL) model for pairwise comparisons to heterogeneous populations of users. Under this framework, we also propose a rank aggregation algorithm based on alternating gradient descent to estimate the underlying item scores and accuracy levels of different users simultaneously fro…
We propose a scalable Bayesian preference learning method for jointly predicting the preferences of individuals as well as the consensus of a crowd from pairwise labels. Peoples' opinions often differ greatly, making it difficult to predict their preferences from small amounts of personal data. Individual biases also make it harder to infer the consensus of a crowd when there are few labels per item. We address these challenges by combining matrix factorisation with Gaussian processes, using a Bayesian approach to account for uncertainty arising from noisy and sparse data. Our method exploits …
Integrating Response Time and Attention Duration in Bayesian Preference Learning for Multiple Criteria Decision Aiding
School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；Institute of Computing Science, Poznan University of Technology（波兹南理工大学计算科学研究所）；Hubei University of Economics（湖北经济学院）
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
School of Computer Science, McGill University, Montreal, Quebec, Canada（麦吉尔大学计算机科学学院）；Mila - Quebec AI Institute, Montreal, Quebec, Canada（魁北克人工智能研究所）；Department of Electrical Engineering, Stanford University, Stanford, California, USA（斯坦福大学电气工程系）
Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；University of Chicago（芝加哥大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Algorithmic machine teaching studies the interaction between a teacher and a learner where the teacher selects labeled examples aiming at teaching a target hypothesis. In a quest to lower teaching complexity and to achieve more natural teacher-learner interactions, several teaching models and complexity measures have been proposed for both the batch settings (e.g., worst-case, recursive, preference-based, and non-clashing models) as well as the sequential settings (e.g., local preference-based model). To better understand the connections between these different batch and sequential models, we …
Automata Learning of Preferences over Temporal Logic Formulas from Pairwise Comparisons
Department of Computer Science, Missouri State University, USA（计算机科学系，密苏里州立大学）；Department Electrical and Computer Engineering , University of Florida, USA（电气与计算机工程系，佛罗里达大学）
Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
Institute of Operations Research and Analytics（运营研究与分析研究所）；National University of Singapore（新加坡国立大学）；Department of Analytics and Operations（分析与运营系）；NUS Business School（新加坡国立大学商学院）
Max Planck Institute for Software Systems (MPI-SWS)（马克斯·普朗克软件系统研究所）；University of Chicago（芝加哥大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Algorithmic machine teaching studies the interaction between a teacher and a learner where the teacher selects labeled examples aiming at teaching a target hypothesis. In a quest to lower teaching complexity, several teaching models and complexity measures have been proposed for both the batch settings (e.g., worst-case, recursive, preference-based, and non-clashing models) and the sequential settings (e.g., local preference-based model). To better understand the connections between these models, we develop a novel framework that captures the teaching process via preference functions $Σ$. In o…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Meituan Inc（美团）；University of Southampton（南安普顿大学）
Modeling the evolution of user preference is essential in recommender systems. Recently, dynamic graph-based methods have been studied and achieved SOTA for recommendation, majority of which focus on user's stable long-term preference. However, in real-world scenario, user's short-term preference evolves over time dynamically. Although there exists sequential methods that attempt to capture it, how to model the evolution of short-term preference with dynamic graph-based methods has not been well-addressed yet. In particular: 1) existing methods do not explicitly encode and capture the evolutio…
Shanghai Jiao Tong University（上海交通大学）；Microsoft Research Asia（微软亚洲研究院）；Fudan University（复旦大学）
Capturing the dynamics in user preference is crucial to better predict user future behaviors because user preferences often drift over time. Many existing recommendation algorithms -- including both shallow and deep ones -- often model such dynamics independently, i.e., user static and dynamic preferences are not modeled under the same latent space, which makes it difficult to fuse them for recommendation. This paper considers the problem of embedding a user's sequential behavior into the latent space of user preferences, namely translating sequence to preference. To this end, we formulate the…
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Baichuan Inc.（百川智能）
IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models
HUJING Digital Media \& Entertainment Group (XingYun Lab), Beijing, China；Department of Automation, Tsinghua University, Beijing, China；Fudan University, Shanghai, China；Beihang University, Beijing, China；Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
INESCTEC；LIACS；HasLab, INESC TEC, Departamento de Informática, Universidade do Minho（Minho大学信息学院HasLab, INESC TEC）；DCC - Faculdade de Ciencias, Universidade do Porto（波尔图大学科学学院DCC）；LIACS, Leiden, Netherlands（莱顿大学LIACS）
In this paper we investigate two variants of association rules for preference data, Label Ranking Association Rules and Pairwise Association Rules. Label Ranking Association Rules (LRAR) are the equivalent of Class Association Rules (CAR) for the Label Ranking task. In CAR, the consequent is a single class, to which the example is expected to belong to. In LRAR, the consequent is a ranking of the labels. The generation of LRAR requires special support and confidence measures to assess the similarity of rankings. In this work, we carry out a sensitivity analysis of these similarity-based measur…
Pairwise and Attribute-Aware Decision Tree-Based Preference Elicitation for Cold-Start Recommendation
KU Leuven, Campus Kulak, Department of Public Health and Primary Care（KU莱顿大学）；Itec, imec research group at KU Leuven（Itec）；KU Leuven, Department of Mathematics（KU莱顿大学）
What Makes LLMs Effective Sequential Recommenders? A Study on Preference Intensity and Temporal Context
Department of Computer Science, Dartmouth College（达特茅斯大学计算机科学系）；ByteDance, US（字节跳动（美国））；Department of Computer Science and Engineering, University of Notre Dame（诺丁汉大学计算机科学与工程系）
School of Elec Eng, Computing and Mathematical Sciences, Curtin University（科廷大学电子工程、计算与数学科学学院）；School of Computer Science, University of Sydney（悉尼大学计算机科学学院）
We propose a novel IaaS composition framework that selects an optimal set of consumer requests according to the provider's qualitative preferences on long-term service provisions. Decision variables are included in the temporal conditional preference networks (TempCP-net) to represent qualitative preferences for both short-term and long-term consumers. The global preference ranking of a set of requests is computed using a \textit{k}-d tree indexing based temporal similarity measure approach. We propose an extended three-dimensional Q-learning approach to maximize the global preference ranking.…
