arXiv 1809.084152018-09-22cs.IR

Differentiable Unbiased Online Learning to Rank

Harrie Oosterhuis, Maarten de Rijke

备注：: Conference on Information and Knowledge Management 2018
DOI：: 10.1145/3269206.3271686

英文摘要：

Online Learning to Rank (OLTR) methods optimize rankers based on user interactions. State-of-the-art OLTR methods are built specifically for linear models. Their approaches do not extend well to non-linear models such as neural networks. We introduce an entirely novel approach to OLTR that constructs a weighted differentiable pairwise loss after each interaction: Pairwise Differentiable Gradient Descent (PDGD). PDGD breaks away from the traditional approach that relies on interleaving or multileaving and extensive sampling of models to estimate gradients. Instead, its gradient is based on inferring preferences between document pairs from user clicks and can optimize any differentiable model. We prove that the gradient of PDGD is unbiased w.r.t. user document pair preferences. Our experiments on the largest publicly available Learning to Rank (LTR) datasets show considerable and significant improvements under all levels of interaction noise. PDGD outperforms existing OLTR methods both in terms of learning speed as well as final convergence. Furthermore, unlike previous OLTR methods, PDGD also allows for non-linear models to be optimized effectively. Our results show that using a neural network leads to even better performance at convergence than a linear model. In summary, PDGD is an efficient and unbiased OLTR approach that provides a better user experience than previously possible.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Unbiased Learning to Rank: Counterfactual and Online Approaches

This tutorial covers and contrasts the two main methodologies in unbiased Learning to Rank (LTR): Counterfactual LTR and Online LTR. There has long been an interest in LTR from user interactions, however, this form of implicit feedback is very biased. In recent years, unbiased LTR methods have been introduced to remove the effect of different types of bias caused by user-behavior in search. For instance, a well addressed type of bias is position bias: the rank at which a document is displayed heavily affects the interactions it receives. Counterfactual LTR methods deal with such types of bias …

2019-07-16cs.IR

Reinforcement Online Learning to Rank with Unbiased Reward Shaping

Online learning to rank (OLTR) aims to learn a ranker directly from implicit feedback derived from users' interactions, such as clicks. Clicks however are a biased signal: specifically, top-ranked documents are likely to attract more clicks than documents down the ranking (position bias). In this paper, we propose a novel learning algorithm for OLTR that uses reinforcement learning to optimize rankers: Reinforcement Online Learning to Rank (ROLTR). In ROLTR, the gradients of the ranker are estimated based on the rewards assigned to clicked and unclicked documents. In order to de-bias the users…

2022-01-05cs.IR

Unbiased Learning to Rank: Online or Offline?

University of Utah（犹他大学）；University of Virginia（弗吉尼亚大学）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）

How to obtain an unbiased ranking model by learning to rank with biased user feedback is an important research question for IR. Existing work on unbiased learning to rank (ULTR) can be broadly categorized into two groups -- the studies on unbiased learning algorithms with logged data, namely the \textit{offline} unbiased learning, and the studies on unbiased parameters estimation with real-time user interactions, namely the \textit{online} learning to rank. While their definitions of \textit{unbiasness} are different, these two types of ULTR algorithms share the same goal -- to find the best m…

2020-12-02cs.IRcs.LG

Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking

University of Amsterdam（阿姆斯特丹大学）；Ahold Delhaize（阿霍德·德尔海兹）

Counterfactual evaluation can estimate Click-Through-Rate (CTR) differences between ranking systems based on historical interaction data, while mitigating the effect of position bias and item-selection bias. We introduce the novel Logging-Policy Optimization Algorithm (LogOpt), which optimizes the policy for logging data so that the counterfactual estimate has minimal variance. As minimizing variance leads to faster convergence, LogOpt increases the data-efficiency of counterfactual estimation. LogOpt turns the counterfactual approach - which is indifferent to the logging policy - into an onli…

2020-07-28cs.IRcs.LG

Unbiased Learning to Rank with Unbiased Propensity Estimation

Learning to rank with biased click data is a well-known challenge. A variety of methods has been explored to debias click data for learning to rank such as click models, result interleaving and, more recently, the unbiased learning-to-rank framework based on inverse propensity weighting. Despite their differences, most existing studies separate the estimation of click bias (namely the \textit{propensity model}) from the learning of ranking algorithms. To estimate click propensities, they either conduct online result randomization, which can negatively affect the user experience, or offline par…

2018-04-23cs.IR

Unbiased Cascade Bandits: Mitigating Exposure Bias in Online Learning to Rank Recommendation

Exposure bias is a well-known issue in recommender systems where items and suppliers are not equally represented in the recommendation results. This is especially problematic when bias is amplified over time as a few popular items are repeatedly over-represented in recommendation lists. This phenomenon can be viewed as a recommendation feedback loop: the system repeatedly recommends certain items at different time points and interactions of users with those items will amplify bias towards those items over time. This issue has been extensively studied in the literature on model-based or neighbo…

2021-08-07cs.IR

更多相关论文收起更多论文

Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm

Although click data is widely used in search systems in practice, so far the inherent bias, most notably position bias, has prevented it from being used in training of a ranker for search, i.e., learning-to-rank. Recently, a number of authors have proposed new techniques referred to as 'unbiased learning-to-rank', which can reduce position bias and train a relatively high-performance ranker using click data. Most of the algorithms, based on the inverse propensity weighting (IPW) principle, first estimate the click bias at each position, and then train an unbiased ranker with the estimated bias…

2019-02-27cs.IR

ConAlign：用于平衡有偏和无偏推荐的条件对齐框架

ConAlign: Conditional Alignment Framework for Balancing Biased and Unbiased Recommendation

Kuaishou Technology（快手科技）

针对工业推荐系统偏差问题，提出ConAlign条件去偏方法，通过离散门控条件对齐机制平衡事实准确性和无偏偏好估计，支持实时流适应，经实验验证有效，提升了快手用户长期参与度和兴趣多样性。

2026-07-28cs.IR

Variance Reduction in Gradient Exploration for Online Learning to Rank

Online Learning to Rank (OL2R) algorithms learn from implicit user feedback on the fly. The key of such algorithms is an unbiased estimation of gradients, which is often (trivially) achieved by uniformly sampling from the entire parameter space. This unfortunately introduces high-variance in gradient estimation, and leads to a worse regret of model estimation, especially when the dimension of parameter space is large. In this paper, we aim at reducing the variance of gradient estimation in OL2R algorithms. We project the selected updating direction into a space spanned by the feature vectors f…

2019-11-15cs.IR

Handling Position Bias for Unbiased Learning to Rank in Hotels Search

TripAdvisor, Inc.（猫途鹰公司）；st Avenue（第一大道400号）；Needham, MA 02494（马萨诸塞州尼德姆市02494）

Nowadays, search ranking and recommendation systems rely on a lot of data to train machine learning models such as Learning-to-Rank (LTR) models to rank results for a given query, and implicit user feedbacks (e.g. click data) have become the dominant source of data collection due to its abundance and low cost, especially for major Internet companies. However, a drawback of this data collection approach is the data could be highly biased, and one of the most significant biases is the position bias, where users are biased towards clicking on higher ranked results. In this work, we will investiga…

2020-02-28cs.IRcs.LG

面向无偏序列推荐的解耦反事实推理

Disentangled Counterfactual Reasoning for Unbiased Sequential Recommendation

本文提出DCR方法，通过解耦流行度与兴趣表示并构建结构因果图，结合反事实推理消除序列推荐中的流行度偏差，离线实验和在线A/B测试验证了其显著性能提升。

2023-08-05cs.IR

论（归一化）折损累计增益作为Top-$n$推荐离线评估指标

On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-$n$ Recommendation

ShareChat

本文从理论上推导了DCG作为在线奖励无偏估计量的前提条件，证明归一化会破坏排序一致性，并通过大规模实验表明无偏DCG与在线奖励强相关而nDCG不成立，从而质疑nDCG在Top-$n$推荐评估中的实际效用。

2024-06-12cs.IRcs.LG

Existence conditions for hidden feedback loops in online recommender systems

Moscow Institute of Physics and Technology（莫斯科物理技术学院）

We explore a hidden feedback loops effect in online recommender systems. Feedback loops result in degradation of online multi-armed bandit (MAB) recommendations to a small subset and loss of coverage and novelty. We study how uncertainty and noise in user interests influence the existence of feedback loops. First, we show that an unbiased additive random noise in user interests does not prevent a feedback loop. Second, we demonstrate that a non-zero probability of resetting user interests is sufficient to limit the feedback loop and estimate the size of the effect. Our experiments confirm the …

2021-12-16cs.IRcs.LG

ULTRA: An Unbiased Learning To Rank Algorithm Toolbox

University of Utah（犹他大学）

Learning to rank systems has become an important aspect of our daily life. However, the implicit user feedback that is used to train many learning to rank models is usually noisy and suffered from user bias (i.e., position bias). Thus, obtaining an unbiased model using biased feedback has become an important research field for IR. Existing studies on unbiased learning to rank (ULTR) can be generalized into two families-algorithms that attain unbiasedness with logged data, offline learning, and algorithms that achieve unbiasedness by estimating unbiased parameters with real-time user interactio…

2021-08-11cs.IRcs.LG