arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.22504cs.IR
收藏

通过硬负样本进行目标塑造:基于RL的LLM推荐系统中的窗口化部分AUC优化

Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders

University of Science and Technology of China(中国科学技术大学);Meta AI;Rochester Institute of Technology(罗切斯特理工学院)

Wentao Shi, Qifan Wang, Chen Chen, Fei Liu, Dongfang Liu, Xu Liu, Wanli Ma, Junfeng Pan, Linhong Zhu, Fuli Feng

AI总结:

本文提出窗口化部分AUC优化方法,通过约束假阳性率提升与Top-K指标的一致性,改进基于RL的LLM推荐系统性能。

摘要:

强化学习(RL)通过对比正负项目有效优化大型语言模型(LLM)推荐系统。实证表明,使用束搜索负样本训练优于随机负样本,但其机制尚不明确。本文通过分析诱导的优化目标,证明:在二元反馈下,使用组相对策略优化(GRPO)优化LLM推荐系统在理论上等同于最大化受试者工作特征曲线(AUC),这常与Top-K推荐不一致;而将随机负样本替换为束搜索负样本可使目标转向部分AUC,从而更直接地与Top-K指标对齐。基于此视角,我们引入窗口化部分AUC(WPAUC),将其假阳性率(FPR)限制在窗口[α,α+d]内,以更直接地与Top-K指标对齐。我们进一步提出一种高效的阈值调整窗口化重加权(TAWin)RL方法用于其优化,实现对目标Top-K性能的显式控制。在四个真实世界数据集上的实验验证了理论并实现了持续的最先进性能。

英文摘要:

Reinforcement learning (RL) effectively optimizes Large Language Model (LLM)-based recommenders by contrasting positive and negative items. Empirically, training with beam-search negatives consistently outperforms random negatives, yet the mechanism is not well understood. We address this gap by analyzing the induced optimization objective and show that: (i) Under binary reward feedback, optimizing LLM recommenders with Group Relative Policy Optimization (GRPO) is theoretically equivalent to maximizing the Area Under the ROC Curve (AUC), which is often misaligned with Top-$K$ recommendation; and (ii) Replacing random negatives with beam-search negatives reshapes the objective toward partial AUC, improving alignment with Top-$K$ metrics. Motivated by this perspective, we introduce Windowed Partial AUC (WPAUC), which constrains the false positive rate (FPR) to a window [$α,α+d$] to more directly align with Top-$K$ metrics. We further propose an efficient Threshold-Adjusted Windowed reweighting (TAWin) RL method for its optimization, enabling explicit control over the targeted Top-$K$ performance. Experiments on four real-world datasets validate the theory and deliver consistent state-of-the-art performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑