arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.10784cs.LG
收藏

MASS-DPO:多负样本主动采样用于直接策略优化

MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization

UC San Diego(UC圣地亚哥大学);Adobe Research(Adobe研究院)

Rohan Surana, Xintong Li, Sheldon Yu, Yiran Jenny Shen, Chuhan Wang, Tong Yu, Prithviraj Ammanabrolu, Jingbo Shang, Julian McAuley, Junda Wu

AI总结:

MASS-DPO通过多负样本主动采样方法,在Plackett-Luce模型下扩展直接偏好优化,通过选择信息丰富的负样本子集提升策略更新效率,减少冗余梯度,提升模型性能。

摘要:

在Plackett-Luce (PL)模型下,多负偏好优化扩展了直接偏好优化(DPO),通过利用一个首选和多个被拒响应之间的比较信号。然而,优化大规模负样本池成本高,许多候选者由于对策略更新的影响相似而贡献冗余梯度。我们引入MASS-DPO,一种多负主动采样方法,推导出特定于PL的Fisher信息目标,以在每个提示中选择紧凑且信息丰富的负样本子集。所得到的对数行列式目标选择对策略更新具有互补信息的负样本,产生保留完整池信息的紧凑子集,同时减少冗余。在实践中,这有利于选择梯度覆盖不同更新方向的负样本,减少近似重复候选者的冗余信号,同时保留最有用的训练信息。在四个涵盖推荐和多选问答的基准以及三个模型家族上,MASS-DPO在准确性上一致优于或匹配现有方法,提高了Recall/NDCG和基于边界的优化动态,并在使用远少于负样本的情况下实现了更强的对齐。

英文摘要:

Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑