打破排斥诅咒:乐观分布鲁棒策略优化用于离线生成推荐
Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates
AI总结:
本文提出DRPO方法,通过乐观分布鲁棒优化解决离线生成推荐中因低质量数据导致的模型崩溃问题,实现高质量行为恢复与噪声过滤。
摘要:
基于策略的强化学习(RL)已在生成推荐中确立为优化顺序用户交互的主导范式。然而,当应用于离线历史日志时,这些方法会遭遇关键失败:低质量数据的主导导致严重的模型崩溃。我们首先建立了排斥优化的发散理论,揭示了负梯度更新本质上会引发离线训练中的指数级爆炸。该理论阐明了现有方法的固有困境,揭示了其无法协调方差减少和噪声模仿的无力。为打破这一诅咒,我们主张解决方案在于严格识别嵌套在嘈杂行为策略中的潜在高质量分布。据此,我们将目标重新表述为乐观分布鲁棒优化(DRO)问题。基于此表述,我们提出了分布鲁棒策略优化(DRPO)。我们证明了硬过滤是解决此DRO目标的精确解,使DRPO能够最优地恢复高质量行为,同时严格剔除引发发散的噪声。大量实验表明,DRPO在混合质量推荐基准上实现了最先进的性能。
英文摘要:
Off-policy policy optimization reuses historical behavior, including negative-advantage samples that suppress known failures. We show that repeated reuse can turn this useful signal into excessive repulsion: as the learner moves away from a historical negative action, subsequent updates make that action increasingly remote without necessarily reducing its update strength. Our aggregate theory characterizes the resulting transition from a stable displacement beyond the positive-only target to persistent drift and the loss of finite stable equilibria; controlled strength sweeps show that an intermediate displacement can improve held-out reward. The relevant learner-relative coordinate is squared standardized distance for Gaussian policies and surprisal for categorical policies. We introduce Dynamic Remoteness-Aware Policy Optimization (DRPO), which leaves the negative update unchanged in the near field and exponentially attenuates its remote tail. DRPO restores eventual inward Gaussian drift for every fixed finite negative-to-positive mass ratio, yields an explicit ultimate-bound radius, and changes categorical support suppression from exponential to polynomial probability decay. External diagnostics and controlled interventions isolate remoteness-dependent policy geometry as a source of negative-update amplification and show that selective tapering can remove its far-field effect without discarding useful local feedback.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。