潜在对抗正则化用于离线偏好优化
Latent Adversarial Regularization for Offline Preference Optimization
AI总结：
本文提出GANPO，通过潜在空间正则化提升语言模型的离线偏好优化性能，增强鲁棒性并减少计算开销。
摘要：
从人类反馈学习通常依赖于通过令牌级正则化约束策略更新的偏好优化。然而，语言模型的偏好优化特别具有挑战性，因为令牌空间相似性不意味着语义或行为相似性。为了解决这一挑战，我们利用潜在空间正则化用于语言模型的偏好优化。我们引入了GANPO，通过惩罚策略模型和参考模型内部表示之间的分歧来实现潜在空间正则化。鉴于潜在表示不与显式概率密度相关联，我们采用受GANs启发的对抗方法来最小化潜在空间分歧。我们将GANPO作为正则化器整合到现有的离线偏好优化目标中。在多个模型架构和任务上的实验显示了潜在空间正则化的一致改进。进一步地，通过比较GANPO引起的推断偏见与令牌级正则化引起的偏见，我们发现GANPO在分布变化和噪声下提供了更稳健的结构反馈，同时在小幅计算开销下保持了与之相当的下游性能。
英文摘要：
Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization. However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity. To address this challenge, we leverage latent-space regularization for language model preference optimization. We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model. Given that latent representations are not associated with explicit probability densities, we adopt an adversarial approach inspired by GANs to minimize latent-space divergence. We integrate GANPO as a regularizer into existing offline preference optimization objectives. Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization. Further, by comparing GANPO-induced inferential biases with those from token-level regularization, we find that GANPO provides more robust structural feedback under distributional shift and noise while maintaining comparable downstream performance with minor computational overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
