理解文本到图像强化学习中的奖励黑客行为
Understanding Reward Hacking in Text-to-Image Reinforcement Learning
AI总结：
本文提出了一种轻量的伪影奖励模型，用于缓解文本到图像强化学习中的奖励黑客问题，通过提高图像真实性和减少低质量生成。
摘要：
强化学习（RL）已成为训练大型语言模型后的一种标准方法，并更近期地用于改进图像生成模型，该方法使用奖励函数来提高生成质量和人类偏好对齐。然而，现有的奖励设计往往是对真实人类判断的不完美的代理，使模型容易产生不现实或低质量的图像，这些图像尽管获得高奖励分数。在本工作中，我们系统地分析了文本到图像（T2I）RL后训练中的奖励黑客行为。我们调查了美学/人类偏好奖励和提示图像一致性奖励如何单独促成奖励黑客，并进一步表明融合多个奖励只能部分缓解这一问题。在多样化的奖励模型中，我们识别出一个共同的失败模式：生成易产生伪影的图像。为了解决这一问题，我们提出了一种轻量且自适应的伪影奖励模型，该模型在一小部分经过精心挑选的无伪影和含伪影样本上进行训练。该模型可以整合到现有的RL流程中，作为常用奖励模型的有效正则化器。实验表明，将我们的伪影奖励纳入多个T2I RL设置中，显著提高了视觉真实性和减少了奖励黑客现象，证明了轻量奖励增强作为防止奖励黑客的保护措施的有效性。
英文摘要：
Reinforcement learning (RL) has become a standard approach for post-training large language models and, more recently, for improving image generation models, which uses reward functions to enhance generation quality and human preference alignment. However, existing reward designs are often imperfect proxies for true human judgment, making models prone to reward hacking--producing unrealistic or low-quality images that nevertheless achieve high reward scores. In this work, we systematically analyze reward hacking behaviors in text-to-image (T2I) RL post-training. We investigate how both aesthetic/human preference rewards and prompt-image consistency rewards individually contribute to reward hacking and further show that ensembling multiple rewards can only partially mitigate this issue. Across diverse reward models, we identify a common failure mode: the generation of artifact-prone images. To address this, we propose a lightweight and adaptive artifact reward model, trained on a small curated dataset of artifact-free and artifact-containing samples. This model can be integrated into existing RL pipelines as an effective regularizer for commonly used reward models. Experiments demonstrate that incorporating our artifact reward significantly improves visual realism and reduces reward hacking across multiple T2I RL setups, demonstrating the effectiveness of lightweight reward augment serving as a safeguard against reward hacking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
