arXiv 2507.013682025-07-02cs.CVcs.LG

用于少样本模型对齐的激活奖励模型

Activation Reward Models for Few-Shot Model Alignment

University of California, Berkeley（加州大学伯克利分校）；Carnegie Mellon University（卡内基梅隆大学）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）

Tianning Chai, Chancharik Mitra, Brandon Huang, Gautam Rajendrakumar Gare, Zhiqiu Lin, Assaf Arbelle, Leonid Karlinsky, Rogerio Feris, Trevor Darrell, Deva Ramanan, Roei Herzig

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出激活奖励模型（Activation RMs），利用激活引导以少样本方式构建奖励信号，无需微调即可超越现有方法，并在新基准 PreferenceHack 上有效缓解奖励黑客行为。

摘要：

将大型语言模型（LLMs）和大型多模态模型（LMMs）与人类偏好对齐，是提升模型生成输出质量以服务于现实应用的核心挑战。一种常见方法是使用奖励建模来编码偏好，从而通过强化学习进行训练后对齐。然而，传统奖励建模难以适应新偏好，因为它需要单独训练一个通常基于大规模偏好数据集的奖励模型。为解决这一问题，我们提出激活奖励模型（Activation RMs）——一种新颖的少样本奖励建模方法，利用激活引导（activation steering）在极少监督且无需额外模型微调的情况下构建良好对齐的奖励信号。在标准奖励建模基准上，Activation RMs 优于现有少样本奖励建模方法，如基于上下文学习的 LLM-as-a-judge、基于投票的评分和基于词元概率的评分。此外，我们证明了 Activation RMs 在缓解奖励黑客（reward hacking）行为方面的有效性，凸显其在安全关键应用中的价值。为此，我们提出 PreferenceHack，一个新颖的少样本设置基准，首次以成对偏好格式测试奖励模型在奖励黑客问题上的表现。最后，我们表明 Activation RM 在该基准上取得了最先进的性能，甚至超越了 GPT-4o。

英文摘要：

Aligning Large Language Models (LLMs) and Large Multimodal Models (LMMs) to human preferences is a central challenge in improving the quality of the models' generative outputs for real-world applications. A common approach is to use reward modeling to encode preferences, enabling alignment via post-training using reinforcement learning. However, traditional reward modeling is not easily adaptable to new preferences because it requires a separate reward model, commonly trained on large preference datasets. To address this, we introduce Activation Reward Models (Activation RMs) -- a novel few-shot reward modeling method that leverages activation steering to construct well-aligned reward signals using minimal supervision and no additional model finetuning. Activation RMs outperform existing few-shot reward modeling approaches such as LLM-as-a-judge with in-context learning, voting-based scoring, and token probability scoring on standard reward modeling benchmarks. Furthermore, we demonstrate the effectiveness of Activation RMs in mitigating reward hacking behaviors, highlighting their utility for safety-critical applications. Toward this end, we propose PreferenceHack, a novel few-shot setting benchmark, the first to test reward models on reward hacking in a paired preference format. Finally, we show that Activation RM achieves state-of-the-art performance on this benchmark, surpassing even GPT-4o.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。