通过因果分解实现大型语言模型的鲁棒奖励建模
Robust Reward Modeling for Large Language Models via Causal Decomposition
AI总结：
本文提出通过因果分解学习解码器，以增强奖励模型对提示意图的建模，从而提升在数学、帮助性和安全性的基准测试中的表现。
摘要：
奖励模型在对齐大型语言模型中至关重要，但通常容易过拟合于虚假线索，如响应长度和过于顺从的语气。大多数先前工作通过惩罚或控制特定伪影来削弱这些线索，但并未明确鼓励模型将偏好建立在提示意图的基础上。我们学习了一个解码器，将候选答案映射到输入的潜在意图嵌入。重构误差被用作信号以正则化奖励模型训练。我们提供了理论证据表明，该信号强调提示依赖信息，同时抑制提示无关的捷径。在数学、帮助性和安全性的基准测试中，解码器在0.877的准确率下选择了较短且不阿谀奉承的候选答案。将此信号纳入Gemma-2-2B-it和Gemma-2-9B-it的RM训练中，将RewardBench准确率从0.832提高到0.868。对于Best-of-N选择，我们的框架在控制长度的情况下提高了胜利率，同时产生较短的输出，并在受控重写测试中保持对长度增加和轻微偏离主题的鲁棒性。
英文摘要：
Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
