句子级奖励模型在基于人类偏好的大语言模型对齐中泛化性更优
Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
AI总结：
针对粗粒度奖励模型稀疏性差、token级奖励模型语义不足问题，提出句子级中粒度奖励模型，通过句子分割、微分运算及新型注意力机制实现，在RewardBench和AlpacaEval上性能更优。
摘要：
从人类偏好数据集中学习奖励模型并随后通过强化学习优化语言模型，已成为使大语言模型（LLM）与人类偏好对齐的基础范式。奖励模型的性能对齐效果起着至关重要的作用。以往的奖励模型在粗粒度层面运行，需要生成完整回复才能获得奖励值。这种稀疏奖励可能给下游强化学习带来挑战。尽管近期研究尝试学习token级奖励模型，但缺乏明确语义信息使得难以对每个单独token的贡献进行建模。本文提出为每个句子分配分数，引入一种中粒度奖励模型。通过将完整回复分割为句子，并对每个句子首尾位置的奖励输出应用微分运算，我们能够有效建模句子的奖励。此外，本文还引入了一种新颖的注意力机制，将所有句子的分数聚合为回复级分数，从而可使用Bradley-Terry模型进行训练。在常用基准上，我们的方法在RewardBench（用于奖励建模评估）上优于回复级奖励模型2.7%，在AlpacaEval（用于对齐评估）上超过所有基线。
英文摘要：
Learning reward models from human preference datasets and subsequently optimizing language models via reinforcement learning has emerged as a fundamental paradigm for aligning LLMs with human preferences. The performance of the reward model plays a crucial role in the effectiveness of alignment. Previous reward models operate at a coarse-grained level, requiring the generation of a complete response to obtain a reward value. The sparse reward may present challenges for downstream reinforcement learning. While recent efforts have attempted to learn token-level reward models, the lack of explicit semantic information makes it difficult to model the credit of every individual token. In this paper, we propose assigning scores to every sentence, introducing an intermediate-grained reward model. By segmenting the complete response into sentences and applying differential operations to reward output at the start and end positions of each sentence, we can effectively model the rewards of sentences. Moreover, a novel attention mechanism is introduced to aggregate the scores of all sentences into a response-level score, which allows it to be trained using the Bradley-Terry model. On common benchmarks, our method outperforms the response-level reward model by 2.7% on RewardBench (for reward modeling evaluation) and surpasses all baselines on AlpacaEval (for alignment evaluation).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
