PIRA：基于双聚合的偏好导向指令调优奖励模型
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
AI总结：
PIRA通过整合三种策略，利用LLM的偏好指令能力，提升奖励模型的鲁棒性和泛化能力，有效缓解奖励过度优化问题。
摘要：
奖励模型在对齐大语言模型（LLM）与人类偏好中起着关键作用。现有方法面临两个关键限制：辨别性奖励模型需要大规模标注数据，因为它们无法利用生成性奖励模型可访问的LLM偏好指令遵循能力。此外，奖励模型特别容易受到奖励过度优化的影响，即LLM利用奖励函数的弱点而不是改进真正的对齐。我们引入PIRA，一种整合三种互补策略的训练范式，以解决这些挑战：（1）将问题-答案对改写成偏好任务指令，以显式利用LLM的偏好指令遵循能力；（2）对每个样本从多样的偏好任务指令中聚合的奖励进行平均，这可以减轻任务特定偏差并增强在评估视角下的鲁棒性；（3）在不同丢弃率下对价值头的输出进行平均，以稳定奖励估计。在公开数据集上的实验表明，PIRA显著提高了性能，增强了泛化能力，并有效缓解了奖励过度优化问题。
英文摘要：
Reward models are pivotal for aligning Large Language Models (LLMs) with human preferences. Existing approaches face two key limitations: Discriminative reward models require large-scale annotated data, as they cannot exploit the preference instruction-following capability of LLMs available to generative reward models. Moreover, reward models are particularly prone to reward overoptimization, where LLMs exploit weaknesses in the reward function instead of improving true alignment. We introduce \textbf{PIRA}, a training paradigm that integrates three complementary strategies to address these challenges: (1) reformulating question-answer pairs into preference-task instructions to explicitly leverage LLMs' preference instruction-following capability, (2) averaging the rewards aggregated from diverse preference-task instructions for each sample, which mitigates task-specific bias and enhances robustness across evaluation perspectives, and (3) averaging outputs from the value head under different dropout rates to stabilize reward estimation. Experiments on public datasets show that PIRA improves performance considerably, enhances generalization, and effectively mitigates reward overoptimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
