工具增强的奖励建模
Tool-Augmented Reward Modeling
AI总结：
本文提出工具增强的偏好建模方法Themis，通过赋予奖励模型访问外部工具的能力，解决算术计算、代码执行和事实查找等基础功能缺陷，在多项任务中显著提升偏好排序和RLHF性能。
摘要：
奖励建模（又称偏好建模）对于将大型语言模型与人类偏好对齐至关重要，特别是在人类反馈强化学习（RLHF）的背景下。尽管传统的奖励模型展现出了显著的可扩展性，但它们在算术计算、代码执行和事实查找等基础功能上往往存在困难。在本文中，我们提出了一种名为Themis的工具增强偏好建模方法，通过赋予奖励模型访问包括计算器和搜索引擎在内的外部环境的能力，来解决这些局限性。这种方法不仅促进了工具使用与奖励评分之间的协同作用，还增强了模型的可解释能力和评分可靠性。我们的研究深入探讨了将外部工具集成到奖励模型中，使其能够与多样化的外部来源进行交互，并以自回归方式构建特定于任务的工具参与和推理轨迹。我们在广泛的领域验证了该方法，结合了七种不同的外部工具。实验结果表明，在八项偏好排序任务中，整体实现了17.7%的显著提升。此外，在TruthfulQA任务的零样本评估中，我们的方法比Gopher 280B高出7.3%。在人类评估中，使用Themis训练的RLHF在四个不同任务上与基线相比，平均胜率达到32%。此外，我们提供了一个全面的工具相关奖励模型数据集集合，包含来自七个不同工具API的数据，总计15,000个实例。我们已公开代码、数据和模型检查点，以促进并激发进一步的研究进展。
英文摘要：
Reward modeling (a.k.a., preference modeling) is instrumental for aligning large language models with human preferences, particularly within the context of reinforcement learning from human feedback (RLHF). While conventional reward models (RMs) have exhibited remarkable scalability, they oft struggle with fundamental functionality such as arithmetic computation, code execution, and factual lookup. In this paper, we propose a tool-augmented preference modeling approach, named Themis, to address these limitations by empowering RMs with access to external environments, including calculators and search engines. This approach not only fosters synergy between tool utilization and reward grading but also enhances interpretive capacity and scoring reliability. Our study delves into the integration of external tools into RMs, enabling them to interact with diverse external sources and construct task-specific tool engagement and reasoning traces in an autoregressive manner. We validate our approach across a wide range of domains, incorporating seven distinct external tools. Our experimental results demonstrate a noteworthy overall improvement of 17.7% across eight tasks in preference ranking. Furthermore, our approach outperforms Gopher 280B by 7.3% on TruthfulQA task in zero-shot evaluation. In human evaluations, RLHF trained with Themis attains an average win rate of 32% when compared to baselines across four distinct tasks. Additionally, we provide a comprehensive collection of tool-related RM datasets, incorporating data from seven distinct tool APIs, totaling 15,000 instances. We have made the code, data, and model checkpoints publicly available to facilitate and inspire further research advancements\footnote{\url{https://github.com/ernie-research/Tool-Augmented-Reward-Model}}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
