揭秘过程奖励建模中的多语言思维链
Demystifying Multilingual Chain-of-Thought in Process Reward Modeling
AI总结：
本文通过训练七种语言的多语言过程奖励模型，在11种语言的推理基准上验证其能提升平均准确率并减少早期错误，为多语言复杂推理提供新途径。
摘要：
大型语言模型（LLMs）被设计用于执行广泛的任务。为了提升其解决需要多步推理的复杂问题的能力，近期研究利用过程奖励建模，在推理过程的每一步为强化学习（RL）提供细粒度反馈，但该研究主要集中于英语。本文中，我们应对将过程奖励模型（PRMs）扩展到多语言环境这一关键挑战。为此，我们在一个涵盖七种语言的数据集上训练多语言PRMs，该数据集由英语翻译而来。通过在11种语言的两个广泛使用的推理基准上进行全面评估，我们证明多语言PRMs不仅提高了平均准确率，还减少了早期阶段的推理错误。此外，我们的结果凸显了多语言PRMs对训练语言数量和英语数据量的敏感性，同时揭示了更多候选响应和可训练参数带来的益处。这项工作为复杂多步推理任务中的稳健多语言应用开辟了有前景的途径。此外，我们发布代码以促进该方向的研究。
英文摘要：
Large language models (LLMs) are designed to perform a wide range of tasks. To improve their ability to solve complex problems requiring multi-step reasoning, recent research leverages process reward modeling to provide fine-grained feedback at each step of the reasoning process for reinforcement learning (RL), but it predominantly focuses on English. In this paper, we tackle the critical challenge of extending process reward models (PRMs) to multilingual settings. To achieve this, we train multilingual PRMs on a dataset spanning seven languages, which is translated from English. Through comprehensive evaluations on two widely used reasoning benchmarks across 11 languages, we demonstrate that multilingual PRMs not only improve average accuracy but also reduce early-stage reasoning errors. Furthermore, our results highlight the sensitivity of multilingual PRMs to both the number of training languages and the volume of English data, while also uncovering the benefits arising from more candidate responses and trainable parameters. This work opens promising avenues for robust multilingual applications in complex, multi-step reasoning tasks. In addition, we release the code to foster research along this line.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
