借助高资源专家模型的可验证语义对齐多语言推理能力
Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model
AI总结：
针对大语言模型推理能力多语言差距问题，提出基于枢轴的语义可验证奖励强化学习框架，借助英语高资源模型迁移推理能力，显著提升多语言推理性能。
摘要：
尽管强化学习已提升了大语言模型（LLM）的推理能力，但这些提升大多局限于英语，造成了不同语言间显著的性能差距。为解决这一问题，我们提出了基于枢轴的语义可验证奖励强化学习（PB-RLSVR），这是一种无需目标语言人工标注数据即可增强多语言推理能力的新型框架。我们的方法采用高性能英语LLM作为“枢轴”模型，为推理任务生成参考回答。随后根据多语言模型的回答与英语参考回答的语义等价性给予奖励，从而将枢轴模型的推理能力有效迁移到不同语言中。我们研究了多种跨语言语义奖励函数，包括基于嵌入和机器翻译的函数。在一组多语言推理基准上开展的大量实验表明，我们的方法显著缩小了英语与其他语言间的性能差距，大幅优于传统PPO基线。具体而言，我们的PB-RLSVR框架将Llama-3.1-8B-Instruct和Qwen3-32B的平均多语言性能分别提升了16.41%和10.17%，证明这是一种构建真正多语言推理智能体的高效且数据节省的方法。
英文摘要：
While reinforcement learning has advanced the reasoning abilities of Large Language Models (LLMs), these gains are largely confined to English, creating a significant performance disparity across languages. To address this, we introduce Pivot-Based Reinforcement Learning with Semantically Verifiable Rewards (PB-RLSVR), a novel framework that enhances multilingual reasoning by circumventing the need for human-annotated data in target languages. Our approach employs a high-performing English LLM as a "pivot" model to generate reference responses for reasoning tasks. A multilingual model is then rewarded based on the semantic equivalence of its responses to the English reference, effectively transferring the pivot model's reasoning capabilities across languages. We investigate several cross-lingual semantic reward functions, including those based on embeddings and machine translation. Extensive experiments on a suite of multilingual reasoning benchmarks show that our method significantly narrows the performance gap between English and other languages, substantially outperforming traditional PPO baselines. Specifically, our PB-RLSVR framework improves the average multilingual performance of Llama-3.1-8B-Instruct and Qwen3-32B by 16.41% and 10.17%, respectively, demonstrating a powerful and data-efficient approach to building truly multilingual reasoning agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
