带可验证奖励的强化学习(RLVR)已成为提升大语言模型(LLMs)推理能力的有效方法。Group Relative Policy Optimization(GRPO)系列在利用RLVR训练LLMs方面展现出强大性能。然而,随着模型训练时间延长和规模扩大,越来越多的训练提示变为残差提示,即奖励方差为零、无法提供训练信号的提示。因此,贡献于训练的提示数量减少,降低了多样性并阻碍了训练效果。为充分利用这些残差提示,我们提出了Explore Residual Prompts in Policy Optimization(ERPO)框架,该框架鼓励对残差提示进行探索并重新激活其训练信号。ERPO为每个提示维护一个历史追踪器,并对先前产生全部正确响应的残差提示自适应地提高采样温度。这促使模型生成更多样化的推理轨迹,引入错误响应从而恢复训练信号。在Qwen2.5系列上的实验结果表明,ERPO在多个数学推理基准上持续超越强基线方法。
英文摘要:
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs). The Group Relative Policy Optimization (GRPO) family has demonstrated strong performance in training LLMs with RLVR. However, as models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal. Consequently, fewer prompts contribute to training, reducing diversity and hindering effectiveness. To fully exploit these residual prompts, we propose the Explore Residual Prompts in Policy Optimization (ERPO) framework, which encourages exploration on residual prompts and reactivates their training signals. ERPO maintains a history tracker for each prompt and adaptively increases the sampling temperature for residual prompts that previously produced all correct responses. This encourages the model to generate more diverse reasoning traces, introducing incorrect responses that revive training signals. Empirical results on the Qwen2.5 series demonstrate that ERPO consistently surpasses strong baselines across multiple mathematical reasoning benchmarks.
No Reader Left Behind: Multi-Agent Summaries Everyone Can Understand
Department of Computer Science and Engineering, Korea University(韩国大学计算机科学与工程系);Department of Computer Science and Engineering, Konkuk University(konkuk大学计算机科学与工程系)
No Language Left Behind: Scaling Human-Centered Machine Translation
Meta AI;UC Berkeley(加州大学伯克利分校);Johns Hopkins University(约翰斯·霍普金斯大学)
在全球范围内消除语言障碍的目标的推动下,机器翻译已经巩固了自己作为当今人工智能研究的一个关键焦点。然而,这些努力都集中在一小部分语言上,留下了绝大多数低资源语言。如何才能在确保安全、高质量的结果的同时,打破200种语言的障碍,同时考虑到道德因素?在《No Language Left Behind》中,我们首先通过与母语人士的探索性访谈,将对低资源语言翻译支持的需求置于情境中,从而接受了这一挑战。然后,我们创建了数据集和模型,旨在缩小低资源语言和高资源语言之间的性能差距。更具体地说,我们开发了一个条件计算模型的基础上稀疏门控的混合专家模型,是训练的数据与新的和有效的数据采掘技术,为低资源的语言。我们提出了多个架构和训练改进,以抵消过度拟合,同时训练数千个任务。重要的是,我们使用人类翻译的基准Flores-200评估了超过40,000种不同翻译方向的性能,并将人类评估与Flores-200中涵盖所有语言的新型毒性基准相结合,以评估翻译安全性。我们的模型实现了44%的BLEU相对于以前的国家的最先进的改进,奠定了重要的基础,实现一个通用的翻译系统。最后,我们开源了本工作中描述的所有贡献,可在https://github.com/facebookresearch/fairseq/tree/nllb上访问。
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
Georgia Institute of Technology(佐治亚理工学院);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学)
Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark
University of Notre Dame(诺特达姆大学);University of Washington(华盛顿大学);Tokyo University of Foreign Studies(东京外国语大学);Nara Institute of Science and Technology(奈良科学技術大学);University of Tübingen(图宾根大学)