arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.02007cs.LG
收藏

阿普里尔-1.5-开放式推理机:基于强化学习的后训练通用高效推理

Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning

ServiceNow

Rafael Pardinas, Ehsan Kamalloo, David Vazquez, Alexandre Drouin

AI总结:

本文提出Apriel-1.5-OpenReasoner,通过可复现的多领域强化学习后训练方法,在Apriel-Base上实现跨五个领域的通用高效推理,提升推理效率并优化不同难度问题的推理长度。

摘要:

通过在Apriel-Base上使用可复现的多领域强化学习后训练方法,构建了跨五个领域的通用推理模型。该模型引入了自适应领域采样机制和难度感知的长度惩罚扩展,以保持目标领域比例并鼓励解决困难问题时进行更长的推理,而对简单问题则生成更短的推理轨迹。在严格16K-token输出预算下训练,Apriel-1.5-OpenReasoner在推理时可扩展至32K tokens,并在AIME 2025、GPQA、MMLU-Pro和LiveCodeBench等任务中优于Apriel-Base,同时生成30-50%更短的推理轨迹。该模型在较低的token成本下与类似规模的强开放式模型相当,从而推动了准确率与token预算的帕累托前沿。

英文摘要:

Building general-purpose reasoning models using reinforcement learning with verifiable rewards (RLVR) across diverse domains has been widely adopted by frontier open-weight models. However, their training recipes and domain mixtures are often not disclosed. Joint optimization across domains poses significant challenges: domains vary widely in rollout length, problem difficulty, and sample efficiency. Further, models with long chain-of-thought traces increase inference cost and latency, making efficiency critical for practical deployment. We present Apriel-Reasoner, trained with a reproducible multi-domain RL post-training recipe on Apriel-Base, a 15B-parameter open-weight LLM, across five domains using public datasets: mathematics, code generation, instruction following, logical puzzles, and function calling. We introduce adaptive domain sampling that preserves target completed-rollout ratios despite heterogeneous rollout dynamics, and a difficulty-aware length penalty that, at no additional training overhead, encourages longer reasoning for difficult problems and shorter traces for easy ones. Trained with a strict 16K-token output budget, Apriel-Reasoner remains effective at a 32K output budget and improves over Apriel-Base on AIME 2025, GPQA, MMLU-Pro, and LiveCodeBench while producing 30-50% shorter reasoning traces. Among the evaluated open-weight models of similar scale, it improves the accuracy-token tradeoff using a fully public-data recipe.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑