arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00815cs.AI
收藏

通过动态单次策略细化实现大规模语言模型推理的资源高效强化学习

Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement

UCAS(中国科学院大学);HKUST(GZ)(香港科技大学(广州));Tsinghua University(清华大学);Sun Yat-Sen University(中山大学);Xidian University(西安电子科技大学);George Mason University(乔治·梅森大学);Peking University(北京大学);Zhejiang University(浙江大学)

Yunjian Zhang, Sudong Wang, Yang Li, Peiran Xu, Conghao Zhou, Xiaoyue Ma, Jianing Li, Yao Zhu

AI总结:

本文提出动态单次策略细化方法,通过减少训练样本数量和计算开销,提升大规模语言模型推理效率。

摘要:

大规模语言模型(LLMs)在复杂推理任务中表现出色,强化学习在可验证奖励下(RLVR)成为对齐模型行为与推理链的系统框架。尽管其有前景,RLVR仍过于资源密集,需要大量奖励信号并在训练过程中产生显著的滚动成本。在本工作中,我们重新审视RLVR中的数据和计算效率问题。我们首先建立了解锁推理能力所需的样本复杂度理论下限,并实证验证强性能可通过惊人的少量训练实例实现。为减轻计算负担,我们提出动态单次策略细化(DoPR),一种基于不确定性的RL策略,通过动态选择每个批次一个信息性的训练样本进行策略更新,由奖励波动和探索驱动的获取引导。DoPR将滚动开销减少近一个数量级,同时保持竞争性的推理准确性,提供了一种可扩展且资源高效的LLM后训练解决方案。这种方法为更高效和可访问的基于强化学习的推理密集型LLM训练提供了实际路径。

英文摘要:

Large language models (LLMs) have exhibited remarkable performance on complex reasoning tasks, with reinforcement learning under verifiable rewards (RLVR) emerging as a principled framework for aligning model behavior with reasoning chains. Despite its promise, RLVR remains prohibitively resource-intensive, requiring extensive reward signals and incurring substantial rollout costs during training. In this work, we revisit the fundamental question of data and compute efficiency in RLVR. We first establish a theoretical lower bound on the sample complexity required to unlock reasoning capabilities, and empirically validate that strong performance can be achieved with a surprisingly small number of training instances. To tackle the computational burden, we propose Dynamic One-Shot Policy Refinement (DoPR), an uncertainty-aware RL strategy that dynamically selects a single informative training sample per batch for policy updates, guided by reward volatility and exploration-driven acquisition. DoPR reduces rollout overhead by nearly an order of magnitude while preserving competitive reasoning accuracy, offering a scalable and resource-efficient solution for LLM post-training. This approach offers a practical path toward more efficient and accessible RL-based training for reasoning-intensive LLM applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑