arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12036cs.CL
收藏

Composition-RL:为大型语言模型的强化学习编纂可验证提示

Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models

The Hong Kong University of Science(香港科学与技术大学);HY, Tencent(HY,腾讯);The University of Hong Kong(香港大学)

Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Cai Yuchen, Yangkun Chen, Weijie Liu, Hao Chen, Yang Wang, Saiyong Yang, Can Yang

AI总结:

Composition-RL通过自动组合多个问题生成新可验证问题,提升有限提示的利用效率,实验表明其能提升推理能力并支持跨领域强化学习。

摘要:

大规模可验证提示是强化学习与可验证奖励(RLVR)成功的关键,但包含大量无信息示例且扩展成本高。近期研究侧重于利用有限训练数据,优先选择通过率0的困难提示,但随着训练进展,通过率1的易提示增多,降低有效数据量。为此,我们提出Composition-RL,一种简单有效的策略,通过自动组合多个问题生成新可验证问题用于RL训练。实验表明,从4B到30B模型规模均显示Composition-RL能提升推理能力。通过渐进增加组合深度的课程变体可进一步提升性能。此外,Composition-RL通过跨领域提示组合实现更有效的跨领域RL。代码、数据集和模型可在https://github.com/XinXU-USTC/Composition-RL获取。

英文摘要:

Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, an offline data-augmentation approach that composes existing prompts before RL to address the pass-rate-1 bottleneck without modifying the RL optimizer. The composed dataset is generated and verified once from the fixed source pool and then used for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. All our code, data, and models are publicly available to facilitate future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑