arXiv 2602.019702026-02cs.AIcs.LG

小规模可泛化提示预测模型可引导大推理模型的高效强化学习后训练

Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models

Department of Automation, Tsinghua University, Beijing, China（自动化系，清华大学，北京，中国）；LLM Department, Tencent, Beijing, China（大模型部门，腾讯，北京，中国）

Yun Qu, Qi Wang, Yixiu Mao, Heming Zou, Yuhang Jiang, Weijie Liu, Clive Bai, Kai Yang, Yangkun Chen, Saiyong Yang, Xiangyang Ji

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出GPS方法，通过轻量级生成模型进行提示难度的贝叶斯推断，结合中间难度优先和历史锚定多样性，提升大模型强化学习后的训练效率和测试效率。

摘要：

强化学习能增强大语言模型的推理能力，但通常因滚动优化而产生高计算成本。在线提示选择通过优先选择信息性提示来提高训练效率。然而，现有方法要么依赖昂贵的精确评估，要么构建缺乏跨提示泛化的提示特定预测模型。本研究引入可泛化的提示选择（GPS），通过轻量级生成模型对提示难度进行贝叶斯推断，利用共享优化历史训练。中间难度优先和历史锚定多样性被纳入批量获取原则中以选择信息性提示批次。小规模预测模型在测试时也具备泛化能力，以实现高效的计算分配。在各种推理基准上的实验表明，GPS在训练效率、最终性能和测试效率上显著优于更优的基线方法。

英文摘要：

Reinforcement learning enhances the reasoning capabilities of large language models but often involves high computational costs due to rollout-intensive optimization. Online prompt selection presents a plausible solution by prioritizing informative prompts to improve training efficiency. However, current methods either depend on costly, exact evaluations or construct prompt-specific predictive models lacking generalization across prompts. This study introduces Generalizable Predictive Prompt Selection (GPS), which performs Bayesian inference towards prompt difficulty using a lightweight generative model trained on the shared optimization history. Intermediate-difficulty prioritization and history-anchored diversity are incorporated into the batch acquisition principle to select informative prompt batches. The small predictive model also generalizes at test-time for efficient computational allocation. Experiments across varied reasoning benchmarks indicate GPS's substantial improvements in training efficiency, final performance, and test-time efficiency over superior baseline methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文