在移动边缘训练：一种在线验证的提示选择方法用于大型推理模型的高效强化学习训练
Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model
AI总结：
本文提出HIVE方法，通过历史奖励轨迹和实时提示熵实现高效RL训练，提升提示选择效率而不牺牲性能。
摘要：
强化学习（RL）已成为在推理任务中训练大型语言模型（LLMs）的关键技术。尽管扩大 rollout 可以稳定训练并提高性能，但计算开销是一个关键问题。在像 GRPO 等算法中，每个提示多个 rollout 会带来极高的成本，因为大量提示提供微不足道的梯度，因此效用较低。为了解决这个问题，我们研究如何在 rollout 阶段之前选择高效用的提示。我们的实验分析揭示了样本效用是非均匀且动态变化的：最强的学习信号集中在「学习边缘」，即中等难度和高不确定性的交界处，随着训练进行而变化。受此启发，我们提出了 HIVE（基于历史和在线验证的提示选择），一种数据高效的 RL 框架。HIVE 利用历史奖励轨迹进行粗略选择，并利用提示熵作为实时代理来修剪效用过时的实例。通过在多个数学推理基准和模型上评估 HIVE，我们证明 HIVE 在不牺牲性能的情况下显著提高了 rollout 的效率。
英文摘要：
Reinforcement learning (RL) has become essential for post-training large language models (LLMs) in reasoning tasks. While scaling rollouts can stabilize training and enhance performance, the computational overhead is a critical issue. In algorithms like GRPO, multiple rollouts per prompt incur prohibitive costs, as a large portion of prompts provide negligible gradients and are thus of low utility. To address this problem, we investigate how to select high-utility prompts before the rollout phase. Our experimental analysis reveals that sample utility is non-uniform and evolving: the strongest learning signals concentrate at the ``learning edge", the intersection of intermediate difficulty and high uncertainty, which shifts as training proceeds. Motivated by this, we propose HIVE (History-Informed and online-VErified prompt selection), a dual-stage framework for data-efficient RL. HIVE utilizes historical reward trajectories for coarse selection and employs prompt entropy as a real-time proxy to prune instances with stale utility. By evaluating HIVE across multiple math reasoning benchmarks and models, we show that HIVE yields significant rollout efficiency without compromising performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
