从数据中心化到样本中心化：通过渐进优化增强大语言模型推理能力
From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
AI总结：
本文提出LPPO框架，通过前缀引导采样和学习进度加权，从样本中心化视角优化RLVR，有效利用少量高质量演示，在数学推理基准上实现更快收敛和更高性能。
摘要：
基于可验证奖励的强化学习（RLVR）近期提升了大型语言模型（LLM）的推理能力。尽管先前的工作侧重于算法设计、数据整理和奖励塑造，我们从样本中心化的视角研究RLVR，并引入LPPO（学习进度与前缀引导优化），一个渐进优化技术的框架。我们的工作解决了一个关键问题：如何最好地利用一小部分可信的高质量演示，而非简单地扩大数据量。首先，受提示如何辅助人类解决问题的启发，我们提出前缀引导采样，一种在线数据增强方法，该方法结合专家演示中的部分解决方案前缀来引导策略，尤其针对具有挑战性的实例。其次，受人类专注于与当前能力相符的重要问题的启发，我们引入学习进度加权，一种动态策略，根据模型进展调整每个训练样本的影响力。我们通过每个样本通过率的指数移动平均来估计样本级学习进度，促进促进学习的样本并弱化停滞不前的样本。在数学推理基准上的实验表明，我们的方法优于强基线，实现更快的收敛和更高的性能上限。
英文摘要：
Reinforcement learning with verifiable rewards (RLVR) has recently advanced the reasoning capabilities of large language models (LLMs). While prior work has emphasized algorithmic design, data curation, and reward shaping, we investigate RLVR from a sample-centric perspective and introduce LPPO (Learning-Progress and Prefix-guided Optimization), a framework of progressive optimization techniques. Our work addresses a critical question: how to best leverage a small set of trusted, high-quality demonstrations, rather than simply scaling up data volume. First, motivated by how hints aid human problem-solving, we propose prefix-guided sampling, an online data augmentation method that incorporates partial solution prefixes from expert demonstrations to guide the policy, particularly for challenging instances. Second, inspired by how humans focus on important questions aligned with their current capabilities, we introduce learning-progress weighting, a dynamic strategy that adjusts each training sample's influence based on model progression. We estimate sample-level learning progress via an exponential moving average of per-sample pass rates, promoting samples that foster learning and de-emphasizing stagnant ones. Experiments on mathematical-reasoning benchmarks demonstrate that our methods outperform strong baselines, yielding faster convergence and a higher performance ceiling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
