RoiRL：基于离线迭代强化学习的高效自监督推理
RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
AI总结：
针对大语言模型推理强化学习依赖真实奖励或在线计算成本高的问题，提出轻量级离线迭代强化学习方法RoiRL，无需参考模型，训练更快且推理性能更优，为无标签自提升大模型提供可扩展路径。
摘要：
强化学习（RL）是提升大语言模型（LLM）推理能力的核心技术，但通常需要真实标签奖励。测试时强化学习（TTRL）利用多数投票奖励消除了这一需求，但依赖繁重的在线RL，会产生大量计算成本。我们提出RoiRL：基于离线迭代强化学习的推理，这是一类轻量级离线学习替代方案，能够针对相同的正则化最优策略。与TTRL不同，RoiRL无需维护参考模型，而是优化加权对数似然目标，可实现稳定训练，且内存与计算需求显著降低。实验结果表明，RoiRL训练速度提升2.5倍，在推理基准上始终优于TTRL，为无标签自提升LLM建立了可扩展路径。
英文摘要：
Reinforcement learning (RL) is central to improving reasoning in large language models (LLMs) but typically requires ground-truth rewards. Test-Time Reinforcement Learning (TTRL) removes this need by using majority-vote rewards, but relies on heavy online RL and incurs substantial computational cost. We propose RoiRL: Reasoning with offline iterative Reinforcement Learning, a family of lightweight offline learning alternatives that can target the same regularized optimal policies. Unlike TTRL, RoiRL eliminates the need to maintain a reference model and instead optimizes weighted log-likelihood objectives, enabling stable training with significantly lower memory and compute requirements. Experimental results show that RoiRL trains to 2.5x faster and consistently outperforms TTRL on reasoning benchmarks, establishing a scalable path to self-improving LLMs without labels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
