arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08237cs.CL
收藏

文档重建解锁可扩展的长上下文强化学习验证奖励

Document Reconstruction Unlocks Scalable Long-Context RLVR

MiroMind AI;Singapore University of Technology and Design(新加坡科技设计大学);Institute for Infocomm Research, A*Star(新加坡科技研究局资讯通信研究院)

Yao Xiao, Lei Wang, Yue Deng, Guanzheng Chen, Ziqi Jin, Jung-jae Kim, Xiaoli Li, Roy Ka-wei Lee, Lidong Bing

AI总结:

本文提出通过文档重建提升大型语言模型长上下文能力的方法,无需人工标注或教师模型监督,有效提升模型在RULER和LongBench~v2上的表现。

摘要:

强化学习与可验证奖励(RLVR)已成为增强大型语言模型(LLMs)能力(即长上下文)的重要范式。然而,它往往依赖于由强大教师模型或人类专家提供的金牌标准答案或显式评估标准,这成本高且耗时。在本工作中,我们研究了无监督方法以增强LLMs的长上下文能力,消除了对大量人工标注或教师模型监督的需要。具体来说,我们首先在长文档中用特殊占位符替换几段文字。通过强化学习训练LLMs,使其能够通过正确识别和排序候选选项中的缺失段落来重建文档。这种训练范式使模型能够捕捉全局叙述连贯性,显著提升长上下文性能。我们在两个广泛使用的基准上验证了我们方法的有效性,RULER和LongBench~v2。虽然在RULER上获得了显著提升,但也能在没有任何人工编纂的长上下文问答数据的情况下,在LongBench~v2上取得合理改进。此外,我们进行了广泛的消融研究,分析奖励设计、数据编纂策略、训练方案和数据扩展效果对模型性能的影响。我们公开发布了我们的代码、数据和模型。

英文摘要:

Reinforcement Learning with Verifiable Rewards~(RLVR) has become a prominent paradigm to enhance the capabilities (i.e.\ long-context) of Large Language Models~(LLMs). However, it often relies on gold-standard answers or explicit evaluation rubrics provided by powerful teacher models or human experts, which are costly and time-consuming. In this work, we investigate unsupervised approaches to enhance the long-context capabilities of LLMs, eliminating the need for heavy human annotations or teacher models' supervision. Specifically, we first replace a few paragraphs with special placeholders in a long document. LLMs are trained through reinforcement learning to reconstruct the document by correctly identifying and sequencing missing paragraphs from a set of candidate options. This training paradigm enables the model to capture global narrative coherence, significantly boosting long-context performance. We validate the effectiveness of our method on two widely used benchmarks, RULER and LongBench~v2. While acquiring noticeable gains on RULER, it can also achieve a reasonable improvement on LongBench~v2 without any manually curated long-context QA data. Furthermore, we conduct extensive ablation studies to analyze the impact of reward design, data curation strategies, training schemes, and data scaling effects on model performance. We publicly release our code, data, and models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑