arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.12465cs.CLcs.AI
收藏

通过过程优势塑造激励深入推理长上下文

Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping

The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tongyi Lab, Alibaba Group(阿里云实验室)

Miao Peng, Weizhou Shen, Nuo Chen, Chenliang Li, Ming Yan, Jia Li

AI总结:

通过过程优势塑造激励深入推理长上下文,提出DeepReasonQA和LongPAS方法,提升长上下文推理性能。

摘要:

可验证奖励的强化学习(RLVR)在增强LLM短上下文推理方面已被证明是有效的,但在需要精确定位和稳健长距离推理的长上下文场景中,其性能会下降。我们识别出长上下文推理中的

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing LLMs short-context reasoning, but its performance degrades in long-context scenarios that require both precise grounding and robust long-range reasoning. We identify the "almost-there" phenomenon in long-context reasoning, where trajectories are largely correct but fail at the final step, and attribute this failure to two factors: (1) the lack of high reasoning density in long-context QA data that push LLMs beyond mere grounding toward sophisticated multi-hop reasoning; and (2) the loss of valuable learning signals during long-context RL training due to the indiscriminate penalization of partially correct trajectories with incorrect outcomes. To overcome this bottleneck, we propose DeepReasonQA, a KG-driven synthesis framework that controllably constructs high-difficulty, multi-hop long-context QA pairs with inherent reasoning chains. Building on this, we introduce Long-context Process Advantage Shaping (LongPAS), a simple yet effective method that performs fine-grained credit assignment by evaluating reasoning steps along Validity and Relevance dimensions, which captures critical learning signals from "almost-there" trajectories. Experiments on three long-context reasoning benchmarks show that our approach substantially outperforms RLVR baselines and matches frontier LLMs while using far fewer parameters. Further analysis confirms the effectiveness of our methods in strengthening long-context reasoning while maintaining stable RL training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑