发表机构
Peking University; Alibaba Group（北京大学; 阿里巴巴集团）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究长上下文推理中语言模型的重复复制问题，提出GEAR奖励塑造方法，通过区分关键证据和干扰上下文来增强奖励信号，经实验验证该方法能提升模型性能，减少重复复制，凸显准确扎根证据对长上下文推理的重要性。
AI 中文摘要
生成逐步推理轨迹的大型语言模型在复杂任务上取得了强大性能，扩展到长上下文设置成为重要前沿。然而，存在关键失败模式：重复复制，即模型大量从输入复制文本到推理轨迹而非有效解决问题。这种行为在前沿长上下文语言模型中普遍存在且随上下文长度加剧。通过将提示分为任务相关关键证据和无关干扰上下文，发现根本原因是扎根不足。为此提出GEAR奖励塑造方法，开发自动化管道构建带证据注释的训练数据。在多个模型规模和基准上验证，显示出比基于准确性奖励的标准强化学习平均提高4.6分，在更长上下文有更大提升，还减少了重复复制和思考长度。研究表明，即使长上下文评估从简单检索转向复杂推理，准确扎根于相关证据仍是不可或缺且有很大改进空间的能力。
英文摘要
Large language models that generate step-by-step reasoning traces have achieved strong performance on complex tasks, and extending them to long-context settings has emerged as an important frontier. However, we identify a critical failure mode in this regime: \emph{repetitive copying}, where models extensively copy text from the input into their reasoning traces rather than productively solving the problem. We show that this behavior is pervasive across frontier long-context LLMs and intensifies with context length. By separating each prompt into task-relevant key evidence and irrelevant distractor context, we further show that the root cause is insufficient grounding: models copy from the prompt indiscriminately, and those that fail to focus on key evidence are far more likely to answer incorrectly. Motivated by this diagnosis, we propose GEAR (Grounding Evidence-Aware Reward), a reward shaping method that augments the accuracy signal with a grounding reward for overlap with key evidence and a distractor penalty for overlap with irrelevant context. To enable GEAR on natural-language data, we develop an automated pipeline that constructs evidence-annotated training data from arbitrary documents. We validate GEAR across multiple model scales and benchmarks, showing consistent improvements of up to +4.6 average points over standard RL with accuracy-based rewards, with larger gains at longer contexts, while also reducing repetitive copying and thinking length. Our findings suggest that, even as long-context evaluation shifts from simple retrieval toward complex reasoning, accurate grounding in relevant evidence remains an indispensable capability with substantial room for improvement.
