更长上下文，更深思考：揭示长上下文能力在推理中的作用
Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
AI总结：
本研究通过实验发现，增强模型的长上下文能力（在监督微调前）能显著提升推理性能，即使对于短输入任务也有泛化收益，表明长上下文建模是推理能力的关键基础。
摘要：
近期语言模型展现出强大的推理能力，但长上下文能力对推理的影响仍未充分探索。在本工作中，我们假设当前推理能力的局限性部分源于长上下文能力不足，这一假设基于经验观察：（1）更高的上下文窗口长度通常带来更强的推理性能，（2）失败的推理案例与失败的长上下文案例相似。为验证这一假设，我们检验了在监督微调（SFT）前增强模型的长上下文能力是否能提升推理性能。具体而言，我们比较了架构和微调数据相同但长上下文能力不同的模型。结果揭示了一致趋势：长上下文能力更强的模型在SFT后，在推理基准上取得了显著更高的准确率。值得注意的是，即使在输入长度较短的任务上，这些增益也持续存在，表明长上下文训练为推理性能提供了可泛化的益处。这些发现表明，长上下文建模不仅对处理长输入至关重要，而且也是推理的关键基础。我们主张将长上下文能力作为未来语言模型设计的首要目标。
英文摘要：
Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored. In this work, we hypothesize that current limitations in reasoning stem, in part, from insufficient long-context capacity, motivated by empirical observations such as (1) higher context window length often leads to stronger reasoning performance, and (2) failed reasoning cases resemble failed long-context cases. To test this hypothesis, we examine whether enhancing a model's long-context ability before Supervised Fine-Tuning (SFT) leads to improved reasoning performance. Specifically, we compared models with identical architectures and fine-tuning data but varying levels of long-context capacity. Our results reveal a consistent trend: models with stronger long-context capacity achieve significantly higher accuracy on reasoning benchmarks after SFT. Notably, these gains persist even on tasks with short input lengths, indicating that long-context training offers generalizable benefits for reasoning performance. These findings suggest that long-context modeling is not just essential for processing lengthy inputs, but also serves as a critical foundation for reasoning. We advocate for treating long-context capacity as a first-class objective in the design of future language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
