arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16069cs.SEcs.LG
收藏

长上下文推理在自动化Bug修复中的局限性

The Limits of Long-Context Reasoning in Automated Bug Fixing

SambaNova Systems(SambaNova系统)

Ravi Raju, Mengmeng Ji, Shubhangi Upasani, Bo Li, Urmish Thakker

AI总结:

本研究发现长上下文推理在自动化Bug修复中存在显著局限,现有模型在长上下文下的表现远低于预期。

摘要:

快速增长的上下文长度导致人们假设大语言模型(LLMs)能够直接对整个代码库进行推理。同时,最近在LLMs上的进展使它们在软件工程基准测试中表现出色,尤其是在与代理工作流结合使用时。在本工作中,我们系统地评估了当前LLMs是否能够可靠地执行长上下文代码调试和补丁生成。使用SWE-bench Verified作为受控实验设置,我们首先在代理Harness(mini-SWE-agent)中评估了最先进的模型,其中性能显著提升:GPT-5-nano在100个样本上实现了高达31%的解决率,而像Deepseek-R1-0528这样的开源模型也取得了具有竞争力的结果。然而,逐个令牌分析显示,成功的代理轨迹通常保持在20k-30k个令牌以下,并且较长的累计上下文与较低的成功率相关,这表明代理的成功主要来自于将任务分解为短上下文步骤,而不是有效的长上下文推理。为了直接测试长上下文能力,我们构建了一个数据管道,其中我们将相关文件放入上下文中(确保完美检索召回率);然后我们研究了在真正长上下文中(64k个令牌)进行单次补丁生成。尽管如此,性能仍然急剧下降:Qwen3-Coder-30B-A3B在64k上下文中仅实现了7%的解决率,而GPT-5-nano解决了零个任务。定性分析揭示了系统性的失败模式,包括幻觉的差异、错误的目标文件和格式不正确的补丁头。总体而言,我们的发现突显了当前LLMs中名义上下文长度与可用上下文容量之间存在显著差距,并表明现有的代理编码基准测试并不真正评估长上下文推理。

英文摘要:

Rapidly increasing context lengths have led to the assumption that large language models (LLMs) can directly reason over entire codebases. Concurrently, recent advances in LLMs have enabled strong performance on software engineering benchmarks, particularly when paired with agentic workflows. In this work, we systematically evaluate whether current LLMs can reliably perform long-context code debugging and patch generation. Using SWE-bench Verified as a controlled experimental setting, we first evaluate state-of-the-art models within an agentic harness (mini-SWE-agent), where performance improves substantially: GPT-5-nano achieves up to a 31\% resolve rate on 100 samples, and open-source models such as Deepseek-R1-0528 obtain competitive results. However, token-level analysis shows that successful agentic trajectories typically remain under 20k-30k tokens, and that longer accumulated contexts correlate with lower success rates, indicating that agentic success primarily arises from task decomposition into short-context steps rather than effective long-context reasoning. To directly test long-context capability, we construct a data pipeline where we artificially inflate the context length of the input by placing the relevant files into the context (ensuring perfect retrieval recall); we then study single-shot patch generation under genuinely long contexts (64k tokens). Despite this setup, performance degrades sharply: Qwen3-Coder-30B-A3B achieves only a 7\% resolve rate at 64k context, while GPT-5-nano solves none of the tasks. Qualitative analysis reveals systematic failure modes, including hallucinated diffs, incorrect file targets, and malformed patch headers. Overall, our findings highlight a significant gap between nominal context length and usable context capacity in current LLMs, and suggest that existing agentic coding benchmarks do not meaningfully evaluate long-context reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑