arXiv 2602.227642026-02-26cs.SE

评估和改进基于LLM代理的仓库级Rust问题解决

Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents

Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（南方科技大学可信自主系统研究院）；Southern University of Science and Technology（南方科技大学）；Ant Group（蚂蚁集团）

Jiahong Xiang, Wenxiao He, Xihua Wang, Hongliang Tian, Yuqun Zhang

备注：: Accepted to the 48th International Conference on Software Engineering (ICSE 2026)
发表信息：: 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE '26), April 12--18, 2026, Rio de Janeiro, Brazil
DOI：: 10.1145/3744916.3773108

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出RUSTFORGER，通过集成自动化测试环境与动态跟踪策略，有效提升Rust问题解决效率，解决率达28.6%。

摘要：

Rust编程语言具有陡峭的学习曲线和显著的编码挑战，使其自动化问题解决成为其更广泛采用的关键。最近，基于LLM的代码代理在解决复杂软件工程任务方面表现出色，但其在Rust上的应用受到缺乏大规模、仓库级基准的限制。为弥合这一差距，我们引入了Rust-SWE-bench，该基准包含来自34个多样且流行的Rust仓库的500个真实世界、仓库级软件工程任务。随后，我们对Rust-SWE-bench进行了全面研究，使用四个代表性代理和四个最先进的LLM进行测试，以建立对其在Rust生态系统中能力和限制的理解。我们的广泛研究发现，虽然ReAct风格的代理具有潜力，即解决21.2%的问题，但它们受到两个主要挑战的限制：理解仓库级代码结构和遵循Rust严格的类型和特质语义。我们还发现，问题重现对于任务解决至关重要。受这些发现的启发，我们提出了RUSTFORGER，一种新的代理方法，它集成了自动测试环境设置与Rust元编程驱动的动态跟踪策略，以促进可靠的故障重现和动态分析。评估显示，RUSTFORGER使用Claude-Sonnet-3.7显著优于所有基线，解决了Rust-SWE-bench中的28.6%的任务，即比最强基线高出34.9%，并且在所有采用的先进LLM中，唯一解决了46个其他代理无法解决的任务。

英文摘要：

The Rust programming language presents a steep learning curve and significant coding challenges, making the automation of issue resolution essential for its broader adoption. Recently, LLM-powered code agents have shown remarkable success in resolving complex software engineering tasks, yet their application to Rust has been limited by the absence of a large-scale, repository-level benchmark. To bridge this gap, we introduce Rust-SWE-bench, a benchmark comprising 500 real-world, repository-level software engineering tasks from 34 diverse and popular Rust repositories. We then perform a comprehensive study on Rust-SWE-bench with four representative agents and four state-of-the-art LLMs to establish a foundational understanding of their capabilities and limitations in the Rust ecosystem. Our extensive study reveals that while ReAct-style agents are promising, i.e., resolving up to 21.2% of issues, they are limited by two primary challenges: comprehending repository-wide code structure and complying with Rust's strict type and trait semantics. We also find that issue reproduction is rather critical for task resolution. Inspired by these findings, we propose RUSTFORGER, a novel agentic approach that integrates an automated test environment setup with a Rust metaprogramming-driven dynamic tracing strategy to facilitate reliable issue reproduction and dynamic analysis. The evaluation shows that RUSTFORGER using Claude-Sonnet-3.7 significantly outperforms all baselines, resolving 28.6% of tasks on Rust-SWE-bench, i.e., a 34.9% improvement over the strongest baseline, and, in aggregate, uniquely solves 46 tasks that no other agent could solve across all adopted advanced LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文