arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.16362cs.SE
收藏

AgentFL: 将基于大语言模型的故障定位扩展到项目级上下文

AgentFL: Scaling LLM-based Fault Localization to Project-Level Context

Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, Xiaoguang Mao

AI总结:

针对现有大语言模型故障定位难以处理大代码范围的问题,提出模拟人类开发者调试流程的多智能体系统AgentFL,在Defects4J基准上取得优于同类方法的Top-1定位效果,且成本较低。

摘要:

故障定位(Fault Localization,FL)是调试过程中的关键步骤。凭借强大的代码理解能力,近期的大语言模型(Large Language Models,LLMs)在代码缺陷诊断方面展现出可观的性能。然而,由于LLM处理长上下文的能力有限,现有基于LLM的故障定位方法仍局限于在小代码范围(即单个方法或类)内定位缺陷,难以对大代码范围(即整个软件系统)进行缺陷诊断。为解决这一局限,本文提出AgentFL——一种基于ChatGPT的多智能体自动化故障定位系统。通过模拟人类开发者的行为,AgentFL将故障定位任务建模为理解、导航、确认三个步骤的流程。在每个步骤中,AgentFL调用具备不同专业能力的智能体,每个智能体使用不同工具处理特定任务。具体而言,我们采用了测试行为追踪(Test Behavior Tracking)、文档引导搜索(Document-Guided Search)、多轮对话(Multi-Round Dialogue)等一系列辅助策略,以克服每个步骤中的挑战。在广泛使用的Defects4J-V1.2.0基准上的评估表明,AgentFL在Top-1指标下可定位395个缺陷中的157个,性能优于其他基于LLM的方法,且与当前最先进的基于学习的技术具有互补性。此外,我们通过消融实验验证了AgentFL各组件的不可或缺性,并通过用户研究证明了AgentFL的可用性。最后,成本分析显示,AgentFL定位单个缺陷的平均花费仅为0.074美元,耗时97秒。

英文摘要:

Fault Localization (FL) is an essential step during the debugging process. With the strong capabilities of code comprehension, the recent Large Language Models (LLMs) have demonstrated promising performance in diagnosing bugs in the code. Nevertheless, due to LLMs' limited performance in handling long contexts, existing LLM-based fault localization remains on localizing bugs within a small code scope (i.e., a method or a class), which struggles to diagnose bugs for a large code scope (i.e., an entire software system). To address the limitation, this paper presents AgentFL, a multi-agent system based on ChatGPT for automated fault localization. By simulating the behavior of a human developer, AgentFL models the FL task as a three-step process, which involves comprehension, navigation, and confirmation. Within each step, AgentFL hires agents with diversified expertise, each of which utilizes different tools to handle specific tasks. Particularly, we adopt a series of auxiliary strategies such as Test Behavior Tracking, Document-Guided Search, and Multi-Round Dialogue to overcome the challenges in each step. The evaluation on the widely used Defects4J-V1.2.0 benchmark shows that AgentFL can localize 157 out of 395 bugs within Top-1, which outperforms the other LLM-based approaches and exhibits complementarity to the state-of-the-art learning-based techniques. Additionally, we confirm the indispensability of the components in AgentFL with the ablation study and demonstrate the usability of AgentFL through a user study. Finally, the cost analysis shows that AgentFL spends an average of only 0.074 dollars and 97 seconds for a single bug.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑