arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.16906cs.SEcs.AIcs.CL
收藏

像人类一样调试:一种通过逐步验证运行时执行的大语言模型调试器

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step

University of California, San Diego(加利福尼亚大学圣迭戈分校)

Li Zhong, Zilong Wang, Jingbo Shang

AI总结:

本文提出LDB调试框架,将程序分割为基本块并跟踪运行时中间变量值,使LLMs能逐块验证代码正确性并定位错误,在多个基准上提升调试性能最高达9.8%。

摘要:

大语言模型(LLMs)正在推动代码生成领域的显著进展。除了一次性代码生成之外,近期工作进一步将单元测试和程序验证器整合到LLMs中,以迭代地改进生成的程序。然而,这些工作将生成的程序视为不可分割的整体,这在LLMs调试程序时存在不足,尤其是当程序包含复杂的逻辑流程和数据操作时。相比之下,当人类开发者调试程序时,他们通常会设置断点并有选择地检查运行时执行信息。执行流程和中间变量在调试过程中起着至关重要的作用,但在现有的代码生成文献中却未被充分利用。在本研究中,我们提出了Large Language Model Debugger(LDB),这是一种新颖的调试框架,使LLMs能够利用运行时执行信息来改进其生成的程序。具体来说,LDB将程序分割为基本块,并在整个运行时执行过程中跟踪每个块之后中间变量的值。这使得LLMs能够专注于整体执行流程中更简单的代码单元,根据任务描述逐块验证其正确性,并高效地定位任何潜在错误。实验表明,LDB在HumanEval、MBPP和TransCoder基准上将基线性能一致提升了最高9.8%,在各种LLM选择中取得了代码调试领域新的最先进性能。

英文摘要:

Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated programs as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments the programs into basic blocks and tracks the values of intermediate variables after each block throughout the runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and efficiently pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑