COAST：通过基于通信智能体的数据合成增强大语言模型（LLMs）的代码调试能力
COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis
AI总结：
针对现有代码调试基准仅聚焦代码修复阶段的局限，本文提出DEBUGEVAL多阶段评估基准，发现7B规模模型调试能力不足；进而提出COAST多智能体数据合成框架，生成的训练数据可使7B规模LLMs调试性能媲美GPT-3.5。
摘要：
代码调试是软件开发的关键阶段，对确保大语言模型（LLMs）在代码生成任务中的可靠性和性能至关重要。人类调试通常遵循多阶段流程，包括错误定位、错误识别、代码修复和代码验证。然而，现有代码调试基准主要聚焦于代码修复阶段，仅能从有限视角评估LLMs的调试能力。本文介绍DEBUGEVAL，这是一个通过模拟人类多阶段调试流程来全面评估LLMs调试能力的基准。通过在DEBUGEVAL上进行评估，我们发现7B规模模型始终落后于更大规模的模型，凸显其在理解代码语义方面的局限性。在此情况下，我们提出基于通信智能体的数据合成（COAST）框架，该框架采用多智能体系统生成高质量训练数据用于监督微调（SFT）。实验结果表明，COAST生成的数据优于人工整理和GPT-4生成的数据，使7B规模LLMs达到与GPT-3.5相当的调试性能。所有数据和代码可在https://github.com/NEUIR/COAST获取。
英文摘要：
Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human debugging typically follows a multi-stage process, which includes Bug Localization, Bug Identification, Code Repair, and Code Recognition. However, existing code debugging benchmarks predominantly focus on the Code Repair stage, which offers only a limited perspective on evaluating the debugging capabilities of LLMs. In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process. Through evaluating on DEBUGEVAL, we observe that 7B-scale models consistently underperform compared to their larger counterparts, highlighting their limitations in comprehending code semantics. In this case, we propose the COmmunicative Agent-based data SynThesis (COAST) framework, which employs a multi-agent system to generate high-quality training data for supervised fine-tuning (SFT). Experimental results demonstrate that COAST-generated data outperform human-curated and GPT-4-generated data, enabling 7B-scale LLMs to achieve debugging performance comparable to GPT-3.5. All data and codes are available at https://github.com/NEUIR/COAST.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
