arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.15658cs.SEcs.AI
收藏

关于数据科学代码生成的自我纠正大型语言模型的实证研究

An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation

Ho Chi Minh City University of Technology(胡志明市理工大学);University of South Eastern Norway(南挪威大学)

Thai Tang Quoc, Duc Ha Minh, Tho Quan Thanh, Anh Nguyen-Duc

AI总结:

本文提出CoT-SelfEvolve方法,通过基于真实编程反馈构建的思维链指导LLM进行迭代自我纠正,显著提升了数据科学代码生成在DS-1000数据集上的准确性与复杂问题解决能力。

摘要:

大型语言模型(LLMs)近期在软件工程任务的许多应用中取得了进展,特别是在代码生成方面的潜力。在当代挑战中,LLMs生成的代码经常存在不准确和幻觉问题,需要外部输入来纠正。解决这些问题的一种最新策略是利用模型自身的输入(自我增强)来细化LLMs生成的代码。在本研究中,我们提出了一种新方法,即CoT-SelfEvolve。CoT-SelfEvolve在由真实世界编程问题反馈构建的思维链指导下,通过自我纠正过程迭代且自动地细化代码。专注于数据科学代码,包括NumPy和Pandas等Python库,我们在DS-1000数据集上的评估表明,CoT-SelfEvolve在解决复杂问题方面显著优于现有模型。该框架在初始代码生成和后续迭代中均显示出实质性改进,模型的准确性随着每次额外迭代而显著提高。这凸显了使用思维链提示来解决程序执行器回溯错误消息所揭示复杂性的有效性。我们还讨论了如何将CoT-SelfEvolve集成到持续软件工程环境中,为改进基于LLM的代码生成提供实用解决方案。

英文摘要:

Large Language Models (LLMs) have recently advanced many applications on software engineering tasks, particularly the potential for code generation. Among contemporary challenges, code generated by LLMs often suffers from inaccuracies and hallucinations, requiring external inputs to correct. One recent strategy to fix these issues is to refine the code generated from LLMs using the input from the model itself (self-augmented). In this work, we proposed a novel method, namely CoT-SelfEvolve. CoT-SelfEvolve iteratively and automatically refines code through a self-correcting process, guided by a chain of thought constructed from real-world programming problem feedback. Focusing on data science code, including Python libraries such as NumPy and Pandas, our evaluations on the DS-1000 dataset demonstrate that CoT-SelfEvolve significantly outperforms existing models in solving complex problems. The framework shows substantial improvements in both initial code generation and subsequent iterations, with the model's accuracy increasing significantly with each additional iteration. This highlights the effectiveness of using chain-of-thought prompting to address complexities revealed by program executor traceback error messages. We also discuss how CoT-SelfEvolve can be integrated into continuous software engineering environments, providing a practical solution for improving LLM-based code generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑