arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2308.08784cs.SEcs.AI
收藏

CodeCoT:应对代码生成中思维链推理的代码语法错误

CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation

The University of Hong Kong(香港大学);Shanghai AI Laboratory(上海人工智能实验室)

Dong Huang, Qingwen Bu, Yuhao Qing, Heming Cui

AI总结:

本文提出CodeCoT方法,将思维链推理与自检过程相结合,通过生成测试用例并在本地环境中迭代修复语法错误,从而在保持代码逻辑正确的同时解决语法错误问题,显著提升了代码生成的pass@1指标。

摘要:

思维链(CoT)已成为自然语言处理中的一项突破性工具,尤其在数学证明等复杂推理任务中表现出显著效果。然而,其在代码生成中的应用面临一个独特挑战:尽管使用CoT推理生成的代码在逻辑上是正确的,但在代码执行过程中会出现语法错误问题(例如无效语法错误报告),这导致CoT结果在HumanEval上的pass@1甚至低于零样本结果。在本文中,我们提出了代码思维链(CodeCoT),它将CoT与自检过程相结合用于代码生成。CodeCoT首先让大语言模型(LLMs)使用CoT进行初始代码开发,以确保生成的代码遵循正确的逻辑流程。然后,CodeCoT生成测试用例来验证代码在执行过程中是否存在语法错误。接着,CodeCoT采用自检阶段,在该阶段中,生成的代码在本地环境中针对这些测试用例执行。如果本地环境报出错误信息(例如无效语法错误),CodeCoT将根据反馈信息迭代地改进代码。在这个循环中,CodeCoT能够确保其生成的代码不仅遵循代码描述的逻辑流程,而且语法错误也通过自检过程得到解决。我们的评估结果表明,CodeCoT提高了代码生成的有效性。例如,CodeCoT将HumanEval数据集上的pass@1从75.6%提高到79.3%。

英文摘要:

Chain-of-thought (CoT) has emerged as a groundbreaking tool in NLP, notably for its efficacy in complex reasoning tasks, such as mathematical proofs. However, its application in code generation faces a distinct challenge, i.e., although the code generated with CoT reasoning is logically correct, it faces the problem of syntax error (e.g., invalid syntax error report) during code execution, which causes the CoT result's pass@1 in HumanEval even lower than the zero-shot result. In this paper, we present Code Chain-of-Thought (CodeCoT) that integrates CoT with a self-examination process for code generation. CodeCoT begins with the LLMs using CoT for initial code development to ensure the generated code follows the correct logic flow. Then, CodeCoT will generate test cases to validate whether the code has syntax errors during the execution. CodeCoT then employs a self-examination phase, in which the generated code is executed against these test cases in the local environment. If the local environment raises error information (e.g., invalid syntax error), CodeCoT will iteratively refine the code based on the feedback information. Within this loop, CodeCoT can make sure their generated codes not only follow the logic flow of the code description, but the syntax error will also be addressed with the self-examination process. Our evaluation results reveal that CodeCoT improves the effectiveness of code generation. For example, CodeCoT increases pass@1 from 75.6% to 79.3% for the HumanEval dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑