arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.13976cs.SE
收藏

筛选谷壳:利用执行反馈对生成的代码候选进行排序

Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates

Zhihong Sun, Yao Wan, Jia Li, Hongyu Zhang, Zhi Jin, Ge Li, Chen Lyu

AI总结:

针对代码生成中候选代码排序难题,本文提出RankEF方法,利用执行反馈结合多任务学习,无需执行代码即可区分正误,在三个基准上显著超越现有最优CodeRanker。

摘要:

大型语言模型(LLMs),如GPT-4、StarCoder和CodeLlama,通过根据给定的自然语言描述自动生成代码,正在改变开发者处理编程的方式。尽管取得了进步,但生成语法和语义上正确的代码仍然具有挑战性,尤其是对于复杂的编程任务。现有方法通常使用LLMs生成多个候选解决方案,以增加产生正确代码的可能性。然而,从这些候选中选择正确的代码——这一过程被称为代码排序——仍然是一个重大挑战。当前关于代码排序的研究可以分为基于执行的方法和基于非执行的方法。基于执行的方法虽然有效,但遇到了显著的局限性,例如高质量单元测试的稀缺性和安全风险。基于非执行的方法如CodeRanker,仅依赖分类标签来训练代码排序器,难以捕捉细微的错误并提供详细的错误见解。认识到两种方法的优势和局限性,我们提出了一种新方法。我们工作的关键见解是,一个有效的代码排序器应该真正理解错误代码的根本原因,因为仅依赖分类标签是不够的。受此启发,本文提出了RankEF,一种利用执行反馈进行代码排序的创新方法。RankEF采用多任务学习,将代码分类与执行反馈生成相结合。这种方法使模型能够理解错误代码背后的原因,在排序阶段无需执行代码即可区分正确和不正确的解决方案。在三个代码生成基准上的实验表明,RankEF显著优于最先进的CodeRanker。

英文摘要:

Large Language Models (LLMs), such as GPT-4, StarCoder, and CodeLlama, are transforming the way developers approach programming by automatically generating code based on given natural language descriptions. Despite advancements, generating syntactically and semantically correct code remains challenging, especially for complex programming tasks. Existing approaches typically generate multiple candidate solutions using LLMs to increase the likelihood of producing correct code. However, selecting the correct code from these candidates-a process known as code ranking-remains a major challenge. Current research on code ranking can be categorized into execution-based and non-execution-based methods. Execution-based methods, although effective, encounter notable limitations, such as scarcity of quality unit tests and security risks. Non-execution-based methods like CodeRanker, which rely solely on classification labels to train a code ranker, struggle to capture subtle errors and provide detailed error insights. Recognizing the strengths and limitations of both approaches, we propose a new method. The key insight of our work is that an effective code ranker is expected to truly comprehend the underlying causes of erroneous code, as relying solely on classification labels is insufficient. Inspired by this, this paper puts forward RankEF, an innovative approach for code ranking that leverages execution feedback. RankEF employs multi-task learning to integrate code classification with execution feedback generation. This approach enables the model to understand the reasons behind incorrect code, distinguishing between correct and incorrect solutions without the need to execute the code during the ranking phase. Experiments on three code generation benchmarks demonstrate that RankEF significantly outperforms the state-of-the-art CodeRanker.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑