arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.10679cs.AIcs.CLcs.SE
收藏

在代码之外思考:头脑风暴提升大型语言模型在代码生成中的表现

Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generation

National Key Laboratory for Novel Software Technology(计算机软件新技术国家重点实验室);Nanjing University(南京大学)

Xin-Ye Li, Jiang-Tian Xue, Zheng Xie, Ming Li

AI总结:

本文提出 Brainstorm 框架,通过生成和选择多样化想法促进算法推理,显著提升大型语言模型解决竞赛级编程问题的能力,在 CodeContests 基准上 pass@$k$ 提升 50% 以上,并使 ChatGPT 达到与人类程序员相当的水平。

摘要:

代码生成旨在根据高层任务规范自动生成源代码,这可以显著提高软件工程的生产力。近期,基于大型语言模型(LLMs)的方法在简单任务上展现出了卓越的代码生成能力。然而,为更复杂的任务(如竞赛级别问题)生成代码仍然具有挑战性。在本文中,我们引入了用于代码生成的 Brainstorm 框架。它利用一个头脑风暴步骤,生成并选择关于该问题的多样化想法以促进算法推理,其中这些想法是解决问题的可能蓝图。我们证明,Brainstorm 显著增强了 LLMs 解决竞赛级别编程问题的能力,使得 ChatGPT 在 CodeContests 基准上的 pass@$k$ 指标提升了 50% 以上,达到了最先进的性能。此外,我们在 LeetCode 竞赛上进行的实验表明,我们的框架将 ChatGPT 的能力提升到了与人类程序员相当的水平。

英文摘要:

Code generation aims to automatically generate source code from high-level task specifications, which can significantly increase productivity of software engineering. Recently, approaches based on large language models (LLMs) have shown remarkable code generation abilities on simple tasks. However, generate code for more complex tasks, such as competition-level problems, remains challenging. In this paper, we introduce Brainstorm framework for code generation. It leverages a brainstorming step that generates and selects diverse thoughts on the problem to facilitate algorithmic reasoning, where the thoughts are possible blueprint of solving the problem. We demonstrate that Brainstorm significantly enhances the ability of LLMs to solve competition-level programming problems, resulting in a more than 50% increase in the pass@$k$ metrics for ChatGPT on the CodeContests benchmark, achieving state-of-the-art performance. Furthermore, our experiments conducted on LeetCode contests show that our framework boosts the ability of ChatGPT to a level comparable to that of human programmers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑