用于代码生成的结构化思维链提示
Structured Chain-of-Thought Prompting for Code Generation
AI总结：
针对CoT提示在代码生成中准确率低的问题，提出SCoT提示，利用顺序、分支和循环三种程序结构构建中间推理步骤，在三个基准上使ChatGPT和Codex的Pass@1提升达13.79%，且更受开发者偏好。
摘要：
大型语言模型（LLMs，例如ChatGPT）在代码生成中展现了出色的性能。LLMs将提示作为输入，而思维链（CoT）提示是最先进的提示技术。CoT提示要求LLMs首先生成CoT（即中间自然语言推理步骤），然后输出代码。然而，CoT提示是为自然语言生成设计的，在代码生成中准确率较低。在本文中，我们提出了结构化思维链（SCoTs），并提出了一种用于代码生成的新型提示技术，名为SCoT提示。我们的动机是源代码包含丰富的结构信息，任何代码都可以由三种程序结构（即顺序、分支和循环结构）组成。直觉上，结构化的中间推理步骤有助于生成结构化的源代码。因此，我们要求LLMs使用程序结构构建CoT，从而获得SCoTs。然后，LLMs基于SCoTs生成最终代码。与CoT提示相比，SCoT提示明确约束LLMs从源代码的角度思考如何解决需求，并进一步提升LLMs在代码生成中的性能。我们将SCoT提示应用于两个LLMs（即ChatGPT和Codex），并在三个基准（即HumanEval、MBPP和MBCPP）上进行评估。(1) SCoT提示在Pass@1上优于最先进的基线——CoT提示，提升高达13.79%。(2) 人类评估表明，人类开发者更偏好SCoT提示生成的程序。(3) SCoT提示对示例具有鲁棒性，并实现了显著的改进。
英文摘要：
Large Language Models (LLMs) (e.g., ChatGPT) have shown impressive performance in code generation. LLMs take prompts as inputs, and Chain-of-Thought (CoT) prompting is the state-of-the-art prompting technique. CoT prompting asks LLMs first to generate CoTs (i.e., intermediate natural language reasoning steps) and then output the code. However, CoT prompting is designed for natural language generation and has low accuracy in code generation. In this paper, we propose Structured CoTs (SCoTs) and present a novel prompting technique for code generation, named SCoT prompting. Our motivation is source code contains rich structural information and any code can be composed of three program structures (i.e., sequence, branch, and loop structures). Intuitively, structured intermediate reasoning steps make for structured source code. Thus, we ask LLMs to use program structures to build CoTs, obtaining SCoTs. Then, LLMs generate the final code based on SCoTs. Compared to CoT prompting, SCoT prompting explicitly constrains LLMs to think about how to solve requirements from the view of source code and further the performance of LLMs in code generation. We apply SCoT prompting to two LLMs (i.e., ChatGPT and Codex) and evaluate it on three benchmarks (i.e., HumanEval, MBPP, and MBCPP). (1) SCoT prompting outperforms the state-of-the-art baseline - CoT prompting by up to 13.79% in Pass@1. (2) Human evaluation shows human developers prefer programs from SCoT prompting. (3) SCoT prompting is robust to examples and achieves substantial improvements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
