arXiv 2406.010062024-06cs.CLcs.AIcs.SE

SemCoder：通过综合语义推理训练代码语言模型

SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning

Columbia University（哥伦比亚大学）

Yangruibo Ding, Jinjun Peng, Marcus J. Min, Gail Kaiser, Junfeng Yang, Baishakhi Ray

备注：: NeurIPS 2024 Camera-ready

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对代码大语言模型缺乏深层语义理解的问题，提出独白推理策略和PyX语料库，训练出67亿参数的SemCoder，在多项代码任务上性能媲美甚至超过GPT-3.5-turbo，还可提升调试与自优化能力。

摘要：

代码大语言模型（Code LLMs）在代码补全等任务上表现出色，但往往缺乏对执行效果、动态状态等深层语义的把握。本文旨在弥合Code LLM对静态文本数据的依赖，与调试、程序修复等复杂任务所需的语义理解能力之间的差距。我们提出一种名为“独白推理”的新策略，用于训练Code LLM进行综合语义推理，涵盖高层功能描述、单条语句的局部执行效果以及整体输入输出行为，从而将静态代码文本与动态执行状态关联起来。我们首先收集了PyX——一个干净的Python语料库，包含带有功能描述和测试用例的完全可执行代码样本。我们提出训练Code LLM不仅要会编写代码，还要通过用自然语言推理关键属性、约束条件和执行行为来理解代码语义，模仿人类的口头调试即橡皮鸭调试法。该方法催生了SemCoder——一个仅含67亿参数的Code LLM，它在代码生成和执行推理任务上展现出与GPT-3.5-turbo相当的性能。SemCoder在HumanEval上达到79.3%（GPT-3.5-turbo为76.8%），在CRUXEval-I上达到63.6%（GPT-3.5-turbo为50.3%），在CRUXEval-O上达到63.9%（GPT-3.5-turbo为59.0%）。我们还研究了SemCoder的独白式执行推理相较于具体草稿板推理的有效性，表明我们的方法能更顺畅地整合多维度语义。最后，我们展示了将习得的语义用于提升Code LLM的调试和自我优化能力的潜力。我们的数据、代码和模型可在以下地址获取：https://github.com/ARiSE-Lab/SemCoder。

英文摘要：

Code Large Language Models (Code LLMs) have excelled at tasks like code completion but often miss deeper semantics such as execution effects and dynamic states. This paper aims to bridge the gap between Code LLMs' reliance on static text data and the need for semantic understanding for complex tasks like debugging and program repair. We introduce a novel strategy, monologue reasoning, to train Code LLMs to reason comprehensive semantics, encompassing high-level functional descriptions, local execution effects of individual statements, and overall input/output behavior, thereby linking static code text with dynamic execution states. We begin by collecting PyX, a clean Python corpus of fully executable code samples with functional descriptions and test cases. We propose training Code LLMs not only to write code but also to understand code semantics by reasoning about key properties, constraints, and execution behaviors using natural language, mimicking human verbal debugging, i.e., rubber-duck debugging. This approach led to the development of SemCoder, a Code LLM with only 6.7B parameters, which shows competitive performance with GPT-3.5-turbo on code generation and execution reasoning tasks. SemCoder achieves 79.3% on HumanEval (GPT-3.5-turbo: 76.8%), 63.6% on CRUXEval-I (GPT-3.5-turbo: 50.3%), and 63.9% on CRUXEval-O (GPT-3.5-turbo: 59.0%). We also study the effectiveness of SemCoder's monologue-style execution reasoning compared to concrete scratchpad reasoning, showing that our approach integrates semantics from multiple dimensions more smoothly. Finally, we demonstrate the potential of applying learned semantics to improve Code LLMs' debugging and self-refining capabilities. Our data, code, and models are available at: https://github.com/ARiSE-Lab/SemCoder.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文