arXiv 2506.035242025-06-05cs.CLcs.SE

Seed-Coder：让代码模型为自己筛选数据

Seed-Coder: Let the Code Model Curate Data for Itself

ByteDance Seed（字节跳动种子）

ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, Jinhua Zhu, Shulin Xin, Don… 展开作者

ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, Jinhua Zhu, Shulin Xin, Dong Huang, Yetao Bai, Lixin Dong, Chao Li, Jianchong Chen, Hanzhi Zhou, Yifan Huang, Guanghan Ning, Xierui Song, Jiaze Chen, Siyao Liu, Kai Shen, Liang Xiang, Yonghui Wu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

Seed-Coder 提出以模型为中心的数据流水线，减少人工参与代码预训练数据构建，并通过指令微调与 LongCoT 强化学习，在 8B 规模下取得领先的代码与软件工程性能。

摘要：

在大语言模型（LLM）预训练中，代码数据不仅对代码相关任务至关重要，而且对提升 LLM 的通用智能也很关键。当前的开源 LLM 通常严重依赖人工来生成代码预训练数据，例如采用针对不同编程语言手工设计的过滤规则，或使用人工标注数据来训练质量过滤器。然而，这些方法在可扩展性方面存在固有局限，容易受到主观偏差影响，并且在不同编程语言之间扩展和维护成本高昂。为应对这些挑战，我们提出了 Seed-Coder，这是一个由 8B 规模的基座、指令和推理模型组成的开源 LLM 系列，在数据构建中最大限度地减少人工参与。我们的代码预训练数据由以模型为中心的数据流水线生成，该流水线主要利用 LLM 对代码数据进行评分和过滤。指令模型进一步通过监督微调和偏好优化进行训练，推理模型则利用长思维链（LongCoT）强化学习来提升多步代码推理能力。Seed-Coder 在同等规模的开源模型中取得了最先进的结果，甚至超越了一些规模大得多的模型，在代码生成、代码补全、代码编辑、代码推理和软件工程任务中展现出卓越性能。

英文摘要：

Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source LLMs often heavily rely on human effort to produce their code pretraining data, such as employing hand-crafted filtering rules tailored to individual programming languages, or using human-annotated data to train quality filters. However, these approaches are inherently limited in scalability, prone to subjective biases, and costly to extend and maintain across diverse programming languages. To address these challenges, we introduce Seed-Coder, a series of open-source LLMs comprising base, instruct and reasoning models of 8B size, minimizing human involvement in data construction. Our code pretraining data is produced by a model-centric data pipeline, which predominantly leverages LLMs for scoring and filtering code data. The instruct model is further trained via supervised fine-tuning and preference optimization, and the reasoning model leverages Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning. Seed-Coder achieves state-of-the-art results among open-source models of similar size and even surpasses some much larger models, demonstrating superior performance in code generation, code completion, code editing, code reasoning, and software engineering tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。