OpenCodeInstruct：面向代码大语言模型的大规模指令微调数据集
OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
AI总结：
针对代码LLM微调数据稀缺问题，提出含500万样本的OpenCodeInstruct数据集，并在多模型多规模上验证其显著提升代码生成等基准性能。
摘要：
大型语言模型（LLMs）通过实现代码生成、自动调试和复杂推理，已经改变了软件开发领域。然而，它们的持续进步受到针对编码任务的高质量、公开可用的监督微调（SFT）数据集稀缺性的制约。为了弥合这一差距，我们推出了OpenCodeInstruct，这是最大的开放访问指令微调数据集，包含500万个多样化的样本。每个样本包括一个编程问题、解决方案、测试用例、执行反馈以及LLM生成的质量评估。我们使用该数据集对多种基础模型（包括LLaMA和Qwen）在多个规模（1B+、3B+和7B+）上进行微调。在流行基准（HumanEval、MBPP、LiveCodeBench和BigCodeBench）上的全面评估表明，使用OpenCodeInstruct进行SFT取得了显著的性能提升。我们还提出了一种详细的方法论，涵盖种子数据整理、合成指令和解决方案生成以及过滤。
英文摘要：
Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
