CodeSimpleQA: 在代码大语言模型中提升事实性
CodeSimpleQA: Scaling Factuality in Code Large Language Models
AI总结：
CodeSimpleQA通过构建双语基准测试和后训练框架，提升代码大语言模型在编程知识事实准确性上的表现。
摘要：
大型语言模型（LLMs）在代码生成方面取得了显著进展，能够从自然语言指令中合成出出色的代码片段。然而，确保LLMs在编程概念、技术实现等方面生成事实准确的响应仍是一个关键挑战。大多数先前的代码相关基准测试侧重于代码执行的正确性，忽略了编程知识的事实准确性。为了解决这一差距，我们提出了CodeSimpleQA，一个全面的双语基准测试，旨在评估代码LLMs在回答代码相关问题时的事实准确性，其中包含精心挑选的英文和中文问题-答案对，涵盖多种编程语言和主要计算机科学领域。进一步，我们创建了CodeSimpleQA-Instruct，一个包含6600万样本的大规模指令语料库，并开发了一个结合监督微调和强化学习的后训练框架。我们对各种LLMs的全面评估表明，即使是前沿LLMs在代码事实性上也存在困难。我们提出的框架在基础模型上表现出显著的改进，凸显了在开发可靠的代码LLMs中事实性意识对齐的重要性。
英文摘要：
Large language models (LLMs) have made significant strides in code generation, achieving impressive capabilities in synthesizing code snippets from natural language instructions. However, a critical challenge remains in ensuring LLMs generate factually accurate responses about programming concepts, technical implementations, etc. Most previous code-related benchmarks focus on code execution correctness, overlooking the factual accuracy of programming knowledge. To address this gap, we present CodeSimpleQA, a comprehensive bilingual benchmark designed to evaluate the factual accuracy of code LLMs in answering code-related questions, which contains carefully curated question-answer pairs in both English and Chinese, covering diverse programming languages and major computer science domains. Further, we create CodeSimpleQA-Instruct, a large-scale instruction corpus with 66M samples, and develop a post-training framework combining supervised fine-tuning and reinforcement learning. Our comprehensive evaluation of diverse LLMs reveals that even frontier LLMs struggle with code factuality. Our proposed framework demonstrates substantial improvements over the base model, underscoring the critical importance of factuality-aware alignment in developing reliable code LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
