大语言模型在语法感知代码中间填充任务上的评估
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
AI总结：
本研究提出语法感知代码中间填充基准SAFIM，含多语言17720个低污染示例与配套评估框架，经15个LLM评估发现FIM预训练可同时提升FIM与L2R能力，预训练方法和数据质量比模型规模更重要。
摘要：
我们提出了语法感知中间填充（Syntax-Aware Fill-In-the-Middle，SAFIM），这是一个用于评估大语言模型（Large Language Models，LLMs）代码中间填充（Fill-in-the-Middle，FIM）任务的新基准。该基准聚焦于代码块、条件表达式等程序结构的语法感知补全，包含来自多种编程语言的17720个示例，均取自2022年4月之后的最新代码提交，以最大程度降低数据污染。SAFIM提供了一个具备多种提示词设计与新型语法感知后处理技术的稳健框架，可实现不同LLM之间准确、公平的比较。我们对15个LLM的综合评估显示，FIM预训练不仅能提升FIM任务能力，还能改善LLM的从左到右（Left-to-Right，L2R）推理效果。我们的发现挑战了传统认知，表明预训练方法与数据质量的影响比模型规模更大。因此，SAFIM可作为未来代码LLM高效预训练策略研究的基础平台。评估工具包与数据集可在https://github.com/gonglinyuan/safim获取，排行榜可在https://safimbenchmark.com查看。
英文摘要：
We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. This benchmark focuses on syntax-aware completions of program structures such as code blocks and conditional expressions, and includes 17,720 examples from multiple programming languages, sourced from recent code submissions after April 2022 to minimize data contamination. SAFIM provides a robust framework with various prompt designs and novel syntax-aware post-processing techniques, facilitating accurate and fair comparisons across LLMs. Our comprehensive evaluation of 15 LLMs shows that FIM pretraining not only enhances FIM proficiency but also improves Left-to-Right (L2R) inference using LLMs. Our findings challenge conventional beliefs and suggest that pretraining methods and data quality have more impact than model size. SAFIM thus serves as a foundational platform for future research in effective pretraining strategies for code LLMs. The evaluation toolkit and dataset are available at https://github.com/gonglinyuan/safim, and the leaderboard is available at https://safimbenchmark.com.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
