NaturalCodeBench：考察 HumanEval 与自然用户提示上的编程性能不匹配
NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts
AI总结：
提出 NaturalCodeBench 真实编程基准及半自动化测试用例构建流水线，通过 39 个 LLM 实验揭示 HumanEval 分数相近模型的实际代码能力仍存在显著差距。
摘要：
大型语言模型（LLM）已展现出为生产活动生成代码的强大能力。然而，当前用于代码合成的基准（如 HumanEval、MBPP 和 DS-1000）主要面向算法和数据科学的入门级任务，无法充分满足现实世界编程中普遍存在的挑战性需求。为填补这一空白，我们提出 NaturalCodeBench（NCB），这是一个旨在反映真实编程任务中场景复杂性与多样性的具有挑战性的代码基准。NCB 包含 402 个 Python 和 Java 高质量问题，这些问题从在线编程服务的自然用户查询中精心筛选，覆盖 6 个不同领域。注意到为现实世界查询创建测试用例异常困难，我们还引入了一条半自动化流水线，以提升测试用例构建的效率。与人工方案相比，其实现了 4 倍以上的效率提升。我们在 39 个 LLM 上进行的系统实验发现，HumanEval 分数接近的模型在 NCB 上的性能差距仍可能十分显著，这表明它们缺乏对实际代码合成场景的关注，或在 HumanEval 上存在过度特定化的优化。另一方面，即使是表现最好的 GPT-4，在 NCB 上仍远未达到令人满意的程度。评估工具包和开发集可在 https://github.com/THUDM/NaturalCodeBench 获取。
英文摘要：
Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the best-performing GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github.com/THUDM/NaturalCodeBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
