arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.19166cs.SEcs.LG
收藏

CodeIF:基准测试大型语言模型在代码生成中的指令遵循能力

CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation

Beihang University(北航大学);Tsinghua University(清华大学);Xiaohongshu(小红书)

Kaiwen Yan, Hongcheng Guo, Xuanqing Shi, Shaosheng Cao, Donglin Di, Zhoujun Li

AI总结:

本文提出CodeIF基准,首个专门评估LLM在代码生成中指令遵循能力的基准,涵盖多任务,实验揭示模型优势与局限,为提升代码生成适应性提供方向。

摘要:

随着大型语言模型(LLMs)的快速发展,在代码生成任务中对强大指令遵循能力的需求显著增长。代码生成不仅促进了更快速的原型设计和自动化测试,还通过提高代码的可维护性和可重用性来增强开发者的效率。在本文中,我们介绍了CodeIF,这是第一个专门设计用于评估LLMs在多样化代码生成场景中遵循任务导向指令能力的基准。CodeIF涵盖了广泛的任务,包括函数合成、错误调试、算法重构和代码解释,从而提供了一个全面的套件,用于评估模型在不同复杂程度和编程领域中的性能。我们使用LLMs进行了广泛的实验,分析了它们在满足这些任务需求方面的优势和局限性。实验结果提供了关于当前模型如何与人类指令对齐,以及它们能在多大程度上生成一致、可维护且上下文相关的代码的宝贵见解。我们的发现不仅强调了遵循指令的LLMs在现代软件开发中可以发挥的关键作用,还为旨在提高其在自动代码生成中的适应性、可靠性和整体有效性的未来研究指明了途径。CodeIF数据和代码公开可用:https://github.com/lin-rany/codeIF

英文摘要:

With the rapid advancement of Large Language Models (LLMs), the demand for robust instruction-following capabilities in code generation tasks has grown significantly. Code generation not only facilitates faster prototyping and automated testing, but also augments developer efficiency through improved maintainability and reusability of code. In this paper, we introduce CodeIF, the first benchmark specifically designed to assess the abilities of LLMs to adhere to task-oriented instructions within diverse code generation scenarios. CodeIF encompasses a broad range of tasks, including function synthesis, error debugging, algorithmic refactoring, and code explanation, thereby providing a comprehensive suite to evaluate model performance across varying complexity levels and programming domains. We conduct extensive experiments with LLMs, analyzing their strengths and limitations in meeting the demands of these tasks. The experimental results offer valuable insights into how well current models align with human instructions, as well as the extent to which they can generate consistent, maintainable, and contextually relevant code. Our findings not only underscore the critical role that instruction-following LLMs can play in modern software development, but also illuminate pathways for future research aimed at enhancing their adaptability, reliability, and overall effectiveness in automated code generation. CodeIF data and code are publicly available: https://github.com/lin-rany/codeIF

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑