评估大型语言模型中面向API代码生成的综合框架
A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models
AI总结：
提出AutoAPIEval框架，系统评估LLM在面向API代码生成中的API推荐和代码示例生成能力，发现不同模型表现差异并验证检索增强生成的提升作用。
摘要：
像GitHub Copilot和ChatGPT这样的大型语言模型（LLM）已经成为代码生成的强大工具，显著提升了生产力并加速了软件开发。然而，现有的基准测试主要关注通用代码生成，而没有考虑面向API的代码生成，即生成调用特定库中API的代码。鉴于面向API代码生成的需求日益增长，迫切需要一种系统化且自动化的方法来评估LLM在面向API代码生成方面的能力。为了填补这一空白，我们提出了AutoAPIEval，这是一个轻量级且自动化的框架，旨在评估LLM在面向API代码生成方面的能力。我们的框架适用于任何提供API文档的库，并专注于两个单元任务：API推荐和代码示例生成，同时采用四个指标来评估生成的API和代码示例，例如任务1中错误API推荐的比例，以及任务2中未调用特定API的代码示例比例和无法编译/无法执行的代码示例比例。此外，我们对三个LLM（ChatGPT、MagiCoder和DeepSeek Coder）以及Java Runtime Environment 8进行了案例研究，以展示该框架的有效性。我们的发现表明，LLM在不同任务中的表现存在显著差异，其中ChatGPT更好地遵循指令，而在代码示例生成方面与同类模型（即MagiCoder和DeepSeek Coder）表现出相似的有效性。我们还识别了与代码质量相关的关键因素，如API流行度和模型置信度，并构建了分类器，在检测错误API推荐和错误代码示例方面实现了高准确率。检索增强生成提升了LLM生成代码的质量，尽管其有效性在不同LLM中有所差异。
英文摘要：
Large language models (LLMs) like GitHub Copilot and ChatGPT have emerged as powerful tools for code generation, significantly enhancing productivity and accelerating software development. However, existing benchmarks primarily focus on general code generation without considering API-oriented code generation, i.e., generating code that invokes APIs from specific libraries. Given the growing demand for API-oriented code generation, there is a pressing need for a systematic and automated approach to evaluate LLM on API-oriented code generation. To address this gap, we propose AutoAPIEval, a lightweight and automated framework designed to evaluate the capabilities of LLMs in API-oriented code generation. Our framework works with any library that provides API documentation and focuses on two unit tasks: API recommendation and code example generation, along with four metrics to evaluate the generated APIs and code examples, such as the proportion of incorrect API recommendations for Task 1, and the proportion of code examples where no specific API is invoked and uncompilable/unexecutable code examples for Task 2. In addition, we conducted a case study on three LLMs (ChatGPT, MagiCoder, and DeepSeek Coder) and Java Runtime Environment 8 to demonstrate the framework's effectiveness. Our findings reveal substantial variability in LLM performance across tasks, with ChatGPT adhering better to instructions, while sharing similar effectiveness in code example generation with its counterparts (i.e., MagiCoder and DeekSeek Coder). We also identify key factors associated with code quality, such as API popularity and model confidence, and build classifiers that achieve high accuracy in detecting incorrect API recommendations and erroneous code examples. Retrieval-augmented generation enhances the quality of code generated by LLMs, though its effectiveness varies across different LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
