arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.09635cs.CL
收藏

评估代码生成中库的上下文学习能力

Evaluating In-Context Learning of Libraries for Code Generation

Mila(米拉);McGill University(麦吉尔大学);Allen Institute for AI(艾伦人工智能研究所);ServiceNow Research(ServiceNow研究院);Facebook(脸书)

Arkil Patel, Siva Reddy, Dzmitry Bahdanau, Pradeep Dasigi

AI总结:

本文系统评估了多种LLMs在仅依赖上下文库规范、自然语言描述或原始代码实现时生成代码的能力,发现即使较小的开源模型也能熟练学习新库,为动态编码环境中的应用奠定基础。

摘要:

当代大型语言模型(LLMs)展现出高度的代码生成与理解能力。其中一个特别有前景的方向是它们能够解读不熟悉库中的代码模块,以解决用户指定的任务。近期研究表明,大型专有LLMs能够从演示中在上下文中学习新颖的库用法。这些结果引出了若干开放性问题:库用法的演示是否必要,更小(且更开放)的模型是否也具备此类能力等。在本工作中,我们采取更广泛的方法,系统性地评估了多种LLMs在三个反映不同领域专业化程度的场景中的表现,以理解它们基于上下文中定义的库生成代码的能力与局限。我们的结果表明,即便是较小的开源LLMs(如Llama-2和StarCoder)也展示出对基于上下文呈现的规范的新颖代码库的熟练理解。我们的发现进一步揭示,LLMs在学习新颖库模块方面表现出令人惊讶的高熟练度,即使仅提供函数的自然语言描述或原始代码实现(这些通常比演示更易获得)也是如此。总体而言,我们的结果为在更具适应性和动态性的编码环境中利用LLMs铺平了道路。

英文摘要:

Contemporary Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. A particularly promising area is their ability to interpret code modules from unfamiliar libraries for solving user-instructed tasks. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations. These results raise several open questions: whether demonstrations of library usage is required, whether smaller (and more open) models also possess such capabilities, etc. In this work, we take a broader approach by systematically evaluating a diverse array of LLMs across three scenarios reflecting varying levels of domain specialization to understand their abilities and limitations in generating code based on libraries defined in-context. Our results show that even smaller open-source LLMs like Llama-2 and StarCoder demonstrate an adept understanding of novel code libraries based on specification presented in-context. Our findings further reveal that LLMs exhibit a surprisingly high proficiency in learning novel library modules even when provided with just natural language descriptions or raw code implementations of the functions, which are often cheaper to obtain than demonstrations. Overall, our results pave the way for harnessing LLMs in more adaptable and dynamic coding environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑