CrossCodeEval:一个多样化的多语言跨文件代码补全基准
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
AI总结:
提出跨文件代码补全基准CrossCodeEval,基于多语言真实仓库,通过静态分析构造需跨文件上下文的示例,实验表明现有模型性能受限,可用于评估补全与检索能力。
摘要:
近年来,代码补全模型取得了显著进展,然而当前流行的评估数据集(如HumanEval和MBPP)主要关注单文件内的代码补全任务。这种过度简化的设置不足以代表现实世界的软件开发场景,在该场景中,代码库跨越多个文件并包含众多跨文件依赖,通常需要访问和理解跨文件上下文才能正确完成代码。为填补这一空白,我们提出了CrossCodeEval,一个多样化的多语言代码补全基准,它要求深入的跨文件上下文理解才能准确完成代码。CrossCodeEval基于一组多样化的真实世界、开源、许可宽松的代码库构建,涵盖四种流行的编程语言:Python、Java、TypeScript和C#。为了创建严格需要跨文件上下文才能准确完成的示例,我们提出了一种简单而高效的基于静态分析的方法,用于定位当前文件中跨文件上下文的使用。对最先进的代码语言模型(如CodeGen和StarCoder)进行的大量实验表明,当相关跨文件上下文缺失时,CrossCodeEval极具挑战性,而将这些上下文添加到提示中后,我们看到了明显的改进。然而,尽管有这些改进,即使使用性能最高的模型,性能的顶峰仍未达到,这表明CrossCodeEval还能够评估模型利用广泛上下文进行更好代码补全的能力。最后,我们评测了各种检索跨文件上下文的方法,并表明CrossCodeEval也可用于衡量代码检索器的能力。
英文摘要:
Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly. To fill in this gap, we propose CrossCodeEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file. Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CrossCodeEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle of performance remains notably unattained even with the highest-performing model, indicating that CrossCodeEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CrossCodeEval can also be used to measure the capability of code retrievers.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。