arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.13383cs.SEcs.AI
收藏

无需执行即可预测代码覆盖率

Predicting Code Coverage without Execution

Microsoft(微软)

Michele Tufano, Shubham Chandel, Anisha Agarwal, Neel Sundaresan, Colin Clement

AI总结:

本研究提出面向大语言模型的代码覆盖率预测基准任务,构建发布COVERAGEEVAL数据集,评测四款前沿代码大模型的表现,指出代码覆盖率对提升LLM软件工程任务能力的价值。

摘要:

代码覆盖率是一种广泛使用的指标,用于量化程序元素(如语句或分支)在测试过程中被执行的程度。计算代码覆盖率需要大量资源,需要进行代码构建和执行,还要承担插桩带来的额外开销。此外,计算任意代码片段的覆盖率都需要完整的程序上下文。利用机器学习来分摊这一高成本过程,仅需源代码上下文即可降低代码覆盖率的计算成本,并且代码覆盖率预测任务可以成为评判模型代码理解能力的新型基准。我们提出了一项名为“面向大语言模型(LLM)的代码覆盖率预测”的新型基准任务。我们对该任务进行了形式化定义,通过判断某个方法的哪些代码行会被给定测试用例及输入执行,来评估大语言模型对代码执行的理解能力。我们通过执行HumanEval数据集中的测试和代码、收集代码覆盖率信息,整理并发布了一个名为COVERAGEEVAL的数据集。我们报告了四款用于代码相关任务的前沿大语言模型在代码覆盖率预测任务上的表现,包括OpenAI的GPT-4和GPT-3.5-Turbo、谷歌的BARD以及Anthropic的Claude。最后,我们认为代码覆盖率作为一种指标和预训练数据源,对于提升大语言模型在软件工程任务上的整体表现具有价值。

英文摘要:

Code coverage is a widely used metric for quantifying the extent to which program elements, such as statements or branches, are executed during testing. Calculating code coverage is resource-intensive, requiring code building and execution with additional overhead for the instrumentation. Furthermore, computing coverage of any snippet of code requires the whole program context. Using Machine Learning to amortize this expensive process could lower the cost of code coverage by requiring only the source code context, and the task of code coverage prediction can be a novel benchmark for judging the ability of models to understand code. We propose a novel benchmark task called Code Coverage Prediction for Large Language Models (LLMs). We formalize this task to evaluate the capability of LLMs in understanding code execution by determining which lines of a method are executed by a given test case and inputs. We curate and release a dataset we call COVERAGEEVAL by executing tests and code from the HumanEval dataset and collecting code coverage information. We report the performance of four state-of-the-art LLMs used for code-related tasks, including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's Claude, on the Code Coverage Prediction task. Finally, we argue that code coverage as a metric and pre-training data source are valuable for overall LLM performance on software engineering tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑