SpecEval：通过程序规约评估大语言模型的代码理解能力
SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specifications
AI总结：
针对现有代码理解评估框架覆盖执行轨迹有限、无法全面评估模型程序理解能力的问题，提出基于形式化程序规约的黑盒评估框架SpecEval，设计4项分层任务并开展反事实分析，实验发现6款先进大语言模型在规约相关任务上表现不佳且对语义保持扰动敏感。
摘要：
大语言模型在自动化软件工程领域已取得令人瞩目的表现。目前已有大量工作从多个维度评估代码大语言模型（code LLM）的能力，相关基准与评估框架数量不断增加。除了最受关注的代码生成能力外，代码理解能力也正获得越来越多的重视。然而，现有评估大语言模型代码理解能力的研究存在诸多局限。CRUXEval、REval等评估框架通常聚焦于特定输入用例下的代码推理任务，导致覆盖的执行轨迹范围有限，所考察的代码语义存在缺失，无法评估大语言模型对目标程序的全面理解能力。为解决这些问题，我们提出SpecEval——一种通过程序规约评估大语言模型代码理解能力的新型黑盒评估框架。受“规约可全面阐述所有可能执行轨迹下的程序行为”这一思路启发，我们采用形式化程序规约表示程序语义，开展全面评估。具体而言，我们精心设计了4项与规约相关的任务，从基础到进阶层级评估大语言模型的能力。我们进一步开展反事实分析，研究大语言模型在语义保持扰动下的性能差异。我们针对6款当前最先进的大语言模型开展了系统实验。大量实验结果显示，大语言模型在规约相关任务上的表现未达预期，揭示出现有大语言模型在利用形式化规约阐述程序语义方面存在局限。反事实分析还表明，大语言模型对语义保持扰动具有敏感性。
英文摘要：
Large Language models have achieved impressive performance in automated software engineering. Extensive efforts have been made to evaluate the abilities of code LLMs in various aspects, with an increasing number of benchmarks and evaluation frameworks proposed. Apart from the most sought-after capability of code generation, the capability of code comprehension is being granted growing attention. Nevertheless, existing works assessing the code comprehension capability of LLMs exhibit varied limitations. Evaluation frameworks like CRUXEval and REval usually focus on code reasoning tasks over a certain input case, leading to a limited range of execution traces covered, resulting in a loss in code semantics examined and the inability to assess the comprehensive understanding of LLMs concerning the target program. To tackle these challenges, we propose SpecEval, a novel black-box evaluation framework to evaluate code comprehension in LLMs via program specifications. Inspired by the idea that specifications can act as a comprehensive articulation of program behaviors concerning all possible execution traces, we employ formalized program specifications to represent program semantics and perform comprehensive evaluations. In particular, four specification-related tasks are designed meticulously to assess the capability of LLMs from basic to advanced levels. Counterfactual analysis is further conducted to study the performance variance of LLMs under semantics-preserving perturbations. Systematic experiments are conducted on six state-of-the-art LLMs. Extensive experimental results present a below-satisfactory performance of LLMs on specification-related tasks, revealing the limitations of existing LLMs in terms of articulating program semantics with formal specifications. Counterfactual analysis also reveals the sensitivity of LLMs towards semantic-preserving perturbations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
