使用变异分析检验大语言模型总结代码的能力
Using Mutation-Analysis to Examine an LLM's Ability to Summarize Code
AI总结：
提出基于变异分析的评估方法，通过注入代码突变并检查摘要是否更新，测试LLM摘要是否反映实际行为；实验表明摘要准确率随复杂度下降，模型常描述意图而非行为。
摘要：
随着开发者越来越依赖LLM生成的代码摘要进行文档、测试和审查，研究这些摘要是否准确反映程序的实际行为变得重要。LLM通常对代码看起来应该做什么（意图）产生自信的描述，而忽略了定义实际行为（行为）的微妙边缘情况或逻辑变化。我们提出了一种基于变异的评估方法，直接测试摘要是否真正匹配代码逻辑。我们的方法生成摘要，向代码注入目标突变，并检查LLM是否更新其摘要以反映新行为。我们通过三个实验验证了该方法，共涉及62个程序的624个突变摘要评估。首先，在12个受控合成程序上，使用324个突变，这些突变在类型（语句、值、决策）和位置（开头、中间、结尾）上有所不同。我们发现摘要准确率随复杂性急剧下降，从单个函数的76.5%降至多线程系统的17.3%，而突变类型和位置的影响较弱。其次，对来自Less Basic Python Problems (LBPP)数据集的50个人工编写程序的150个突变样本进行测试，确认了相同的失败模式持续存在，模型通常描述算法意图而非实际突变行为，摘要准确率为49.3%。此外，虽然GPT-4和GPT-5.2之间的比较显示出显著的性能飞跃（从49.3%到85.3%），并且将突变识别为“错误”的能力有所提高，但两个模型在区分实现细节与标准算法模式方面仍然存在困难。这项工作将变异分析确立为一种系统方法，用于评估LLM生成的摘要是否反映程序行为而非表面文本模式。
英文摘要：
As developers increasingly rely on LLM-generated code summaries for documentation, testing, and review, it is important to study whether these summaries accurately reflect what the program actually does. LLMs often produce confident descriptions of what the code looks like it should do (intent), while missing subtle edge cases or logic changes that define what it actually does (behavior). We present a mutation-based evaluation methodology that directly tests whether a summary truly matches the code's logic. Our approach generates a summary, injects a targeted mutation into the code, and checks if the LLM updates its summary to reflect the new behavior. We validate it through three experiments totalling 624 mutation-summary evaluations across 62 programs. First, on 12 controlled synthetic programs with 324 mutations varying in type (statement, value, decision) and location (beginning, middle, end). We find that summary accuracy decreases sharply with complexity from 76.5% for single functions to 17.3% for multi-threaded systems, while mutation type and location exhibit weaker effects. Second, testing 150 mutated samples on 50 human-written programs from the Less Basic Python Problems (LBPP) dataset confirms the same failure patterns persist as models often describe algorithmic intent rather than actual mutated behavior with a summary accuracy rate of 49.3%. Furthermore, while a comparison between GPT-4 and GPT-5.2 shows a substantial performance leap (from 49.3% to 85.3%) and an improved ability to identify mutations as "bugs", both models continue to struggle with distinguishing implementation details from standard algorithmic patterns. This work establishes mutation analysis as a systematic approach for assessing whether LLM-generated summaries reflect program behavior rather than superficial textual patterns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
