当名称消失:揭示 LLM 对代码的真实理解
When Names Disappear: Revealing What LLMs Actually Understand About Code
AI总结:
本文通过保持语义的代码命名混淆揭示 LLM 对标识符模式的依赖,并发布 ClassEval-Obf 基准以更可靠地评估真实代码理解与泛化能力。
摘要:
大语言模型(LLM)在代码任务上取得了出色结果,但它们如何推导程序含义仍不清楚。我们认为,代码通过两个渠道传递信息:定义形式化行为的结构语义,以及传达意图的、人类可解释的命名。移除命名渠道会严重削弱摘要生成等意图层面任务,此时模型会退化为逐行描述。令人惊讶的是,我们还观察到本应只依赖结构的执行任务出现一致性下降,这表明当前基准奖励的是对命名模式的记忆,而非真正的语义推理。为分离这些影响,我们提出一套保持语义的混淆方法,并证明它们揭示了摘要生成和执行任务中的标识符泄漏。基于这些洞察,我们发布 ClassEval-Obf,这是一个通过混淆增强的基准,在保留行为的同时系统性抑制命名线索。结果表明,ClassEval-Obf 缩小了虚高的性能差距,削弱了记忆捷径,并为评估 LLM 的代码理解和泛化能力提供了更可靠的基础。
英文摘要:
Large Language Models (LLMs) achieve strong results on code tasks, but how they derive program meaning remains unclear. We argue that code communicates through two channels: structural semantics, which define formal behavior, and human-interpretable naming, which conveys intent. Removing the naming channel severely degrades intent-level tasks such as summarization, where models regress to line-by-line descriptions. Surprisingly, we also observe consistent reductions on execution tasks that should depend only on structure, revealing that current benchmarks reward memorization of naming patterns rather than genuine semantic reasoning. To disentangle these effects, we introduce a suite of semantics-preserving obfuscations and show that they expose identifier leakage across both summarization and execution. Building on these insights, we release ClassEval-Obf, an obfuscation-enhanced benchmark that systematically suppresses naming cues while preserving behavior. Our results demonstrate that ClassEval-Obf reduces inflated performance gaps, weakens memorization shortcuts, and provides a more reliable basis for assessing LLMs' code understanding and generalization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。