发表机构
Nanjing University（南京大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过Face/Off重命名框架揭示LLMs在代码理解中过度依赖词汇线索，即使误导性命名也影响输出，且难以通过干预消除，呼吁平衡词汇与结构语义。
AI 中文摘要
大型语言模型（LLMs）的最新进展使其被广泛用于代码相关任务。在自然产生的代码中，标识符名称在统计上具有信息性，但其信息并不总是可靠的。我们研究了当重命名保留程序结构时，当前LLMs是否对词汇线索赋予了不成比例的权重。我们引入了Face/Off，一个保持语义的标识符重命名框架，并在多个模型和代码理解任务上评估了渐进式命名条件。在该框架内，词汇过度强调在评估的模型和主要任务中普遍存在：随着标识符信息被移除或变得误导，性能通常会下降，输出往往被导向误导性名称所暗示的含义。这种模式在代表性的提示和微调干预下持续存在，表明词汇过度强调是一个根深蒂固的问题。一个类型推断对照确认了一个边界：当答案可以在没有目标名称的情况下局部恢复时，命名效应较小。这些结果并不意味着标识符无用；相反，它们揭示了当前LLMs在平衡词汇线索与程序结构方面的系统性脆弱性。我们的发现促使评估和建模方法在保留自然代码规律益处的同时，使结论基于准确、形式化的代码语义。
英文摘要
Recent advances in large language models (LLMs) have made them widely used for code-related tasks. Identifier names are statistically informative in naturally occurring code, but their information is not always reliable. We investigate whether current LLMs assign disproportionate weight to lexical cues when renaming preserves program structure. We introduce Face/Off, a semantics-preserving identifier-renaming framework, and evaluate progressive naming conditions across multiple models and code-comprehension tasks. Within this framework, lexical overemphasis is pervasive across the evaluated models and primary tasks: performance generally decreases as identifier information is removed or made misleading, and outputs are often directed toward the meanings suggested by misleading names. The pattern persists under representative prompt- and fine-tuning-based interventions, suggesting that lexical overemphasis is an entrenched problem. A type-inference control confirms a boundary: naming effects are smaller when the answer is locally recoverable without the target name. These results do not imply that identifiers are unhelpful; rather, they reveal a systematic vulnerability in how current LLMs balance lexical cues against program structure. Our findings motivate evaluations and modeling methods that preserve the benefits of natural code regularities while keeping conclusions grounded in accurate, formalized code semantics.
Comments27 pages, 9 figures, 12 tables. Submitted to an ACM journal in September 2025. Preprint; manuscript under review. Corresponding author: Ming Li
