arXiv 2510.034152025-10cs.PLcs.AIcs.CLcs.SE

LLMs 依赖先验而非编程语言语义

LLMs Lean on Priors, Not Programming Language Semantics

The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Cisco Research（思科研究）

Aditya Thimmaiah, Jiyang Zhang, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric

备注：: Accepted at ICML 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

通过 PLSemanticsBench 基准测试，发现前沿大语言模型在程序执行任务中依赖预训练统计规律而非形式语义规则，语义变异和结构复杂度导致准确率大幅下降。

摘要：

近期工作探究大语言模型（LLMs）是否基于显式规则而非预训练统计规律进行推理。程序执行提供了一个典型实例：形式语义通过符号转换规则定义行为，这些规则在分布偏移下可被系统性改变。我们研究 LLMs 能否通过程序执行基于形式语义进行推理，并引入 PLSemanticsBench，将轻量级 C 程序与两种语义系统（小步操作语义和 K 语义）配对，探测四种能力：组合规则得到最终状态、状态未变时选择规则、在长轨迹上维持这种条件推理、以及在新语义下遵循提供的规则。为解耦语义推理与语法熟悉度，我们重新定义熟悉运算符以引发符号-含义冲突，并引入仅通过提供规则定义的新符号，同时在人类编写、LLM 翻译和模糊生成的分割上以递增的结构复杂度进行压力测试。在 11 个前沿 LLM 上，标准语义下的最终状态准确率（高达 90%）在语义变异和结构复杂度增加时急剧下降，降幅达 40-60 个百分点。仅少数模型实现了非零的长程条件推理准确率，即使最佳系统也仅达到 35%。这些结果表明，当代 LLMs 往往依赖预训练的词汇关联，而非系统地基于提供的正式规则进行推理。PLSemanticsBench 公开于 https://EngineeringSoftware.github.io/PLSemanticsBench。

英文摘要：

Recent work asks whether large language models (LLMs) condition their reasoning on explicit rules rather than statistical regularities from pretraining. Program execution provides a canonical instance: formal semantics define behavior through symbolic transition rules that can be systematically altered under distribution shift. We investigate whether LLMs can condition their reasoning on formal semantics through program execution and introduce PLSemanticsBench, pairing featherweight C programs with two semantic systems -- small-step operational semantics and K semantics -- and probing four capabilities: composing rules for final states, selecting rules when state is unmutated, sustaining such conditioning over long traces, and following supplied rules under novel semantics. To decouple semantic reasoning from syntactic familiarity, we redefine familiar operators to induce symbol-meaning conflict and introduce novel symbols defined only through the supplied rules, and stress-test models on Human-Written, LLM-Translated, and Fuzzer-Generated splits with increasing structural complexity. Across 11 frontier LLMs, strong final-state accuracy under standard semantics (up to 90%) drops sharply -- by as much as 40--60% points -- under semantic mutations and increasing structural complexity. Only a handful of models achieve non-zero long-horizon conditioning accuracy, and even the best systems reach just 35%. Together, these results suggest that contemporary LLMs often rely on pretrained lexical associations rather than systematically conditioning on supplied formal rules. PLSemanticsBench is publicly available at https://EngineeringSoftware.github.io/PLSemanticsBench.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文