LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains open. We study LLMs through the lens of robustness of their code understanding using a standard program-output prediction task. Our results reveal a stark divergence in model behavior: while open-source reasoning models (DeepSeek-R1 family) maintain stable, albeit somewhat lower accuracies (38% to 67%) under code transformations & input perturbations, the frontier model GPT-5.2 exhibits significant brittleness. Despite achieving a near-perfect score of 99% on the original, unperturbed CRUXEval benchmark, perturbed inputs trigger accuracy declines between 20% and 24%. In addition, we find that many models perform much worse at predicting behavior on perturbed inputs that raise exceptions, and that prediction performance depends on the kind of exception. We study remedies to address this deficiency in exception prediction, and evaluate the effect of these remedies on the ability to predict non-exception behaviors. Our findings both point to limitations in the way all models understand code, and establish the value of using perturbation to evaluate code models.
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)
Gistify! Codebase-Level Understanding via Runtime Execution
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Microsoft Research(微软研究院);Cornell University(康奈尔大学);University of California San Diego(加州大学圣地亚哥分校);University of Texas at Austin(德克萨斯大学奥斯汀分校)
CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学);The Hong Kong University of Science and Technology(香港科技大学);Huazhong University of Science and Technology(华中科技大学)
DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Weiyang College, Tsinghua University(清华大学未央学院);Shanghai Qi Zhi Institute(上海期智研究院)
Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);China Mobile Online Services Co., Ltd.(中移在线服务有限公司)
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Shenzhen Institute of Advanced Technology Chinese Academy of Sciences(深圳先进技术研究院中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Dalian University of Technology(大连理工大学);UNSW Sydney(悉尼大学);Shenzhen University of Advanced Technology(深圳先进技术大学)
Integrating Symbolic Execution with LLMs for Automated Generation of Program Specifications
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);UCAS(中国科学院大学);Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学计算机科学与技术学院与人工智能学院);School of Computer Science, Peking University(北京大学计算机学院);Beijing Institute of Control Engineering(北京控制工程研究所)
Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity
Columbia University(哥伦比亚大学);University of California, Riverside(加州大学河滨分校)
Detecting semantically similar functions -- a crucial analysis capability with broad real-world security usages including vulnerability detection, malware lineage, and forensics -- requires understanding function behaviors and intentions. This task is challenging as semantically similar functions can be implemented differently, run on different architectures, and compiled with diverse compiler optimizations or obfuscations. Most existing approaches match functions based on syntactic features without understanding the functions' execution semantics. We present Trex, a transfer-learning-based fr…
SpaceTime Programming: Live and Omniscient Exploration of Code and Execution
Université de Rennes, IRISA(雷恩大学,IRISA);Université de Rennes, CNRS, IRISA(雷恩大学,CNRS,IRISA);Université de Rennes, Inria, CNRS, IRISA(雷恩大学,Inria,CNRS,IRISA)
From Brain Models to Executable Digital Twins: Execution Semantics and Neuro-Neuromorphic Systems
International Laboratory on Learning Systems (ILLS)(学习与系统国际实验室);CNRS(法国国家科学研究中心);McGill(麦吉尔大学);ETS(工程技术高等学院);MILA(蒙特利尔人工智能研究院);Université Paris-Saclay(巴黎萨克雷大学)
CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
Hong Kong Baptist University(香港浸会大学);UC Santa Barbara(加州大学圣巴巴拉分校);UC Berkeley(加州大学伯克利分校);Alibaba Group(阿里巴巴集团);The University of Tokyo(东京大学);University of Alberta(阿尔伯塔大学)