A promising research direction in enabling LLMs to generate consistently correct code involves addressing their inability to properly estimate program execution, particularly for code they generate. In this work, we demonstrate that Code LLMs can be trained to simulate program execution in a step-by-step manner and that this capability can be leveraged to improve competitive programming performance. Our approach combines supervised fine-tuning on natural language execution traces, textual explanations grounded in true execution, with reinforcement learning using verifiable rewards. We introduce two complementary objectives: output prediction given code and inputs, and solving competitive programming tasks with either ground-truth or self-predicted execution feedback. These objectives enable models to perform self-verification over multiple candidate solutions, and iterative self-fixing by simulating test execution. Across multiple competitive programming benchmarks, our method yields consistent improvements over standard reasoning approaches. We further present ablations and analysis to elucidate the role of execution simulation and its limitations.
Gistify! Codebase-Level Understanding via Runtime Execution
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Microsoft Research(微软研究院);Cornell University(康奈尔大学);University of California San Diego(加州大学圣地亚哥分校);University of Texas at Austin(德克萨斯大学奥斯汀分校)
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Monash University(墨尔本大学);CSIRO’s Data61(CSIRO的数据61);Purdue University(普渡大学);Independent(独立);HKUST (Guangzhou)(香港科技大学(广州));UCSD(加州大学圣地亚哥分校);UVA(弗吉尼亚大学);CNRS, France(法国国家科学研究中心);IBM;Cisco(思科);Comenius University in Bratislava(布拉提斯拉瓦康门纽斯大学);University of Notre Dame(Notre Dame大学);Uber;Tano Labs(Tano实验室);NUS(国立大学新加坡);Institute of Automation, CAS(中国科学院自动化研究所);Tencent AI Lab(腾讯AI实验室);University of Washington(华盛顿大学);Nevsky Collective(Nevsky集体);ETH Zurich(苏黎世联邦理工学院);Detomo Inc(Detomo公司);University of Oxford(牛津大学);UIUC(伊利诺伊大学香槟分校);Google(谷歌);NVIDIA;Singapore Management University(新加坡管理学院);ServiceNow Research(ServiceNow研究);Hugging Face
Leveraging Multimodal Self-Consistency Reasoning in Coding Motivational Interviewing for Alcohol Use Reduction
Department of Computer Science, University of Memphis(密苏里大学计算机科学系);Veterans Affairs Health Care System(退伍军人事务医疗系统);Department of Psychiatry and Behavioral Sciences, University of California San Francisco(旧金山大学精神病学与行为科学系);Department of Psychology, University of Memphis(密苏里大学心理学系);Department of Psychology, Washington State University Vancouver(华盛顿州立大学温哥华分校心理学系)