Large language models (LLMs) are commonly evaluated on tasks that test their knowledge or reasoning abilities. In this paper, we explore a different type of evaluation: whether an LLM can predict aspects of its own responses. Since LLMs lack the ability to execute themselves, we introduce the Self-Execution Benchmark, which measures a model's ability to anticipate properties of its output, such as whether a question will be difficult for it, whether it will refuse to answer, or what kinds of associations it is likely to produce. Our experiments show that models generally perform poorly on this benchmark, and that increased model size or capability does not consistently lead to better performance. These results suggest a fundamental limitation in how LLMs represent and reason about their own behavior.
Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
Xi’an Jiaotong University（西安交通大学）；Institute of AI for Industries, Chinese Academy of Sciences（中国科学院人工智能产业研究院）；The University of Tokyo（东京大学）；University of Alberta（阿尔伯塔大学）
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
University of Cambridge（剑桥大学）；Institute for AI, University of Stuttgart（斯图加特大学人工智能研究所）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；University of Southampton（南安普顿大学）；Tübingen AI Center（图宾根人工智能中心）
CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
School of Computer Science, Peking University（北京大学计算机科学系）；Tongyi Lab, Alibaba Group（阿里集团通义实验室）；School of Computer Science, Wuhan University（武汉大学计算机科学系）
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Alibaba Group（阿里巴巴集团）；TU Wien（维也纳技术大学）；University of Chinese Academy of Sciences（中国科学院大学）
GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis
School of Geosciences and Info-Physics, Central South University（中南大学地球科学与信息物理学院）；School of Earth Sciences and Spatial Information Engineering, Hunan University of Science and Technology（湖南科技大学地球科学与空间信息工程学院）
How can we test whether state-of-the-art generative models, such as Blender and GPT-3, are good AI teachers, capable of replying to a student in an educational dialogue? Designing an AI teacher test is challenging: although evaluation methods are much-needed, there is no off-the-shelf solution to measuring pedagogical ability. This paper reports on a first attempt at an AI teacher test. We built a solution around the insight that you can run conversational agents in parallel to human teachers in real-world dialogues, simulate how different agents would respond to a student, and compare these c…
