大型语言模型(LLM)智能体如今可在数据集上运行因果发现算法并输出结果。但它们是否能完成更多任务,比如注意到算法出错的地方并通过进一步测试修复,从当前基准中难以判断。多数基准会给变量赋予有意义的名称,仅看到图宾根因果对名称的LLM就能正确判断99%的方向;去掉名称后,数据通常足够干净,选择比PC算法更优的算法可缩小大部分差距。我们推出Heuresis,这是一个匿名模拟数据集的基准,其机制范围从线性高斯到重尾分布。对于每个数据集系列,它区分了更好的现有算法能带来的增益与仅通过更仔细观察数据才能获得的增益,并追踪智能体更改的每条边。每个系列都存在一个差距,我们运行的所有算法、路由器以及两个因果发现基础模型都无法缩小,因此得分高于所选最佳算法必须来自对数据的利用。我们用智能体框架和三个LLM实例化该基准,分别是GPT-6.1 Sol、Claude Opus 5.5和Qwen 3.8 Flash。GPT-6.1 Sol在所有六个系列上都超过了所选最佳算法,在最难的系列上相邻F1值高出14%,Claude Opus 5.5在该系列上高出7.4%。在两个最难的系列上,这两个智能体恢复的真实边中,有三分之一未出现在任何算法的输出中。代码和数据将公开发布,以促进未来研究。
英文摘要:
LLM agents can now run causal discovery algorithms on a dataset and report what comes out. Whether they do more than that, such as noticing where an algorithm went wrong and fixing it with further tests, is hard to tell from current benchmarks. Most of them give the variables meaningful names, and an LLM shown only the names of the Tübingen cause-effect pairs already gets 99% of the directions right. With the names removed, the data are usually clean enough that picking a better algorithm than PC closes most of the gap. We introduce Heuresis, a benchmark of anonymous simulated datasets whose mechanisms range from linear Gaussian to heavy-tailed. For each family of datasets it separates what a better existing algorithm can gain from what only a closer look at the data can gain, and it traces every edge an agent changes. Every family leaves a gap that none of the algorithms, the router and the two causal-discovery foundation models we ran can close, so a score above the best chosen algorithm has to come from using the data. We instantiate the benchmark with an agent harness and three LLMs, GPT-6.1 Sol, Claude Opus 5.5 and Qwen 3.8 Flash. GPT-6.1 Sol surpasses the best chosen algorithm on all six families, by 14% adjacency F1 on the most difficult one, and Claude Opus 5.5 does so by 7.4% there. A third of the true edges the two agents restore on the two hardest families appear in no algorithm's output. The code and data will be publicly released to facilitate future work.
Linear-LLM-SCM: Benchmarking LLMs for Coefficient Elicitation in Linear-Gaussian Causal Models
Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI)(德国人工智能研究中心数据科学与应用部门);Dept. of Computer Science, University of Kaiserslautern–Landau (RPTU)(科隆-兰道大学计算机科学系);Digital Health - Machine Learning Research Group, Hasso Plattner Institute for Digital Engineering(哈索·普朗纳研究所数字工程学院数字健康-机器学习研究组);Institute of Informatics, University of Munich (LMU)(慕尼黑大学信息学院);Hasso Plattner Institute for Digital Health at Mount Sinai, Icahn School of Medicine at Mount Sinai(西奈山医学院哈索·普朗纳研究所数字健康中心);Munich Center for Machine Learning (MCML), Germany(慕尼黑机器学习中心)
Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation
Technical University of Munich(慕尼黑工业大学);Mila - Quebec AI Institute(米拉 - 魁北克人工智能研究所);ServiceNow Research(ServiceNow研究院);Université de Montréal(蒙特利尔大学);Helmholtz AI(亥姆霍兹人工智能研究所)
$\texttt{causalAssembly}$: Generating Realistic Production Data for Benchmarking Causal Discovery
Technical University of Munich(慕尼黑工业大学);Robert Bosch GmbH(罗伯特·博世有限公司);University of Applied Sciences Kempten(肯普滕应用科学大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学现代软件工程国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)
KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware
Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom(计算机科学与技术系,剑桥大学,剑桥,英国);Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom(电气与电子工程系,伦敦帝国理工学院,伦敦,英国);School of Informatics, University of Edinburgh, Edinburgh, United Kingdom(信息学院,爱丁堡大学,爱丁堡,英国)
CausalGame: Benchmarking Causal Thinking of LLM Agents in Games
MBZUAI(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);Hong Kong Baptist University(香港浸会大学);University of Oxford(牛津大学);New York University, Abu Dhabi(纽约大学阿布扎比分校)
Physics-Audited Agentic Discovery in Scientific Machine Learning
New York University Abu Dhabi(纽约大学阿布扎比分校);National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校国家超级计算应用中心)
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
Shanghai Institute of AI for Education(上海人工智能教育研究院);School of Computer Science(计算机科学学院);East China Normal University(东华大学);Tencent Inc.(腾讯公司);Shanghai Innovation Institute(上海创新研究院)