arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.22846cs.LG
收藏

智能体因果发现的基准测试与增强

Benchmarking and Enhancing Agentic Causal Discovery

Microsoft(微软公司)

Pingchuan Ma, Rui Ding, Shuai Wang

AI总结:

该研究推出Heuresis基准,测试发现GPT-6.1 Sol等LLM智能体在因果发现任务中优于现有算法,部分恢复的边未被算法覆盖,代码数据将公开。

摘要:

大型语言模型(LLM)智能体如今可在数据集上运行因果发现算法并输出结果。但它们是否能完成更多任务,比如注意到算法出错的地方并通过进一步测试修复,从当前基准中难以判断。多数基准会给变量赋予有意义的名称,仅看到图宾根因果对名称的LLM就能正确判断99%的方向;去掉名称后,数据通常足够干净,选择比PC算法更优的算法可缩小大部分差距。我们推出Heuresis,这是一个匿名模拟数据集的基准,其机制范围从线性高斯到重尾分布。对于每个数据集系列,它区分了更好的现有算法能带来的增益与仅通过更仔细观察数据才能获得的增益,并追踪智能体更改的每条边。每个系列都存在一个差距,我们运行的所有算法、路由器以及两个因果发现基础模型都无法缩小,因此得分高于所选最佳算法必须来自对数据的利用。我们用智能体框架和三个LLM实例化该基准,分别是GPT-6.1 Sol、Claude Opus 5.5和Qwen 3.8 Flash。GPT-6.1 Sol在所有六个系列上都超过了所选最佳算法,在最难的系列上相邻F1值高出14%,Claude Opus 5.5在该系列上高出7.4%。在两个最难的系列上,这两个智能体恢复的真实边中,有三分之一未出现在任何算法的输出中。代码和数据将公开发布,以促进未来研究。

英文摘要:

LLM agents can now run causal discovery algorithms on a dataset and report what comes out. Whether they do more than that, such as noticing where an algorithm went wrong and fixing it with further tests, is hard to tell from current benchmarks. Most of them give the variables meaningful names, and an LLM shown only the names of the Tübingen cause-effect pairs already gets 99% of the directions right. With the names removed, the data are usually clean enough that picking a better algorithm than PC closes most of the gap. We introduce Heuresis, a benchmark of anonymous simulated datasets whose mechanisms range from linear Gaussian to heavy-tailed. For each family of datasets it separates what a better existing algorithm can gain from what only a closer look at the data can gain, and it traces every edge an agent changes. Every family leaves a gap that none of the algorithms, the router and the two causal-discovery foundation models we ran can close, so a score above the best chosen algorithm has to come from using the data. We instantiate the benchmark with an agent harness and three LLMs, GPT-6.1 Sol, Claude Opus 5.5 and Qwen 3.8 Flash. GPT-6.1 Sol surpasses the best chosen algorithm on all six families, by 14% adjacency F1 on the most difficult one, and Claude Opus 5.5 does so by 7.4% there. A third of the true edges the two agents restore on the two hardest families appear in no algorithm's output. The code and data will be publicly released to facilitate future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑