arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.04556cs.CLcs.AIcs.LG
收藏

语言模型中的组合因果推理评估

Compositional Causal Reasoning Evaluation in Language Models

Harvard University(哈佛大学);Microsoft Research Cambridge(微软研究院剑桥)

Jacqueline R. M. A. Maasch, Alihan Hüyük, Xinnuo Xu, Aditya V. Nori, Javier Gonzalez

AI总结:

本文提出组合因果推理(CCR)统一视角与评估框架,系统测试LLama、Phi和GPT系列模型,发现除o1外错误随因果路径复杂度上升。

摘要:

因果推理和组合推理是人工智能的两大核心追求。衡量这些行为的程度需要有原则的评估方法。我们探索了一种同时考虑这两种行为的统一视角,称为组合因果推理(CCR):即推断因果测度如何组合,以及等价地,因果量如何在图中传播的能力。我们实例化了一个框架,用于针对平均处理效应和必要充分概率系统性评估CCR。作为概念验证,我们展示了对LLama、Phi和GPT系列语言模型的CCR评估。在一道数学应用题上,我们的框架揭示了一系列分类上不同的错误模式。除o1外,所有模型的CCR错误都随因果路径复杂度的增加而增加。

英文摘要:

Causal reasoning and compositional reasoning are two core aspirations in AI. Measuring the extent of these behaviors requires principled evaluation methods. We explore a unified perspective that considers both behaviors simultaneously, termed compositional causal reasoning (CCR): the ability to infer how causal measures compose and, equivalently, how causal quantities propagate through graphs. We instantiate a framework for the systematic evaluation of CCR for the average treatment effect and the probability of necessity and sufficiency. As proof of concept, we demonstrate CCR evaluation for language models in the LLama, Phi, and GPT families. On a math word problem, our framework revealed a range of taxonomically distinct error patterns. CCR errors increased with the complexity of causal paths for all models except o1.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑