CLEAR：语言模型真的能理解因果图吗？
CLEAR: Can Language Models Really Understand Causal Graphs?
AI总结：
本研究率先探究语言模型的因果图理解能力，构建多维度评估框架与含20项任务的三级复杂度基准CLEAR，实验发现6个主流语言模型仅具备初步理解能力，仍有较大提升空间。
摘要：
因果推理是人类解读世界的核心能力之一。为了对因果关系进行建模与推理，因果图提供了一种简洁而有效的解决方案。随着语言模型取得令人瞩目的进展，一个关键问题随之而来：它们真的能理解因果图吗？为此，我们率先开展了语言模型对因果图理解能力的研究。具体而言，我们构建了一个定义因果图理解的框架，通过源自哲学、心理学等多学科的四项实用标准评估语言模型的行为。随后，我们开发了新型基准CLEAR，该基准设定了三个复杂度层级，涵盖分布在这些层级中的20项基于因果图的任务。最后，基于我们的框架和基准，我们对6个主流语言模型开展了大量实验，并总结出五项实证发现。结果表明，尽管语言模型展现出了对因果图的初步理解，但仍存在巨大的提升空间。我们的项目网站为https://github.com/OpenCausaLab/CLEAR。
英文摘要：
Causal reasoning is a cornerstone of how humans interpret the world. To model and reason about causality, causal graphs offer a concise yet effective solution. Given the impressive advancements in language models, a crucial question arises: can they really understand causal graphs? To this end, we pioneer an investigation into language models' understanding of causal graphs. Specifically, we develop a framework to define causal graph understanding, by assessing language models' behaviors through four practical criteria derived from diverse disciplines (e.g., philosophy and psychology). We then develop CLEAR, a novel benchmark that defines three complexity levels and encompasses 20 causal graph-based tasks across these levels. Finally, based on our framework and benchmark, we conduct extensive experiments on six leading language models and summarize five empirical findings. Our results indicate that while language models demonstrate a preliminary understanding of causal graphs, significant potential for improvement remains. Our project website is at https://github.com/OpenCausaLab/CLEAR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
