CARL-GT：评估大型语言模型的因果推理能力
CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models
AI总结：
研究提出CARL-GT基准，利用图和表格数据从因果图推理、知识发现和决策制定评估LLM的因果推理能力，发现LLM在该能力上仍较弱，且不同类别任务间表现相关性更强。
摘要：
因果推理能力对于大型语言模型（LLMs）在教育、医疗等广泛领域的应用至关重要。但目前仍缺乏能更好理解这些能力的基准。现有的LLM基准主要基于对话任务、学术数学测试和编程测试。这些基准在规则化良好的环境中评估LLM，但在评估解决现实世界问题的技能和能力方面存在局限。在本研究中，我们提供了一个名为CARL-GT的基准，它利用图和表格数据评估大型语言模型的因果推理能力。该基准包含多样化的任务，从因果图推理、知识发现和决策制定方面对LLM进行评估。此外，我们还为这些任务开发了有效的零样本学习提示。在实验中，我们利用该基准评估开源LLM，并对LLM的因果推理能力进行了详细比较。我们发现LLM在因果推理方面仍然较弱，特别是在使用表格数据发现新见解时。此外，我们通过分析LLM的表现，研究并讨论了不同基准任务之间的关系。实验结果表明，LLM在不同任务上具有不同的优势，并且它们在不同类别任务（即因果图推理、知识发现和决策制定）上的表现，比同类别任务之间的表现显示出更强的相关性。
英文摘要：
Causal reasoning capabilities are essential for large language models (LLMs) in a wide range of applications, such as education and healthcare. But there is still a lack of benchmarks for a better understanding of such capabilities. Current LLM benchmarks are mainly based on conversational tasks, academic math tests, and coding tests. Such benchmarks evaluate LLMs in well-regularized settings, but they are limited in assessing the skills and abilities to solve real-world problems. In this work, we provide a benchmark, named by CARL-GT, which evaluates CAusal Reasoning capabilities of large Language models using Graphs and Tabular data. The benchmark has a diverse range of tasks for evaluating LLMs from causal graph reasoning, knowledge discovery, and decision-making aspects. In addition, effective zero-shot learning prompts are developed for the tasks. In our experiments, we leverage the benchmark for evaluating open-source LLMs and provide a detailed comparison of LLMs for causal reasoning abilities. We found that LLMs are still weak in casual reasoning, especially with tabular data to discover new insights. Furthermore, we investigate and discuss the relationships of different benchmark tasks by analyzing the performance of LLMs. The experimental results show that LLMs have different strength over different tasks and that their performance on tasks in different categories, i.e., causal graph reasoning, knowledge discovery, and decision-making, shows stronger correlation than tasks in the same category.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
