arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09529cs.CLcs.AI
收藏

重新审视大型语言模型的图推理能力:翻译、连通性和最短路径的案例研究

Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path

Michigan State University(密歇根州立大学);Microsoft Research(微软研究院)

Xinnan Dai, Qihao Wen, Yifei Shen, Hongzhi Wen, Dongsheng Li, Jiliang Tang, Caihua Shan

AI总结:

本文通过图描述翻译、连通性和最短路径三类任务及知识图谱实测,重新评估LLMs的图推理能力,揭示其难以从文本理解图结构且性能不稳定。

摘要:

大型语言模型(LLMs)已在各类推理任务中取得巨大成功。本研究聚焦于LLMs的图推理能力。尽管理论研究证明LLMs能够处理图推理任务,但实证评估却揭示了大量失败案例。为加深对这一差异的理解,我们重新审视了LLMs在三项基本图任务上的能力:图描述翻译、图连通性和最短路径问题。研究结果表明,LLMs可能无法通过文本描述理解图结构,并且在这三项基本任务上均表现出参差不齐的性能。同时,我们在知识图谱上开展了真实场景调查,并得到了与上述发现一致的观察结果。代码和数据集均已公开。

英文摘要:

Large Language Models (LLMs) have achieved great success in various reasoning tasks. In this work, we focus on the graph reasoning ability of LLMs. Although theoretical studies proved that LLMs are capable of handling graph reasoning tasks, empirical evaluations reveal numerous failures. To deepen our understanding on this discrepancy, we revisit the ability of LLMs on three fundamental graph tasks: graph description translation, graph connectivity, and the shortest-path problem. Our findings suggest that LLMs can fail to understand graph structures through text descriptions and exhibit varying performance for all these three fundamental tasks. Meanwhile, we perform a real-world investigation on knowledge graphs and make consistent observations with our findings. The codes and datasets are available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑