arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.15992cs.CL
收藏

LLM图推理能否超越模式记忆实现泛化?

Can LLM Graph Reasoning Generalize beyond Pattern Memorization?

University of Washington(华盛顿大学);Xi’an Jiaotong University(西安交通大学);University of Notre Dame(圣母大学);Tsinghua University(清华大学)

Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, Yulia Tsvetkov

AI总结:

该研究提出NLGift基准评估LLM图推理泛化能力,发现现有模型在简单模式上泛化尚可,但在推理和现实世界模式上表现不佳,并探索了三种改进策略,指出超越模式记忆仍是开放问题。

摘要:

大型语言模型(LLMs)在具有隐式图结构的问题上展现出巨大潜力,而近期的研究试图通过专门的指令微调来增强LLMs的图推理能力。由此产生的“图LLMs”仅在分布内设置下进行评估,因此LLMs究竟是学习了可泛化的图推理技能,还是仅仅记住了合成训练数据中的模式,仍有待探究。为此,我们提出NLGift基准,这是一个评估LLM图推理泛化能力的测试套件:即LLMs能否超越合成训练数据中的语义、数值、结构和推理模式,并提升在现实世界基于图的任务上的实用性。在两个LLMs和四个图推理任务上的大量实验表明,尽管在简单模式(语义、数值)上的泛化在一定程度上令人满意,但LLMs在推理和现实世界模式上的泛化仍显吃力,这让人对合成图微调在具有底层网络结构的现实世界任务中的益处产生怀疑。我们探索了三种策略来改善LLM图推理泛化,并发现虽然训练后对齐在现实世界任务中最具前景,但赋能LLM图推理以超越模式记忆仍是一个开放的研究问题。

英文摘要:

Large language models (LLMs) demonstrate great potential for problems with implicit graphical structures, while recent works seek to enhance the graph reasoning capabilities of LLMs through specialized instruction tuning. The resulting 'graph LLMs' are evaluated with in-distribution settings only, thus it remains underexplored whether LLMs are learning generalizable graph reasoning skills or merely memorizing patterns in the synthetic training data. To this end, we propose the NLGift benchmark, an evaluation suite of LLM graph reasoning generalization: whether LLMs could go beyond semantic, numeric, structural, reasoning patterns in the synthetic training data and improve utility on real-world graph-based tasks. Extensive experiments with two LLMs across four graph reasoning tasks demonstrate that while generalization on simple patterns (semantic, numeric) is somewhat satisfactory, LLMs struggle to generalize across reasoning and real-world patterns, casting doubt on the benefit of synthetic graph tuning for real-world tasks with underlying network structures. We explore three strategies to improve LLM graph reasoning generalization, and we find that while post-training alignment is most promising for real-world tasks, empowering LLM graph reasoning to go beyond pattern memorization remains an open research question.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑