Large language models (LLMs) demonstrate great potential for problems with implicit graphical structures, while recent works seek to enhance the graph reasoning capabilities of LLMs through specialized instruction tuning. The resulting 'graph LLMs' are evaluated with in-distribution settings only, thus it remains underexplored whether LLMs are learning generalizable graph reasoning skills or merely memorizing patterns in the synthetic training data. To this end, we propose the NLGift benchmark, an evaluation suite of LLM graph reasoning generalization: whether LLMs could go beyond semantic, numeric, structural, reasoning patterns in the synthetic training data and improve utility on real-world graph-based tasks. Extensive experiments with two LLMs across four graph reasoning tasks demonstrate that while generalization on simple patterns (semantic, numeric) is somewhat satisfactory, LLMs struggle to generalize across reasoning and real-world patterns, casting doubt on the benefit of synthetic graph tuning for real-world tasks with underlying network structures. We explore three strategies to improve LLM graph reasoning generalization, and we find that while post-training alignment is most promising for real-world tasks, empowering LLM graph reasoning to go beyond pattern memorization remains an open research question.
KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
Huawei Poisson Laboratory(华为泊松实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Hong Kong University of Science and Technology(香港科技大学)
Reason to Rote: Rethinking Memorization in Reasoning
Department of ICS, Utrecht University(信息科学与计算系,乌特勒支大学);MaiNLP, Center for Information and Language Processing, LMU Munich(MaiNLP,信息与语言处理中心,慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Saarland Informatics Campus, Saarland University(萨尔兰州信息科学校区,萨尔兰州大学)
The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction
New York University(纽约大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);McGill University(麦吉尔大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Toronto(多伦多大学);Vector Institute(矢量研究所)
Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学);SynthLabs;Carnegie Mellon University(卡内基梅隆大学)
LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);London School of Economics(伦敦经济学院);BUPT(北京邮电大学);Harbin Institute of Technology(哈尔滨工业大学);SUTD(新加坡科技设计大学);University of Oxford(牛津大学)
Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing
School of Information and Communication Technology, Griffith University(格里菲斯大学信息与通信技术学院);Department of Data Science and AI, Monash University(莫纳什大学数据科学与人工智能系);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Department of Computer Science, Emory University(埃默里大学计算机科学系)
ReverseMath: Answer Inversion for Scalable and Verifiable Mathematical Problem Generation
Center for Information and Language Processing(信息与语言处理中心);LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心 (MCML));Saarland University(萨尔兰州大学)
Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS University of Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院,中国科学院大学)
Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦大学玛丽女王学院电子工程与计算机科学学院);The Alan Turing Institute(艾伦·图灵研究所)
Quantifying Generalization Complexity for Large Language Models
Harvard University(哈佛大学);Massachusetts Institute of Technology(麻省理工学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Meta;University of Chicago(芝加哥大学)