Large language models (LLMs) have revolutionized many areas (e.g. natural language processing, software engineering, etc.) by achieving state-of-the-art performance on extensive downstream tasks. Aiming to achieve robust and general artificial intelligence, there has been a surge of interest in investigating the reasoning ability of the LLMs. Whereas the textual and numerical reasoning benchmarks adopted by previous works are rather shallow and simple, it is hard to conclude that the LLMs possess strong reasoning ability by merely achieving positive results on these benchmarks. Recent efforts have demonstrated that the LLMs are poor at solving sequential decision-making problems that require common-sense planning by evaluating their performance on the reinforcement learning benchmarks. In this work, we conduct an in-depth assessment of several state-of-the-art LLMs' reasoning ability based on the inductive logic programming (ILP) benchmark, which is broadly recognized as a representative and challenging measurement for evaluating logic program induction/synthesis systems as it requires inducing strict cause-effect logic to achieve robust deduction on independent and identically distributed (IID) and out-of-distribution (OOD) test samples. Our evaluations illustrate that compared with the neural program induction systems which are much smaller in model size, the state-of-the-art LLMs are much poorer in terms of reasoning ability by achieving much lower performance and generalization using either natural language prompting or truth-value matrix prompting.
LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；School of Information Science and Engineering, Hebei University of Science and Technology（河北科技大学信息科学与工程学院）
Technical University of Denmark（丹麦技术大学）；Tradeshift；DeepMind
本文关注的是学习解决需要一系列相互依赖的关系推理步骤的任务，比如回答关于对象之间关系的复杂问题，或者解决解决方案中较小元素相互约束的难题。我们介绍了经常性的关系网络，一个通用的模块，操作对象的图形表示。作为Santoro et al. [2017]关系网络的推广，它可以增强任何神经网络模型，使其具有进行多步关系推理的能力。我们使用递归关系网络在bAbI文本问答数据集上实现了最先进的结果，始终如一地解决了20/20任务。由于bAbI从关系推理的角度来看并不是特别具有挑战性，因此我们引入了Pretty-CLEVR，这是一种新的关系推理诊断数据集。在Pretty-CLEVR设置中，我们可以改变问题，以控制获得答案所需的关系推理步骤的数量。使用Pretty-CLEVR，我们探讨了多层感知器，关系和递归关系网络的局限性。最后，我们展示了循环关系网络如何从有监督的训练数据中学习解决数独难题，这是一项具有挑战性的任务，需要64个以上的关系推理步骤。我们通过解决96.6%最难的数独谜题，在可比方法中取得了最先进的结果。
IRRGN: An Implicit Relational Reasoning Graph Network for Multi-turn Response Selection
College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；yz-intelligence Inc（元智智能公司）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons
School of Professional Education and Executive Development（专业教育与执行发展学院）；The Hong Kong Polytechnic University（香港理工大学）；School of Mathematics and Statistics（数学与统计学学院）；Beijing Institute of Technology（北京理工大学）；Department of Mathematics（数学系）；The University of Hong Kong（香港大学）
RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
UCLA Center for Vision, Cognition, Learning and Autonomy（加州大学洛杉矶分校视觉、认知、学习与自主性中心）；International Center for AI and Robot Autonomy (CARA)（人工智能与机器人自主性国际中心（CARA））
Reasoning-Oriented and Analogy-Based Methods for Locating and Editing in Zero-Shot Event-Relational Reasoning
School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；School of Computer Science and Artificial Intelligence, Liaoning Normal University（辽宁师范大学计算机与人工智能学院）
本文提出 ROLE 和 ABLE 两种定位与编辑方法，分别通过定位语言模型关键模块和利用任务间类比关系，高效提升零样本事件关系推理的可解释性与性能。
Compositional Language Understanding with Text-based Relational Reasoning
Mila（米拉研究所）；McGill University（麦吉尔大学）；Université de Montréal（蒙特利尔大学）；Facebook AI Research (FAIR)（脸书人工智能研究院）；Samsung Advanced Institute of Technology (SAIT)（三星综合技术院）
Relational Temporal Graph Reasoning for Dual-task Dialogue Language Understanding
Australian Artificial Intelligence Institute (AAII)（澳大利亚人工智能研究所）；University of Technology Sydney（悉尼科技大学）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）
Temporal Fact Reasoning over Hyper-Relational Knowledge Graphs
LMU Munich（慕尼黑大学）；University of Cambridge（剑桥大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Stuttgart（斯图加特大学）；Technical University of Munich（慕尼黑工业大学）
