Evaluating Logical Generalization in Graph Neural Networks
Facebook AI Research（Facebook AI研究）；McGill University（麦吉尔大学）；Montreal Institute of Learning Algorithms (Mila)（蒙特利尔学习算法研究所（Mila））；School of Computer Science（计算机科学学院）
Koustuv Sinha, Shagun Sodhani, Joelle Pineau, William L. Hamilton
Recent research has highlighted the role of relational inductive biases in building learning agents that can generalize and reason in a compositional manner. However, while relational learning algorithms such as graph neural networks (GNNs) show promise, we do not understand how effectively these approaches can adapt to new tasks. In this work, we study the task of logical generalization using GNNs by designing a benchmark suite grounded in first-order logic. Our benchmark suite, GraphLog, requires that learning algorithms perform rule induction in different synthetic logics, represented as knowledge graphs. GraphLog consists of relation prediction tasks on 57 distinct logical domains. We use GraphLog to evaluate GNNs in three different setups: single-task supervised learning, multi-task pretraining, and continual learning. Unlike previous benchmarks, our approach allows us to precisely control the logical relationship between the different tasks. We find that the ability for models to generalize and adapt is strongly determined by the diversity of the logical rules they encounter during training, and our results highlight new challenges for the design of GNN models. We publicly release the dataset and code used to generate and interact with the dataset at https://www.cs.mcgill.ca/~ksinha4/graphlog.
SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models
Zhejiang University（浙江大学）；The University of Sydney（悉尼大学）；ManyCore；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning
Department of Computer Science, Informatics Institute, Istanbul Technical University（伊斯坦布尔技术大学信息学研究所计算机科学系）；School of Information Technology, Deakin University（迪肯大学信息技术学院）
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）；Central South University（中南大学）；University of Electronic Science and Technology of China（电子科技大学）；Microsoft Research Asia（微软亚洲研究院）
CIIRC, Faculty of Electrical Engineering, Czech Technical University in Prague（捷克技术大学布拉格分校电子工程学院智能信息研究中心）；Seminar for Applied Mathematics, Department of Mathematics, ETH Zurich（苏黎世联邦理工学院应用数学研讨会）；Institute for Computational Life Sciences, Zurich University of Applied Sciences（苏黎世应用科学大学计算生命科学研究所）；Loschmidt Laboratories, Faculty of Science, Masaryk University, Brno（马萨里克大学布拉格分校科学学院洛施米特实验室）；ICRC, St. Anne’s University Hospital, Brno（布拉格大学医院ICRC）
