Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph's features -- rationale -- which guides the model prediction. Unfortunately, the leading rationalization models often rely on data biases, especially shortcut features, to compose rationales and make predictions without probing the critical and causal patterns. Moreover, such data biases easily change outside the training distribution. As a result, these models suffer from a huge drop in interpretability and predictive performance on out-of-distribution data. In this work, we propose a new strategy of discovering invariant rationale (DIR) to construct intrinsically interpretable GNNs. It conducts interventions on the training distribution to create multiple interventional distributions. Then it approaches the causal rationales that are invariant across different distributions while filtering out the spurious patterns that are unstable. Experiments on both synthetic and real-world datasets validate the superiority of our DIR in terms of interpretability and generalization ability on graph classification over the leading baselines. Code and datasets are available at https://github.com/Wuyxin/DIR-GNN.
University of Groningen（格罗宁根大学）；Utrecht University（乌得勒支大学）
In AI and law, systems that are designed for decision support should be explainable when pursuing justice. In order for these systems to be fair and responsible, they should make correct decisions and make them using a sound and transparent rationale. In this paper, we introduce a knowledge-driven method for model-agnostic rationale evaluation using dedicated test cases, similar to unit-testing in professional software development. We apply this new method in a set of machine learning experiments aimed at extracting known knowledge structures from artificial datasets from fictional and non-fic…
University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）
Leading graph contrastive learning (GCL) methods perform graph augmentations in two fashions: (1) randomly corrupting the anchor graph, which could cause the loss of semantic information, or (2) using domain knowledge to maintain salient features, which undermines the generalization to other domains. Taking an invariance look at GCL, we argue that a high-performing augmentation should preserve the salient semantics of anchor graphs regarding instance-discrimination. To this end, we relate GCL with invariant rationale discovery, and propose a new framework, Rationale-aware Graph Contrastive Lea…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究）；CSAIL MIT（麻省理工学院计算机科学与人工智能实验室）
Selective rationalization improves neural network interpretability by identifying a small subset of input features -- the rationale -- that best explains or supports the prediction. A typical rationalization criterion, i.e. maximum mutual information (MMI), finds the rationale that maximizes the prediction performance based only on the rationale. However, MMI can be problematic because it picks up spurious correlations between the input features and the output. Instead, we introduce a game-theoretic invariant rationalization criterion where the rationales are constrained to enable the same pre…
Perimeter Institute for Theoretical Physics（圆周理论物理研究所）；University of Waterloo（滑铁卢大学）
In this paper, we introduce interpretable Siamese Neural Networks (SNN) for similarity detection to the field of theoretical physics. More precisely, we apply SNNs to events in special relativity, the transformation of electromagnetic fields, and the motion of particles in a central potential. In these examples, the SNNs learn to identify datapoints belonging to the same events, field configurations, or trajectory of motion. It turns out that in the process of learning which datapoints belong to the same event or field configuration, these SNNs also learn the relevant symmetry invariants and c…
Rethinking Dimensional Rationale in Graph Contrastive Learning from Causal Perspective
Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences（中国科学院软件研究所综合信息系统技术国家级重点实验室）；State Key Laboratory of Intelligent Game（智能博弈国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；State Key Laboratory of Computer Science, Institute of Software Chinese Academy of Sciences（中国科学院软件研究所计算机科学国家重点实验室）
Multimodal demonstrations provide robots with an abundance of information to make sense of the world. However, such abundance may not always lead to good performance when it comes to learning sensorimotor control policies from human demonstrations. Extraneous data modalities can lead to state over-specification, where the state contains modalities that are not only useless for decision-making but also can change data distribution across environments. State over-specification leads to issues such as the learned policy not generalizing outside of the training data distribution. In this work, we …
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Invariants and conservation laws convey critical information about the underlying dynamics of a system, yet it is generally infeasible to find them from large-scale data without any prior knowledge or human insight. We propose ConservNet to achieve this goal, a neural network that spontaneously discovers a conserved quantity from grouped data where the members of each group share invariants, similar to a general experimental setting where trajectories from different trials are observed. As a neural network trained with a novel and intuitive loss function called noise-variance loss, ConservNet …
We study the invariance characteristics of pre-trained predictive models by empirically learning transformations on the input that leave the prediction function approximately unchanged. To learn invariant transformations, we minimize the Wasserstein distance between the predictive distribution conditioned on the data instances and the predictive distribution conditioned on the transformed data instances. To avoid finding degenerate or perturbative transformations, we add a similarity regularization to discourage similarity between the data and its transformed values. We theoretically analyze t…
Discovering Multiscale Deep Formulas in Complex Systems via Neural-Guided Lambda Calculus
National Engineering Laboratory for Big Data Analytics（大数据分析国家工程实验室）；Xi’an Jiaotong University（西安交通大学）；School of Mathematics and Statistics（数学与统计学学院）；School of Computer Science and Technology, Faculty of Electronic and Information Engineering（计算机科学与技术学院，电子与信息工程学院）
Discovering Hierarchy-Grounded Domains with Adaptive Granularity for Clinical Domain Generalization
Department of Computer Science, Stevens Institute of Technology（斯蒂文斯理工学院计算机科学系）；Weill Cornell Medicine Graduate School of Medical Sciences, Cornell University（韦尔医学院医学研究生院，康奈尔大学）
Imperial College London（伦敦帝国理工学院）；DeepMind（深度思维）；Lancaster University（兰卡斯特大学）；University of Oxford（牛津大学）
Deep neural networks have been extremely successful as highly accurate wave function ansätze for variational Monte Carlo calculations of molecular ground states. We present an extension of one such ansatz, FermiNet, to calculations of the ground states of periodic Hamiltonians, and study the homogeneous electron gas. FermiNet calculations of the ground-state energies of small electron gas systems are in excellent agreement with previous initiator full configuration interaction quantum Monte Carlo and diffusion Monte Carlo calculations. We investigate the spin-polarized homogeneous electron gas…
In this work, we consider one-shot imitation learning for object rearrangement tasks, where an AI agent needs to watch a single expert demonstration and learn to perform the same task in different environments. To achieve a strong generalization, the AI agent must infer the spatial goal specification for the task. However, there can be multiple goal specifications that fit the given demonstration. To address this, we propose a reward learning approach, Graph-based Equivalence Mappings (GEM), that can discover spatial goal representations that are aligned with the intended goal specification, e…
Tabular data synthesis is an emerging approach to circumvent strict regulations on data privacy while discovering knowledge through big data. Although state-of-the-art AI-based tabular data synthesizers, e.g., table-GAN, CTGAN, TVAE, and CTAB-GAN, are effective at generating synthetic tabular data, their training is sensitive to column permutations of input data. In this paper, we first conduct an extensive empirical study to disclose such a property of permutation invariance and an in-depth analysis of the existing synthesizers. We show that changing the input column order worsens the statist…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Samsung Medical Center（三星医疗中心）；Sungkyunkwan University School of Medicine（成均馆大学医学院）；AITRICS
Explaining generalizations and preventing over-confident predictions are central goals of studies on the loss landscape of neural networks. Flatness, defined as loss invariability on perturbations of a pre-trained solution, is widely accepted as a predictor of generalization in this context. However, the problem that flatness and generalization bounds can be changed arbitrarily according to the scale of a parameter was pointed out, and previous studies partially solved the problem with restrictions: Counter-intuitively, their generalization bounds were still variant for the function-preserving…
Discovering Causal Relations and Equations from Data
Universitat de València（瓦伦西亚大学）；German Aerospace Center（德国航空航天中心）；Technische Universität Berlin（柏林工业大学）；University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Bournemouth University（伯恩茅斯大学）；Heidelberg University（海德堡大学）；KTH Royal Institute of Technology（KTH皇家理工学院）；New York University（纽约大学）
Soft causal learning for generalized molecule property prediction: An environment perspective
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）
Learning Topological Features of $\widehat Z$-invariants
Institute of Physics, University of Amsterdam（阿姆斯特丹大学物理研究所）；School of Mathematics, Statistics and Computer Science, University of KwaZulu-Natal（夸祖鲁-纳塔尔大学数学、统计与计算机科学学院）；Northeastern University（东北大学）；Richard N. Merkin Center for Pure and Applied Mathematics, California Institute of Technology（加州理工学院理查德·N·默金纯数学与应用数学中心）；Ulsan National Institute of Science and Technology（蔚山国家科学技术研究院）；Institute for Mathematics, Academia Sinica（中央研究院数学研究所）；Korteweg-de Vries Institute for Mathematics, University of Amsterdam（阿姆斯特丹大学科特韦格-德弗里斯数学研究所）
A generalized dual potential for inelastic Constitutive Artificial Neural Networks: A JAX implementation at finite strains
Institute of Applied Mechanics, RWTH Aachen University, Germany（应用力学研究所，亚琛工业大学，德国）；Institute for Continuum and Material Mechanics, Hamburg University of Technology, Germany（连续介质与材料力学研究所，汉堡技术大学，德国）；Department of Mechanical Engineering, Stanford University, United States（机械工程系，斯坦福大学，美国）
Learning-based stereo matching and depth estimation networks currently excel on public benchmarks with impressive results. However, state-of-the-art networks often fail to generalize from synthetic imagery to more challenging real data domains. This paper is an attempt to uncover hidden secrets of achieving domain robustness and in particular, discovering the important ingredients of generalization success of stereo matching networks by analyzing the effect of synthetic image learning on real data performance. We provide evidence that demonstrates that learning of features in the synthetic dom…
Systematic construction of continuous-time neural networks for linear dynamical systems
Institute of Advanced Study, Technical University of Munich（慕尼黑工业大学高等研究院）；Institute of Data-Science Foundations, Technische Universität Hamburg（汉堡工业大学数据科学基础研究所）；TUM School of Computation, Information, and Technology（慕尼黑工业大学计算、信息与技术学院）；Eindhoven University of Technology（埃因霍温理工大学）
