In order to communicate, humans flatten a complex representation of ideas and their attributes into a single word or a sentence. We investigate the impact of representation learning in artificial agents by developing graph referential games. We empirically show that agents parametrized by graph neural networks develop a more compositional language compared to bag-of-words and sequence models, which allows them to systematically generalize to new combinations of familiar features.
Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of {\em augmentations} and the value of {\em contrastive loss} of representations. We demonstrate that such analyses, that ignore {\em inductive biases} of the function class and training algorithm, cannot adequately explain the success of …
A fascinating hypothesis is that human and animal intelligence could be explained by a few principles (rather than an encyclopedic list of heuristics). If that hypothesis was correct, we could more easily both understand our own intelligence and build intelligent machines. Just like in physics, the principles themselves would not be sufficient to predict the behavior of complex systems like brains, and substantial computation might be needed to simulate human-like intelligence. This hypothesis would suggest that studying the kind of inductive biases that humans and animals exploit could help b…
With the resurgence of interest in neural networks, representation learning has re-emerged as a central focus in artificial intelligence. Representation learning refers to the discovery of useful encodings of data that make domain-relevant information explicit. Factorial representations identify underlying independent causal factors of variation in data. A factorial representation is compact and faithful, makes the causal factors explicit, and facilitates human interpretation of data. Factorial representations support a variety of applications, including the generation of novel examples, index…
Département d’informatique et de recherche opérationnelle（计算机科学与运筹学系）
In this thesis, we try to build a connection between the two schools by introducing syntactic inductive biases for deep learning models. We propose two families of inductive biases, one for constituency structure and another one for dependency structure. The constituency inductive bias encourages deep learning models to use different units (or neurons) to separately process long-term and short-term information. This separation provides a way for deep learning models to build the latent hierarchical representations from sequential inputs, that a higher-level representation is composed of and ca…
SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biases
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯AI实验室）
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT）人类工程学与机器人研究所）；Stanford University（斯坦福大学）
Endowing robots with human-like physical reasoning abilities remains challenging. We argue that existing methods often disregard spatio-temporal relations and by using Graph Neural Networks (GNNs) that incorporate a relational inductive bias, we can shift the learning process towards exploiting relations. In this work, we learn action-conditional forward dynamics models of a simulated manipulation task from visual observations involving cluttered and irregularly shaped objects. We investigate two GNN approaches and empirically assess their capability to generalize to scenarios with novel and a…
DeepMind（深度思维）；Google Brain（谷歌大脑）；MIT（麻省理工学院）；University of Edinburgh（爱丁堡大学）
Artificial intelligence (AI) has undergone a renaissance recently, making major progress in key domains such as vision, language, control, and decision-making. This has been due, in part, to cheap data and cheap compute resources, which have fit the natural strengths of deep learning. However, many defining characteristics of human intelligence, which developed under much different pressures, remain out of reach for current approaches. In particular, generalizing beyond one's experiences--a hallmark of human intelligence from infancy--remains a formidable challenge for modern AI. The following…
Harnessing cortical geometry, wiring, and function as inductive biases for recurrent neural networks
Neuromatch Academy, Neuromatch, Inc., USA（Neuromatch学院，Neuromatch公司，美国）；McGovern Institute for Brain Research, Massachusetts Institute of Technology (MIT)（麦戈文脑科学研究所，麻省理工学院（MIT））
The Hong Kong Polytechnic University（香港理工大学）；University of Hong Kong（香港大学）；The Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）；Research Institute for Sustainable Urban Development, The Hong Kong Polytechnic University（香港理工大学可持续城市发展研究院）
Accurate short-term traffic prediction plays a pivotal role in various smart mobility operation and management systems. Currently, most of the state-of-the-art prediction models are based on graph neural networks (GNNs), and the required training samples are proportional to the size of the traffic network. In many cities, the available amount of traffic data is substantially below the minimum requirement due to the data collection expense. It is still an open question to develop traffic prediction models with a small size of training data on large-scale networks. We notice that the traffic sta…
The ability to discriminate between large and small quantities is a core aspect of basic numerical competence in both humans and animals. In this work, we examine the extent to which the state-of-the-art neural networks designed for vision exhibit this basic ability. Motivated by studies in animal and infant numerical cognition, we use the numerical bisection procedure to test number discrimination in different families of neural architectures. Our results suggest that vision-specific inductive biases are helpful in numerosity discrimination, as models with such biases have lowest test errors …
Recently, graph neural networks have been gaining a lot of attention to simulate dynamical systems due to their inductive nature leading to zero-shot generalizability. Similarly, physics-informed inductive biases in deep-learning frameworks have been shown to give superior performance in learning the dynamics of physical systems. There is a growing volume of literature that attempts to combine these two approaches. Here, we evaluate the performance of thirteen different graph neural networks, namely, Hamiltonian and Lagrangian graph neural networks, graph neural ODE, and their variants with ex…
Delft University of Technology（代尔夫特理工大学）；Gradiant；Bosch Security Systems B.V.（博世安防系统有限公司）；Synergy Sports
The second edition of the "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" challenges featured five data-impaired challenges, where models are trained from scratch on a reduced number of training samples for various key computer vision tasks. To encourage new and creative ideas on incorporating relevant inductive biases to improve the data efficiency of deep learning models, we prohibited the use of pre-trained checkpoints and other transfer learning techniques. The provided baselines are outperformed by a large margin in all five challenges, mainly thanks to extensive data…
University of Edinburgh（爱丁堡大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Cambridge（剑桥大学）；DeepMind（深度思维）
Referential games and reconstruction games are the most common game types for studying emergent languages. We investigate how the type of the language game affects the emergent language in terms of: i) language compositionality and ii) transfer of an emergent language to a task different from its origin, which we refer to as language expressivity. With empirical experiments on a handcrafted symbolic dataset, we show that languages emerged from different games have different compositionality and further different expressivity.
Johns Hopkins University（约翰斯·霍普金斯大学）；University of California, Berkeley（加州大学伯克利分校）；Microsoft Research AI（微软人工智能研究院）；Princeton University（普林斯顿大学）
How do learners acquire languages from the limited data available to them? This process must involve some inductive biases - factors that affect how a learner generalizes - but it is unclear which inductive biases can explain observed patterns in language acquisition. To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases. This framework disentangles universal inductive biases, which are…
We study the problem of emergent communication, in which language arises because speakers and listeners must communicate information in order to solve tasks. In temporally extended reinforcement learning domains, it has proved hard to learn such communication without centralized training of agents, due in part to a difficult joint exploration problem. We introduce inductive biases for positive signalling and positive listening, which ease this problem. In a simple one-step environment, we demonstrate how these biases ease the learning problem. We also apply our methods to a more extended envir…
Sequence-to-sequence (seq2seq) learners are widely used, but we still have only limited knowledge about what inductive biases shape the way they generalize. We address that by investigating how popular seq2seq learners generalize in tasks that have high ambiguity in the training data. We use SCAN and three new tasks to study learners' preferences for memorization, arithmetic, hierarchical, and compositional reasoning. Further, we connect to Solomonoff's theory of induction and propose to use description length as a principled and sensitive measure of inductive biases. In our experimental study…
With the starting point that implicit human biases are reflected in the statistical regularities of language, it is possible to measure biases in English static word embeddings. State-of-the-art neural language models generate dynamic word embeddings dependent on the context in which the word appears. Current methods measure pre-defined social and intersectional biases that appear in particular contexts defined by sentence templates. Dispensing with templates, we introduce the Contextualized Embedding Association Test (CEAT), that can summarize the magnitude of overall bias in neural language …
We show that it is possible to craft transformations that, applied to compositional grammars, result in grammars that neural networks can learn easily, but humans do not. This could explain the disconnect between current metrics of compositionality, that are arguably human-centric, and the ability of neural networks to generalize to unseen examples. We propose to use the transformations as a benchmark, ICY, which could be used to measure aspects of the compositional inductive bias of networks, and to search for networks with similar compositional inductive biases to humans. As an example of th…
People use rich prior knowledge about the world in order to efficiently learn new concepts. These priors - also known as "inductive biases" - pertain to the space of internal models considered by a learner, and they help the learner make inferences that go beyond the observed data. A recent study found that deep neural networks optimized for object recognition develop the shape bias (Ritter et al., 2017), an inductive bias possessed by children that plays an important role in early word learning. However, these networks use unrealistically large quantities of training data, and the conditions …
Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communication
Leiden Institute of Advanced Computer Science, Leiden University, Netherlands（莱顿高级计算机科学研究所，莱顿大学，荷兰）；School of Computing, University of Kent, United Kingdom（计算学院，肯特大学，英国）；Cognitive Psychology Unit, Institute of Psychology, Leiden, the Netherlands（认知心理学单元，心理学研究所，莱顿，荷兰）
Carnegie Mellon University（卡内基梅隆大学）；University of Haifa（海法大学）；University of Pittsburgh（匹兹堡大学）；University of Washington（华盛顿大学）
Natural language processing (NLP) models trained on people-generated data can be unreliable because, without any constraints, they can learn from spurious correlations that are not relevant to the task. We hypothesize that enriching models with speaker information in a controlled, educated way can guide them to pick up on relevant inductive biases. For the speaker-driven task of predicting code-switching points in English--Spanish bilingual dialogues, we show that adding sociolinguistically-grounded speaker features as prepended prompts significantly improves accuracy. We find that by adding i…
University of Cambridge（剑桥大学）；ETH Zürich（苏黎世联邦理工学院）
Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages. Investigation of these biases has proved complicated due to the many variables that appear in the experimental setup. Languages vary in many typological dimensions, and it is difficult to single out one or two to investigate without the others acting as confounders. We propose a novel method for investigating the inductive biases of language models using artificial languages. These lang…
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
Mila & Université de Montréal（Mila与蒙特利尔大学）；Noah’s Ark Lab（Noah’s Ark实验室）；Advanced Micro Devices；Chandar Research Lab（Chandar研究实验室）；Polytechnique Montréal（蒙特利尔理工学院）；CIFAR AI Chair（CIFAR人工智能主席）
Learners that are exposed to the same training data might generalize differently due to differing inductive biases. In neural network models, inductive biases could in theory arise from any aspect of the model architecture. We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection. For both tasks, the training set is consistent with a generalization based on hierarchical structure and a generalization based on linear order. All architectural factor…
Facebook A.I. Research（Facebook人工智能研究院）；ENS - EHESS - PSL Research University - CNRS - INRIA（巴黎高等师范学院-社会科学高等研究院-巴黎文理研究大学-法国国家科学研究中心-法国国家信息与自动化研究所）；ICREA（加泰罗尼亚研究与高等教育学院）
Sequence-processing neural networks led to remarkable progress on many NLP tasks. As a consequence, there has been increasing interest in understanding to what extent they process language as humans do. We aim here to uncover which biases such models display with respect to "natural" word-order constraints. We train models to communicate about paths in a simple gridworld, using miniature languages that reflect or violate various natural language trends, such as the tendency to avoid redundancy or to minimize long-distance dependencies. We study how the controlled characteristics of our miniatu…
We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and (ii) recursive syntactic compositions, which here are implemented through a special attention mask and deterministic transformation of the linearized tree. We find that TGs outperform various strong baselines on sentence-level language modeling perplexity, as well as on multiple syntax-sensitive language modeling evaluation metrics. Additionally, we find that the recursive syntactic composition bottleneck which repr…
LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）
Bar Ilan University（巴伊兰大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）
How do typological properties such as word order and morphological case marking affect the ability of neural sequence models to acquire the syntax of a language? Cross-linguistic comparisons of RNNs' syntactic performance (e.g., on subject-verb agreement prediction) are complicated by the fact that any two languages differ in multiple typological properties, as well as by differences in training corpus. We propose a paradigm that addresses these issues: we create synthetic versions of English, which differ from English in one or more typological parameters, and generate corpora for those langu…
Learning and communication pressures in neural networks: Lessons from emergent communication
Centre for Machine Learning（机器学习中心）；University of Southern Denmark (SDU)（南丹麦大学）；Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）；University of Glasgow（格拉斯哥大学）
Facebook AI Research（Facebook AI研究院）；Université de Paris（巴黎大学）；CoML, ENS/CNRS/EHESS/INRIA/PSL University（计算语言学实验室，巴黎高等师范学院/法国国家科学研究中心/法国社会科学高等研究院/法国国家信息与自动化研究所/巴黎文理研究大学）；CRI, Département Frontières du Vivant et de l’Apprendre, IIFR, Université de Paris（认知研究所，生命与学习前沿系，法国医学研究院，巴黎大学）；École normale supérieure, PSL University, CNRS（巴黎高等师范学院，巴黎文理研究大学，法国国家科学研究中心）
Our ability to comprehend speech remains, to date, unrivaled by deep learning models. This feat could result from the brain's ability to fine-tune generic sound representations for speech-specific processes. To test this hypothesis, we compare i) five types of deep neural networks to ii) human brain responses elicited by spoken sentences and recorded in 102 Dutch subjects using functional Magnetic Resonance Imaging (fMRI). Each network was either trained on an acoustics scene classification, a speech-to-text task (based on Bengali, English, or Dutch), or not trained. The similarity between eac…
