Modern Artificial Intelligence (AI) systems excel at diverse tasks, from image classification to strategy games, even outperforming humans in many of these domains. After making astounding progress in language learning in the recent decade, AI systems, however, seem to approach the ceiling that does not reflect important aspects of human communicative capacities. Unlike human learners, communicative AI systems often fail to systematically generalize to new data, suffer from sample inefficiency, fail to capture common-sense semantic knowledge, and do not translate to real-world communicative situations. Cognitive Science offers several insights on how AI could move forward from this point. This paper aims to: (1) suggest that the dominant cognitively-inspired AI directions, based on nativist and symbolic paradigms, lack necessary substantiation and concreteness to guide progress in modern AI, and (2) articulate an alternative, "grounded", perspective on AI advancement, inspired by Embodied, Embedded, Extended, and Enactive Cognition (4E) research. I review results on 4E research lines in Cognitive Science to distinguish the main aspects of naturalistic learning conditions that play causal roles for human language development. I then use this analysis to propose a list of concrete, implementable components for building "grounded" linguistic intelligence. These components include embodying machines in a perception-action cycle, equipping agents with active exploration mechanisms so they can build their own curriculum, allowing agents to gradually develop motor abilities to promote piecemeal language development, and endowing the agents with adaptive feedback from their physical and social environment. I hope that these ideas can direct AI research towards building machines that develop human-like language abilities through their experiences with the world.
Universidad Nacional de Córdoba（科尔多瓦国立大学）；CONICET（阿根廷国家科学与技术研究委员会）；Roskilde University（罗斯基勒大学）
In order to interpret the communicative intents of an utterance, it needs to be grounded in something that is outside of language; that is, grounded in world modalities. In this paper, we argue that dialogue clarification mechanisms make explicit the process of interpreting the communicative intents of the speaker's utterances by grounding them in the various modalities in which the dialogue is situated. This paper frames dialogue clarification mechanisms as an understudied research problem and a key missing piece in the giant jigsaw puzzle of natural language understanding. We discuss both th…
Institute for Logic, Language and Computation（逻辑、语言与计算研究所）；University of Amsterdam（阿姆斯特丹大学）
This position paper proposes a conceptual framework for the design of Natural Language Generation (NLG) systems that follow efficient and effective production strategies in order to achieve complex communicative goals. In this general framework, efficiency is characterised as the parsimonious regulation of production and comprehension costs while effectiveness is measured with respect to task-oriented and contextually grounded communicative goals. We provide concrete suggestions for the estimation of goals, costs, and utility via modern statistical methods, demonstrating applications of our fr…
Describe me an Aucklet: Generating Grounded Perceptual Category Descriptions
Centre for Linguistic Theory and Studies in Probability (CLASP)（语言理论与概率研究中心）；Department of Philosophy, Linguistics, and Theory of Science (FLoV)（哲学、语言学与理论科学系）；University of Gothenburg（哥德堡大学）
Recent hype surrounding the increasing sophistication of language processing models has renewed optimism regarding machines achieving a human-like command of natural language. Research in the area of natural language understanding (NLU) in artificial intelligence claims to have been making great strides in this area, however, the lack of conceptual clarity/consistency in how 'understanding' is used in this and other disciplines makes it difficult to discern how close we actually are. In this interdisciplinary research thesis, I integrate insights from cognitive science/psychology, philosophy o…
In recent years, automatic speech-to-speech and speech-to-text translation has gained momentum thanks to advances in artificial intelligence, especially in the domains of speech recognition and machine translation. The quality of such applications is commonly tested with automatic metrics, such as BLEU, primarily with the goal of assessing improvements of releases or in the context of evaluation campaigns. However, little is known about how the output of such systems is perceived by end users or how they compare to human performances in similar communicative tasks. In this paper, we present th…
Emergent communication research often focuses on optimizing task-specific utility as a driver for communication. However, human languages appear to evolve under pressure to efficiently compress meanings into communication signals by optimizing the Information Bottleneck tradeoff between informativeness and complexity. In this work, we study how trading off these three factors -- utility, informativeness, and complexity -- shapes emergent communication, including compared to human communication. To this end, we propose Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for t…
National Institute of Information and Communications Technology（国立信息与通信技术研究所）
The target task of this study is grounded language understanding for domestic service robots (DSRs). In particular, we focus on instruction understanding for short sentences where verbs are missing. This task is of critical importance to build communicative DSRs because manipulation is essential for DSRs. Existing instruction understanding methods usually estimate missing information only from non-grounded knowledge; therefore, whether the predicted action is physically executable or not was unclear. In this paper, we present a grounded instruction understanding method to estimate appropriate …
We introduce GODEL (Grounded Open Dialogue Language Model), a large pre-trained language model for dialog. In contrast with earlier models such as DialoGPT, GODEL leverages a new phase of grounded pre-training designed to better support adapting GODEL to a wide range of downstream dialog tasks that require information external to the current conversation (e.g., a database or document) to produce good responses. Experiments against an array of benchmarks that encompass task-oriented dialog, conversational QA, and grounded open-domain dialog show that GODEL outperforms state-of-the-art pre-train…
More Victories, Less Cooperation: Assessing Cicero's Diplomacy Play
University of Maryland（马里兰大学）；Princeton University（普林斯顿大学）；University of Sydney（悉尼大学）；Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）
A distinguishing property of human intelligence is the ability to flexibly use language in order to communicate complex ideas with other humans in a variety of contexts. Research in natural language dialogue should focus on designing communicative agents which can integrate themselves into these contexts and productively collaborate with humans. In this abstract, we propose a general situated language learning paradigm which is designed to bring about robust language agents able to cooperate productively with humans.
LinguaGame: A Linguistically Grounded Game-Theoretic Paradigm for Multi-Agent Dialogue Generation
Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）；Peking University（北京大学）；East China University of Political Science and Law（中国政法大学）
The question of how an effective and efficient communication system can emerge in a population of agents that need to solve a particular task attracts more and more attention from researchers in many fields, including artificial intelligence, linguistics and statistical physics. A common methodology for studying this question consists of carrying out multi-agent experiments in which a population of agents takes part in a series of scripted and task-oriented communicative interactions, called 'language games'. While each individual language game is typically played by two agents in the populati…
Microsoft Research（微软研究院）；University of Waterloo（滑铁卢大学）；Meta Research（Meta研究院）；Moscow Institute of Physics Technology（莫斯科物理技术学院）；Artificial Intelligence Research Institute（人工智能研究院）
Human intelligence can remarkably adapt quickly to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research which can enable similar capabilities in machines, we made the following contributions (1) formalized the collaborative embodied agent using natural language task; (2) developed a tool for extensive and scalable data collection; and (3) collected the first dataset for interactive grounded language understan…
EiCAP: Beyond Fluency, Probing and Improving Emotional Intelligence in LLMs via Psychologically Grounded Multi-Turn Dialogue
Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University（卡塔尔计算研究所（QCRI），哈马德·本·卡伊夫大学）；University of Illinois Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）
EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
Tianjin University（天津大学）；Fuzhou University（福州大学）；Shanghai Jiaotong University（上海交通大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
MedRLM: Recursive Multimodal Health Intelligence for Long-Context Clinical Reasoning, Sensor-Guided Screening, Evidence-Grounded Decision Support, and Community-to-Tertiary Referral Optimization
School of Information, Computer；Communication Technology Sirindhorn International Institute of Technology, Thammasat University Pathum Thani, Thailand 1
An overarching goal of natural language processing is to enable machines to communicate seamlessly with humans. However, natural language can be ambiguous or unclear. In cases of uncertainty, humans engage in an interactive process known as repair: asking questions and seeking clarification until their uncertainty is resolved. We propose a framework for building a visually grounded question-asking model capable of producing polar (yes-no) clarification questions to resolve misunderstandings in dialogue. Our model uses an expected information gain objective to derive informative questions from …
Microsoft Research（微软研究院）；University of Amsterdam（阿姆斯特丹大学）；Alexa AI（Alexa人工智能）；MIPT（莫斯科物理技术学院）；AIRI（人工智能研究院）；Facebook AI（脸书人工智能）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Kalashnikov Izhevsk State Technical University（卡拉什尼科夫伊热夫斯克国立技术大学）；Universiteit Leiden（莱顿大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）
Human intelligence has the remarkable ability to quickly adapt to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research in this direction, we propose \emph{IGLU: Interactive Grounded Language Understanding in a Collaborative Environment}. The primary goal of the competition is to approach the problem of how to build interactive agents that learn to solve a task while provided with grounded natural language ins…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Theorem proving in natural mathematical language - the mixture of symbolic and natural language used by humans - plays a central role in mathematical advances and education, and tests aspects of reasoning that are core to intelligence. Yet it has remained underexplored with modern generative models. We study large-scale language models on two new generation tasks: suggesting the next step in a mathematical proof, and full proof generation. We develop NaturalProver, a language model that generates proofs by conditioning on background references (e.g. theorems and definitions that are either ret…
University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Emotional language generation is one of the keys to human-like artificial intelligence. Humans use different type of emotions depending on the situation of the conversation. Emotions also play an important role in mediating the engagement level with conversational partners. However, current conversational agents do not effectively account for emotional content in the language generation process. To address this problem, we develop a language modeling approach that generates affective content when the dialogue is situated in a given context. We use the recently released Empathetic-Dialogues cor…
Facebook AI Research（Facebook AI 研究院）；University of Cambridge（剑桥大学）
Meaning has been called the "holy grail" of a variety of scientific disciplines, ranging from linguistics to philosophy, psychology and the neurosciences. The field of Artifical Intelligence (AI) is very much a part of that list: the development of sophisticated natural language semantics is a sine qua non for achieving a level of intelligence comparable to humans. Embodiment theories in cognitive science hold that human semantic representation depends on sensori-motor experience; the abundant evidence that human meaning representation is grounded in the perception of physical reality leads to…
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
Meituan（美团）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University（南京大学）；Harbin Institute of Technology（哈尔滨工业大学）；Australian Institute for Machine Learning, Adelaide University（阿德莱德大学澳大利亚机器学习研究所）；Ludwig Maximilian University of Munich（慕尼黑大学）；University of Science and Technology of China（中国科学技术大学）；Queen Mary University of London（伦敦玛丽女王大学）
Human intelligence has the remarkable ability to adapt to new tasks and environments quickly. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research in this direction, we propose IGLU: Interactive Grounded Language Understanding in a Collaborative Environment. The primary goal of the competition is to approach the problem of how to develop interactive embodied agents that learn to solve a task while provided with grounded natural language…
Schemata are structured representations of complex tasks that can aid artificial intelligence by allowing models to break down complex tasks into intermediate steps. We propose a novel system that induces schemata from web videos and generalizes them to capture unseen tasks with the goal of improving video retrieval performance. Our system proceeds in three major phases: (1) Given a task with related videos, we construct an initial schema for a task using a joint video-text model to match video segments with text representing steps from wikiHow; (2) We generalize schemata to unseen tasks by le…
We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we …
