Things not Written in Text: Exploring Spatial Commonsense from Visual Signals
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；The MOE Key Laboratory of Computational Linguistics, Peking University（北京大学计算语言学教育部重点实验室）；Artificial Intelligence Institute of Peking University（北京大学人工智能研究院）；State Key Laboratory of Media Convergence Production Technology and Systems（媒体融合生产技术与系统国家重点实验室）
Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part of commonsense knowledge. Although pretrained language models (PLMs) succeed in many NLP tasks, they are shown to be ineffective in spatial commonsense reasoning. Starting from the observation that images are more likely to exhibit spatial commonsense than texts, we explore whether models with visual signals learn more spatial commonsense than text-based PLMs. We propose a spatial commonsense benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We probe PLMs and models with visual signals, including vision-language pretrained models and image synthesis models, on this benchmark, and find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models. The spatial knowledge from image synthesis models also helps in natural language understanding tasks that require spatial commonsense.
Hong Kong University of Science and Technology（香港科技大学）；Tencent（腾讯）
Commonsense knowledge graphs (CKGs) like Atomic and ASER are substantially different from conventional KGs as they consist of much larger number of nodes formed by loosely-structured text, which, though, enables them to handle highly diverse queries in natural language related to commonsense, leads to unique challenges for automatic KG construction methods. Besides identifying relations absent from the KG between nodes, such methods are also expected to explore absent nodes represented by text, in which different real-world things, or entities, may appear. To deal with the innumerable entities…
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Seoul National University（首尔国立大学）；University of Washington（华盛顿大学）；University of Southern California（南加州大学）；University of Pittsburgh（匹兹堡大学）；Carnegie Mellon University（卡内基梅隆大学）
Existing commonsense knowledge bases often organize tuples in an isolated manner, which is deficient for commonsense conversational models to plan the next steps. To fill the gap, we curate a large-scale multi-turn human-written conversation corpus, and create the first Chinese commonsense conversation knowledge graph which incorporates both social commonsense knowledge and dialog flow information. To show the potential of our graph, we develop a graph-conversation matching approach, and benchmark two graph-grounded conversational tasks.
UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations
Cornell University（康奈尔大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）；University of Pittsburgh（匹兹堡大学）；University of California, Berkeley（加州大学伯克利分校）
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Humans understand language based on the rich background knowledge about how the physical world works, which in turn allows us to reason about the physical world through language. In addition to the properties of objects (e.g., boats require fuel) and their affordances, i.e., the actions that are applicable to them (e.g., boats can be driven), we can also reason about if-then inferences between what properties of objects imply the kind of actions that are applicable to them (e.g., that if we can drive something then it likely requires fuel). In this paper, we investigate the extent to which sta…
IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor data
MARS Lab, School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（MARS实验室，机械与航空航天工程学院，南洋理工大学，新加坡）；School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（机械与航空航天工程学院，南洋理工大学，新加坡）
It is rare that texts or entire books written in a Controlled Natural Language (CNL) become very popular, but exactly this has happened with a book that has been published last year. Randall Munroe's Thing Explainer uses only the 1'000 most often used words of the English language together with drawn pictures to explain complicated things such as nuclear reactors, jet engines, the solar system, and dishwashers. This restricted language is a very interesting new case for the CNL community. I describe here its place in the context of existing approaches on Controlled Natural Languages, and I pro…
University of Pennsylvania（宾夕法尼亚大学）；Tencent AI Lab（腾讯人工智能实验室）
Large-scale pretrained language models have made significant advances in solving downstream language understanding tasks. However, they generally suffer from reporting bias, the phenomenon describing the lack of explicit commonsense knowledge in written text, e.g., ''an orange is orange''. To overcome this limitation, we develop a novel approach, Z-LaVI, to endow language models with visual imagination capabilities. Specifically, we leverage two complementary types of ''imaginations'': (i) recalling existing images through retrieval and (ii) synthesizing nonexistent images via text-to-image ge…
University of Washington（华盛顿大学）；Allen Institute for AI（艾伦人工智能研究所）
Modern neural language models can produce remarkably fluent and grammatical text. So much, in fact, that recent work by Clark et al. (2021) has reported that conventional crowdsourcing can no longer reliably distinguish between machine-authored (GPT-3) and human-authored writing. As errors in machine generations become ever subtler and harder to spot, it poses a new challenge to the research community for robust machine text evaluation. We propose a new framework called Scarecrow for scrutinizing machine text via crowd annotation. To support the broad range of real machine errors that can be i…
Institute for Cross-Disciplinary Physics and Complex Systems IFISC (UIB-CSIC)（跨学科物理与复杂系统研究所IFISC（巴利阿里群岛大学-西班牙国家研究委员会））；Centro de Investigaciones Ópticas (CONICET La Plata-CIC-UNLP)（光学研究中心（拉普拉塔国家科学技术研究委员会-拉普拉塔文化与科学研究中心-拉普拉塔国立大学））；Departamento de Ciencias Básicas, Facultad de Ingeniería, Universidad Nacional de La Plata (UNLP)（拉普拉塔国立大学工程学院基础科学系）
Words are fundamental linguistic units that connect thoughts and things through meaning. However, words do not appear independently in a text sequence. The existence of syntactic rules induces correlations among neighboring words. Using an ordinal pattern approach, we present an analysis of lexical statistical connections for 11 major languages. We find that the diverse manners that languages utilize to express word relations give rise to unique pattern structural distributions. Furthermore, fluctuations of these pattern distributions for a given language can allow us to determine both the his…
