Recently, transformer-based methods such as RoBERTa and GPT-3 have led to significant experimental advances in natural language processing tasks such as question answering and commonsense reasoning. The latter is typically evaluated through multiple benchmarks framed as multiple-choice instances of the former. According to influential leaderboards hosted by the Allen Institute (evaluating state-of-the-art performance on commonsense reasoning benchmarks), models based on such transformer methods are approaching human-like performance and have average accuracy well over 80% on many benchmarks. Since these are commonsense benchmarks, a model that generalizes on commonsense reasoning should not experience much performance loss across multiple commonsense benchmarks. In this paper, we study the generalization issue in detail by designing and conducting a rigorous scientific study. Using five common benchmarks, multiple controls and statistical analysis, we find clear evidence that fine-tuned commonsense language models still do not generalize well, even with moderate changes to the experimental setup, and may, in fact, be susceptible to dataset bias. We also perform selective studies, including qualitative and consistency analyses, to gain deeper insight into the problem.
Universitat Politècnica de València（瓦伦西亚理工大学）；Izmir Institute of Technology（伊兹密尔理工学院）；bytes Corp.（2bytes公司）；University of Koblenz-Landau（科布伦茨-兰道大学）
Widespread and rapid dissemination of false news has made fact-checking an indispensable requirement. Given its time-consuming and labor-intensive nature, the task calls for an automated support to meet the demand. In this paper, we propose to leverage commonsense knowledge for the tasks of false news classification and check-worthy claim detection. Arguing that commonsense knowledge is a factor in human believability, we fine-tune the BERT language model with a commonsense question answering task and the aforementioned tasks in a multi-task learning environment. For predicting fine-grained fa…
Georgia Institute of Technology（佐治亚理工学院）；Microsoft Research Asia（微软亚洲研究院）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
Question Answering (QA) has been a long-standing research topic in AI and NLP fields, and a wealth of studies have been conducted to attempt to equip QA systems with human-level reasoning capability. To approximate the complicated human reasoning process, state-of-the-art QA systems commonly use pre-trained language models (LMs) to access knowledge encoded in LMs together with elaborately designed modules based on Graph Neural Networks (GNNs) to perform reasoning over knowledge graphs (KGs). However, many problems remain open regarding the reasoning functionality of these GNN-based modules. Ca…
Tel-Aviv University（特拉维夫大学）；Allen Institute for AI（艾伦人工智能研究所）
Large pre-trained language models (LMs) have been shown to perform surprisingly well when fine-tuned on tasks that require commonsense and world knowledge. However, in end-to-end architectures, it is difficult to explain what is the knowledge in the LM that allows it to make a correct prediction. In this work, we propose a model for multi-choice question answering, where a LM-based generator generates a textual hypothesis that is later used by a classifier to answer the question. The hypothesis provides a window into the information used by the fine-tuned LM that can be inspected by humans. A …
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>…
Transformers have been showing near-human performance on a variety of tasks, but they are not without their limitations. We discuss the issue of conflating results of transformers that are instructed to do multiple tasks simultaneously. In particular, we focus on the domain of commonsense reasoning within story prose, which we call contextual commonsense inference (CCI). We look at the GLUCOSE (Mostafazadeh et al. 2020) dataset and task for predicting implicit commonsense inferences between story sentences. Since the GLUCOSE task simultaneously generates sentences and predicts the CCI relation…
Generating step-by-step "chain-of-thought" rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering. However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracy by using only few-shot inference. We propose a technique to iteratively leverage a small number of rationale examples and a large dataset without rationales, to bootstrap the ability to perform successively more complex reasoning. This technique, the "Self-Taught Reasoner" (STaR), relie…
Allen Institute for AI（艾伦人工智能研究所）；EPFL（洛桑联邦理工学院）；University of Washington（华盛顿大学）
Recently, commonsense knowledge models - pretrained language models (LM) fine-tuned on knowledge graph (KG) tuples - showed that considerable amounts of commonsense knowledge can be encoded in the parameters of large language models. However, as parallel studies show that LMs are poor hypothesizers of declarative commonsense relationships on their own, it remains unclear whether this knowledge is learned during pretraining or from fine-tuning on KG examples. To investigate this question, we train commonsense knowledge models in few-shot settings to study the emergence of their commonsense repr…
Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
Huazhong University of Science and Technology（华中科技大学）；Griffith University（格里菲斯大学）；Squirrel Ai Learning（squirrel ai 学习）；Shanghai Jiaotong University（上海交通大学）；Lehigh University（莱斯大学）
GSAM: A Generalizable and Safe Robotic Framework for Articulated Object Manipulation
College of Computer Science, Chongqing University, Chongqing, China（重庆大学计算机学院）；Lumos Robotics, China（Lumos机器人中国）；Xi'an Jiaotong-Liverpool University, China（西安交通大学利物浦大学）；Fudan University, China（复旦大学）；Department of Information Engineering and Computer Science, University of Trento, Trento, Italy（特伦托大学信息工程与计算机科学系）
University of Science and Technology of China（中国科学技术大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
The state-of-the-art pre-trained language representation models, such as Bidirectional Encoder Representations from Transformers (BERT), rarely incorporate commonsense knowledge or other knowledge explicitly. We propose a pre-training approach for incorporating commonsense knowledge into language representation models. We construct a commonsense-related multi-choice question answering dataset for pre-training a neural language representation model. The dataset is created automatically by our proposed "align, mask, and select" (AMS) method. We also investigate different pre-training tasks. Expe…
Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models
School of Computer Science and Statistics, Trinity College Dublin, Dublin, Ireland（特里尼蒂学院都柏林分校计算机科学与统计学学院）；School of Computer Science, University College Dublin, Dublin, Ireland（都柏林大学学院计算机科学学院）
How much information do NLP tasks really need from a transformer's attention mechanism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that the floating-points within its computation can be discretized to fewer values with minimal loss to task accuracies. However, this requires retraining or even creating entirely new models, both of which can be expensive and carbon-emitting. Focused on optimizations that do not require training, we systematically study the full range of typical attention values necessary. This informs the design of an inf…
CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation
Department of Computer Science, University of Copenhagen（哥本哈根大学计算机科学系）；Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
We address the general task of structured commonsense reasoning: given a natural language input, the goal is to generate a graph such as an event -- or a reasoning-graph. To employ large language models (LMs) for this task, existing approaches ``serialize'' the output graph as a flat list of nodes and edges. Although feasible, these serialized graphs strongly deviate from the natural language corpora that LMs were pre-trained on, hindering LMs from generating them correctly. In this paper, we show that when we instead frame structured commonsense reasoning tasks as code generation tasks, pre-t…
Outlier-weighed Layerwise Sampling for LLM Fine-tuning
Dalian University of Technology（大连理工大学）；University of Surrey（Surrey大学）；Eindhoven University of Technology（埃因霍温理工大学）；University College London（伦敦大学学院）；University of Oxford（牛津大学）
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
York University（约克大学）；Nanyang Technological University（南洋理工大学）；Dialpad Canada Inc.（Dialpad加拿大公司）；Royal Bank of Canada（加拿大皇家银行）；Salesforce Research（赛富时研究院）
Western Michigan University（西密歇根大学）；Virginia Commonwealth University（弗吉尼亚联邦大学）
In this paper, we investigate a commonsense inference task that unifies natural language understanding and commonsense reasoning. We describe our attempt at SemEval-2020 Task 4 competition: Commonsense Validation and Explanation (ComVE) challenge. We discuss several state-of-the-art deep learning architectures for this challenge. Our system uses prepared labeled textual datasets that were manually curated for three different natural language inference subtasks. The goal of the first subtask is to test whether a model can distinguish between natural language statements that make sense and those…
Crystal: Introspective Reasoners Reinforced with Self-Feedback
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；FAIR, Meta（Meta 基础人工智能研究院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. Most of the existing approaches rely on a randomly initialized classifier on top of such networks. We argue that this fine-tuning procedure is sub-optimal as the pre-trained model has no prior on the specific classifier labels, while it might have already learned an intrinsic textual representation of the task. In this paper, we introduce a new scoring method that casts a plausibility ranking task in a full-text format and leverages the masked language modeling head tuned during the pre…
University of Oxford（牛津大学）；Imperial College London（帝国理工学院）；Alan Turing Institute（艾伦·图灵研究所）
The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. In this paper, we show that the performance of three language models on WSC273 strongly improves when fine-tuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We additionally generate a large unsupervised WSC-like dataset. By fine-tuning the BERT language model both on the introduced and on the WSCR dataset, we achieve overall accuracies of 72.5% and 74.7% on WSC273 and WNLI, improving the previous state…
Mila – Quebec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Google Research（谷歌研究院）；University of Cambridge（剑桥大学）；Facebook（脸书）
While achieving state-of-the-art results in multiple tasks and languages, translation-based cross-lingual transfer is often overlooked in favour of massively multilingual pre-trained encoders. Arguably, this is due to its main limitations: 1) translation errors percolating to the classification phase and 2) the insufficient expressiveness of the maximum-likelihood translation. To remedy this, we propose a new technique that integrates both steps of the traditional pipeline (translation and classification) into a single model, by treating the intermediate translations as a latent random variabl…
