Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks. Previously, researchers have proposed to use language model pre-training and multi-task learning to learn robust representations. However, these methods can achieve sub-optimal performance in low-resource scenarios. Inspired by the recent success of optimization-based meta-learning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong baselines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively.
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Ursinus College（乌尔辛纳斯学院）
Sports broadcasters inject drama into play-by-play commentary by building team and player narratives through subjective analyses and anecdotes. Prior studies based on small datasets and manual coding show that such theatrics evince commentator bias in sports broadcasts. To examine this phenomenon, we assemble FOOTBALL, which contains 1,455 broadcast transcripts from American football games across six decades that are automatically annotated with 250K player mentions and linked with racial metadata. We identify major confounding factors for researchers examining racial bias in FOOTBALL, and per…
Investigating Hallucination in Conversations for Low Resource Languages
University of North Alabama（北阿拉巴马大学）；Wichita State University（威奇托州立大学）；Murray State University（默里州立大学）；California State Polytechnic University Pomona（加州州立理工大学波莫纳分校）；Auburn University at Montgomery（奥本大学蒙哥马利分校）；Auburn University（奥本大学）；Gustavus Adolphus College（古斯塔夫阿道夫学院）；Stanford University（斯坦福大学）；Meta；Amazon GenAI（亚马逊生成式人工智能）
Lelapa AI（莱拉帕人工智能公司）；Luleå University of Technology（吕勒奥理工大学）；Lanfrica（兰弗里卡）；Instituto Politécnico Nacional（墨西哥国立理工学院）；USC Information Sciences Institute（南加州大学信息科学研究所）；University of California, Davis（加州大学戴维斯分校）；New Mexico State University（新墨西哥州立大学）；Technical University of Munich（慕尼黑工业大学）；University of Porto（波尔图大学）；ML Collective（机器学习集体）；University of Toronto（多伦多大学）；University of Dayton（代顿大学）；McGill University（麦吉尔大学）；Mila Quebec AI Institute（米拉魁北克人工智能研究所）
Many natural language processing (NLP) tasks make use of massively pre-trained language models, which are computationally expensive. However, access to high computational resources added to the issue of data scarcity of African languages constitutes a real barrier to research experiments on these languages. In this work, we explore the applicability of low-compute approaches such as language adapters in the context of this low-resource double-bind. We intend to answer the following question: do language adapters allow those who are doubly bound by data and compute to practically build useful m…
University of Pretoria（比勒陀利亚大学）；CSIR（科学与工业研究理事会）；University of Zululand（祖鲁兰大学）；University of Cape Town（开普敦大学）；University of Limpopo（林波波大学）；North-West University（西北大学）；University of the Witwatersrand（威特沃特斯兰德大学）
The recent advances in Natural Language Processing have been a boon for well-represented languages in terms of available curated data and research resources. One of the challenges for low-resourced languages is clear guidelines on the collection, curation and preparation of datasets for different use-cases. In this work, we take on the task of creation of two datasets that are focused on news headlines (i.e short text) for Setswana and Sepedi and creation of a news topic classification task. We document our work and also present baselines for classification. We investigate an approach on data …
LMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Models
Ludwig-Maximilians-University (LMU)（路德维希-马克西米利安大学（慕尼黑大学））；Institute for Information and Language Processing (CIS)（信息与语言处理研究所）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Identification of the languages written using cuneiform symbols is a difficult task due to the lack of resources and the problem of tokenization. The Cuneiform Language Identification task in VarDial 2019 addresses the problem of identifying seven languages and dialects written in cuneiform; Sumerian and six dialects of Akkadian language: Old Babylonian, Middle Babylonian Peripheral, Standard Babylonian, Neo-Babylonian, Late Babylonian, and Neo-Assyrian. This paper describes the approaches taken by SharifCL team to this problem in VarDial 2019. The best result belongs to an ensemble of Support…
Boston College（波士顿学院）；University of California, Davis（加州大学戴维斯分校）
Many automatic speech recognition (ASR) data sets include a single pre-defined test set consisting of one or more speakers whose speech never appears in the training set. This "hold-speaker(s)-out" data partitioning strategy, however, may not be ideal for data sets in which the number of speakers is very small. This study investigates ten different data split methods for five languages with minimal ASR training resources. We find that (1) model performance varies greatly depending on which speaker is selected for testing; (2) the average word error rate (WER) across all held-out speakers is co…
Queen’s University（女王大学）；iFLYTEK Research（科大讯飞研究院）；University of Science and Technology of China（中国科学技术大学）
Natural language inference (NLI) is among the most challenging tasks in natural language understanding. Recent work on unsupervised pretraining that leverages unsupervised signals such as language-model and sentence prediction objectives has shown to be very effective on a wide range of NLP problems. It would still be desirable to further understand how it helps NLI; e.g., if it learns artifacts in data annotation or instead learn true inference knowledge. In addition, external knowledge that does not exist in the limited amount of NLI training data may be added to NLI models in two typical wa…
Institute of Mathematics and Computer Science, University of Latvia（拉脱维亚大学数学与计算机科学研究所）
In this paper we present an ongoing research investigating the possibility and potential of integrating frame semantics, particularly FrameNet, in the Grammatical Framework (GF) application grammar development. An important component of GF is its Resource Grammar Library (RGL) that encapsulates the low-level linguistic knowledge about morphology and syntax of currently more than 20 languages facilitating rapid development of multilingual applications. In the ideal case, porting a GF application grammar to a new language would only require introducing the domain lexicon - translation equivalent…
The Less the Merrier? Investigating Language Representation in Multilingual Models
University of California Berkeley（加州大学伯克利分校）；University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）；Instituto Politécnico Nacional（墨西哥国立理工学院）；Lelapa AI（莱拉帕人工智能公司）
For languages with no annotated resources, transferring knowledge from rich-resource languages is an effective solution for named entity recognition (NER). While all existing methods directly transfer from source-learned model to a target language, in this paper, we propose to fine-tune the learned model with a few similar examples given a test case, which could benefit the prediction by leveraging the structural and semantic information conveyed in such similar examples. To this end, we present a meta-learning algorithm to find a good model parameter initialization that could fast adapt to th…
MetaMT,a MetaLearning Method Leveraging Multiple Domain Data for Low Resource Machine Translation
School of Computer Science, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校计算机科学学院）；Department of Computer Science, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校计算机科学系）；Bedford Veterans Affairs Medical Center（贝德福德退伍军人事务医疗中心）；Department of Medicine, University of Massachusetts Medical School（马萨诸塞大学医学院医学系）
As language data and associated technologies proliferate and as the language resources community expands, it is becoming increasingly difficult to locate and reuse existing resources. Are there any lexical resources for such-and-such a language? What tool works with transcripts in this particular format? What is a good format to use for linguistic data of this type? Questions like these dominate many mailing lists, since web search engines are an unreliable way to find language resources. This paper reports on a new digital infrastructure for discovering language resources being developed by t…
École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Tsinghua University（清华大学）；Hangzhou Dianzi University（杭州电子科技大学）
Natural language generation (NLG) is an essential component of task-oriented dialogue systems. Despite the recent success of neural approaches for NLG, they are typically developed for particular domains with rich annotated training examples. In this paper, we study NLG in a low-resource setting to generate sentences in new scenarios with handful training examples. We formulate the problem from a meta-learning perspective, and propose a generalized optimization-based approach (Meta-NLG) based on the well-recognized model-agnostic meta-learning (MAML) algorithm. Meta-NLG defines a set of meta t…
Indiana University Bloomington（印第安纳大学伯明顿分校）；Brigham Young University（杨百翰大学）；Northwestern University（西北大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Allen Institute for AI（艾伦人工智能研究所）；Shanghai Jiaotong Univ.（上海交通大学）
Multilingual transformers (XLM, mT5) have been shown to have remarkable transfer skills in zero-shot settings. Most transfer studies, however, rely on automatically translated resources (XNLI, XQuAD), making it hard to discern the particular linguistic knowledge that is being transferred, and the role of expert annotated monolingual datasets when developing task-specific models. We investigate the cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference (NLI), with a focus on the recent large-scale Chinese dataset OCNLI. To better understand linguistic trans…
Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an under-studied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages, bringing additional challenges for transfer onto extremely low-resource languages. …
The University of Hong Kong（香港大学）；New York University（纽约大学）
In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML) for low-resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation~\citep{gu2018universal} to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, …
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Stanford University（斯坦福大学）；University of Science and Technology of China（中国科学技术大学）；Carnegie Mellon University（卡内基梅隆大学）
Meta-learning has achieved great success in leveraging the historical learned knowledge to facilitate the learning process of the new task. However, merely learning the knowledge from the historical tasks, adopted by current meta-learning algorithms, may not generalize well to testing tasks when they are not well-supported by training tasks. This paper studies a low-resource text classification problem and bridges the gap between meta-training and meta-testing tasks by leveraging the external knowledge bases. Specifically, we propose KGML to introduce additional representation for each sentenc…
Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning
Mila - Quebec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；The University of Tokyo（东京大学）；Western University（西方大学）；Vector Institute（向量研究所）；Polytechnique Montréal（蒙特利尔理工学院）；CIFAR AI Chair（CIFAR人工智能主席）；AIST（日本产业技术综合研究所）
As language data and associated technologies proliferate and as the language resources community rapidly expands, it has become difficult to locate and reuse existing resources. Are there any lexical resources for such-and-such a language? What tool can work with transcripts in this particular format? What is a good format to use for linguistic data of this type? Questions like these dominate many mailing lists, since web search engines are an unreliable way to find language resources. This paper describes a new digital infrastructure for language resource discovery, based on the Open Archives…
In this paper, we proposed to apply meta learning approach for low-resource automatic speech recognition (ASR). We formulated ASR for different languages as different tasks, and meta-learned the initialization parameters from many pretraining languages to achieve fast adaptation on unseen target language, via recently proposed model-agnostic meta learning algorithm (MAML). We evaluated the proposed approach using six languages as pretraining tasks and four languages as target tasks. Preliminary results showed that the proposed method, MetaASR, significantly outperforms the state-of-the-art mul…
Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
Technical University of Munich, Germany（慕尼黑技术大学，德国）；Helmholtz Munich, Munich Center for Machine Learning, Germany（海德堡慕尼黑，慕尼黑机器学习中心，德国）；University of Tübingen, Tübingen AI Center, Germany（图宾根大学，图宾根人工智能中心，德国）；University of Trento, Italy（特伦托大学，意大利）；Beijing University of Posts and Telecommunications, China（北京邮电大学，中国）
Institute for Natural Language Processing（自然语言处理研究所）；University of Stuttgart（斯图加特大学）
While neural text-to-speech systems perform remarkably well in high-resource scenarios, they cannot be applied to the majority of the over 6,000 spoken languages in the world due to a lack of appropriate training data. In this work, we use embeddings derived from articulatory vectors rather than embeddings derived from phoneme identities to learn phoneme representations that hold across languages. In conjunction with language agnostic meta learning, this enables us to fine-tune a high-quality text-to-speech model on just 30 minutes of data in a previously unseen language spoken by a previously…
NAVER；Korea University（高丽大学）；KAIST（韩国科学技术院）；Upstage AI Research（Upstage AI 研究院）
Unsupervised machine translation, which utilizes unpaired monolingual corpora as training data, has achieved comparable performance against supervised machine translation. However, it still suffers from data-scarce domains. To address this issue, this paper presents a novel meta-learning algorithm for unsupervised neural machine translation (UNMT) that trains the model to adapt to another domain by utilizing only a small amount of training data. We assume that domain-general knowledge is a significant factor in handling data-scarce domains. Hence, we extend the meta-learning algorithm, which u…
We propose a new meta learning based framework for low resource speech recognition that improves the previous model agnostic meta learning (MAML) approach. The MAML is a simple yet powerful meta learning approach. However, the MAML presents some core deficiencies such as training instabilities and slower convergence speed. To address these issues, we adopt multi-step loss (MSL). The MSL aims to calculate losses at every step of the inner loop of MAML and then combines them with a weighted importance vector. The importance vector ensures that the loss at the last step has more importance than t…
