The adaption of multilingual pre-trained LLMs into eloquent and helpful assistants is essential to facilitate their use across different language regions. In that spirit, we are the first to conduct an extensive study of the performance of multilingual models instruction-tuned on different language compositions on parallel instruction-tuning benchmarks across a selection of the most spoken Indo-European languages. We systematically examine the effects of language and instruction dataset size on a mid-sized and a large, multilingual LLMs by instruction-tuning them on parallel instruction-tuning datasets. Our results demonstrate that instruction-tuning on parallel instead of monolingual corpora benefits cross-lingual instruction following capabilities by up to 9.9%. Furthermore, we show that the Superficial Alignment Hypothesis does not hold in general, as the investigated multilingual 7B parameter model presents a counter-example requiring large-scale instruction-tuning datasets. Finally, we conduct a human annotation study to understand the alignment between human-based and GPT-4-based evaluation within multilingual chat scenarios.
This paper aims for a potential architectural improvement for multilingual learning and asks: Can different tasks from different languages be modeled in a monolithic framework, i.e. without any task/language-specific module? The benefit of achieving this could open new doors for future multilingual research, including allowing systems trained on low resources to be further assisted by other languages as well as other tasks. We approach this goal by developing a learning framework named Polyglot Prompting to exploit prompting methods for learning a unified semantic space for different languages…
Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this simple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consistently underperforms models trained on monolingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to t…
Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心（CIS））
Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT's performance as knowledge base language-independ…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages. Our method combines the advantages of contextual word representations with those of multilingual representation learning. We produce language models from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, semantic role labeling, and named entity recognition, with comparisons to monolingual and non-contextual variants. Our results provide further evidence for the benefits of polyglot learning, i…
This article investigates multilingual evidence retrieval and fact verification as a step to combat global disinformation, a first effort of this kind, to the best of our knowledge. The goal is building multilingual systems that retrieve in evidence-rich languages to verify claims in evidence-poor languages that are more commonly targeted by disinformation. To this end, our EnmBERT fact verification system shows evidence of transfer learning ability and 400 example mixed English-Romanian dataset is made available for cross-lingual transfer learning evaluation.
University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）
Previous approaches to multilingual semantic dependency parsing treat languages independently, without exploiting the similarities between semantic structures across languages. We experiment with a new approach where we combine resources from a pair of languages in the CoNLL 2009 shared task to build a polyglot semantic role labeler. Notwithstanding the absence of parallel data, and the dissimilarity in annotations between languages, our approach results in an improvement in SRL performance on multiple languages over a monolingual baseline. Analysis of the polyglot model shows it to be advanta…
Distributed word representations (word embeddings) have recently contributed to competitive performance in language modeling and several NLP tasks. In this work, we train word embeddings for more than 100 languages using their corresponding Wikipedias. We quantitatively demonstrate the utility of our word embeddings by using them as the sole features for training a part of speech tagger for a subset of these languages. We find their performance to be competitive with near state-of-art methods in English, Danish and Swedish. Moreover, we investigate the semantic features captured by these embed…
University of Information Technology（信息科技大学）；Vietnam National University（越南国家大学）
Aspect category detection (ACD) is one of the challenging tasks in the Aspect-based sentiment Analysis problem. The purpose of this task is to identify the aspect categories mentioned in user-generated reviews from a set of pre-defined categories. In this paper, we investigate the performance of various monolingual pre-trained language models compared with multilingual models on the Vietnamese aspect category detection problem. We conduct the experiments on two benchmark datasets for the restaurant and hotel domain. The experimental results demonstrated the effectiveness of the monolingual Pho…
Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain Setups
National University of Science and Technology POLITEHNICA Bucharest（布加勒斯特理工国立大学）；National Institute for Research & Development in Informatics - ICI Bucharest（布加勒斯特国家信息学研究与发展研究所）；Academy of Romanian Scientists（罗马尼亚科学院）；Paris 1 Panthéon-Sorbonne University（巴黎第一大学先贤祠-索邦大学）；University of Bucharest（布加勒斯特大学）
The increasing diversity of languages used on the web introduces a new level of complexity to Information Retrieval (IR) systems. We can no longer assume that textual content is written in one language or even the same language family. In this paper, we demonstrate how to build massive multilingual annotators with minimal human expertise and intervention. We describe a system that builds Named Entity Recognition (NER) annotators for 40 major languages using Wikipedia and Freebase. Our approach does not require NER human annotated datasets or language specific resources like treebanks, parallel…
Investigating the Multilingual Calibration Effects of Language Model Instruction-Tuning
Mila - Quebec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；The University of Tokyo（东京大学）；Western University（西方大学）；Vector Institute（向量研究所）；Polytechnique Montréal（蒙特利尔理工学院）；CIFAR AI Chair（CIFAR人工智能主席）；AIST（日本产业技术综合研究所）
On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning
Department of Computer Science, TU Dortmund University（图恩-多特蒙德大学计算机科学系）；Bonn-Aachen International Center for Information Technology, University of Bonn（波恩-亚琛国际信息科技中心，波恩大学）；Computer Science School, University of Sheffield（谢菲尔德大学计算机科学学院）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔人工智能与机器学习研究所）
Indiana University Bloomington（印第安纳大学伯明顿分校）；Brigham Young University（杨百翰大学）；Northwestern University（西北大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；Allen Institute for AI（艾伦人工智能研究所）；Shanghai Jiaotong Univ.（上海交通大学）
Multilingual transformers (XLM, mT5) have been shown to have remarkable transfer skills in zero-shot settings. Most transfer studies, however, rely on automatically translated resources (XNLI, XQuAD), making it hard to discern the particular linguistic knowledge that is being transferred, and the role of expert annotated monolingual datasets when developing task-specific models. We investigate the cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference (NLI), with a focus on the recent large-scale Chinese dataset OCNLI. To better understand linguistic trans…
Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Department of Linguistics, Harvard University（哈佛大学语言学系）；Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University（哈佛大学凯普纳自然与人工智能研究 institute）
An essential design decision for multilingual Neural Text-To-Speech (NTTS) systems is how to represent input linguistic features within the model. Looking at the wide variety of approaches in the literature, two main paradigms emerge, unified and separate representations. The former uses a shared set of phonetic tokens across languages, whereas the latter uses unique phonetic tokens for each language. In this paper, we conduct a comprehensive study comparing multilingual NTTS systems models trained with both representations. Our results reveal that the unified approach consistently achieves be…
Evaluating Zero-Shot Multilingual Aspect-Based Sentiment Analysis with Large Language Models
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials, School of Electronic Science and Engineering (School of Microelectronics)（广东量子工程与材料重点实验室，电子科学与工程学院（微电子学院））；South China Normal University（华南师范大学）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东智能信息处理重点实验室）；Ludwig Maximilian University of Munich（慕尼黑路德维希-马克西米利安大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Technical University of Munich（慕尼黑技术大学）
Salesforce Research Asia（赛富时亚洲研究院）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Multilingual models have demonstrated impressive cross-lingual transfer performance. However, test sets like XNLI are monolingual at the example level. In multilingual communities, it is common for polyglots to code-mix when conversing with each other. Inspired by this phenomenon, we present two strong black-box adversarial attacks (one word-level, one phrase-level) for multilingual models that push their ability to handle code-mixed sentences to the limit. The former uses bilingual dictionaries to propose perturbations and translations of the clean example for sense disambiguation. The latter…
Training multilingual Neural Text-To-Speech (NTTS) models using only monolingual corpora has emerged as a popular way for building voice cloning based Polyglot NTTS systems. In order to train these models, it is essential to understand how the composition of the training corpora affects the quality of multilingual speech synthesis. In this context, it is common to hear questions such as "Would including more Spanish data help my Italian synthesis, given the closeness of both languages?". Unfortunately, we found existing literature on the topic lacking in completeness in this regard. In the pre…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Despite advances in dependency parsing, languages with small treebanks still present challenges. We assess recent approaches to multilingual contextual word representations (CWRs), and compare them for crosslingual transfer from a language with a large treebank to a language with a small or nonexistent treebank, by sharing parameters between languages in the parser itself. We experiment with a diverse selection of languages in both simulated and truly low-resource scenarios, and show that multilingual CWRs greatly facilitate low-resource dependency parsing even without crosslingual supervision…
Breaking Boundaries: Investigating the Effects of Model Editing on Cross-linguistic Performance
Indian Institute of Technology Kharagpur（印度理工学院克勒格布尔分校）；National Institute of Standards and Technology（美国国家标准与技术研究院）；INSAIT（保加利亚索菲亚大学INSAIT研究院）；Sofia University “St. Kliment Ohridski”（圣基里曼·奥赫里德斯基索菲亚大学）
PALO: A Polyglot Large Multimodal Model for 5B People
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；Aalto University（阿尔托大学）；The University of Melbourne（墨尔本大学）；Linköping University（林雪平大学）
Counter Turing Test ($CT^2$): Investigating AI-Generated Text Detection for Hindi -- Ranking LLMs based on Hindi AI Detectability Index ($ADI_{hi}$)
International Institute of Information Technology, Hyderabad（国际信息技术学院（海得拉巴））；Massachusetts Institute of Technology（麻省理工学院）；AI Institute, University of South Carolina（南卡罗来纳大学人工智能学院）
Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision
National Taiwan University（国立台湾大学）；ASUS Intelligent Cloud Services（华硕智能云服务）
University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；San Diego State University（圣地亚哥州立大学）
This paper describes our system for the SemEval2022 task of matching dictionary glosses to word embeddings. We focus on the Reverse Dictionary Track of the competition, which maps multilingual glosses to reconstructed vector representations. More specifically, models convert the input of sentences to three types of embeddings: SGNS, Char, and Electra. We propose several experiments for applying neural network cells, general multilingual and multitask structures, and language-agnostic tricks to the task. We also provide comparisons over different types of word embeddings and ablation studies to…
VinAI Research（VinAI研究院）；The University of Melbourne（墨尔本大学）
We present the first empirical study investigating the influence of disfluency detection on downstream tasks of intent detection and slot filling. We perform this study for Vietnamese -- a low-resource language that has no previous study as well as no public dataset available for disfluency detection. First, we extend the fluent Vietnamese intent detection and slot filling dataset PhoATIS by manually adding contextual disfluencies and annotating them. Then, we conduct experiments using strong baselines for disfluency detection and joint intent detection and slot filling, which are based on pre…
Investigating the Influence of Prompt and Response Languages on LLM Content Generation
Université de Technologie de Compiègne（贡比涅技术大学）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）；SimulaMet（西穆拉梅特研究院）；HUTECH University（胡志明市技术大学）
