A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models
LMU Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Instituto Superior Técnico, Universidade de Lisboa（里斯本大学高等技术学院）；Instituto de Telecomunicações（电信研究所）；Unbabel（昂巴贝尔公司）
Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models, improving performance in both bilingual tasks, e.g., machine translation, and general-purpose tasks, e.g., text classification. Building upon these findings, our comprehensive study aims to identify the most effective strategies for leveraging parallel corpora. We investigate the impact of parallel corpora quality and quantity, training objectives, and model size on the performance of multilingual large language models enhanced with parallel corpora across diverse languages and tasks. Our analysis reveals several key insights: (i) filtering noisy translations is essential for effectively exploiting parallel corpora, while language identification and short sentence filtering have little effect; (ii) even a corpus with just 10K parallel sentences can yield results comparable to those obtained from much larger datasets; (iii) employing only the machine translation objective yields the best results among various training objectives and their combinations; (iv) larger multilingual language models benefit more from parallel corpora than smaller models. Our study offers valuable insights into the optimal utilization of parallel corpora to enhance multilingual large language models, extending the generalizability of previous findings from limited languages and tasks to a broader range of scenarios.
Pretrained language models are typically trained on massive web-based datasets, which are often "contaminated" with downstream test sets. It is not clear to what extent models exploit the contaminated data for downstream tasks. We present a principled method to study this question. We pretrain BERT models on joint corpora of Wikipedia and labeled downstream datasets, and fine-tune them on the relevant task. Comparing performance between samples seen and unseen during pretraining enables us to define and quantify levels of memorization and exploitation. Experiments with two models and three dow…
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；KTH Royal Institute of Technology（皇家理工学院）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
National Center for AI and Big Data Technology（国家人工智能与大数据技术中心）；King Abdulaziz City for Science and Technology (KACST)（阿卜杜勒阿齐兹国王科技城（KACST））
In this paper, we present a recipe for building a good Arabic-English neural machine translation. We compare neural systems with traditional phrase-based systems using various parallel corpora including UN, ISI and Ummah. We also investigate the importance of special preprocessing of the Arabic script. The presented results are based on test sets from NIST MT 2005 and 2012. The best neural system produces a gain of +13 BLEU points compared to an equivalent simple phrase-based system in NIST MT12 test set. Unexpectedly, we find that tuning a model trained on the whole data using a small high qu…
ADAPT Research Centre（ADAPT研究中心）；Insight Centre for Data Analytics（洞察数据分析中心）；Dublin City University（都柏林城市大学）
Multi-word expressions (MWEs) are a hot topic in research in natural language processing (NLP), including topics such as MWE detection, MWE decomposition, and research investigating the exploitation of MWEs in other NLP fields such as Machine Translation. However, the availability of bilingual or multi-lingual MWE corpora is very limited. The only bilingual MWE corpora that we are aware of is from the PARSEME (PARSing and Multi-word Expressions) EU Project. This is a small collection of only 871 pairs of English-German MWEs. In this paper, we present multi-lingual and bilingual MWE corpora tha…
Models trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior and unwanted biases. We investigate a variety of methods to mitigate these issues in the context of open-domain generative dialogue models. We introduce a new human-and-model-in-the-loop framework for both training safer models and for evaluating them, as well as a novel method to distill safety considerations inside generative models without the use of an external classifier at deployment time. We conduct experiments comparing the…
Cookpad Ltd.（Cookpad有限公司）；The University of Liverpool（利物浦大学）
Cross-lingual text representations have gained popularity lately and act as the backbone of many tasks such as unsupervised machine translation and cross-lingual information retrieval, to name a few. However, evaluation of such representations is difficult in the domains beyond standard benchmarks due to the necessity of obtaining domain-specific parallel language data across different pairs of languages. In this paper, we propose an automatic metric for evaluating the quality of cross-lingual textual representations using images as a proxy in a paired image-text evaluation dataset. Experiment…
DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing
VGG, Dept. of Engineering Science, University of Oxford（视觉感知与计算实验室，工程科学系，牛津大学）；University of Zurich（苏黎世大学）；KAIST（韩国科学技术院）；LIGM, CNRS, Univ Gustave Eiffel, ENPC, IP Paris（LIGM，国家科学研究中心，古斯塔夫·埃菲尔大学，巴黎理工大学，IP巴黎）
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Allen Institute for AI（艾伦人工智能研究所）；University of California, Berkeley（加州大学伯克利分校）；Carnegie Mellon University（卡内基梅隆大学）；Spiffy AI（Spiffy AI公司）；Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）
Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO
National Tsing Hua University（台湾清华大学）；Accton Technology Corporation（智邦科技公司）；National Yang Ming Chiao Tung University（阳明交通大学）；University of Taipei（台北市立大学）
LongForm: Effective Instruction Tuning with Reverse Instructions
Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
McGill University（麦吉尔大学）；Mila-Quebec AI Institute（魁北克AI研究所）；LMU Munich & Munich Center for Machine Learning（慕尼黑大学及慕尼黑机器学习中心）；Microsoft AI for Good Research Lab（微软AI for Good研究实验室）
SignDATA: Data Pipeline for Sign Language Translation
Computer Science and Information Engineering National Central University（计算机科学与信息工程国家级中央大学）；Electrical Engineering National Changhua University Of Education（电气工程国家彰化教育大学）
Recently, there is increasing interest in multilingual automatic speech recognition (ASR) where a speech recognition system caters to multiple low resource languages by taking advantage of low amounts of labeled corpora in multiple languages. With multilingualism becoming common in today's world, there has been increasing interest in code-switching ASR as well. In code-switching, multiple languages are freely interchanged within a single sentence or between sentences. The success of low-resource multilingual and code-switching ASR often depends on the variety of languages in terms of their aco…
