本文提出 DistilWhisper，通过语言特定专家的轻量级模块化微调与 whisper-large-v2 知识蒸馏相结合，在保持多任务多语言能力的同时，高效提升低资源语言的 ASR 性能。
摘要：
Whisper 是一个覆盖 99 种语言的多任务多语言语音模型。它在其覆盖语言的一个子集上取得了值得称赞的自动语音识别（ASR）结果，但该模型在不可忽视数量的低资源语言上仍然表现不佳，这一问题在较小模型版本中更加严重。在这项工作中，我们考察了其局限性，证明了与说话人相关的（性别、年龄）和与模型相关的（资源丰富程度和模型规模）偏差的存在。尽管如此，我们表明只有与模型相关的偏差会被量化放大，从而对低资源语言和较小模型产生更大影响。为了寻找更好的压缩方法，我们提出了 DistilWhisper，该方法能够在保持多任务和多语言能力优势的同时，弥合这些语言在 ASR 上的性能差距。我们的方法涉及两个关键策略：使用语言特定专家对 whisper-small 进行轻量级模块化 ASR 微调，以及从 whisper-large-v2 进行知识蒸馏。这种双重方法使我们能够有效提升 ASR 性能，同时保持从多任务和多语言预训练中继承的鲁棒性。结果表明，我们的方法比标准微调或 LoRA 适配器更有效，在域内和域外测试集上均提升了目标语言的性能，同时在推理时仅引入可忽略不计的参数开销。
英文摘要：
Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still underperforms on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we examine its limitations, demonstrating the presence of speaker-related (gender, age) and model-related (resourcefulness and model size) bias. Despite that, we show that only model-related bias are amplified by quantization, impacting more low-resource languages and smaller models. Searching for a better compression approach, we propose DistilWhisper, an approach that is able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.
When learning a new skill, you take advantage of your preexisting skills and knowledge. For instance, if you are a skilled violinist, you will likely have an easier time learning to play cello. Similarly, when learning a new language you take advantage of the languages you already speak. For instance, if your native language is Norwegian and you decide to learn Dutch, the lexical overlap between these two languages will likely benefit your rate of language acquisition. This thesis deals with the intersection of learning multiple tasks and learning multiple languages in the context of Natural L…
Transformer language models (TLMs) are critical for most NLP tasks, but they are difficult to create for low-resource languages because of how much pretraining data they require. In this work, we investigate two techniques for training monolingual TLMs in a low-resource setting: greatly reducing TLM size, and complementing the masked language modeling objective with two linguistically rich supervised tasks (part-of-speech tagging and dependency parsing). Results from 7 diverse languages indicate that our model, MicroBERT, is able to produce marked improvements in downstream task evaluations re…
On Multilingual Encoder Language Model Compression for Low-Resource Languages
Saarland University（萨尔兰州大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Kempelen Institute of Intelligent Technologies (KInIT)（Kempelen智能技术研究所）；Centre for European Research in Trusted AI (CERTAIN)（可信人工智能欧洲研究中心）
This paper aims for a potential architectural improvement for multilingual learning and asks: Can different tasks from different languages be modeled in a monolithic framework, i.e. without any task/language-specific module? The benefit of achieving this could open new doors for future multilingual research, including allowing systems trained on low resources to be further assisted by other languages as well as other tasks. We approach this goal by developing a learning framework named Polyglot Prompting to exploit prompting methods for learning a unified semantic space for different languages…
University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；San Diego State University（圣地亚哥州立大学）
This paper describes our system for the SemEval2022 task of matching dictionary glosses to word embeddings. We focus on the Reverse Dictionary Track of the competition, which maps multilingual glosses to reconstructed vector representations. More specifically, models convert the input of sentences to three types of embeddings: SGNS, Char, and Electra. We propose several experiments for applying neural network cells, general multilingual and multitask structures, and language-agnostic tricks to the task. We also provide comparisons over different types of word embeddings and ablation studies to…
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）；Microsoft Research Redmond（微软研究院（雷德蒙德））
We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-swi…
We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
In Natural Language Processing (NLP), one traditionally considers a single task (e.g. part-of-speech tagging) for a single language (e.g. English) at a time. However, recent work has shown that it can be beneficial to take advantage of relatedness between tasks, as well as between languages. In this work I examine the concept of relatedness and explore how it can be utilised to build NLP models that require less manually annotated data. A large selection of NLP tasks is investigated for a substantial language sample comprising 60 languages. The results show potential for joint multitask and mu…
Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translation
Auditory Cognition and Computational Acoustics Lab MoE Key Lab of Artificial Intelligence, AI Institute School of Computer Science, Shanghai Jiao Tong University, Shanghai, China（听觉认知与计算声学实验室、人工智能国家重点实验室、人工智能学院、上海交通大学）
We propose data and knowledge-driven approaches for multilingual training of the automated speech recognition (ASR) system for a target language by pooling speech data from multiple source languages. Exploiting the acoustic similarities between Indian languages, we implement two approaches. In phone/senone mapping, deep neural network (DNN) learns to map senones or phones from one language to the others, and the transcriptions of the source languages are modified such that they can be used along with the target language data to train and fine-tune the target language ASR system. In the other a…
The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. To overcome the challenges and embrace the opportunities of MoE, we first develop a system capable of scaling MoE models efficiently to tr…
We present Maestro, a self-supervised training method to unify representations learnt from speech and text modalities. Self-supervised learning from speech signals aims to learn the latent structure inherent in the signal, while self-supervised learning from text attempts to capture lexical information. Learning aligned representations from unpaired speech and text sequences is a challenging task. Previous work either implicitly enforced the representations learnt from these two modalities to be aligned in the latent space through multitasking and parameter sharing or explicitly through conver…
VK Lab（VK实验室）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Tinkoff
In this work, we present a novel approach for simultaneous knowledge transfer and model compression called Weight Squeezing. With this method, we perform knowledge transfer from a teacher model by learning the mapping from its weights to smaller student model weights. We applied Weight Squeezing to a pre-trained text classification model based on BERT-Medium model and compared our method to various other knowledge transfer and model compression methods on GLUE multitask benchmark. We observed that our approach produces better results while being significantly faster than other methods for trai…
While few-shot learning as a transfer learning paradigm has gained significant traction for scenarios with limited data, it has primarily been explored in the context of building unimodal and unilingual models. Furthermore, a significant part of the existing literature in the domain of few-shot multitask learning perform in-context learning which requires manually generated prompts as the input, yielding varying outcomes depending on the level of manual prompt-engineering. In addition, in-context learning suffers from substantial computational, memory, and storage costs which eventually leads …
Qatar Computing Research Institute（卡塔尔计算研究所）；Sofia University "St. Kliment Ohridski"（索菲亚大学"圣克莱孟·奥赫里迪斯"）；University of Padova（帕多瓦大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Alibaba Group（阿里巴巴集团）；TU Wien（维也纳技术大学）；University of Chinese Academy of Sciences（中国科学院大学）
M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
Mohamed bin Zayed University of Artificial Intelligence（Mohamed bin Zayed人工智能大学）；Ubiquitous Knowledge Processing Lab（ubiquitous知识处理实验室）；Department of Computer Science, TU Darmstadt（TU Darmstadt计算机科学系）；National Research Center for Applied Cybersecurity ATHENE（应用网络安全国家研究中心ATHENE）；Department of Electrical Engineering, KU Leuven（KU Leuven电气工程系）；Department of Computer Science, KU Leuven（KU Leuven计算机科学系）
University of Waterloo（滑铁卢大学）；University of Washington（华盛顿大学）；Cohere For AI Community（Cohere人工智能社区）；Google Research（谷歌研究院）；Cohere For AI（Cohere人工智能研究院）
Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages. Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for low-resource languages. It is thus crucial to understand the trade-offs between scale, multilingualism, and compression. In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. Applying this framewor…
CreoleVal: Multilingual Multitask Benchmarks for Creoles
Aalborg University（奥尔堡大学）；KU Leuven（鲁汶大学）；National Institute of Information and Communications Technology（日本信息通信研究机构）；University of Copenhagen（哥本哈根大学）；University of Electronic Science and Technology of China（电子科技大学）；Meta；Zamboanga State College of Marine Sciences and Technology（三宝颜加国立海洋科学与技术学院）；University of Newcastle（纽卡斯尔大学）；University of Surrey（萨里大学）；Kean University（肯恩大学）；Linköping University（林雪平大学）；Université Paris Nanterre（巴黎第十大学）；Massachusetts Institute of Technology（麻省理工学院）
Microsoft Egypt Development Center（微软埃及开发中心）；Microsoft Corporation（微软公司）
This paper describes our submission to the constrained track of WMT21 shared news translation task. We focus on the three relatively low resource language pairs Bengali to and from Hindi, English to and from Hausa, and Xhosa to and from Zulu. To overcome the limitation of relatively low parallel data we train a multilingual model using a multitask objective employing both parallel and monolingual data. In addition, we augment the data using back translation. We also train a bilingual model incorporating back translation and knowledge distillation then combine the two models using sequence-to-s…
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval
Islamic University of Technology (IUT)（伊斯兰科技大学）；Nanyang Technological University (NTU)（南洋理工大学）；Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；Bosch Research（博世研究院）；Salesforce Research（赛富时研究院）
The Hong Kong University of Science and Technology（香港科技大学）；Department of Computer Science and Engineering（计算机科学与工程系）
Current research on hate speech analysis is typically oriented towards monolingual and single classification tasks. In this paper, we present a new multilingual multi-aspect hate speech analysis dataset and use it to test the current state-of-the-art multilingual multitask learning approaches. We evaluate our dataset in various classification settings, then we discuss how to leverage our annotations in order to improve hate speech detection and classification in general.
GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek
LIX, Ecole Polytechnique（巴黎综合理工学院LIX实验室）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；National Technical University of Athens（雅典国家技术大学）；University of Ioannina（约阿尼纳大学）；University of Peloponnese（伯罗奔尼撒大学）
In this paper, we propose a robust multilingual model to improve the quality of search results. Our model not only leverage the processed class-balanced dataset, but also benefit from multitask pre-training that leads to more general representations. In pre-training stage, we adopt mlm task, classification task and contrastive learning task to achieve considerably performance. In fine-tuning stage, we use confident learning, exponential moving average method (EMA), adversarial training (FGM) and regularized dropout strategy (R-Drop) to improve the model's generalization and robustness. Moreove…
University of Oslo（奥斯陆大学）；University of Copenhagen（哥本哈根大学）
In order to preserve word-order information in a non-autoregressive setting, transformer architectures tend to include positional knowledge, by (for instance) adding positional encodings to token embeddings. Several modifications have been proposed over the sinusoidal positional encodings used in the original transformer architecture; these include, for instance, separating position encodings and token embeddings, or directly modifying attention weights based on the distance between word pairs. We first show that surprisingly, while these modifications tend to improve monolingual language mode…
Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind
X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China（X-LANCE实验室，计算机科学与工程系，MoE人工智能关键实验室，上海交通大学人工智能研究院，上海，中国）；Suzhou Laboratory, Suzhou, China（苏州实验室，苏州，中国）
We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations. We also provid…
Multilingual Generative Retrieval via Cross-lingual Semantic Compression
Faculty of Information Engineering and Automation, Kunming University of Science and Technology（信息工程与自动化学院，昆明理工大学）；Yunnan Key Laboratory of Artificial Intelligence（云南省人工智能重点实验室）
Leveraging shared learning through Massively Multilingual Models, state-of-the-art machine translation models are often able to adapt to the paucity of data for low-resource languages. However, this performance comes at the cost of significantly bloated models which are not practically deployable. Knowledge Distillation is one popular technique to develop competitive, lightweight models: In this work, we first evaluate its use to compress MT models focusing on languages with extremely limited training data. Through our analysis across 8 languages, we find that the variance in the performance o…
