Mu$^{2}$SLAM：多任务、多语言语音与语言模型
Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language Models
AI总结：
提出多任务多语言模型Mu$^{2}$SLAM，联合预训练语音与文本并利用监督任务对齐跨模态表示，在多语言语音翻译、识别及文本理解任务上取得显著提升。
摘要：
我们提出了Mu$^{2}$SLAM，这是一个在超过100种语言的无标签语音、无标签文本以及涵盖自动语音识别（ASR）、自动语音翻译（AST）和机器翻译（MT）的监督数据上联合预训练的多语言序列到序列模型。通过利用量化的语音表示作为目标，Mu$^{2}$SLAM在无标签语音和文本上，于解码器端使用类似于T5的序列到序列掩码去噪目标，于编码器端使用掩码语言建模（MLM）目标来训练语音文本模型，同时利用监督任务来改善模型内部的跨语言和跨模态表示对齐。在CoVoST AST上，Mu$^{2}$SLAM为在公开数据集上训练的模型建立了新的最先进水平，在xx-en翻译上比之前的最佳结果提高了1.9个BLEU点，在en-xx翻译上提高了1.1个BLEU点。在Voxpopuli ASR上，尽管使用了相对较弱的序列到序列架构，我们的模型仍匹配了使用RNN-T解码器微调的mSLAM模型的性能。在文本理解任务上，我们的模型在XNLI上比mSLAM提高了6%以上，在XNLI和TydiQA上逐渐接近同等容量的mT5模型的性能，为所有语音和文本理解任务走向单一模型铺平了道路。
英文摘要：
We present Mu$^{2}$SLAM, a multilingual sequence-to-sequence model pre-trained jointly on unlabeled speech, unlabeled text and supervised data spanning Automatic Speech Recognition (ASR), Automatic Speech Translation (AST) and Machine Translation (MT), in over 100 languages. By leveraging a quantized representation of speech as a target, Mu$^{2}$SLAM trains the speech-text models with a sequence-to-sequence masked denoising objective similar to T5 on the decoder and a masked language modeling (MLM) objective on the encoder, for both unlabeled speech and text, while utilizing the supervised tasks to improve cross-lingual and cross-modal representation alignment within the model. On CoVoST AST, Mu$^{2}$SLAM establishes a new state-of-the-art for models trained on public datasets, improving on xx-en translation over the previous best by 1.9 BLEU points and on en-xx translation by 1.1 BLEU points. On Voxpopuli ASR, our model matches the performance of an mSLAM model fine-tuned with an RNN-T decoder, despite using a relatively weaker sequence-to-sequence architecture. On text understanding tasks, our model improves by more than 6\% over mSLAM on XNLI, getting closer to the performance of mT5 models of comparable capacity on XNLI and TydiQA, paving the way towards a single model for all speech and text understanding tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
