语音翻译的跨语言迁移学习
Cross-Lingual Transfer Learning for Speech Translation
AI总结：
针对数据受限下扩展多语言基础模型的语音翻译能力问题，以Whisper为例，利用语音检索揭示其共享语义空间，通过仅英中数据微调解码器，实现零样本跨语言迁移并提升多语言到中文的翻译性能。
摘要：
自然语言处理和语音研究中构建多语言基础模型的兴趣日益浓厚。本文探讨了如何在数据受限的情况下扩展这些模型的语音翻译能力，以Whisper（在语音识别和英语翻译上表现出色的语音基础模型）为例。通过语音到语音检索分析编码器生成的音频表示，我们发现不同语言的话语被映射到共享语义空间。该共享嵌入空间可用于语音翻译的零样本跨语言迁移：仅用英中语音翻译数据微调Whisper解码器，除英语外，多种语言到中文的翻译性能均获提升；对于训练中未见过或无法转录的相关语言，也能实现语音翻译。
英文摘要：
There has been increasing interest in building multilingual foundation models for NLP and speech research. This paper examines how to expand the speech translation capability of these models with restricted data. Whisper, a speech foundation model with strong performance on speech recognition and English translation, is used as the example model. Using speech-to-speech retrieval to analyse the audio representations generated by the encoder, we show that utterances from different languages are mapped to a shared semantic space. This shared embedding space can then be leveraged for zero-shot cross-lingual transfer in speech translation. By fine-tuning the Whisper decoder with only English-to-Chinese speech translation data, improved performance for translation to Chinese can be obtained for multiple languages, in addition to English. Furthermore, for languages related to those seen in training it is possible to perform speech translation, despite the model never seeing the language in training, or being able to perform transcription.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
