基于自监督和弱监督的近期基础模型在汉英语码转换ASR和语音到文本翻译中的零样本泛化能力研究
Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision
AI总结：
本研究在三个语码转换语料库上评估了SeamlessM4T和Whisper等前沿基础模型，发现自监督模型接近监督模型性能，但句内语码转换表现仍有待提升，且Whisper变体在语码转换场景中保持有效。
摘要：
本工作在三个语码转换语料库上评估了基于自监督或弱监督的多个前沿大规模基础模型，包括SeamlessM4T、SeamlessM4T v2和Whisper-large-v3。我们发现自监督模型能达到接近监督模型的性能，表明了多语言自监督预训练的有效性。我们还观察到，这些模型仍有改进空间，因为它们不断犯类似错误，且在建模句内语码转换方面表现不佳。此外，探索了Whisper多个变体的有效性，我们得出结论，它们在语码转换场景中仍然有效，且用于自监督模型的类似技术值得研究以提升语码转换任务的性能。
英文摘要：
This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found that self-supervised models can achieve performances close to the supervised model, indicating the effectiveness of multilingual self-supervised pre-training. We also observed that these models still have room for improvement as they kept making similar mistakes and had unsatisfactory performances on modeling intra-sentential code-switching. In addition, the validity of several variants of Whisper was explored, and we concluded that they remained effective in a code-switching scenario, and similar techniques for self-supervised models are worth studying to boost the performance of code-switched tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
