DistilXLSR：一种轻量级跨语言语音表示模型
DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model
AI总结：
本文提出轻量级蒸馏模型 DistilXLSR，通过音素随机打乱和跳层初始化，仅用英语数据压缩跨语言语音表示模型，在15种低资源语言上实现50%参数削减并保持表示能力。
摘要：
多语言自监督语音表示模型已大幅提升了低资源语言的语音识别性能，而压缩这些庞大模型也成为其工业应用的关键前提。本文提出 DistilXLSR，这是一种蒸馏后的跨语言语音表示模型。通过随机打乱现有语音中的音素，我们减少语言信息，并仅使用英语数据蒸馏跨语言模型。我们还设计了一种跳层初始化方法，以充分利用教师模型的预训练权重。在2类教师模型和15种低资源语言上的实验表明，我们的方法可将参数量减少50%，同时保持跨语言表示能力。该方法被证明可泛化到多种语言和教师模型，并具有提升英语预训练模型跨语言性能的潜力。
英文摘要：
Multilingual self-supervised speech representation models have greatly enhanced the speech recognition performance for low-resource languages, and the compression of these huge models has also become a crucial prerequisite for their industrial application. In this paper, we propose DistilXLSR, a distilled cross-lingual speech representation model. By randomly shuffling the phonemes of existing speech, we reduce the linguistic information and distill cross-lingual models using only English data. We also design a layer-jumping initialization method to fully leverage the teacher's pre-trained weights. Experiments on 2 kinds of teacher models and 15 low-resource languages show that our method can reduce the parameters by 50% while maintaining cross-lingual representation ability. Our method is proven to be generalizable to various languages/teacher models and has the potential to improve the cross-lingual performance of the English pre-trained models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
