CoLLD：用于压缩多语言预训练语音编码器的对比层到层蒸馏
CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders
AI总结：
提出对比层到层蒸馏（CoLLD）方法，利用掩码预测和对比学习压缩多语言预训练语音编码器，在多语言语音翻译和识别基准上优于先前方法，缩小了小型与大型模型的差距。
摘要：
大规模自监督预训练语音编码器在语音识别和翻译任务中优于传统方法。由于开发这些大型模型的成本高昂，为新任务构建新编码器并将其部署到设备端应用是不可行的。先前的研究提出了模型压缩方法来解决这一问题，但这些工作侧重于较小的模型和不太现实的任务。因此，我们提出了对比层到层蒸馏（CoLLD），一种新颖的知识蒸馏方法，通过利用掩码预测和对比学习来训练学生模型模仿大型教师模型的行为，从而压缩预训练语音编码器。CoLLD在多语言语音到文本翻译和识别基准上优于先前的方法，并缩小了小型与大型模型之间的差距。
英文摘要：
Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new tasks and deploying them to on-device applications are infeasible. Prior studies propose model compression methods to address this issue, but those works focus on smaller models and less realistic tasks. Thus, we propose Contrastive Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to compress pre-trained speech encoders by leveraging masked prediction and contrastive learning to train student models to copy the behavior of a large teacher model. CoLLD outperforms prior methods and closes the gap between small and large models on multilingual speech-to-text translation and recognition benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
