SKILL：面向语音自监督学习的相似性感知知识蒸馏
SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
AI总结：
SKILL提出基于层相似性层次聚类的跨层组知识蒸馏方法，用于压缩语音自监督模型，在30M参数级别多个SUPERB任务上超越DPHuBERT并达到最先进性能。
摘要：
自监督学习（SSL）已在各类语音处理任务中取得显著成功。为提升其效率，先前工作通常采用压缩技术。近期一项值得关注的尝试是DPHuBERT，它联合应用知识蒸馏（KD）与结构化剪枝，学习得到显著更小的SSL模型。本文通过提出SKILL为该研究领域作出贡献，这是一种新方法，它在教师网络内部跨层组进行蒸馏，而非对任意选取的单个层进行蒸馏。待蒸馏层的识别通过应用于层相似性度量的层次聚类过程实现。大量实验表明，我们蒸馏得到的WavLM Base+版本不仅优于DPHuBERT，还在多个SUPERB任务上于30M参数模型类别中取得了最先进的结果。
英文摘要：
Self-supervised learning (SSL) has achieved remarkable success across various speech-processing tasks. To enhance its efficiency, previous works often leverage the use of compression techniques. A notable recent attempt is DPHuBERT, which applies joint knowledge distillation (KD) and structured pruning to learn a significantly smaller SSL model. In this paper, we contribute to this research domain by introducing SKILL, a novel method that conducts distillation across groups of layers instead of distilling individual arbitrarily selected layers within the teacher network. The identification of the layers to distill is achieved through a hierarchical clustering procedure applied to layer similarity measures. Extensive experiments demonstrate that our distilled version of WavLM Base+ not only outperforms DPHuBERT but also achieves state-of-the-art results in the 30M parameters model class across several SUPERB tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
