UNFUSED:基于自监督蒸馏的无监督微调
UNFUSED: UNsupervised Finetuning Using SElf supervised Distillation
AI总结:
提出UnFuSeD方法,通过自监督预训练编码器生成伪标签并指导随机初始化模型的自蒸馏以实现无监督微调,在低资源音频分类任务上以更少参数取得最优性能。
摘要:
本文介绍了UnFuSeD,这是一种利用自监督学习并减少音频分类对大量标注数据需求的新方法。与以往直接在目标数据集上微调自监督预训练编码器的工作不同,我们在实际微调步骤之前,先使用编码器生成伪标签以进行无监督微调。我们首先使用一种新颖的自监督学习算法(SSL)在无标签音频数据集上训练编码器。然后,通过聚类提取的特征表示,利用该编码器在目标任务数据集上生成伪标签。这些伪标签随后被用于指导随机初始化模型的自蒸馏,我们将其称为无监督微调。最后,将得到的编码器在目标任务数据集上进行微调。通过UnFuSeD,我们提出了首个摆脱文献中预训练和微调同一编码器的通用SSL范式的系统,并展示了一种基于自蒸馏的新颖系统,以利用SSL预训练进行低资源音频分类。在实践中,UnFuSeD在LAPE Benchmark上取得了最先进的结果,显著优于所有基线方法。此外,与之前的最先进系统相比,UnFuSeD在参数量减少40%的情况下实现了这一目标。我们公开了所有代码。
英文摘要:
In this paper, we introduce UnFuSeD, a novel approach to leverage self-supervised learning and reduce the need for large amounts of labeled data for audio classification. Unlike prior works, which directly fine-tune a self-supervised pre-trained encoder on a target dataset, we use the encoder to generate pseudo-labels for unsupervised fine-tuning before the actual fine-tuning step. We first train an encoder using a novel self-supervised learning algorithm (SSL) on an unlabeled audio dataset. Then, we use that encoder to generate pseudo-labels on our target task dataset via clustering the extracted representations. These pseudo-labels are then used to guide self-distillation on a randomly initialized model, which we call unsupervised fine-tuning. Finally, the resultant encoder is then fine-tuned on our target task dataset. Through UnFuSeD, we propose the first system that moves away from generic SSL paradigms in literature, which pre-train and fine-tune the same encoder, and present a novel self-distillation-based system to leverage SSL pre-training for low-resource audio classification. In practice, UnFuSeD achieves state-of-the-art results on the LAPE Benchmark, significantly outperforming all our baselines. Additionally, UnFuSeD allows us to achieve this at a 40% reduction in the number of parameters over the previous state-of-the-art system. We make all our codes publicly available.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。