使用自预训练 Transformer 模型改进说话人验证
Improving Speaker Verification with Self-Pretrained Transformer Models
AI总结：
提出自预训练方法，在同一数据集上预训练和微调 Transformer 模型，在说话人验证任务中仅用三分之一数据即达竞争力性能。
摘要：
近期，使用下游数据集微调大型预训练 Transformer 模型引起了广泛关注。尽管取得了成功，但如何将大规模数据集和 Transformer 结构的优势与预训练的局限性区分开来仍然具有挑战性。本文引入了一种名为自预训练的分层训练方法，其中 Transformer 模型在同一数据集上进行预训练和微调。包括 HuBERT、Conformer 和 WavLM 在内的三个预训练模型在四个不同规模的说话人验证数据集上进行了评估。实验表明，与 Librispeech 预训练相比，这些自预训练模型仅需三分之一的数据即可在下游说话人验证任务（如 VoxCeleb1 和 CNCeleb1）上取得具有竞争力的性能。此外，仅在 VoxCeleb2-dev 上进行预训练时，Conformer 模型在使用相同微调设置下优于在 94k 小时数据上预训练的模型。
英文摘要：
Recently, fine-tuning large pre-trained Transformer models using downstream datasets has received a rising interest. Despite their success, it is still challenging to disentangle the benefits of large-scale datasets and Transformer structures from the limitations of the pre-training. In this paper, we introduce a hierarchical training approach, named self-pretraining, in which Transformer models are pretrained and finetuned on the same dataset. Three pre-trained models including HuBERT, Conformer and WavLM are evaluated on four different speaker verification datasets with varying sizes. Our experiments show that these self-pretrained models achieve competitive performance on downstream speaker verification tasks with only one-third of the data compared to Librispeech pretraining, such as VoxCeleb1 and CNCeleb1. Furthermore, when pre-training only on the VoxCeleb2-dev, the Conformer model outperforms the one pre-trained on 94k hours of data using the same fine-tuning settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
