arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.19333cs.CLcs.SDeess.AS
收藏

通过离散语音单元预训练实现紧凑语音翻译模型

Compact Speech Translation Models via Discrete Speech Units Pretraining

School of Informatics, University of Edinburgh(爱丁堡大学信息学院)

Tsz Kin Lam, Alexandra Birch, Barry Haddow

AI总结:

提出一种基于离散语音单元预训练的紧凑语音翻译模型方法,无需转录文本,在CoVoST-2上以SSS模型一半大小实现三个指标的稳定提升。

摘要:

我们提出一种预训练方法,利用自监督语音(SSS)模型来构建更紧凑的语音到文本翻译模型。与使用SSS模型进行初始化不同,我们的方法更适合内存受限的场景,如设备端部署。我们的方法基于从SSS模型中提取的离散语音单元(DSU)。第一步,我们的方法在两个较小的编码器-解码器模型上分别预训练,数据为:1) 滤波器组到DSU(Fbk-to-DSU)和2) DSU到翻译(DSU-to-Trl)。因此,DSU成为较小模型的蒸馏输入。随后,从Fbk-to-DSU模型中提取编码器,从DSU-to-Trl模型中提取解码器,用于初始化紧凑模型。最后,紧凑模型在配对的Fbk-Trl数据上进行微调。除了紧凑性外,我们的方法不需要转录文本,使其适用于低资源设置。它还避免了推理中的语音离散化,并且对DSU分词更加鲁棒。在CoVoST-2(X-En)上的评估表明,我们的方法在三个指标上相对于基线有一致的改进,同时保持紧凑,即仅为SSS模型大小的一半。

英文摘要:

We propose a pretraining method to use Self-Supervised Speech (SSS) model to creating more compact Speech-to-text Translation. In contrast to using the SSS model for initialization, our method is more suitable to memory constrained scenario such as on-device deployment. Our method is based on Discrete Speech Units (DSU) extracted from the SSS model. In the first step, our method pretrains two smaller encoder-decoder models on 1) Filterbank-to-DSU (Fbk-to-DSU) and 2) DSU-to-Translation (DSU-to-Trl) data respectively. The DSU thus become the distillation inputs of the smaller models. Subsequently, the encoder from the Fbk-to-DSU model and the decoder from the DSU-to-Trl model are taken to initialise the compact model. Finally, the compact model is finetuned on the paired Fbk-Trl data. In addition to being compact, our method requires no transcripts, making it applicable to low-resource settings. It also avoids speech discretization in inference and is more robust to the DSU tokenization. Evaluation on CoVoST-2 (X-En) shows that our method has consistent improvement over the baseline in three metrics while being compact i.e., only half the SSS model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑