arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2011.08238cs.CLcs.SDeess.AS
收藏

使用Transformer网络和自监督预训练特征的端到端语言理解

End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features

IBM Research(IBM研究院)

Edmilson Morais, Hong-Kwang J. Kuo, Samuel Thomas, Zoltan Tuske, Brian Kingsbury

摘要:

Transformer网络和自监督预训练在自然语言处理(NLP)领域一直提供最先进的结果;然而,它们在口语语言理解(SLU)领域的优点仍需要进一步研究。在本文中,我们介绍了一个模块化的端到端(E2 E)SLU Transformer网络的基础架构,允许使用自我监督的预训练的声学特征,预训练模型初始化和多任务训练。使用ATIS数据集进行了几个SLU实验,用于预测意图和实体标签/值。这些实验研究了预训练模型初始化和多任务训练与传统滤波器组或自监督预训练声学特征的相互作用。结果表明,自监督预训练的声学特征不仅在几乎所有的实验中都优于滤波器组特征,而且当这些特征与多任务训练相结合时,它们几乎消除了预训练模型初始化的必要性。

英文摘要:

Transformer networks and self-supervised pre-training have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of spoken language understanding (SLU) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SLU transformer network based architecture which allows the use of self-supervised pre-trained acoustic features, pre-trained model initialization and multi-task training. Several SLU experiments for predicting intent and entity labels/values using the ATIS dataset are performed. These experiments investigate the interaction of pre-trained model initialization and multi-task training with either traditional filterbank or self-supervised pre-trained acoustic features. Results show not only that self-supervised pre-trained acoustic features outperform filterbank features in almost all the experiments, but also that when these features are used in combination with multi-task training, they almost eliminate the necessity of pre-trained model initialization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑