用于抗噪声语言理解的变形语言模型
Warped Language Models for Noise Robust Language Understanding
摘要:
Masked语言模型(MLM)是一种自我监督的神经网络,它被训练成用掩码标记来填充给定句子中的空白。尽管MLM对于各种基于文本的任务取得了巨大的成功,但是它们对于口语语言理解,特别是对于自发的会话语音识别噪声,并不鲁棒。在这项工作中,我们介绍了扭曲的语言模型(WLM),其中输入的句子在训练时经过相同的修改,在传销,加上两个额外的修改,即插入和丢弃随机令牌。这两个修改除了在MLMs中的修改之外,还扩展和收缩了句子,因此名称中的单词“warped”。在WLM的训练期间输入文本的插入和丢弃修改类似于由于自动语音识别(ASR)错误而引起的噪声类型,并且因此WLM可能对ASR噪声更鲁棒。通过计算结果,我们表明,建立在WLM之上的自然语言理解系统比建立在MLM之上的系统表现得更好,特别是在存在ASR错误的情况下。
英文摘要:
Masked Language Models (MLM) are self-supervised neural networks trained to fill in the blanks in a given sentence with masked tokens. Despite the tremendous success of MLMs for various text based tasks, they are not robust for spoken language understanding, especially for spontaneous conversational speech recognition noise. In this work we introduce Warped Language Models (WLM) in which input sentences at training time go through the same modifications as in MLM, plus two additional modifications, namely inserting and dropping random tokens. These two modifications extend and contract the sentence in addition to the modifications in MLMs, hence the word "warped" in the name. The insertion and drop modification of the input text during training of WLM resemble the types of noise due to Automatic Speech Recognition (ASR) errors, and as a result WLMs are likely to be more robust to ASR noise. Through computational results we show that natural language understanding systems built on top of WLMs perform better compared to those built based on MLMs, especially in the presence of ASR errors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。