在这项工作中,我们重新审视了基于transformer的预训练语言模型,并分别在位置编码和模型表示中识别出两种不同类型的信息混淆。首先,我们表明,在相对位置编码,相对距离和方向的联合建模带来了两个异构信息之间的混淆。这可能会使模型无法捕获相同距离和相反方向的关联语义,进而影响下游任务的性能。其次,我们注意到带有MLM(Mask Language Modeling)预训练目标的BERT输出相似的token表示(不同token的最后隐藏状态)和head表示(不同head的注意力权重),这可能使不同token和head所表达的信息的多样性受到限制。基于上述研究,我们提出了两种改进预训练语言模型的新技术:解耦方向相对位置(DDRP)编码和MTH预训练目标。DDRP是对经典相对位置编码中的相对距离特征和方向特征的继承。MTH除了MLM之外,还应用了两个新的辅助正则化器来扩大(a)不同令牌的最后隐藏状态和(b)不同头部的注意力权重之间的差异。这些设计允许模型更清晰地捕捉不同类别的信息,作为减轻表示学习中信息混乱的一种方式,以实现更好的优化。在GLUE基准上的大量实验和烧蚀研究证明了所提出方法的有效性。
英文摘要:
In this work, we revisit the Transformer-based pre-trained language models and identify two different types of information confusion in position encoding and model representations, respectively. Firstly, we show that in the relative position encoding, the joint modeling about relative distances and directions brings confusion between two heterogeneous information. It may make the model unable to capture the associative semantics of the same distance and the opposite directions, which in turn affects the performance of downstream tasks. Secondly, we notice the BERT with Mask Language Modeling (MLM) pre-training objective outputs similar token representations (last hidden states of different tokens) and head representations (attention weights of different heads), which may make the diversity of information expressed by different tokens and heads limited. Motivated by the above investigation, we propose two novel techniques to improve pre-trained language models: Decoupled Directional Relative Position (DDRP) encoding and MTH pre-training objective. DDRP decouples the relative distance features and the directional features in classical relative position encoding. MTH applies two novel auxiliary regularizers besides MLM to enlarge the dissimilarities between (a) last hidden states of different tokens, and (b) attention weights of different heads. These designs allow the model to capture different categories of information more clearly, as a way to alleviate information confusion in representation learning for better optimization. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of our proposed methods.
Reducing language context confusion for end-to-end code-switching automatic speech recognition
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心);Huawei Noah's Ark Lab(华为诺亚方舟实验室)
语码转换是指在交际过程中出现的替代语言。训练用于代码切换的端到端(E2 E)自动语音识别(ASR)系统尤其具有挑战性,因为代码切换训练数据总是不足以对抗由于存在多于一种语言而增加的多语言上下文混淆。本文基于等价约束理论,提出了一种与语言相关的注意力机制,以减少E2 E语码转换ASR模型中的多语言语境混淆。语言学理论要求任何出现在语码转换句中的单语片段必须出现在单语句子中。该理论在单语数据和语码转换数据之间架起了一座桥梁。我们利用这一语言学理论来设计语码转换E2 E ASR模型。该模型有效地从丰富的单语数据中转移语言知识,以提高代码转换ASR模型的性能。我们在ASRU 2019普通话-英语码转换挑战数据集上评估了我们的模型。与基线模型相比,我们提出的模型实现了17.12%的相对误差减少。