arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.04246cs.CL
收藏

通过减少表示混淆来改善预训练

Better Pre-Training by Reducing Representation Confusion

Tencent(腾讯);Peking University(北京大学)

Haojie Zhang, Mingfei Liang, Ruobing Xie, Zhenlong Sun, Bo Zhang, Leyu Lin

摘要:

在这项工作中,我们重新审视了基于transformer的预训练语言模型,并分别在位置编码和模型表示中识别出两种不同类型的信息混淆。首先,我们表明,在相对位置编码,相对距离和方向的联合建模带来了两个异构信息之间的混淆。这可能会使模型无法捕获相同距离和相反方向的关联语义,进而影响下游任务的性能。其次,我们注意到带有MLM(Mask Language Modeling)预训练目标的BERT输出相似的token表示(不同token的最后隐藏状态)和head表示(不同head的注意力权重),这可能使不同token和head所表达的信息的多样性受到限制。基于上述研究,我们提出了两种改进预训练语言模型的新技术:解耦方向相对位置(DDRP)编码和MTH预训练目标。DDRP是对经典相对位置编码中的相对距离特征和方向特征的继承。MTH除了MLM之外,还应用了两个新的辅助正则化器来扩大(a)不同令牌的最后隐藏状态和(b)不同头部的注意力权重之间的差异。这些设计允许模型更清晰地捕捉不同类别的信息,作为减轻表示学习中信息混乱的一种方式,以实现更好的优化。在GLUE基准上的大量实验和烧蚀研究证明了所提出方法的有效性。

英文摘要:

In this work, we revisit the Transformer-based pre-trained language models and identify two different types of information confusion in position encoding and model representations, respectively. Firstly, we show that in the relative position encoding, the joint modeling about relative distances and directions brings confusion between two heterogeneous information. It may make the model unable to capture the associative semantics of the same distance and the opposite directions, which in turn affects the performance of downstream tasks. Secondly, we notice the BERT with Mask Language Modeling (MLM) pre-training objective outputs similar token representations (last hidden states of different tokens) and head representations (attention weights of different heads), which may make the diversity of information expressed by different tokens and heads limited. Motivated by the above investigation, we propose two novel techniques to improve pre-trained language models: Decoupled Directional Relative Position (DDRP) encoding and MTH pre-training objective. DDRP decouples the relative distance features and the directional features in classical relative position encoding. MTH applies two novel auxiliary regularizers besides MLM to enlarge the dissimilarities between (a) last hidden states of different tokens, and (b) attention weights of different heads. These designs allow the model to capture different categories of information more clearly, as a way to alleviate information confusion in representation learning for better optimization. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of our proposed methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑