arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.05274cs.CL
收藏

学习移除:各向同性预训练BERT嵌入

Learning to Remove: Towards Isotropic Pre-trained BERT Embedding

Northwest University(西北大学);Shannxi Normal University(陕西师范大学)

Yuxin Liang, Rui Cao, Jie Zheng, Jie Ren, Ling Gao

摘要:

预先训练的语言模型,如BERT,已成为自然语言处理(NLP)任务的更常见选择。单词表示的研究表明,各向同性嵌入可以显着提高下游任务的性能。然而,我们测量和分析预训练的BERT嵌入的几何形状,发现它远非各向同性。我们发现词向量并不以原点为中心,两个随机词之间的平均余弦相似度远高于零,这表明词向量分布在一个狭窄的圆锥体中,恶化了词嵌入的表示能力。我们提出了一个简单而有效的方法来解决这个问题:用一组可学习的权重去除BERT嵌入的几个主导方向。我们训练单词相似性任务的权重,并表明处理后的嵌入更各向同性。我们的方法进行评估三个标准化的任务:词相似性,词的类比,语义文本相似性。在所有任务中,我们的方法处理的词嵌入始终优于原始嵌入(词类比平均提高13%,语义文本相似性平均提高16%)和两个基线方法。我们的方法也被证明是更强大的超参数的变化。

英文摘要:

Pre-trained language models such as BERT have become a more common choice of natural language processing (NLP) tasks. Research in word representation shows that isotropic embeddings can significantly improve performance on downstream tasks. However, we measure and analyze the geometry of pre-trained BERT embedding and find that it is far from isotropic. We find that the word vectors are not centered around the origin, and the average cosine similarity between two random words is much higher than zero, which indicates that the word vectors are distributed in a narrow cone and deteriorate the representation capacity of word embedding. We propose a simple, and yet effective method to fix this problem: remove several dominant directions of BERT embedding with a set of learnable weights. We train the weights on word similarity tasks and show that processed embedding is more isotropic. Our method is evaluated on three standardized tasks: word similarity, word analogy, and semantic textual similarity. In all tasks, the word embedding processed by our method consistently outperforms the original embedding (with average improvement of 13% on word analogy and 16% on semantic textual similarity) and two baseline methods. Our method is also proven to be more robust to changes of hyperparameter.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑