Pre-trained language models such as BERT have become a more common choice of natural language processing (NLP) tasks. Research in word representation shows that isotropic embeddings can significantly improve performance on downstream tasks. However, we measure and analyze the geometry of pre-trained BERT embedding and find that it is far from isotropic. We find that the word vectors are not centered around the origin, and the average cosine similarity between two random words is much higher than zero, which indicates that the word vectors are distributed in a narrow cone and deteriorate the representation capacity of word embedding. We propose a simple, and yet effective method to fix this problem: remove several dominant directions of BERT embedding with a set of learnable weights. We train the weights on word similarity tasks and show that processed embedding is more isotropic. Our method is evaluated on three standardized tasks: word similarity, word analogy, and semantic textual similarity. In all tasks, the word embedding processed by our method consistently outperforms the original embedding (with average improvement of 13% on word analogy and 16% on semantic textual similarity) and two baseline methods. Our method is also proven to be more robust to changes of hyperparameter.
Comparative Study of Pre-Trained BERT and Large Language Models for Code-Mixed Named Entity Recognition
Dept. of Computer Engineering, Pune Institute of Computer Technology, Pune, India(计算机工程系,普那计算机技术研究所,普那,印度);Indian Institute of Technology Madras, Chennai, India(印度理工学院马德拉斯学院,钦奈,印度);L3Cube Labs, Pune, Maharashtra, India(L3Cube实验室,普那,马哈拉施特拉邦,印度)
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning
University of Cambridge(剑桥大学);Beijing Institute of Technology(北京理工大学);Amazon(亚马逊);Monash University(莫纳什大学)
在过去的几年里,BERT和RoBERTa等掩蔽语言模型(Masked language models,MLMs)彻底改变了自然语言理解领域。然而,现有的预先训练的MLM通常输出令牌表示的各向异性分布,其占据整个表示空间的窄子集。这样的标记表示并不理想,特别是对于需要不同标记的区别性语义的任务。在这项工作中,我们提出了TaCL(令牌感知对比学习),一种新的连续预训练方法,鼓励BERT学习令牌表示的各向同性和歧视性分布。TaCL是完全无监督的,不需要额外的数据。我们在广泛的英语和中文基准上广泛测试我们的方法。实验结果表明,TaCL模型较之原有的BERT模型有了一致的、显著的改进。此外,我们进行详细的分析,以揭示我们的方法的优点和内部工作。