arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.13304cs.CL
收藏

关于Transformers的各向同性校验

On Isotropy Calibration of Transformers

ETH Zürich(苏黎世联邦理工学院);University of Zürich(苏黎世大学)

Yue Ding, Karolis Martinkus, Damian Pascual, Simon Clematide, Roger Wattenhofer

摘要:

对Transformer模型的嵌入空间的不同研究表明,上下文表示的分布是高度各向异性的-嵌入分布在一个狭窄的圆锥体中。同时,静态单词表示(例如,Word 2 Vec或GloVe)已经被证明受益于各向同性空间。因此,以前的工作已经开发了方法来校准的嵌入空间的Transformers器,以确保各向同性。然而,最近的一项研究(Cai et al. 2021)表明,Transformers的嵌入空间是局部各向同性的,这表明这些模型已经能够利用其嵌入空间的表达能力。在这项工作中,我们对Transformers各向同性校准的最先进方法进行了实证评估,发现它们不能在模型和任务中提供一致的改进。这些结果支持的论点,即局部各向同性,Transformers不受益于额外的各向同性校准。

英文摘要:

Different studies of the embedding space of transformer models suggest that the distribution of contextual representations is highly anisotropic - the embeddings are distributed in a narrow cone. Meanwhile, static word representations (e.g., Word2Vec or GloVe) have been shown to benefit from isotropic spaces. Therefore, previous work has developed methods to calibrate the embedding space of transformers in order to ensure isotropy. However, a recent study (Cai et al. 2021) shows that the embedding space of transformers is locally isotropic, which suggests that these models are already capable of exploiting the expressive capacity of their embedding space. In this work, we conduct an empirical evaluation of state-of-the-art methods for isotropy calibration on transformers and find that they do not provide consistent improvements across models and tasks. These results support the thesis that, given the local isotropy, transformers do not benefit from additional isotropy calibration.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑