arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.01183cs.CL
收藏

一种改进上下文嵌入空间各向同性的算法

A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space

Iran University of Science and Technology(伊朗科技大学);Tehran Institute for Advanced Studies(德黑兰高等研究院)

Sara Rajaee, Mohammad Taher Pilehvar

摘要:

上下文词表示(Contextual Word Representations,CWRs)中的表示退化问题通过形成各向异性的圆锥来损害嵌入空间的表达能力,其中即使不相关的词也具有过度的正相关性。解决这个问题的现有技术需要一个学习过程来重新训练具有额外目标的模型,并且大多采用全局评估来研究各向同性。我们对各向同性的定量分析表明,由于CWRs的聚集结构,局部评估可能更准确。基于这一观察,我们提出了一种基于局部聚类的方法来解决上下文嵌入空间中的退化问题。我们发现,在包括标点符号和停用词的聚类中,局部主导方向编码结构信息,去除这些信息可以提高CWRs在语义任务上的性能。此外,我们还发现动词表征中的时态信息支配着语义。我们发现,去除动词表征的主导方向可以改变空间,以更好地适应语义应用。我们的实验表明,所提出的基于聚类的方法可以减轻退化问题的多个任务。

英文摘要:

The representation degeneration problem in Contextual Word Representations (CWRs) hurts the expressiveness of the embedding space by forming an anisotropic cone where even unrelated words have excessively positive correlations. Existing techniques for tackling this issue require a learning process to re-train models with additional objectives and mostly employ a global assessment to study isotropy. Our quantitative analysis over isotropy shows that a local assessment could be more accurate due to the clustered structure of CWRs. Based on this observation, we propose a local cluster-based method to address the degeneration issue in contextual embedding spaces. We show that in clusters including punctuations and stop words, local dominant directions encode structural information, removing which can improve CWRs performance on semantic tasks. Moreover, we find that tense information in verb representations dominates sense semantics. We show that removing dominant directions of verb representations can transform the space to better suit semantic applications. Our experiments demonstrate that the proposed cluster-based method can mitigate the degeneration problem on multiple tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑