arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.09663cs.CL
收藏

词嵌入的范数编码信息增益

Norm of Word Embedding Encodes Information Gain

Kyoto University(京都大学);RIKEN AIP(理化学研究所革新智能统合研究中心);Tohoku University(东北大学)

Momose Oyama, Sho Yokoi, Hidetoshi Shimodaira

AI总结:

本文证明跳元模型词嵌入的平方范数编码由 KL 散度定义的信息增益,并基于指数族理论解释,将其推广到上下文嵌入及 softmax 网络,并验证其在关键词提取等任务中的有效性。

摘要:

词的分布式表示编码了词汇语义信息,但编码的是何种信息以及如何编码?本文聚焦于带负采样的跳元模型(skip-gram with negative-sampling),发现静态词嵌入的平方范数编码了该词所传达的信息增益;信息增益由该词共现分布相对于一元分布的 Kullback-Leibler 散度定义。我们的发现通过指数族概率分布的理论框架得到解释,并通过精确实验得到证实,这些实验消除了由词频引起的虚假相关。该理论还扩展到语言模型中的上下文词嵌入或任何具有 softmax 输出层的神经网络。我们还证明,KL 散度和词嵌入的平方范数在关键词提取、专有名词判别和上位词判别等任务中提供了有用的词信息量度量。

英文摘要:

Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain conveyed by the word; the information gain is defined by the Kullback-Leibler divergence of the co-occurrence distribution of the word to the unigram distribution. Our findings are explained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. This theory also extends to contextualized word embeddings in language models or any neural networks with the softmax output layer. We also demonstrate that both the KL divergence and the squared norm of embedding provide a useful metric of the informativeness of a word in tasks such as keyword extraction, proper-noun discrimination, and hypernym discrimination.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑