arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.02433cs.LGstat.ML
收藏

被盗概率:神经语言模型的结构弱点

Stolen Probability: A Structural Weakness of Neural Language Models

Northwestern University(西北大学);H. Lee Moffitt Cancer Center(H·李·莫菲特癌症中心);Allen Institute for AI(艾伦人工智能研究所)

David Demeter, Gregory Kimmel, Doug Downey

摘要:

神经网络语言模型(NNLM)通过将softmax函数应用于距离度量来生成概率分布,该距离度量通过在高维嵌入空间中取预测向量与所有词向量的点积来形成。点积距离度量形成NNLM的归纳偏差的一部分。虽然NNLM优化与此归纳偏见,我们表明,这导致在一个次优的嵌入空间,结构上的一些字在牺牲其他分配概率时,排序。我们目前的数值,理论和经验分析表明,在嵌入空间的凸包的内部的话有自己的概率壳上的话的概率有界。

英文摘要:

Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this results in a sub-optimal ordering of the embedding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses showing that words on the interior of the convex hull in the embedding space have their probability bounded by the probabilities of the words on the hull.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑