为何如此低落？点态互信息在分布语义中的作用
Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics
摘要：
在分布式语义中，逐点互信息（$\mathit{PMI}$）加权的共生矩阵的性能远远优于原始计数。然而，当$\mathit{PMI}$趋于负无穷大时，存在未观察到的对同现的问题。这个问题是加剧了不可靠的统计数据，从有限的语料库，导致大量的这种对。常见的做法是在$0$处夹持阴性$\mathit{PMI}$（$\mathit{\texttt{-} PMI}$），也称为阳性$\mathit{PMI}$（$\mathit{PPMI}$）。在本文中，我们探讨了其他方法来处理$\mathit{\texttt{-} PMI}$，更重要的是，研究的作用，负信息发挥的性能低秩，加权分解不同的$\mathit{PMI}$矩阵。使用各种语义和句法任务作为探针模型，使用消极或积极的$\mathit{PMI}$（或两者），我们发现，大多数编码的语义和语法来自积极的$\mathit{PMI}$，相反，$\mathit{\texttt{-} PMI}$，它几乎只贡献语法信息。我们的研究结果加深了我们对分布语义的理解，同时也引入了新的$PMI$变体，并为流行的$PPMI$测量奠定了基础。
英文摘要：
In distributional semantics, the pointwise mutual information ($\mathit{PMI}$) weighting of the cooccurrence matrix performs far better than raw counts. There is, however, an issue with unobserved pair cooccurrences as $\mathit{PMI}$ goes to negative infinity. This problem is aggravated by unreliable statistics from finite corpora which lead to a large number of such pairs. A common practice is to clip negative $\mathit{PMI}$ ($\mathit{\texttt{-} PMI}$) at $0$, also known as Positive $\mathit{PMI}$ ($\mathit{PPMI}$). In this paper, we investigate alternative ways of dealing with $\mathit{\texttt{-} PMI}$ and, more importantly, study the role that negative information plays in the performance of a low-rank, weighted factorization of different $\mathit{PMI}$ matrices. Using various semantic and syntactic tasks as probes into models which use either negative or positive $\mathit{PMI}$ (or both), we find that most of the encoded semantics and syntax come from positive $\mathit{PMI}$, in contrast to $\mathit{\texttt{-} PMI}$ which contributes almost exclusively syntactic information. Our findings deepen our understanding of distributional semantics, while also introducing novel $PMI$ variants and grounding the popular $PPMI$ measure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
