发表机构
Beijing Normal-Hong Kong Baptist University（北京师范大学-香港浸会大学联合国际学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文为PMI估计与SGNS词嵌入建立统计推断框架，证明估计量性质并分解方差，实验揭示SGNS的系统性偏差，为理论与实践提供启示。
AI 中文摘要
点互信息（PMI）是检验词关联的核心度量，而负采样跳元模型（SGNS）本质上是一种隐式分解移位PMI矩阵的方法。然而，关于PMI估计中有限样本不确定性的系统且全面的刻画仍然缺失，亟需深入探究。我们为PMI估计及其与SGNS的联系提供了一个统计框架。我们证明了经验PMI估计量的一致性、渐近无偏性和渐近正态性，通过Delta方法推导了其方差，并应用随机近似理论，获得了基于SGNS的PMI估计的方差分解，将数据方差与优化方差分离开来。模拟实验验证了Delta方法的近似效果。在Brown语料库（d=100）上的真实数据实验表明，SGNS系统地偏离了理论关系PMI + log K。经验关系显示出衰减的PMI系数、放大的log K效应以及正截距，表明存在系统性偏差。词类比验证确认了模型的有效性。在低维条件下方差分解未能得到验证，但这并不削弱其理论价值；相反，它指出了无偏性假设是关键瓶颈，并阐明了渐近理论与实践之间的差距，为实践和理论提供了启示。
英文摘要
Pointwise Mutual Information (PMI) is a core measure of testing word association, and Skip-gram with Negative Sampling (SGNS) is essentially a method that implicitly factorizes a shifted PMI matrix. However, a systematic and well-rounded characterization of finite-sample uncertainty in PMI estimation remains absent and imperative to venture into. We provide a statistical framework for PMI estimation and its connection to SGNS. We prove consistency, asymptotic unbiasedness, and asymptotic normality of the empirical PMI estimator, derive its variance via the Delta method, and, applying stochastic approximation theory, obtain a variance decomposition for SGNS-based PMI estimation that separates data variance from optimization variance. Simulation experiments validate the Delta method approximation. Real-data experiments on the Brown Corpus (d = 100) reveal that SGNS systematically deviates from the theoretical relationship PMI + log K. The empirical relationship shows an attenuated PMI coefficient, an amplified log K effect, and a positive intercept, indicating systematic bias. Word analogy validation confirms the models are effective. The failure to validate the variance decomposition under low-dimensional conditions does not diminish its theoretical value; rather, it identifies the unbiasedness assumption as the key bottleneck and clarifies the gap between asymptotic theory and practice, providing implications for both practice and theory.
Comments18 pages
