Zipfian白化
Zipfian Whitening
AI总结:
本研究针对词嵌入空间偏斜问题,提出基于Zipf定律经验词频加权的PCA白化方法,性能超越基线,并从理论上解释了该方法及多种主流NLP方法有效的核心原因。
摘要:
神经模型中的词嵌入空间存在偏斜,校正这种偏斜可提升任务性能。我们指出,大多数用于建模、校正和衡量嵌入空间对称性的方法都隐含假设词频是均匀的;但实际上,词频遵循高度非均匀的分布,即Zipf定律(齐普夫定律)。令人意外的是,仅通过遵循Zipf定律的经验词频加权执行PCA白化(主成分分析白化),就能显著提升任务性能,超越已有的基线方法。从理论角度看,我们的方法与现有方法可被清晰归类:词表示服从指数族分布,其基础测度要么是均匀的要么是Zipfian的。采用后者时,我们能从向量范数层面自然凸显具有信息价值的低频词,这一点从信息几何视角以及针对不平衡分类的损失函数中均可得到印证。此外,我们的理论证实,skip-gram negative sampling(跳字负采样)、WhiteningBERT、headless language models(无头语言模型)等流行的自然语言处理方法效果良好,正是因为它们的词嵌入将经验词频编码到了底层概率模型中。
英文摘要:
The word embedding space in neural models is skewed, and correcting this can improve task performance. We point out that most approaches for modeling, correcting, and measuring the symmetry of an embedding space implicitly assume that the word frequencies are uniform; in reality, word frequencies follow a highly non-uniform distribution, known as Zipf's law. Surprisingly, simply performing PCA whitening weighted by the empirical word frequency that follows Zipf's law significantly improves task performance, surpassing established baselines. From a theoretical perspective, both our approach and existing methods can be clearly categorized: word representations are distributed according to an exponential family with either uniform or Zipfian base measures. By adopting the latter approach, we can naturally emphasize informative low-frequency words in terms of their vector norm, which becomes evident from the information-geometric perspective, and in terms of the loss functions for imbalanced classification. Additionally, our theory corroborates that popular natural language processing methods, such as skip-gram negative sampling, WhiteningBERT, and headless language models, work well just because their word embeddings encode the empirical word frequency into the underlying probabilistic model.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。