arXiv 1506.041472015-06-18stat.MLcs.CLcs.LGstat.ME

关于自归一化对数线性模型的精度

On the accuracy of self-normalized log-linear models

University of California, Berkeley（加州大学伯克利分校）

Jacob Andreas, Maxim Rabinovich, Dan Klein, Michael I. Jordan

摘要：

对数归一化子的计算是对数线性模型在大输出空间应用中的一个主要计算障碍。因此，快速归一化计算的问题在理论和应用机器学习文献中引起了极大的关注。在本文中，我们分析了最近提出的一种称为“自归一化”的技术，该技术在训练中引入了正则化项，以惩罚偏离零的对数归一化器。这使得可以使用非标准化模型分数作为近似概率。经验证据表明，自我正常化是非常有效的，但在理论上理解为什么它应该工作，以及如何普遍适用，在很大程度上缺乏。我们证明了泛化界的估计方差的归一化和上限损失的准确性，由于自我规范化，描述类的输入分布，自我规范化容易，并构建明确的例子高方差输入分布。我们的理论结果预测拟合自归一化模型的几类分布的困难，我们得出结论，这些预测的实证验证。

英文摘要：

Calculation of the log-normalizer is a major computational obstacle in applications of log-linear models with large output spaces. The problem of fast normalizer computation has therefore attracted significant attention in the theoretical and applied machine learning literature. In this paper, we analyze a recently proposed technique known as "self-normalization", which introduces a regularization term in training to penalize log normalizers for deviating from zero. This makes it possible to use unnormalized model scores as approximate probabilities. Empirical evidence suggests that self-normalization is extremely effective, but a theoretical understanding of why it should work, and how generally it can be applied, is largely lacking. We prove generalization bounds on the estimated variance of normalizers and upper bounds on the loss in accuracy due to self-normalization, describe classes of input distributions that self-normalize easily, and construct explicit examples of high-variance input distributions. Our theoretical results make predictions about the difficulty of fitting self-normalized models to several classes of distributions, and we conclude with empirical validation of these predictions.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文