语言建模的一点进展
A Bit of Progress in Language Modeling
摘要:
在过去的几年中,已经发现了许多不同的语言建模改进,包括缓存,高阶n-gram,跳过,插值Kneser-Ney平滑和聚类。我们目前的变化,或限制,这些技术的探索,包括显示,句子混合模型可能有更多的潜力。虽然所有这些技术都已单独研究,但很少结合起来研究。我们发现一些重要的相互作用,特别是平滑和聚类技术。我们比较了所有技术的组合在一起的卡茨平滑的三元模型,没有计数截止。根据训练数据的大小,我们实现了38%到50%(1位熵)的困惑减少,以及8.9%的字出错率减少。我们的困惑减少可能是最高的报告相比,公平的基线。这是本文的扩展版本;它包含了更多的细节和证明,旨在很好地介绍语言建模的最新技术。
英文摘要:
In the past several years, a number of different language modeling improvements over simple trigram models have been found, including caching, higher-order n-grams, skipping, interpolated Kneser-Ney smoothing, and clustering. We present explorations of variations on, or of the limits of, each of these techniques, including showing that sentence mixture models may have more potential. While all of these techniques have been studied separately, they have rarely been studied in combination. We find some significant interactions, especially with smoothing and clustering techniques. We compare a combination of all techniques together to a Katz smoothed trigram model with no count cutoffs. We achieve perplexity reductions between 38% and 50% (1 bit of entropy), depending on training data size, as well as a word error rate reduction of 8.9%. Our perplexity reductions are perhaps the highest reported compared to a fair baseline. This is the extended version of the paper; it contains additional details and proofs, and is designed to be a good introduction to the state of the art in language modeling.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。