印度尼西亚语统计语言模型中修正Kneser-Ney和Witten-Bell平滑技术的比较
Comparison of Modified Kneser-Ney and Witten-Bell Smoothing Techniques in Statistical Language Model of Bahasa Indonesia
摘要：
平滑是统计语言模型中克服数据稀疏性的一种技术。虽然在其数学定义中，没有明确的依赖于特定的自然语言，不同的自然语言的性质导致不同的平滑技术的效果。Whittaker（1998）的研究表明，俄语也是如此。本文比较了印度尼西亚语统计语言模型中的修正Kneser-Ney和Witten-Bell平滑技术。我们使用了训练集，总共22万字，我们从印尼语版的维基百科提取。据我们所知，这是用于为印度尼西亚语建立统计语言模型的最大训练集。对3-gram，5-gram和7-gram的实验表明，修正Kneser-Ney在困惑值方面始终优于Witten-Bell平滑技术。有趣的是，我们的实验表明，5-gram模型的改进Kneser-Ney平滑技术优于7-gram。同时，Witten-Bell平滑随着n-gram阶数的增加而不断改善。
英文摘要：
Smoothing is one technique to overcome data sparsity in statistical language model. Although in its mathematical definition there is no explicit dependency upon specific natural language, different natures of natural languages result in different effects of smoothing techniques. This is true for Russian language as shown by Whittaker (1998). In this paper, We compared Modified Kneser-Ney and Witten-Bell smoothing techniques in statistical language model of Bahasa Indonesia. We used train sets of totally 22M words that we extracted from Indonesian version of Wikipedia. As far as we know, this is the largest train set used to build statistical language model for Bahasa Indonesia. The experiments with 3-gram, 5-gram, and 7-gram showed that Modified Kneser-Ney consistently outperforms Witten-Bell smoothing technique in term of perplexity values. It is interesting to note that our experiments showed 5-gram model for Modified Kneser-Ney smoothing technique outperforms that of 7-gram. Meanwhile, Witten-Bell smoothing is consistently improving over the increase of n-gram order.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
