一种改进的基于贝叶斯TRIE的SMS文本规范化模型
An improved Bayesian TRIE based model for SMS text normalization
摘要:
短信文本(俗称短信语言)的规范化已经进行了十多年。在文献中提出了一种基于Trie数据结构的概率方法,该方法被发现在预测词典外单词的正确替代方面比先前提出的基于HMM的方法性能更好。然而,基于Trie的方法的成功在很大程度上取决于如何正确地估计单词出现的潜在概率。在这项工作中,我们提出了一个结构修改现有的Trie为基础的模型,以及一个新的训练算法和概率生成方案。我们证明了两个定理的统计性质的建议Trie,并使用它们来声称是一个无偏的和一致的估计的出现概率的话。我们进一步融合我们的模型到基于噪声信道的纠错的范例,并提供了一个启发式的Damerau Levenshtein距离超过1。我们还进行了模拟,以支持我们的索赔,并显示优越性的建议计划比以前的作品。
英文摘要:
Normalization of SMS text, commonly known as texting language, is being pursued for more than a decade. A probabilistic approach based on the Trie data structure was proposed in literature which was found to be better performing than HMM based approaches proposed earlier in predicting the correct alternative for an out-of-lexicon word. However, success of the Trie based approach depends largely on how correctly the underlying probabilities of word occurrences are estimated. In this work we propose a structural modification to the existing Trie-based model along with a novel training algorithm and probability generation scheme. We prove two theorems on statistical properties of the proposed Trie and use them to claim that is an unbiased and consistent estimator of the occurrence probabilities of the words. We further fuse our model into the paradigm of noisy channel based error correction and provide a heuristic to go beyond a Damerau Levenshtein distance of one. We also run simulations to support our claims and show superiority of the proposed scheme over previous works.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。