马拉雅拉姆语可音译词和母语词的无监督分离
Unsupervised Separation of Transliterable and Native Words for Malayalam
摘要：
区分固有语言词和可译词是辅助涉及不同自然语言的文本处理任务的关键步骤。我们考虑的问题，无监督的分离的可音译的话，从母语的文字在马拉雅拉姆语。概述了一个关键的观察字符的多样性超出词干，我们开发了一种优化方法来评分的基础上，他们的原生词。我们的方法依赖于使用的概率分布在字符n-gram中，在迭代优化公式中与原生评分的步骤中进行细化。使用实证评估，我们说明，我们的方法，DTIM，提供了显着的改善，马拉雅拉姆语的本土得分，建立DTIM作为任务的首选方法。
英文摘要：
Differentiating intrinsic language words from transliterable words is a key step aiding text processing tasks involving different natural languages. We consider the problem of unsupervised separation of transliterable words from native words for text in Malayalam language. Outlining a key observation on the diversity of characters beyond the word stem, we develop an optimization method to score words based on their nativeness. Our method relies on the usage of probability distributions over character n-grams that are refined in step with the nativeness scorings in an iterative optimization formulation. Using an empirical evaluation, we illustrate that our method, DTIM, provides significant improvements in nativeness scoring for Malayalam, establishing DTIM as the preferred method for the task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
