基于隐马尔可夫模型的中东语言语境分析
Contextual Analysis for Middle Eastern Languages with Hidden Markov Models
摘要:
由于字母表中每个字符的表示形式不同,因此中东语言的文档需要进行上下文分析。文档的单词将通过连接表示字符的相应表示形式的正确位置字形来形成。一组规则定义字形的连接。通常,这些规则因语言而异,并取决于软件开发人员的解释。在本文中,我们提出了一个机器学习方法的上下文分析的基础上,一阶隐马尔可夫模型。我们将设计和建立一个波斯语模型来展示这项技术。波斯语模型达到94%的准确率与训练的基础上的一个简短的列表,由2780个波斯语字符组成的89个波斯语词汇。该实验可以很容易地扩展到许多语言,包括阿拉伯语,乌尔都语和信德语。此外,这种方法的优点是,可以使用相同的软件来执行上下文分析,而无需为每种特定语言编写复杂的规则。特别令人感兴趣的是,使用者较少的语言在网络上的代表性更大,因为它们通常由于缺乏经济激励而被软件开发人员忽略。
英文摘要:
Displaying a document in Middle Eastern languages requires contextual analysis due to different presentational forms for each character of the alphabet. The words of the document will be formed by the joining of the correct positional glyphs representing corresponding presentational forms of the characters. A set of rules defines the joining of the glyphs. As usual, these rules vary from language to language and are subject to interpretation by the software developers. In this paper, we propose a machine learning approach for contextual analysis based on the first order Hidden Markov Model. We will design and build a model for the Farsi language to exhibit this technology. The Farsi model achieves 94 \% accuracy with the training based on a short list of 89 Farsi vocabularies consisting of 2780 Farsi characters. The experiment can be easily extended to many languages including Arabic, Urdu, and Sindhi. Furthermore, the advantage of this approach is that the same software can be used to perform contextual analysis without coding complex rules for each specific language. Of particular interest is that the languages with fewer speakers can have greater representation on the web, since they are typically ignored by software developers due to lack of financial incentives.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。