arXivDaily arXiv每日学术速递 周一至周五更新
arXiv cs/0411074cs.CLcs.IR
收藏

基于无监督短文档自切分的汉语词典构建

Building Chinese Lexicons from Scratch by Unsupervised Short Document Self-Segmentation

Daniel Gayo-Avello

摘要:

中文文本分割是一个众所周知的难题。一方面,汉语中没有一个简单的“词”的概念,这使得基于规则的系统很难实现书面文本的分割,因此通常使用词汇和统计信息来实现这样的任务。另一方面,任何一段中文文本通常包括既不存在于词典中也不存在于训练数据中的段。更糟糕的是,这种看不见的序列可以被分割成许多完全不相关的单词,使得后续处理阶段变得困难。例如,使用基于词典的系统,序列?(欧盟委员会现任候任主席巴罗佐)可以分割成?(ba,希望,希望)和??(loozuo,一个未定义的词)完全改变了句子的意义。本文介绍了一种新的、非常简单的、特别适合于处理短中文文档的算法。这种新的算法执行文本“自我分割”产生的结果相比,母语人士所取得的,而不使用任何词典或任何统计信息以外的输入文本。此外,它在寻找新的“单词”,特别是专有名词方面非常强大,并且非常适合从头开始构建词典。除了它的就业的例子提供了一些初步的结果。

英文摘要:

Chinese text segmentation is a well-known and difficult problem. On one side, there is not a simple notion of "word" in Chinese language making really hard to implement rule-based systems to segment written texts, thus lexicons and statistical information are usually employed to achieve such a task. On the other side, any piece of Chinese text usually includes segments present neither in the lexicons nor in the training data. Even worse, such unseen sequences can be segmented into a number of totally unrelated words making later processing phases difficult. For instance, using a lexicon-based system the sequence ???(Baluozuo, Barroso, current president-designate of the European Commission) can be segmented into ?(ba, to hope, to wish) and ??(luozuo, an undefined word) changing completely the meaning of the sentence. A new and extremely simple algorithm specially suited to work over short Chinese documents is introduced. This new algorithm performs text "self-segmentation" producing results comparable to those achieved by native speakers without using either lexicons or any statistical information beyond the obtained from the input text. Furthermore, it is really robust for finding new "words", especially proper nouns, and it is well suited to build lexicons from scratch. Some preliminary results are provided in addition to examples of its employment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑