arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1405.5474cs.CL
收藏

汉字结构处理的新视角

New Perspectives in Sinographic Language Processing Through the Use of Character Structure

Institut Télécom - Télécom Bretagne(电信学院 - 布列塔尼电信学院)

Yannis Haralambous

摘要:

汉字具有复杂的层次结构,承载着语义和语音信息。我们使用这种结构来增强文本模型,并在标准NLP操作中获得更好的结果。首先,为了解决图形变化的问题,我们定义了异体字类。其次,一个字符中包含一个子字符的关系,为我们提供了一个异体字类的有向图。我们为这个图提供两个权重:语义性(子字符和字符之间的语义关系)和语音性(语音关系),并计算每个字符的“最语义子字符路径”。最后,将这些路径中包含的信息添加到一元语法中,我们声称可以提高文本挖掘方法的效率。我们在两个语料库(中文和日文)的文本分类任务上评估了我们的方法,总共有1800万个字符,并在线性SVM分类器获得的89.6%精度的高基线上提高了3%。其他可能的应用和前景的系统进行了讨论。

英文摘要:

Chinese characters have a complex and hierarchical graphical structure carrying both semantic and phonetic information. We use this structure to enhance the text model and obtain better results in standard NLP operations. First of all, to tackle the problem of graphical variation we define allographic classes of characters. Next, the relation of inclusion of a subcharacter in a characters, provides us with a directed graph of allographic classes. We provide this graph with two weights: semanticity (semantic relation between subcharacter and character) and phoneticity (phonetic relation) and calculate "most semantic subcharacter paths" for each character. Finally, adding the information contained in these paths to unigrams we claim to increase the efficiency of text mining methods. We evaluate our method on a text classification task on two corpora (Chinese and Japanese) of a total of 18 million characters and get an improvement of 3% on an already high baseline of 89.6% precision, obtained by a linear SVM classifier. Other possible applications and perspectives of the system are discussed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑