基于语言建模的电子词典结构异常检测
Detecting Structural Irregularity in Electronic Dictionaries Using Language Modeling
摘要:
字典通常使用保存到基于可扩展置标语言(XML)的标准的工具来开发。这些标准通常允许高级重复元素表示词汇条目,并利用这些重复元素的后代以XML树的形式表示每个词汇条目内的结构。在许多情况下,出版的字典中有错误和不一致之处,手动查找这些错误和不一致之处的成本很高。本文讨论了一种利用统计语言模型快速审计词典词条结构规则性的方法。该方法学习XML树中可能出现的XML节点的模式,然后计算字典中每个XML树与这些模式的概率,以查找偏离规范的条目。
英文摘要:
Dictionaries are often developed using tools that save to Extensible Markup Language (XML)-based standards. These standards often allow high-level repeating elements to represent lexical entries, and utilize descendants of these repeating elements to represent the structure within each lexical entry, in the form of an XML tree. In many cases, dictionaries are published that have errors and inconsistencies that are expensive to find manually. This paper discusses a method for dictionary writers to quickly audit structural regularity across entries in a dictionary by using statistical language modeling. The approach learns the patterns of XML nodes that could occur within an XML tree, and then calculates the probability of each XML tree in the dictionary against these patterns to look for entries that diverge from the norm.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。