基于统计异常检测的XML电子词典数据清洗
Data Cleaning for XML Electronic Dictionaries via Statistical Anomaly Detection
摘要：
许多重要形式的数据都以XML格式进行数字存储。XML字段中数据的文本内容可能会出现错误。手动修复这些错误既耗时又昂贵，尤其是对于大量数据。人们对研究、开发和使用自动化技术来帮助数据清理越来越感兴趣。电子词典是经常以XML格式存储的数据的重要形式，其经常具有通过人工印刷输入错误和光学字符识别错误的混合而引入的错误。在本文中，我们描述的方法标记统计异常可能的错误存储在XML格式的电子词典。我们描述了六个系统的基础上，不同的信息来源。该系统使用数据中的各种信号来检测错误，包括不常见字符、文本长度、基于字符的语言模型、基于单词的语言模型、绑定字段长度比和绑定字段音译模型。其中四个系统基于从单个字段类型的元素内的内容自动推断的期望来检测错误。我们称之为单场系统。其中两个系统基于从多个相关字段类型的元素内的内容自动推断出的对应期望来检测错误。我们称之为束缚场系统。对于每个系统，我们提供了一个直观的分析，它是成功检测的错误类型。最后，我们描述了两个大规模的评估使用众包与亚马逊的土耳其机械平台，并使用领域专家的注释。评价一致表明，该系统是有用的，以提高效率，在XML电子词典中的错误可以检测。
英文摘要：
Many important forms of data are stored digitally in XML format. Errors can occur in the textual content of the data in the fields of the XML. Fixing these errors manually is time-consuming and expensive, especially for large amounts of data. There is increasing interest in the research, development, and use of automated techniques for assisting with data cleaning. Electronic dictionaries are an important form of data frequently stored in XML format that frequently have errors introduced through a mixture of manual typographical entry errors and optical character recognition errors. In this paper we describe methods for flagging statistical anomalies as likely errors in electronic dictionaries stored in XML format. We describe six systems based on different sources of information. The systems detect errors using various signals in the data including uncommon characters, text length, character-based language models, word-based language models, tied-field length ratios, and tied-field transliteration models. Four of the systems detect errors based on expectations automatically inferred from content within elements of a single field type. We call these single-field systems. Two of the systems detect errors based on correspondence expectations automatically inferred from content within elements of multiple related field types. We call these tied-field systems. For each system, we provide an intuitive analysis of the type of error that it is successful at detecting. Finally, we describe two larger-scale evaluations using crowdsourcing with Amazon's Mechanical Turk platform and using the annotations of a domain expert. The evaluations consistently show that the systems are useful for improving the efficiency with which errors in XML electronic dictionaries can be detected.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
