一种用于词法错误恢复的鲁棒文本处理技术
A Robust Text Processing Technique Applied to Lexical Error Recovery
摘要：
本论文主要研究词法错误的自动恢复和错误文本的标记化。我们提出了一种技术，可以自动纠正拼写错误，分割错误和真正的字错误，在一个统一的框架，使用语言生产模型和打字行为模型，并使令牌化的恢复过程的一部分。打字过程被建模为噪声信道，其中隐马尔可夫模型被用于对信道特性进行建模。弱统计语言模型用于预测哪些句子可能通过通道传输。这些组件在令牌传递框架中保持在一起，该框架提供了正交模式匹配和语言期望之间的期望的紧密耦合。该系统，CTR（连接文本识别），已被测试两个语料库来自两个不同的应用程序，自然语言对话系统和转录打字的情况。实验表明，CTR可以在不引入太多噪声的情况下自动纠正测试集中相当一部分错误。分割错误纠正率几乎是完美的。
英文摘要：
This thesis addresses automatic lexical error recovery and tokenization of corrupt text input. We propose a technique that can automatically correct misspellings, segmentation errors and real-word errors in a unified framework that uses both a model of language production and a model of the typing behavior, and which makes tokenization part of the recovery process. The typing process is modeled as a noisy channel where Hidden Markov Models are used to model the channel characteristics. Weak statistical language models are used to predict what sentences are likely to be transmitted through the channel. These components are held together in the Token Passing framework which provides the desired tight coupling between orthographic pattern matching and linguistic expectation. The system, CTR (Connected Text Recognition), has been tested on two corpora derived from two different applications, a natural language dialogue system and a transcription typing scenario. Experiments show that CTR can automatically correct a considerable portion of the errors in the test sets without introducing too much noise. The segmentation error correction rate is virtually faultless.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
