arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.12281cs.CL
收藏

基于分层嵌入自适应滤波的中文拼写检查域移位调节算法

Domain-shift Conditioning using Adaptable Filtering via Hierarchical Embeddings for Robust Chinese Spell Check

Minh Nguyen, Gia H. Ngo, Nancy F. Chen

摘要:

拼写检查是一个有用的应用程序,它处理嘈杂的人类生成的文本。中文拼写检查提出了未解决的问题,由于大量的字符,稀疏分布的错误,以及缺乏足够的覆盖异构和移动的错误域的资源。对于中文拼写检查,使用混淆集进行筛选缩小了搜索空间,并使查找更正更容易。然而,大多数(如果不是全部的话)迄今为止使用的混淆集是固定的,因此不包括新的、移动的错误域。我们提出了一个可扩展的自适应过滤器,利用分层字符嵌入(1)消除手工制作混淆集的需要,(2)解决与罕见错误相关的稀疏性问题。我们的方法与竞争对手的基线相比毫不逊色,并在2014年和2015年的中文拼写检查烘焙测试数据集上获得了SOTA结果。

英文摘要:

Spell check is a useful application which processes noisy human-generated text. Spell check for Chinese poses unresolved problems due to the large number of characters, the sparse distribution of errors, and the dearth of resources with sufficient coverage of heterogeneous and shifting error domains. For Chinese spell check, filtering using confusion sets narrows the search space and makes finding corrections easier. However, most, if not all, confusion sets used to date are fixed and thus do not include new, shifting error domains. We propose a scalable adaptable filter that exploits hierarchical character embeddings to (1) obviate the need to handcraft confusion sets, and (2) resolve sparsity problems related to infrequent errors. Our approach compares favorably with competitive baselines and obtains SOTA results on the 2014 and 2015 Chinese Spelling Check Bake-off datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑