arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.08037cs.CLcs.LG

TiSpell:一种通过数据增强覆盖多层级错误的藏文拼写纠错半掩码方法

TiSpell: A Semi-Masked Methodology for Tibetan Spelling Correction covering Multi-Level Error with Data Augmentation

School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);School of Information Science and Technology, Tibet University(西藏大学信息科学与技术学院);Department of Ophthalmology, University of Texas Southwestern Medical Center(德克萨斯西南医学中心眼科部门)
Yutong Liu, Feng Xiao, Ziyue Zhang, Yongbin Yu, Cheng Huang, Fan Gao, Xiangxiang Wang, Ma-bao Ban, Manping Fan, Thupten Tsering, Cheng Huang, Gadeng Luosang, Re… 展开作者

Yutong Liu, Feng Xiao, Ziyue Zhang, Yongbin Yu, Cheng Huang, Fan Gao, Xiangxiang Wang, Ma-bao Ban, Manping Fan, Thupten Tsering, Cheng Huang, Gadeng Luosang, Renzeng Duojie, Nyima Tashi

AI总结:

针对藏文字符级与音节级纠错缺乏统一模型及专用数据的问题,提出无标注多层级损坏增强方法与半掩码模型 TiSpell,实验证明其性能优异。

摘要:

多层级藏文拼写纠错旨在在一个统一模型中同时处理字符级和音节级错误。现有方法主要聚焦于单一层级的纠错,缺乏对两个层级的有效整合;此外,藏语中还没有针对该任务定制的开源数据集或增强方法。为解决这些问题,我们提出一种利用无标注文本生成多层级损坏的数据增强方法,并推出 TiSpell,这是一种能够同时纠正字符级和音节级错误的半掩码模型。尽管音节级纠错因依赖全局语境而更具挑战性,但我们的半掩码策略简化了这一过程。我们在干净句子上合成九类损坏,以构建稳健的训练集。在模拟数据和真实数据上的实验均表明,使用我们数据集训练的 TiSpell 优于基线模型,并达到与最先进方法相当的性能,证实了其有效性。

英文摘要:

Multi-level Tibetan spelling correction addresses errors at both the character and syllable levels within a unified model. Existing methods focus mainly on single-level correction and lack effective integration of both levels. Moreover, there are no open-source datasets or augmentation methods tailored for this task in Tibetan. To tackle this, we propose a data augmentation approach using unlabeled text to generate multi-level corruptions, and introduce TiSpell, a semi-masked model capable of correcting both character- and syllable-level errors. Although syllable-level correction is more challenging due to its reliance on global context, our semi-masked strategy simplifies this process. We synthesize nine types of corruptions on clean sentences to create a robust training set. Experiments on both simulated and real-world data demonstrate that TiSpell, trained on our dataset, outperforms baseline models and matches the performance of state-of-the-art approaches, confirming its effectiveness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑