arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~

跨语言迁移的通用型还是语族特定型文字统一?以突厥语族为例的研究

Universal or Language-Family-Specific Script Unification for Cross-Lingual Transfer? A Case Study on Turkic Languages

Zijie Zhang

arXiv 2608.09356首次发表:更新:

发表机构

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

机构由 AI 辅助整理,请以论文原文为准。

AI 中文总结

本研究以突厥语族为例,对比通用uroman与语族特定CTS两种文字统一方法的跨语言迁移效果,发现二者在NER任务表现相当且均优于基线,POS任务效果受语言、子词重叠和监督数据影响,fastText系统具竞争力。

AI 中文摘要

使用不同文字书写的亲缘语言在表层几乎没有重叠,这限制了多语言模型的跨语言迁移能力。本研究对比两种文字统一方法:通用型的uroman罗马化工具,以及语族特定的通用突厥语文字(Common Turkic Script, CTS)。我们在来自11种突厥语族语言的转写维基百科语料上训练匹配的fastText模型,并在WikiANN命名实体识别(NER)和通用依存句法(Universal Dependencies)词性标注(POS)任务上进行评估。CTS与uroman在NER任务上无显著差异,但两者均显著优于官方单语言fastText基线模型。POS结果显示不存在通用最优方案:语言间差异与每种表示法诱导的跨语言字符n-gram覆盖率相关,而当目标语言有监督数据可用时,语言内覆盖率变得更重要。尽管CANINE-c在POS任务上取得更高的整体平均分数,但更简单的基于fastText的系统在多个树库上仍具有竞争力。总体而言,文字统一的有效性取决于具体语言、诱导的子词重叠程度以及可用的监督数据。

英文摘要

Closely related languages written in different scripts expose little surface overlap to multilingual models, limiting cross-lingual transfer. We compare two approaches to script unification: the general-purpose uroman romanizer and the family-specific Common Turkic Script (CTS). We train matched fastText models on transliterated Wikipedia corpora from 11 Turkic languages and evaluate them on WikiANN named entity recognition and Universal Dependencies part-of-speech tagging. CTS and uroman show no significant difference on NER, while both substantially outperform the official monolingual fastText baselines. POS results reveal no universal winner: language-specific differences are associated with the cross-lingual character n-gram coverage induced by each representation, while within-language coverage becomes more important when target-language supervision is available. Although CANINE-c achieves higher overall POS averages, the substantially simpler fastText-based systems remain competitive on several treebanks. Overall, the effectiveness of script unification depends on the language, the induced subword overlap, and the available supervision.

论文原文

arXiv 摘要页 · PDF 原文 · HTML 原文

↑