词化与乌拉尔语NLP中的形态保真度：跨语言评估
Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluation
AI总结：
本文通过比较三种子词方法在六个乌拉尔语中的表现，发现OBPE在形态对齐和标注准确性上优于传统方法，尤其在拉丁字母语中表现突出。
摘要：
子词词化对自然语言处理性能影响重大，但其在形态丰富且资源有限的语言家族中的行为尚待深入研究。本研究系统比较了三种子词范式——字节对编码（BPE）、重叠BPE（OBPE）和单字语言模型——在六个资源状况和语言类型各异的乌拉尔语中的表现。通过词性标注作为受控下游任务，研究发现OBPE在形态对齐和标注准确性上优于传统方法，尤其在拉丁字母语中表现突出。这些优势源于开放类词的碎片化减少以及频率谱的更好平衡。转移效果进一步取决于下游标注架构，与训练数据量和基因亲缘关系相互作用。综合来看，这些发现表明，对形态敏感的词化不仅是预处理选择，更是使聚合型、低资源语言实现有效跨语言转移的关键因素。
英文摘要：
Subword tokenization critically affects Natural Language Processing (NLP) performance, yet its behavior in morphologically rich and low-resource language families remains under-explored. This study systematically compares three subword paradigms -- Byte Pair Encoding (BPE), Overlap BPE (OBPE), and Unigram Language Model -- across six Uralic languages with varying resource availability and typological diversity. Using part-of-speech (POS) tagging as a controlled downstream task, we show that OBPE consistently achieves stronger morphological alignment and higher tagging accuracy than conventional methods, particularly within the Latin-script group. These gains arise from reduced fragmentation in open-class categories and a better balance across the frequency spectrum. Transfer efficacy further depends on the downstream tagging architecture, interacting with both training volume and genealogical proximity. Taken together, these findings highlight that morphology-sensitive tokenization is not merely a preprocessing choice but a decisive factor in enabling effective cross-lingual transfer for agglutinative, low-resource languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
