面向形态多样语言的子词切分学习动态研究
The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
AI总结：
本文扩展支持预训练微调的SSLM框架，针对三类形态差异语言研究动态子词切分的学习演化规律，验证该方法可提升低资源形态复杂语言的生成与跨语言迁移效果。
摘要：
子词切分通常在预处理阶段应用，且在训练过程中保持固定。作为替代方案，它也可以在训练期间学习得到，以优化训练目标。本文研究子词切分的学习动态：如果语言模型能够动态优化分词方式，其子词在预训练和微调阶段会如何演化？为探究这一问题，我们扩展了可在训练期间学习子词的框架——子词切分语言模型（SSLM），使其支持预训练与微调流程。我们为三种类型学差异显著的语言训练模型，以研究覆盖全形态谱系的学习动态：科萨语（Isi-Xhosa）为黏着型语言（长词形由多个语素构成），塞茨瓦纳语（Setswana）为分离型语言（语素以独立词形式书写），英语则代表类型学上的中间形态。我们从语言学视角分析子词动态，追踪形态、能产性与 fertility 指标的变化，识别出子词学习的四个阶段，其中形态复杂的科萨语表现出更强的不稳定性。微调阶段，子词边界会向更细粒度偏移。最后我们证明，可学习子词为改善低资源形态复杂语言的文本生成效果与跨语言迁移能力提供了极具前景的路径。
英文摘要：
Subword segmentation is typically applied in preprocessing and stays fixed during training. Alternatively, it can be learned during training to optimise the training objective. In this paper we study the learning dynamics of subword segmentation: if a language model can dynamically optimise tokenisation, how do its subwords evolve during pretraining and finetuning? To explore this, we extend the subword segmental language model (SSLM), a framework for learning subwords during training, to support pretraining and finetuning. We train models for three typologically diverse languages to study learning dynamics across the morphological spectrum: Isi-Xhosa is conjunctive (long word forms composed of many morphemes), Setswana is disjunctive (morphemes written as separate words), and English represents a typological middle ground. We analyse subword dynamics from a linguistic perspective, tracking morphology, productivity, and fertility. We identify four stages of subword learning, with the morphologically complex isi-Xhosa exhibiting greater instability. During finetuning, subword boundaries shift to become finer-grained. Lastly, we show that learnable subwords offers a promising approach to improve text generation and cross-lingual transfer for low-resource, morphologically complex languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
