一个脚本而不是数百个？关于预训练罗马化编码语言模型
One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models
AI总结：
研究通过预训练罗马化和原始文本，探讨罗马化在多语言模型中的有效性，发现分段脚本语言性能损失小，而形态音节脚本语言需更高保真度罗马化以缓解退化。
摘要：
揭示潜在的词汇重叠，罗马化作为一种有效策略，已逐渐成为多语言语言模型（mLMs）中跨语言迁移（XLT）的改进手段。然而，大多数先前的工作都集中在那些最有利于罗马化的设置上：（1）从高资源拉丁语到低资源非拉丁语语言的迁移，以及/或（2）在不同脚本的基因学上密切相关的语言之间。因此，罗马化是否是预训练通用多语言模型的良好表示选择仍不清楚，更准确地说，罗马化是否会导致高资源语言的性能下降。我们通过从头预训练编码语言模型，使用罗马化和原始文本对六种类型多样且高资源的语言进行预训练，探讨了两种潜在的退化来源：（i）脚本特定信息的丢失，以及（ii）由于词汇重叠增加而产生的负面跨语言干扰。使用两种具有不同保真度的罗马化工具，我们发现对于具有分段脚本的语言，性能损失几乎可以忽略不计，而具有形态音节脚本的语言（中文和日语）则出现了退化，更高保真度的罗马化可以缓解但无法完全恢复。重要的是，比较单语语言模型与其多语语言模型的对应物，我们发现没有证据表明增加的子词重叠会引发负面干扰。我们进一步表明，罗马化在分段脚本中提高了编码效率（即生育率），且在可忽略的性能成本下实现。
英文摘要：
Exposing latent lexical overlap, script romanization has emerged as an effective strategy for improving cross-lingual transfer (XLT) in multilingual language models (mLMs). Most prior work, however, focused on setups that favor romanization the most: (1) transfer from high-resource Latin-script to low-resource non-Latin-script languages and/or (2) between genealogically closely related languages with different scripts. It thus remains unclear whether romanization is a good representation choice for pretraining general-purpose mLMs, or, more precisely, if information loss associated with romanization harms performance for high-resource languages. We address this gap by pretraining encoder LMs from scratch on both romanized and original texts for six typologically diverse high-resource languages, investigating two potential sources of degradation: (i) loss of script-specific information and (ii) negative cross-lingual interference from increased vocabulary overlap. Using two romanizers with different fidelity profiles, we observe negligible performance loss for languages with segmental scripts, whereas languages with morphosyllabic scripts (Chinese and Japanese) suffer degradation that higher-fidelity romanization mitigates but cannot fully recover. Importantly, comparing monolingual LMs with their mLM counterpart, we find no evidence that increased subword overlap induces negative interference. We further show that romanization improves encoding efficiency (i.e., fertility) for segmental scripts at a negligible performance cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
