扩展、简化与适应：基于机器翻译文本预训练的经验教训
Scaling, Simplification, and Adaptation: Lessons from Pretraining on Machine-Translated Text
AI总结：
本文通过将英语机器翻译为印尼语和泰米尔语并预训练GPT-2模型，研究了MT衍生数据的扩展性、源端简化的影响及对母语文本的适应能力，发现MT预训练模型可从扩展和适应中获益，但源端简化有害，且文化敏感任务仍需更多母语数据。
摘要：
大多数语言缺乏足够的数据用于大规模单语预训练，从而形成了一道“数据墙”。多语言预训练虽有所帮助，但受限于语言不平衡和“多语言诅咒”。一种替代方案是利用机器翻译（MT）翻译高资源文本，这引出了三个问题：（1）MT衍生数据如何随模型容量扩展？（2）源端变换（例如，用LLM简化英语）能否提升对母语文本的泛化能力？（3）在有限的母语文本上持续训练时，基于MT衍生数据预训练的模型适应能力如何？我们通过将英语翻译成印度尼西亚语和泰米尔语——两种类型学上相距较远、资源较少的语言——并在来自原始英语和经LLM简化英语的母语或MT衍生语料上预训练GPT-2模型（124M-774M）来研究这些问题。我们评估了母语文本上的交叉熵损失，以及句法探针和下游任务上的准确率。结果表明：（1）MT预训练模型能从扩展中受益；（2）源端简化会损害对母语文本的泛化；（3）在母语文本上适应MT预训练模型通常比仅用母语训练的模型表现更好，即使母语数据更少。然而，需要文化细微差别的任务（例如毒性检测）需要更多接触母语数据。
英文摘要：
Most languages lack sufficient data for large-scale monolingual pretraining, creating a "data wall." Multilingual pretraining helps but is limited by language imbalance and the "curse of multilinguality." An alternative is to translate high-resource text with machine translation (MT), which raises three questions: (1) How does MT-derived data scale with model capacity? (2) Can source-side transformations (e.g., simplifying English with an LLM) improve generalization to native text? (3) How well do models pretrained on MT-derived data adapt when continually trained on limited native text? We investigate these questions by translating English into Indonesian and Tamil--two typologically distant, lower-resource languages--and pretraining GPT-2 models (124M-774M) on native or MT-derived corpora from raw and LLM-simplified English. We evaluate cross-entropy loss on native text, along with accuracy on syntactic probes and downstream tasks. Our results show that (1) MT-pretrained models benefit from scaling; (2) source-side simplification harms generalization to native text; and (3) adapting MT-pretrained models on native text often yields better performance than native-only models, even with less native data. However, tasks requiring cultural nuance (e.g., toxicity detection) demand more exposure to native data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
