语言变迁下的持续学习
Continual Learning Under Language Shift
AI总结：
研究语言变迁下持续更新语言模型的利弊，发现前向迁移普遍积极，后向迁移取决于语言顺序与特征，语言污染与句法相似性共同解释结果。
摘要：
近期，语言模型预训练在数据和模型规模上的增加带来了巨大的训练成本。在随时间推移出现新数据的场景中，更新模型而非完全重新训练将带来显著收益。我们研究了当新数据来自新语言时更新语言模型的利弊——即语言变迁下的持续学习。从一个单语英语模型出发，我们逐步添加丹麦语、冰岛语和挪威语数据，以探究在三种不同模型规模下，前向和后向迁移效应如何依赖于预训练顺序和语言特征。结果表明，前向迁移总体上是积极的且与语言顺序无关，而后向迁移则可能为积极或消极，取决于新语言的顺序和特征。我们探讨了多个潜在解释因素，发现语言污染与句法相似性的结合最能解释我们的结果。
英文摘要：
The recent increase in data and model scale for language model pre-training has led to huge training costs. In scenarios where new data become available over time, updating a model instead of fully retraining it would therefore provide significant gains. We study the pros and cons of updating a language model when new data comes from new languages -- the case of continual learning under language shift. Starting from a monolingual English language model, we incrementally add data from Danish, Icelandic, and Norwegian to investigate how forward and backward transfer effects depend on pre-training order and characteristics of languages, for three different model sizes. Our results show that, while forward transfer is largely positive and independent of language order, backward transfer can be positive or negative depending on the order and characteristics of new languages. We explore a number of potentially explanatory factors and find that a combination of language contamination and syntactic similarity best fits our results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
