利用跨语言专家语言模型打破多语言诅咒
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models
AI总结：
针对多语言语言模型因参数竞争性能不及单语言模型的问题，提出跨语言专家语言模型X-ELM，通过语料子集独立训练缓解竞争，相同算力下性能更优且易扩展新语言、降低训练硬件门槛。
摘要：
尽管多语言语言模型在非英语自然语言处理领域应用广泛，但由于不同语言对模型参数的竞争，其性能往往不及单语言模型。我们提出跨语言专家语言模型（X-ELM），通过在多语言语料的子集上独立训练语言模型来缓解这种竞争。这一过程使X-ELM针对不同语言形成专长，同时作为多语言集成模型仍保持有效性。我们的实验表明，在相同计算预算下，X-ELM在所有测试语言上的表现均优于联合训练的多语言模型，且这些性能提升可迁移至下游任务。除性能提升外，X-ELM还具备额外优势：可迭代添加新的专家模型，使X-ELM适配新语言且不会发生灾难性遗忘。此外，其训练是异步的，降低了多语言训练的硬件要求，推动了多语言建模的普及。
英文摘要：
Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which mitigate this competition by independently training language models on subsets of the multilingual corpus. This process specializes X-ELMs to different languages while remaining effective as a multilingual ensemble. Our experiments show that when given the same compute budget, X-ELM outperforms jointly trained multilingual models across all considered languages and that these gains transfer to downstream tasks. X-ELM provides additional benefits over performance improvements: new experts can be iteratively added, adapting X-ELM to new languages without catastrophic forgetting. Furthermore, training is asynchronous, reducing the hardware requirements for multilingual training and democratizing multilingual modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
