LOLA——一个开源的大规模多语言大语言模型
LOLA -- An Open-Source Massively Multilingual Large Language Model
AI总结：
本文提出开源大规模多语言大语言模型LOLA，基于稀疏MoE架构在160余种语言上训练，通过专家路由机制缓解多语言诅咒，在多项任务表现优异并促进未来研究。
摘要：
本文提出了LOLA，这是一个使用稀疏Mixture-of-Experts Transformer架构在超过160种语言上训练的大规模多语言大语言模型。我们的架构和实现选择解决了利用语言多样性的挑战，同时保持了效率，并避免了多语言性的常见陷阱。我们对评估结果的分析表明，该模型在自然语言生成和理解任务中具有竞争力的性能。此外，我们展示了学习到的专家路由机制如何利用隐式系统发育语言模式来潜在地缓解多语言诅咒。我们深入探讨了训练过程，分析了数据集，并平衡地探索了模型的优势和局限性。作为一个开源模型，LOLA促进了可复现性，并为未来的研究提供了坚实的基础。我们的发现使得开发在多种语言上具有强大且可扩展性能的计算高效多语言模型成为可能。
英文摘要：
This paper presents LOLA, a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
