混合模块化专家：从多语言教师模型中蒸馏知识到专业化模块化语言模型
Mixture of Modular Experts: Distilling Knowledge from a Multilingual Teacher into Specialized Modular Language Models
AI总结：
本研究结合知识蒸馏与混合专家构建高效多语言模型，通过对比KD方法和MoE架构，发现组合损失与MoE能有效防止灾难性遗忘并保留跨语言知识，同时开源了相关数据集与代码。
摘要：
本研究结合了知识蒸馏（KD）和混合专家（MoE）来开发模块化、高效的多语言语言模型。关键目标包括评估KD中的自适应与固定alpha方法，并比较模块化MoE架构以处理多领域输入和防止灾难性遗忘。KD将大型语言模型（LLMs）压缩为更小、高效的模型，而MoE通过专业化任务增强了模块性。实验表明，两种KD方法表现相似，自适应alpha带来了微小的改进。组合损失方法提供了更稳定的学习。被训练用于将输入序列分类为英语、法语、德语或Python的路由器，达到了99.95%的精确率、召回率和F1分数，其中Logistic Regression是最有效的分类器。对模块化MoE架构的评估显示，预训练语言专家（PLE）和联合专家嵌入训练（JEET）表现相似，而带有公共专家的MoE（MoE-CE）设置表现略低。在MoE-CE中包含公共专家改善了其性能。关于灾难性遗忘的研究表明，顺序训练导致了显著的遗忘，而使用平衡批次的单次会话训练和MoE方法缓解了这一问题。MoE架构有效地保留了跨多种语言的知识。该研究贡献了开源资源，包括数据集（https://zenodo.org/doi/10.5281/zenodo.12677631）、平衡数据集创建工具（https://github.com/padas-lab-de/multi-language-dataset-creator）和研究代码库（https://github.com/ModMaamari/mixture-modular-experts）。
英文摘要：
This research combines Knowledge Distillation (KD) and Mixture of Experts (MoE) to develop modular, efficient multilingual language models. Key objectives include evaluating adaptive versus fixed alpha methods in KD and comparing modular MoE architectures for handling multi-domain inputs and preventing catastrophic forgetting. KD compresses large language models (LLMs) into smaller, efficient models, while MoE enhances modularity with specialized tasks. Experiments showed similar performance for both KD methods, with marginal improvements from adaptive alpha. A combined loss approach provided more stable learning. The router, trained to classify input sequences into English, French, German, or Python, achieved 99.95% precision, recall, and F1 score, with Logistic Regression being the most effective classifier. Evaluations of modular MoE architectures revealed that Pre-trained Language Experts (PLE) and Joint Expert Embedding Training (JEET) performed similarly, while the MoE with Common Expert (MoE-CE) setup showed slightly lower performance. Including a common expert in MoE-CE improved its performance. Studies on catastrophic forgetting indicated that sequential training led to significant forgetting, while single-session training with balanced batches and the MoE approach mitigated this issue. The MoE architecture preserved knowledge across multiple languages effectively. The research contributes open-sourced resources including the dataset (https://zenodo.org/doi/10.5281/zenodo.12677631), a balanced dataset creation tool (https://github.com/padas-lab-de/multi-language-dataset-creator), and the research codebase (https://github.com/ModMaamari/mixture-modular-experts).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
