通过语言家族专家混合高效普及50种语言的医疗大语言模型
Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts
AI总结：
针对医疗大语言模型适配低资源语言时的数据稀缺问题，提出语言家族专家混合架构，通过语言特定专家、跨语言路由及Post-MoE架构，在保留可解释性的同时增强泛化能力，无需额外参数即可扩展至50种语言。
摘要：
将医疗大语言模型（LLMs）适配到本地语言可降低获取医疗服务的障碍，但数据稀缺仍是重大挑战，尤其对低资源语言而言。为此，我们首先构建高质量医疗数据集并开展分析以确保其质量。为利用多语言LLMs的泛化能力高效扩展至更多资源受限语言，我们借助专家混合（MoE）模块化从多语言视角探索LLMs的内部信息流。技术上，我们提出新颖的MoE路由方法，采用语言特定专家与跨语言路由。受电路理论启发，我们的路由分析揭示“末端扩散”信息流机制：早期层集中跨语言信息流，后期层呈现语言特定分化。该洞见直接推动Post-MoE架构的开发，仅在后期层应用稀疏路由，其余层保持密集。实验结果表明，该方法在保留可解释性的同时增强多语言模型对其他语言的泛化能力。最后，我们引入语言家族专家概念，利用语言先验，实现无需增加额外参数即可将模型扩展至50种语言。
英文摘要：
Adapting medical Large Language Models to local languages can reduce barriers to accessing healthcare services, but data scarcity remains a significant challenge, particularly for low-resource languages. To address this, we first construct a high-quality medical dataset and conduct analysis to ensure its quality. In order to leverage the generalization capability of multilingual LLMs to efficiently scale to more resource-constrained languages, we explore the internal information flow of LLMs from a multilingual perspective using Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE routing method that employs language-specific experts and cross-lingual routing. Inspired by circuit theory, our routing analysis revealed a Spread Out in the End information flow mechanism: while earlier layers concentrate cross-lingual information flow, the later layers exhibit language-specific divergence. This insight directly led to the development of the Post-MoE architecture, which applies sparse routing only in the later layers while maintaining dense others. Experimental results demonstrate that this approach enhances the generalization of multilingual models to other languages while preserving interpretability. Finally, to efficiently scale the model to 50 languages, we introduce the concept of language family experts, drawing on linguistic priors, which enables scaling the number of languages without adding additional parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
