MoECollab：通过协作式混合专家实现大语言模型开发的民主化
MoECollab: Democratizing LLM Development Through Collaborative Mixture of Experts
AI总结：
提出 MoECollab 框架，以可训练门控网络协调分布式专家模块，支持低资源协作开发 LLM，并在准确率、计算成本和专家利用率上取得显著提升。
摘要：
大语言模型（LLM）开发日益集中化，使参与主体局限于资源充足的组织。本文提出 MoECollab，这是一个利用混合专家（MoE）架构实现分布式、协作式 LLM 开发的新颖框架。通过将单体模型分解为由可训练门控网络协调的专业化专家模块，该框架使不同贡献者无论计算资源多少都能参与。我们提供了完整技术实现，涵盖专家动态、门控机制和集成策略的数学基础。在多个数据集上的实验表明，本方法相较基线模型取得 3%–7% 的准确率提升，同时将计算需求降低 34%。专家专业化带来显著的领域特定增益：通用分类任务的 F1 分数从 51% 提升至 88%，新闻分类的准确率从 23% 提升至 44%。我们形式化了路由熵优化问题，并证明适当的正则化技术可使专家利用率提高 14%。这些结果验证了 MoECollab 是通过架构支持的协作实现 LLM 开发民主化的有效方法。
英文摘要：
Large Language Model (LLM) development has become increasingly centralized, limiting participation to well-resourced organizations. This paper introduces MoECollab, a novel framework leveraging Mixture of Experts (MoE) architecture to enable distributed, collaborative LLM development. By decomposing monolithic models into specialized expert modules coordinated by a trainable gating network, our framework allows diverse contributors to participate regardless of computational resources. We provide a complete technical implementation with mathematical foundations for expert dynamics, gating mechanisms, and integration strategies. Experiments on multiple datasets demonstrate that our approach achieves accuracy improvements of 3-7% over baseline models while reducing computational requirements by 34%. Expert specialization yields significant domain-specific gains, with improvements from 51% to 88% F1 score in general classification and from 23% to 44% accuracy in news categorization. We formalize the routing entropy optimization problem and demonstrate how proper regularization techniques lead to 14% higher expert utilization rates. These results validate MoECollab as an effective approach for democratizing LLM development through architecturally-supported collaboration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
