FlexMoRE: 一种灵活的秩异质专家混合模型，用于高效联邦训练的大型语言模型
FlexMoRE: A Flexible Mixture of Rank-heterogeneous Experts for Efficient Federatedly-trained Large Language Models
AI总结：
FlexMoRE通过灵活的秩异质专家混合模型，在联邦训练的大型语言模型中实现更高效的性能，使用低秩适配器替代完整专家，提升任务表现并减少参数量。
摘要：
最近混合专家架构的进步表明，个体专家模型可以联邦式训练，即在与其他专家隔离的情况下，通过使用共同的基础模型来促进协调。然而，我们假设并非所有领域都需要完整的专家模型，低秩适配器可能就足够了。在这里，我们介绍了FlexMoRE，一种灵活的秩异质专家混合模型，它可以是完整的专家或合适秩的适配器。我们系统地研究了专家秩与下游任务性能之间的权衡，通过评估6个秩从2^0到2^14的专家，产生了覆盖150种混合（96种2专家，54种7专家）的实验，这些实验在120个任务上进行评估。在我们的实验中，我们基于FlexOlmo构建，将其预训练的专家转换为低秩版本。我们的回归分析从专家秩到下游任务性能揭示，最佳性能的秩在推理密集型基准上比知识密集型基准显著更高。这些关于秩敏感性的发现对内存效率有直接影响：使用最优秩，FlexMoRE在参数量仅为FlexOlmo的三分之一（10.75B vs 33.27B）的情况下，比基线FlexOlmo风格的完整专家混合模型在下游任务上表现更好（平均得分47.18 vs 45.46）。所有代码都将公开。
英文摘要：
Recent advances in mixture-of-experts architectures have shown that individual experts models can be trained federatedly, i.e., in isolation from other experts by using a common base model to facilitate coordination. However, we hypothesize that full-sized experts may not be necessary for all domains and that instead low-rank adapters may be sufficient. Here, we introduce FlexMoRE, a Flexible Mixture of Rank-heterogenous Experts, which may be either full-sized experts or adapters of a suitable rank. We systematically investigate the trade-off between expert rank and downstream task performance by evaluating $6$ experts with ranks $2^0$ to $2^{14}$ resulting in experiments covering 150 mixtures (96 with 2 experts, 54 with 7 experts) that are evaluated across $120$ tasks. For our experiments, we build on FlexOlmo and turn its pre-trained experts into low-rank versions. Our regression analysis from expert rank to downstream task performance reveals that the best-performing rank is substantially higher for reasoning-heavy benchmarks than for knowledge-heavy benchmarks. These findings on rank sensitivity come with direct implications for memory efficiency: Using optimal ranks, FlexMoRE yields improved downstream task performance (average score $47.18$) compared to the baseline FlexOlmo-style mixture of full-sized experts (average score $45.46$) at less than one third the parameters ($10.75$B for FlexMoRE vs. $33.27$B for FlexOlmo). All code will be made available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
