通过在混合专家模型中合并专家来学习更泛化的专家
Learning More Generalized Experts by Merging Experts in Mixture-of-Experts
AI总结：
针对混合专家模型中共享特征学习导致性能下降的问题，提出基于专家使用频率的合并与更新机制，以学习更通用特征并改善多域增量学习。
摘要：
我们观察到，在混合专家模型中引入共享层可能会导致性能下降。这使我们假设，学习共享特征在深度学习中存在挑战，其潜在原因是同一特征被当作各种不同特征来学习。为解决该问题，我们追踪每个专家的使用频率，并合并被选择最频繁的两个专家。随后，我们利用这些专家的组合来更新被选择最少的专家。该方法与随后对路由器专家选择的学习相结合，使模型能够判断被选择最频繁的专家是否以不同方式学习了同一特征。如果确实如此，组合后的专家可以进一步训练，以学习更通用的特征。因此，在应用于多域任务增量学习时，我们的算法增强了迁移学习，并缓解了灾难性遗忘。
英文摘要：
We observe that incorporating a shared layer in a mixture-of-experts can lead to performance degradation. This leads us to hypothesize that learning shared features poses challenges in deep learning, potentially caused by the same feature being learned as various different features. To address this issue, we track each expert's usage frequency and merge the two most frequently selected experts. We then update the least frequently selected expert using the combination of experts. This approach, combined with the subsequent learning of the router's expert selection, allows the model to determine if the most frequently selected experts have learned the same feature differently. If they have, the combined expert can be further trained to learn a more general feature. Consequently, our algorithm enhances transfer learning and mitigates catastrophic forgetting when applied to multi-domain task incremental learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
