arXiv 2402.125502024-10-16cs.CVcs.LG

多线性专家混合：通过因子分解实现可扩展的专家专业化

Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization

Queen Mary University of London（伦敦玛丽女王大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；City University of London（伦敦城市大学）；National and Kapodistrian University of Athens（雅典国立卡波迪斯特里亚大学）；Archimedes AI, Athena RC（阿基米德人工智能公司，雅典娜研究中心）；The Cyprus Institute（塞浦路斯研究所）；Imperial College London（帝国理工学院）

James Oldfield, Markos Georgopoulos, Grigorios G. Chrysos, Christos Tzelepis, Yannis Panagakis, Mihalis A. Nicolaou, Jiankang Deng, Ioannis Patras

备注：: Accepted at NeurIPS 2024. Github: https://github.com/james-oldfield/muMoE. Project page: https://james-oldfield.github.io/muMoE

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出多线性专家混合（μMoE）层，通过因子分解形式对超大权重张量进行隐式计算，以低成本实现可扩展的专家专业化，并在视觉模型微调与预训练中验证了其有效性。

摘要：

专家混合（MoE）范式提供了一种强大的方式，将稠密层分解为更小、模块化的计算，这些计算通常更易于人类解释、调试和编辑。然而，一个主要挑战在于将专家数量扩展到足够高以实现细粒度专业化时的计算成本。在本文中，我们提出多线性专家混合（μMoE）层来解决这一问题，重点关注视觉模型。μMoE层通过完全以因子分解形式对极其庞大的权重张量执行隐式计算，从而实现可扩展的专家专业化。因此，μMoE（1）避免了稠密MoE在推理时限制性的高昂成本，同时（2）不会继承流行的稀疏MoE离散（不可微分）专家路由的训练问题。我们提供了定性和定量证据，表明在为视觉任务微调基础模型时扩展μMoE层能够在类别级别产生更专业化的专家，进一步支持在CelebA属性分类中进行手动偏差校正。最后，我们展示了定性结果，证明在每一层使用参数匹配的μMoE块预训练大型GPT2和MLP-Mixer模型时所实现的专家专业性，同时保持可比的准确率。我们的代码可在以下地址获取：https://github.com/james-oldfield/muMoE。

英文摘要：

The Mixture of Experts (MoE) paradigm provides a powerful way to decompose dense layers into smaller, modular computations often more amenable to human interpretation, debugging, and editability. However, a major challenge lies in the computational cost of scaling the number of experts high enough to achieve fine-grained specialization. In this paper, we propose the Multilinear Mixture of Experts ($μ$MoE) layer to address this, focusing on vision models. $μ$MoE layers enable scalable expert specialization by performing an implicit computation on prohibitively large weight tensors entirely in factorized form. Consequently, $μ$MoEs (1) avoid the restrictively high inference-time costs of dense MoEs, yet (2) do not inherit the training issues of the popular sparse MoEs' discrete (non-differentiable) expert routing. We present both qualitative and quantitative evidence that scaling $μ$MoE layers when fine-tuning foundation models for vision tasks leads to more specialized experts at the class-level, further enabling manual bias correction in CelebA attribute classification. Finally, we show qualitative results demonstrating the expert specialism achieved when pre-training large GPT2 and MLP-Mixer models with parameter-matched $μ$MoE blocks at every layer, maintaining comparable accuracy. Our code is available at: https://github.com/james-oldfield/muMoE.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。