MetaMoE：面向隐私保护的混合专家统一中的代理选择
MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
AI总结：
MetaMoE通过利用公共代理数据统一独立训练的领域专家，解决隐私约束下混合专家训练的挑战，采用多样性感知的代理选择方法提升专家协调性。
摘要：
混合专家（MoE）模型通过结合专业专家扩展容量，但现有方法大多假设训练数据集中存储。实际上，数据分布于客户端，隐私限制下无法共享，使统一MoE训练困难。我们提出MetaMoE，一种隐私保护框架，利用公共代理数据作为不可访问私有数据的替代品，将独立训练的领域专家统一为单一MoE。核心是多样性感知的代理选择，选择客户端领域相关且多样化的样本，有效近似私有数据分布并监督路由器学习。这些代理进一步对齐专家训练，提高统一时的专家协调性，同时上下文感知的路由器增强异构输入下的专家选择。在计算机视觉和自然语言处理基准测试中，MetaMoE一致优于最近的隐私保护MoE统一方法。代码可在https://github.com/ws-jiang/MetaMoE获取。
英文摘要：
Mixture-of-Experts (MoE) models scale capacity by combining specialized experts, but most existing approaches assume centralized access to training data. In practice, data are distributed across clients and cannot be shared due to privacy constraints, making unified MoE training challenging. We propose MetaMoE, a privacy-preserving framework that unifies independently trained, domain-specialized experts into a single MoE using public proxy data as surrogates for inaccessible private data. Central to MetaMoE is diversity-aware proxy selection, which selects client-domain-relevant and diverse samples from public data to effectively approximate private data distributions and supervise router learning. These proxies are further used to align expert training, improving expert coordination at unification time, while a context-aware router enhances expert selection across heterogeneous inputs. Experiments on computer vision and natural language processing benchmarks demonstrate that MetaMoE consistently outperforms recent privacy-preserving MoE unification methods. Code is available at https://github.com/ws-jiang/MetaMoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
