混合专家模型中的块级路由对卷积神经网络具有可证明的样本效率
Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks
AI总结：
本文首次从理论上证明，在卷积神经网络混合专家模型中，块级路由（pMoE）能以 n/l 的多项式量级降低样本复杂度，并通过过滤无关块、路由判别性块到同一专家来提升泛化能力，实验验证其可避免学习虚假相关性。
摘要：
在深度学习中，混合专家模型（MoE）按样本或按词元激活一个或少数几个专家（子网络），从而显著减少计算量。最近提出的混合专家模型中的块级路由（pMoE）将每个输入划分为 n 个块（或词元），并通过优先级路由将 l 个块（l≪n）发送给每个专家。pMoE 在降低训练和推理成本的同时保持测试精度方面已展现出显著的实证成功。然而，pMoE 以及一般 MoE 的理论解释仍然不明确。针对使用两层卷积神经网络（CNN）混合模型的监督分类任务，我们首次证明 pMoE 能够以 n/l 的多项式量级因子可证明地降低达到理想泛化能力所需的训练样本数量（即样本复杂度），并且优于具有相同甚至更大容量的单专家对应模型。这一优势源于判别性路由特性，该特性在理论和实践中都得到验证：pMoE 路由器能够过滤与标签无关的块，并将相似的类别判别性块路由到同一专家。我们在 MNIST、CIFAR-10 和 CelebA 上的实验结果支持了关于 pMoE 泛化能力的理论发现，并表明 pMoE 可以避免学习虚假相关性。
英文摘要：
In deep learning, mixture-of-experts (MoE) activates one or few experts (sub-networks) on a per-sample or per-token basis, resulting in significant computation reduction. The recently proposed \underline{p}atch-level routing in \underline{MoE} (pMoE) divides each input into $n$ patches (or tokens) and sends $l$ patches ($l\ll n$) to each expert through prioritized routing. pMoE has demonstrated great empirical success in reducing training and inference costs while maintaining test accuracy. However, the theoretical explanation of pMoE and the general MoE remains elusive. Focusing on a supervised classification task using a mixture of two-layer convolutional neural networks (CNNs), we show for the first time that pMoE provably reduces the required number of training samples to achieve desirable generalization (referred to as the sample complexity) by a factor in the polynomial order of $n/l$, and outperforms its single-expert counterpart of the same or even larger capacity. The advantage results from the discriminative routing property, which is justified in both theory and practice that pMoE routers can filter label-irrelevant patches and route similar class-discriminative patches to the same expert. Our experimental results on MNIST, CIFAR-10, and CelebA support our theoretical findings on pMoE's generalization and show that pMoE can avoid learning spurious correlations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
