用于CNN可解释性的稀疏门控混合专家层
Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability
AI总结：
本文将稀疏门控MoE层引入视觉CNN，提出软、硬约束训练方法以稳定训练并揭示专家按语义类别或目标尺寸自发专业化，从而增强模型可解释性。
摘要：
稀疏门控混合专家（MoE）层最近已成功应用于扩展大型transformer，尤其是语言建模任务。稀疏MoE层一个有趣的副作用是，它们通过自然的专家专业化为模型赋予内在可解释性。在本工作中，我们将稀疏MoE层应用于面向计算机视觉任务的CNN，并分析由此对模型可解释性产生的影响。为稳定MoE训练，我们提出了基于软约束和硬约束的两种方法。在硬约束下，某些专家的权重被允许变为零；而软约束则通过额外的辅助损失来平衡各专家的贡献。结果表明，软约束能更好地处理专家利用问题并支持专家专业化过程，而硬约束维持更具泛化性的专家，并提升整体模型性能。我们的发现证明，专家能够隐式地聚焦于输入空间的各个子域。例如，为CIFAR-100图像分类训练的专家在没有预先进行数据聚类的情况下，会专门识别花卉或动物等不同领域。使用RetinaNet和COCO数据集的实验进一步表明，目标检测专家也可以专门检测不同尺寸的对象。
英文摘要：
Sparsely-gated Mixture of Expert (MoE) layers have been recently successfully applied for scaling large transformers, especially for language modeling tasks. An intriguing side effect of sparse MoE layers is that they convey inherent interpretability to a model via natural expert specialization. In this work, we apply sparse MoE layers to CNNs for computer vision tasks and analyze the resulting effect on model interpretability. To stabilize MoE training, we present both soft and hard constraint-based approaches. With hard constraints, the weights of certain experts are allowed to become zero, while soft constraints balance the contribution of experts with an additional auxiliary loss. As a result, soft constraints handle expert utilization better and support the expert specialization process, while hard constraints maintain more generalized experts and increase overall model performance. Our findings demonstrate that experts can implicitly focus on individual sub-domains of the input space. For example, experts trained for CIFAR-100 image classification specialize in recognizing different domains such as flowers or animals without previous data clustering. Experiments with RetinaNet and the COCO dataset further indicate that object detection experts can also specialize in detecting objects of distinct sizes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
