arXiv 2308.101102023-08-19cs.CVcs.AIcs.LG

面向卷积神经网络的鲁棒混合专家训练

Robust Mixture-of-Expert Training for Convolutional Neural Networks

Michigan State University（密歇根州立大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；MIT（麻省理工学院）；Harvard University（哈佛大学）；UC Santa Barbara（加州大学圣巴巴拉分校）；Carnegie Mellon University（卡内基梅隆大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）

Yihua Zhang, Ruisi Cai, Tianlong Chen, Guanhua Zhang, Huan Zhang, Pin-Yu Chen, Shiyu Chang, Zhangyang Wang, Sijia Liu

备注：: ICCV 2023

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对 MoE-CNN 在传统对抗训练下路由器与专家难以协同的问题，提出交替训练框架 AdvMoE，在 4 个数据集和 4 种 CNN 上提升鲁棒性并降低逾半推理成本。

摘要：

稀疏门控混合专家（Mixture of Expert，MoE）是一种新兴的深度模型架构，已展现出实现高精度和超高效模型推理的巨大潜力。尽管 MoE 日益流行，但很少有工作研究其在推进卷积神经网络（CNN）方面的潜力，尤其是在对抗鲁棒性方面。由于缺乏鲁棒性已成为 CNN 的主要障碍之一，本文提出问题：如何在对抗层面增强基于 CNN 的 MoE 模型？能否像训练普通 CNN 模型一样鲁棒地训练它？我们的初步研究表明，为普通 CNN 设计的传统对抗训练（AT）机制在增强 MoE-CNN 时不再有效。为更好理解这一现象，我们将 MoE-CNN 的鲁棒性分解为两个维度：路由器的鲁棒性（即用于选择特定数据专家的门控函数）和专家的鲁棒性（即由骨干 CNN 子网络定义、受路由器引导的路径）。分析表明，在普通 AT 中，路由器和专家难以相互适应。因此，我们提出一种新的用于 MoE 的路由器—专家交替对抗训练框架，称为 AdvMoE。在 4 个基准数据集上对 4 种常用 CNN 模型架构的实验验证了该方法的有效性。我们发现，AdvMoE 相比原始密集 CNN 取得 1%～4% 的对抗鲁棒性提升，同时享有稀疏门控 MoE 的效率优势，使推理成本降低超过 50%。代码可在 https://github.com/OPTML-Group/Robust-MoE-CNN 获取。

英文摘要：

Sparsely-gated Mixture of Expert (MoE), an emerging deep model architecture, has demonstrated a great promise to enable high-accuracy and ultra-efficient model inference. Despite the growing popularity of MoE, little work investigated its potential to advance convolutional neural networks (CNNs), especially in the plane of adversarial robustness. Since the lack of robustness has become one of the main hurdles for CNNs, in this paper we ask: How to adversarially robustify a CNN-based MoE model? Can we robustly train it like an ordinary CNN model? Our pilot study shows that the conventional adversarial training (AT) mechanism (developed for vanilla CNNs) no longer remains effective to robustify an MoE-CNN. To better understand this phenomenon, we dissect the robustness of an MoE-CNN into two dimensions: Robustness of routers (i.e., gating functions to select data-specific experts) and robustness of experts (i.e., the router-guided pathways defined by the subnetworks of the backbone CNN). Our analyses show that routers and experts are hard to adapt to each other in the vanilla AT. Thus, we propose a new router-expert alternating Adversarial training framework for MoE, termed AdvMoE. The effectiveness of our proposal is justified across 4 commonly-used CNN model architectures over 4 benchmark datasets. We find that AdvMoE achieves 1% ~ 4% adversarial robustness improvement over the original dense CNN, and enjoys the efficiency merit of sparsity-gated MoE, leading to more than 50% inference cost reduction. Codes are available at https://github.com/OPTML-Group/Robust-MoE-CNN.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。