arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.10253cs.LGcs.AIcs.CRcs.CV
收藏

关于混合专家模型的对抗鲁棒性

On the Adversarial Robustness of Mixture of Experts

Google Research(谷歌研究院)

Joan Puigcerver, Rodolphe Jenatton, Carlos Riquelme, Pranjal Awasthi, Srinadh Bhojanapalli

摘要:

对抗鲁棒性是神经网络的一个重要特性。经验表明,它受到其规模的影响,较大的网络通常更强大。最近,Bubeck和Sellke证明了函数的Lipschitz常数的下界,该函数适合训练数据的参数数量。这就提出了一个有趣的开放性问题,具有更多参数但不一定需要更多计算成本的函数是否具有更好的鲁棒性?我们研究这个问题的稀疏混合专家模型(MoEs),这使得有可能扩大模型的大小大致恒定的计算成本。我们从理论上表明,在一定的条件下的路由和数据的结构,MOE可以有显着较小的Lipschitz常数比他们的密集同行。当输入的最高权重专家实现足够不同的功能时,MoE的鲁棒性可能会受到影响。接下来,我们使用对抗攻击在ImageNet上实证评估MoE的鲁棒性,并证明它们确实比具有相同计算成本的密集模型更鲁棒。我们进行了关键的观察,显示了MoEs对专家选择的鲁棒性,突出了在实践中训练的模型中专家的冗余性。

英文摘要:

Adversarial robustness is a key desirable property of neural networks. It has been empirically shown to be affected by their sizes, with larger networks being typically more robust. Recently, Bubeck and Sellke proved a lower bound on the Lipschitz constant of functions that fit the training data in terms of their number of parameters. This raises an interesting open question, do -- and can -- functions with more parameters, but not necessarily more computational cost, have better robustness? We study this question for sparse Mixture of Expert models (MoEs), that make it possible to scale up the model size for a roughly constant computational cost. We theoretically show that under certain conditions on the routing and the structure of the data, MoEs can have significantly smaller Lipschitz constants than their dense counterparts. The robustness of MoEs can suffer when the highest weighted experts for an input implement sufficiently different functions. We next empirically evaluate the robustness of MoEs on ImageNet using adversarial attacks and show they are indeed more robust than dense models with the same computational cost. We make key observations showing the robustness of MoEs to the choice of experts, highlighting the redundancy of experts in models trained in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑