混合专家模型对特征噪声的鲁棒性
Robustness of Mixtures of Experts to Feature Noise
AI总结：
研究混合专家模型在特征噪声下的鲁棒性，发现稀疏专家激活能作为噪声滤波器，相比密集网络具有更低的泛化误差、更强的鲁棒性和更快的收敛速度。
摘要：
尽管混合专家（MoE）模型在实践中取得了成功，但其为何能在参数规模相当的情况下超越密集网络仍不清楚。我们研究了一个等参数设置，其中输入具有潜在的模块化结构但被特征噪声破坏，这作为内部激活噪声的代理。我们表明，稀疏专家激活起到了噪声滤波器的作用：与密集估计器相比，MoE在特征噪声下实现了更低的泛化误差、对扰动的更强鲁棒性以及更快的收敛速度。在合成数据和真实语言任务上的实验结果证实了理论见解，展示了稀疏模块化计算带来的持续鲁棒性和效率提升。
英文摘要：
Despite their practical success, it remains unclear why Mixture of Experts (MoE) models can outperform dense networks beyond sheer parameter scaling. We study an iso-parameter regime where inputs exhibit latent modular structure but are corrupted by feature noise, a proxy for noisy internal activations. We show that sparse expert activation acts as a noise filter: compared to a dense estimator, MoEs achieve lower generalization error under feature noise, improved robustness to perturbations, and faster convergence speed. Empirical results on synthetic data and real-world language tasks corroborate the theoretical insights, demonstrating consistent robustness and efficiency gains from sparse modular computation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
