通过稀疏混合专家中的无监督训练探索专家特化
Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
AI总结：
本文提出SMoE-VAE架构，通过无监督专家路由在QuickDraw数据集上发现超越人类定义的子类别结构，揭示数据量与专家特化的权衡，为MoE架构设计提供指导。
摘要：
理解神经网络的内部组织仍然是深度学习可解释性中的一个基本挑战。我们通过探索一种新颖的稀疏混合专家变分自编码器（SMoE-VAE）架构来应对这一挑战。我们在QuickDraw数据集上测试了我们的模型，将无监督专家路由与由真实标签引导的有监督基线进行了比较。令人惊讶的是，我们发现无监督路由始终能实现更优的重建性能。专家学习识别有意义的子类别结构，这些结构往往超越人类定义的类别边界。通过t-SNE可视化和重建分析，我们研究了MoE模型如何揭示与模型目标比预定义标签更一致的基础数据结构。此外，我们对数据集规模影响的研究提供了关于数据量与专家特化之间权衡的见解，为设计高效的MoE架构提供了指导。
英文摘要：
Understanding the internal organization of neural networks remains a fundamental challenge in deep learning interpretability. We address this challenge by exploring a novel Sparse Mixture of Experts Variational Autoencoder (SMoE-VAE) architecture. We test our model on the QuickDraw dataset, comparing unsupervised expert routing against a supervised baseline guided by ground-truth labels. Surprisingly, we find that unsupervised routing consistently achieves superior reconstruction performance. The experts learn to identify meaningful sub-categorical structures that often transcend human-defined class boundaries. Through t-SNE visualizations and reconstruction analysis, we investigate how MoE models uncover fundamental data structures that are more aligned with the model's objective than predefined labels. Furthermore, our study on the impact of dataset size provides insights into the trade-offs between data quantity and expert specialization, offering guidance for designing efficient MoE architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
