DECO：稀疏专家混合模型在端侧设备上实现密集级性能
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
AI总结：
本文提出DECO，一种稀疏的专家混合模型，能够在相同的总参数预算和训练token数量下，实现与密集Transformer相当的性能，并在端侧设备上实现高效部署。
摘要：
尽管混合专家（MoE）可以在不按比例增加计算的情况下扩展模型容量，但其巨大的总参数足迹造成了显著的存储和内存访问瓶颈，阻碍了同时需要高性能、低计算成本和小存储开销的端侧部署。为了实现这些特性，我们提出了DECO，一种稀疏的MoE架构，旨在在相同的总参数预算和训练token数量下匹配密集Transformer的性能。DECO利用可微且灵活的基于ReLU的路由增强，通过可学习的专家级缩放，自适应地平衡路由和共享专家的贡献。此外，我们引入了NormSiLU，一种在SiLU操作前对输入进行归一化的激活函数，产生更稳定的路由专家激活比率趋势和更高的内在稀疏性水平。我们还发现使用非门控MLP专家与基于ReLU的路由具有经验优势，表明MoE架构可能简化。实验表明，DECO仅激活20%的路由专家，能够实现密集性能，并优于现有的MoE基线。我们的专用加速内核在Jetson AGX Orin上相比密集推理实现了2.93倍的加速。代码和检查点可在https://github.com/thunlp/DECO获取。
英文摘要：
While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computational cost, and small storage overhead. To achieve these properties, we present DECO, a sparse MoE architecture designed to match the performance of dense Transformers under identical total parameter budgets and training tokens. DECO utilizes the differentiable and flexible ReLU-based routing enhanced by learnable expert-wise scaling, which adaptively balances the contributions of routed and shared experts. Furthermore, we introduce NormSiLU, an activation function that normalizes inputs prior to SiLU operators, producing a more stable trend of routed-expert activation ratio and a higher intrinsic sparsity level. We also identify an empirical advantage in using non-gated MLP experts with ReLU-based routing, indicating the possibility of MoE architecture simplification. Experiments demonstrate that DECO, activating only 20% of routed experts, matches dense performance and outperforms established MoE baselines. Our specialized acceleration kernel delivers a 2.93$\times$ speedup on Jetson AGX Orin compared with dense inference. Code and checkpoints are available at https://github.com/thunlp/DECO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
