BlockFFN：面向端侧加速友好、具备块级激活稀疏性的混合专家模型
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
AI总结：
针对传统MoE路由不灵活、块级稀疏性低不利于端侧加速的问题，提出BlockFFN架构，通过可微路由、CLS感知训练及结合稀疏性与投机解码的加速内核，实现更优性能与端侧3.67倍最高加速。
摘要：
为缓解大语言模型（LLM）的计算负担，以混合专家（MoE）为代表的具备激活稀疏性的架构受到越来越多关注。然而，原生MoE的路由不可微且缺乏灵活性，会损害模型性能。此外，尽管每个token仅激活少量参数，这类稀疏激活架构的块级稀疏性较低，即多个连续token激活的参数并集占比很大。这种稀疏模式不利于低资源场景（如端侧设备）下的加速，且与主流加速技术（如投机解码）不兼容。为解决这些挑战，我们提出了一种新型MoE架构BlockFFN，以及其高效训练与部署技术。具体而言，我们使用融合了ReLU激活与RMSNorm的路由器，实现可微且灵活的路由。其次，为同时提升token级稀疏性（TLS）与块级稀疏性（CLS），我们设计了感知CLS的训练目标，使BlockFFN更适配加速需求。最后，我们实现了高效加速内核，首次将激活稀疏性与投机解码相结合。实验结果表明，BlockFFN性能优于其他MoE基线，实现了超过80%的TLS与70%的8-token CLS。我们的内核在真实端侧设备上相比稠密模型最高实现3.67倍加速。所有代码与检查点均公开可用（https://github.com/thunlp/BlockFFN）。
英文摘要：
To alleviate the computational burden of large language models (LLMs), architectures with activation sparsity, represented by mixture-of-experts (MoE), have attracted increasing attention. However, the non-differentiable and inflexible routing of vanilla MoE hurts model performance. Moreover, while each token activates only a few parameters, these sparsely-activated architectures exhibit low chunk-level sparsity, indicating that the union of multiple consecutive tokens activates a large ratio of parameters. Such a sparsity pattern is unfriendly for acceleration under low-resource conditions (e.g., end-side devices) and incompatible with mainstream acceleration techniques (e.g., speculative decoding). To address these challenges, we introduce a novel MoE architecture, BlockFFN, as well as its efficient training and deployment techniques. Specifically, we use a router integrating ReLU activation and RMSNorm for differentiable and flexible routing. Next, to promote both token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training objectives are designed, making BlockFFN more acceleration-friendly. Finally, we implement efficient acceleration kernels, combining activation sparsity and speculative decoding for the first time. The experimental results demonstrate the superior performance of BlockFFN over other MoE baselines, achieving over 80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67$\times$ speedup on real end-side devices than dense models. All codes and checkpoints are available publicly (https://github.com/thunlp/BlockFFN).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
