arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.12928cs.CL
收藏

Finedeep:通过多层细粒度专家缓解密集 LLM 中的稀疏激活

Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts

College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Chinese Academy of Sciences(中国科学院);Kuaishou Technology(快手科技);Tsinghua University(清华大学)

Leiyu Pan, Zhenpeng Su, Minxuan Lv, Yizhe Xiong, Xiangwen Zhang, Zijia Lin, Hui Chen, Jungong Han, Guiguang Ding, Cheng Luo, Di Zhang, Kun Gai, Deyi Xiong

AI总结:

针对密集 LLM 激活稀疏问题,提出多层细粒度专家架构 Finedeep 与新型路由机制,在相近参数和算力下提升困惑度及基准表现。

摘要:

大语言模型已在广泛任务中展现出卓越性能。然而,密集模型通常受到稀疏激活的困扰,即许多激活值趋向于零(也就是处于未激活状态)。我们认为,这可能限制对模型表示空间的高效探索。为缓解该问题,我们提出了 Finedeep,这是一种面向密集模型的深层细粒度专家架构。我们的框架将传统密集模型的前馈神经网络层划分为小型专家,并将它们排列在多个子层中。文中还提出了一种新颖的路由机制,用于确定每个专家的贡献。我们在多种模型规模上开展了大量实验,结果表明,在参数量和浮点运算量保持相当的情况下,我们的方法在困惑度和基准性能方面显著优于传统密集架构。此外,我们发现 Finedeep 在平衡深度与宽度时可取得最佳结果,具体是通过调整专家子层数量以及每个子层中的专家数量实现的。实证结果证实,Finedeep 能有效缓解稀疏激活,并高效利用密集模型中的表示能力。

英文摘要:

Large language models have demonstrated exceptional performance across a wide range of tasks. However, dense models usually suffer from sparse activation, where many activation values tend towards zero (i.e., being inactivated). We argue that this could restrict the efficient exploration of model representation space. To mitigate this issue, we propose Finedeep, a deep-layered fine-grained expert architecture for dense models. Our framework partitions the feed-forward neural network layers of traditional dense models into small experts, arranges them across multiple sub-layers. A novel routing mechanism is proposed to determine each expert's contribution. We conduct extensive experiments across various model sizes, demonstrating that our approach significantly outperforms traditional dense architectures in terms of perplexity and benchmark performance while maintaining a comparable number of parameters and floating-point operations. Moreover, we find that Finedeep achieves optimal results when balancing depth and width, specifically by adjusting the number of expert sub-layers and the number of experts per sub-layer. Empirical results confirm that Finedeep effectively alleviates sparse activation and efficiently utilizes representation capacity in dense models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑