arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.26182cs.CL
收藏

MossNet:状态空间专家混合即多头注意力

MossNet: Mixture of State-Space Experts is a Multi-Head Attention

Samsung Research America(三星美国研究院)

Shikhar Tuli, James Seale Smith, Haris Jeelani, Chi-Heng Lin, Abhishek Patel, Vasili Ramanishka, Yen-Chang Hsu, Hongxia Jin

AI总结:

针对现有SSM/GRM方法仅模拟单注意力头限制表达能力的缺陷,提出MossNet架构,在MLP块和SSM内核中均采用MoE以模拟线性多头注意力,实验证明其性能与可扩展性优于Transformer和SSM基线。

摘要:

大型语言模型(LLMs)显著推进了自然语言处理(NLP)中的生成式应用。模型架构的最新趋势围绕Transformer的高效变体或状态空间/门控循环模型(SSMs, GRMs)展开。然而,现有的基于SSM/GRM的方法通常仅模拟单个注意力头,这可能限制了其表达能力。在这项工作中,我们提出了MossNet,一种模拟线性多头注意力(MHA)的新型状态空间专家混合架构。MossNet不仅在通道混合多层感知机(MLP)块中利用混合专家(MoE)实现,还在时间混合SSM内核中采用MoE来实现多个“注意力头”。在语言建模和下游评估上的大量实验表明,MossNet优于具有相似模型大小和数据预算的基于Transformer和SSM的架构。在数万亿tokens上训练的MossNet更大变体进一步证实了其可扩展性和卓越性能。此外,在Samsung Galaxy S24 Ultra和Nvidia A100 GPU上的真实设备性能分析表明,与相似规模的基线相比,其具有更佳的运行时速度和资源使用率。我们的结果表明,MossNet是高效、高性能循环LLM架构的一个极具吸引力的新方向。

英文摘要:

Large language models (LLMs) have significantly advanced generative applications in natural language processing (NLP). Recent trends in model architectures revolve around efficient variants of transformers or state-space/gated-recurrent models (SSMs, GRMs). However, prevailing SSM/GRM-based methods often emulate only a single attention head, potentially limiting their expressiveness. In this work, we propose MossNet, a novel mixture-of-state-space-experts architecture that emulates a linear multi-head attention (MHA). MossNet leverages a mixture-of-experts (MoE) implementation not only in channel-mixing multi-layered perceptron (MLP) blocks but also in the time-mixing SSM kernels to realize multiple "attention heads." Extensive experiments on language modeling and downstream evaluations show that MossNet outperforms both transformer- and SSM-based architectures of similar model size and data budgets. Larger variants of MossNet, trained on trillions of tokens, further confirm its scalability and superior performance. In addition, real-device profiling on a Samsung Galaxy S24 Ultra and an Nvidia A100 GPU demonstrate favorable runtime speed and resource usage compared to similarly sized baselines. Our results suggest that MossNet is a compelling new direction for efficient, high-performing recurrent LLM architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑