Large language models (LLMs) have significantly advanced generative applications in natural language processing (NLP). Recent trends in model architectures revolve around efficient variants of transformers or state-space/gated-recurrent models (SSMs, GRMs). However, prevailing SSM/GRM-based methods often emulate only a single attention head, potentially limiting their expressiveness. In this work, we propose MossNet, a novel mixture-of-state-space-experts architecture that emulates a linear multi-head attention (MHA). MossNet leverages a mixture-of-experts (MoE) implementation not only in channel-mixing multi-layered perceptron (MLP) blocks but also in the time-mixing SSM kernels to realize multiple "attention heads." Extensive experiments on language modeling and downstream evaluations show that MossNet outperforms both transformer- and SSM-based architectures of similar model size and data budgets. Larger variants of MossNet, trained on trillions of tokens, further confirm its scalability and superior performance. In addition, real-device profiling on a Samsung Galaxy S24 Ultra and an Nvidia A100 GPU demonstrate favorable runtime speed and resource usage compared to similarly sized baselines. Our results suggest that MossNet is a compelling new direction for efficient, high-performing recurrent LLM architectures.
Mixture of In-Context Experts Enhance LLMs' Long Context Awareness
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);XiaoMi AI Lab(小米AI实验室);Career Science Lab, BOSS Zhipin(BOSS直聘职业科学实验室);NLP Center, BOSS Zhipin(BOSS直聘自然语言处理中心)
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Institute for Artificial Intelligence, Peking University, Beijing(人工智能研究院,北京大学,北京);School of Integrated Circuits, Peking University, Beijing(集成电路学院,北京大学,北京)
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA(计算机科学与软件工程系,阿伯拉罕大学,阿伯丁,阿拉巴马州,美国);Department of Information Management, National Central University, Taoyuan, Taiwan(信息管理系,国立中央大学,桃园,台湾)
Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
Advanced Micro Devices, Inc.(超威半导体公司);Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University(西安交通大学人工智能与机器人研究所);University of HongKong(香港大学)
Improving Transformers with Probabilistic Attention Keys
FPT Software AI Center(FPT软件人工智能中心);University of California, Los Angeles(加州大学洛杉矶分校);VinUniversity(文大);Deezer Research(Deezer研究院);Rice University(莱斯大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
多头注意力是最先进的Transformers背后的驱动力,它在各种自然语言处理(NLP)和计算机视觉任务中实现了卓越的性能。已经观察到,对于许多应用,这些注意力头学习冗余嵌入,并且其中大多数可以在不降低模型性能的情况下被移除。受此启发,我们提出了Transformer与混合高斯密钥(transformer-MGK),一种新的Transformer架构,取代冗余的头在每个头的混合密钥的变压器。这些键的混合遵循高斯混合模型,并允许每个注意力头有效地集中在输入序列的不同部分。与传统的Transformer相比,Transformer-MGK加速了训练和推理,参数更少,需要计算的FLOP更少,同时在任务中实现了相当或更好的准确性。transformer-MGK也可以很容易地扩展到使用线性注意力。我们经验证明了Transformer-MGK在一系列实际应用中的优势,包括语言建模和涉及很长序列的任务。在Wikitext-103和Long Range Arena基准测试中,具有4个头的Transformer-MGK获得了与具有8个头的基线Transformers相当或更好的性能。
Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室);Ant Group(蚂蚁集团);School of Software Technology, Zhejiang University(浙江大学软件学院);Zhejiang Key Laboratory of Big Data Intelligent Computing(浙江省大数据智能计算重点实验室)
MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Washington University in St. Louis(圣路易斯华盛顿大学)
Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);Microsoft(微软公司);Microsoft Research Asia(微软亚洲研究院)