State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency. However, most existing state-based methods typically apply only per-block control updates, which limits inter-block information exchange and restricts representational adaptation. Meanwhile, prior mechanisms that enable cross-block communication often introduce considerable computational overhead, reducing their practicality for efficient fine-tuning. We introduce Mixture-of-Control (MoC), an efficient cross-layer communication that adaptively integrates local and global control signals to enhance representation learning during adaptation. MoC treats block-wise control states as experts in a sparse mixture-of-experts process, enabling efficient communication across transformer blocks. Experiments on 33 datasets and 10 pretrained models across diverse transformer-based benchmarks demonstrate that MoC consistently outperforms existing state-based fine-tuning methods while maintaining comparable efficiency in memory and compute.
Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)(深圳市人工智能与机器人研究院);Zhejiang University(浙江大学);Westlake University(西湖大学);SII;Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);Tencent AI Lab(腾讯AI Lab)
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning
Korea University(高丽大学);Johns Hopkins University(约翰斯·霍普金斯大学);Texas A&M University-Commerce(德州农工大学 Commerce 分校);Qualcomm AI Research(高通人工智能研究部);Stanford(斯坦福大学);KAIST(韩国科学技术院);Qualcomm Vietnam Company Limited(高通越南有限公司)
Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
Department of ECE The Ohio State University(电子工程系,俄亥俄州立大学);Department of CSE The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Department of Computer Science University of Houston(计算机科学系,休斯顿大学);Department of ECE Department of CSE The Ohio State University(电子工程系、计算机科学与工程系,俄亥俄州立大学)
Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixtures
Shanghai Center for Mathematical Sciences, Fudan University(复旦大学上海数学中心);Department of Statistics and Data Science, School of Management, Fudan University(复旦大学管理学院统计与数据科学系)
Memory Limitations of Prompt Tuning in Transformers
Department of Mathematics, National University of Singapore(新加坡国立大学数学系);IPAL, IRL2955, Singapore(新加坡IPAL);School of Computing, National University of Singapore(新加坡国立大学计算机学院);Department of Electrical and Computer Engineering, National University of Singapore(新加坡国立大学电气与计算机工程系)
Improving Transformers with Probabilistic Attention Keys
FPT Software AI Center(FPT软件人工智能中心);University of California, Los Angeles(加州大学洛杉矶分校);VinUniversity(文大);Deezer Research(Deezer研究院);Rice University(莱斯大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
多头注意力是最先进的Transformers背后的驱动力,它在各种自然语言处理(NLP)和计算机视觉任务中实现了卓越的性能。已经观察到,对于许多应用,这些注意力头学习冗余嵌入,并且其中大多数可以在不降低模型性能的情况下被移除。受此启发,我们提出了Transformer与混合高斯密钥(transformer-MGK),一种新的Transformer架构,取代冗余的头在每个头的混合密钥的变压器。这些键的混合遵循高斯混合模型,并允许每个注意力头有效地集中在输入序列的不同部分。与传统的Transformer相比,Transformer-MGK加速了训练和推理,参数更少,需要计算的FLOP更少,同时在任务中实现了相当或更好的准确性。transformer-MGK也可以很容易地扩展到使用线性注意力。我们经验证明了Transformer-MGK在一系列实际应用中的优势,包括语言建模和涉及很长序列的任务。在Wikitext-103和Long Range Arena基准测试中,具有4个头的Transformer-MGK获得了与具有8个头的基线Transformers相当或更好的性能。
MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
Rochester Institute of Technology(罗切斯特理工学院);University of Rochester(罗切斯特大学);Meta AI;Rice University(莱斯大学);Case Western Reserve University(凯斯西储大学);U.S. DEVCOM Army Research Laboratory(美国陆军作战能力发展司令部陆军研究实验室);University of Missouri - Kansas City(密苏里大学堪萨斯城分校);UC Davis(加州大学戴维斯分校)
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
Department of Computer Science and Engineering, Korea University, Republic of Korea(韩国大学计算机科学与工程系);KT, Republic of Korea(韩国KT);School of Software, Soongsil University, Republic of Korea(顺天大学软件学院)
Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
University of California San Diego(加州大学圣迭戈分校);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Fermi National Accelerator Laboratory(费米国家加速器实验室);University of California Davis(加州大学戴维斯分校)
Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography
Concordia Institute for Information Systems Engineering (CIISE)(康科德信息系统工程研究所);Concordia University(康科德大学);Thomas Jefferson University Hospital(托马斯·杰斐逊大学医院)