arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.31397cs.LGcs.AI
收藏

控制混合:基于Transformer模型的状态感知微调

Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models

Qualcomm AI Research(高通人工智能研究院)

Duc Anh Nguyen, Tien Ngoc Luu, Tung Pham, Toan Tran

AI总结:

提出Mixture-of-Control(MoC)框架,通过稀疏混合专家机制自适应融合局部和全局控制信号,增强Transformer块间信息交换,实现高效微调。

摘要:

基于状态的微调已成为Transformer权重适应的一种引人注目的替代方案,它将轻量级控制更新到状态而非模型权重,在保持参数效率的同时大幅节省内存。然而,现有的大多数基于状态的方法通常仅应用每块控制更新,这限制了块间信息交换并限制了表示适应。同时,先前实现跨块通信的机制通常引入大量计算开销,降低了它们用于高效微调的实用性。我们引入了控制混合(MoC),一种轻量级微调框架,自适应地整合局部和全局控制信号以增强表示学习。MoC将块级控制状态视为稀疏混合专家过程中的专家,实现跨Transformer块的高效通信。在多种基于Transformer的基准测试上的实证结果表明,MoC在保持可比的内存和计算效率的同时,优于基于状态的方法。

英文摘要:

State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency. However, most existing state-based methods typically apply only per-block control updates, which limits inter-block information exchange and restricts representational adaptation. Meanwhile, prior mechanisms that enable cross-block communication often introduce considerable computational overhead, reducing their practicality for efficient fine-tuning. We introduce Mixture-of-Control (MoC), an efficient cross-layer communication that adaptively integrates local and global control signals to enhance representation learning during adaptation. MoC treats block-wise control states as experts in a sparse mixture-of-experts process, enabling efficient communication across transformer blocks. Experiments on 33 datasets and 10 pretrained models across diverse transformer-based benchmarks demonstrate that MoC consistently outperforms existing state-based fine-tuning methods while maintaining comparable efficiency in memory and compute.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑