arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.36009cs.LGcs.AI
收藏

DuplexTune:用于内存高效的混合专家模型适配的耦合状态重放

DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

Texas A&M University-Commerce(德州农工大学 Commerce 分校);Michigan State University(密歇根州立大学)

Paul Raj, Simon Caldwell, Daniel Whitmore, Mutlu Mete

AI总结:

DuplexTune是一种耦合状态重放框架,通过宽度桥接层实现混合专家模型的内存高效适配,在Qwen1.5-MoE-A2.7B上较检查点微调降低峰值内存、提升吞吐量,仅多语言分数略逊于基线。

摘要:

稀疏专家执行不会移除适配混合专家语言模型所需的存储状态。我们提出DuplexTune,一种通过预训练注意力模块和专家模块重放耦合隐藏状态的框架。可训练的宽度桥接层连接两个残差流与主干网络,桥接层预热先于主干网络适配,且路由器权重保持冻结。我们分析隐式状态恢复方程,确定重建所需的收敛与重放条件。在Qwen1.5-MoE-A2.7B上,单H800评估显示峰值内存为38.8 GB,而使用检查点的监督微调峰值内存为64.8 GB,吞吐量为25.1样本/秒,对比基线的19.9样本/秒。DuplexTune取得67.2%的MMLU、75.6%的GSM8K和7.69的MT-Bench,仅多语言分数弱于检查点基线。结果表明状态存储、吞吐量与适配质量间存在依赖配置的权衡关系。

英文摘要:

Sparse expert execution does not remove the stored states needed to adapt a mixture-of-experts language model. We present DuplexTune, a framework for replaying coupled hidden states through pretrained attention and expert modules. Trainable width bridges connect two residual streams to the backbone. Bridge warm-up precedes backbone adaptation, with router weights kept frozen. We analyze the implicit state-recovery equation and identify the convergence and replay conditions needed for reconstruction. On Qwen1.5-MoE-A2.7B, the single-H800 evaluation reports 38.8 GB peak memory, versus 64.8 GB for supervised fine-tuning with checkpointing, and throughput of 25.1 versus 19.9 samples/s. DuplexTune achieves 67.2% MMLU, 75.6% GSM8K, and 7.69 MT-Bench, with a weaker multilingual score than the checkpointed baseline. The results characterize a configuration-dependent tradeoff among state storage, throughput, and adaptation quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑