arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01762cs.LGcs.AIcs.CLcs.DC
收藏

FourierMoE:大型语言模型的傅里叶混合专家适应

FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models

The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))

Juyong Jiang, Fan Wang, Hong Qi, Sunghun Kim, Jing Tang

AI总结:

本文提出傅里叶MoE,通过频域分析提升大语言模型在多任务适应中的性能,采用频域适应方法减少参数开销,实验证明其在单任务和多任务场景中均优于基线方法。

摘要:

参数高效微调(PEFT)已成为在受限计算预算下适应大型语言模型(LLMs)的关键范式。然而,标准PEFT方法在多任务微调设置中常常遇到困难,因为多样化的优化目标导致任务干扰和有限的参数预算引起表示不足。尽管最近的方法引入混合专家(MoE)来缓解这些问题,但它们大多在空域操作,这可能会引入结构冗余和参数开销。为克服这些限制,我们重新表述适应在频域中。我们的频域分析揭示了不同任务表现出不同的频率能量分布,且LLM层具有异质的频率敏感性。受这些见解的启发,我们提出了FourierMoE,它将MoE架构与反向离散傅里叶变换(IDFT)结合,用于频率感知的适应。具体而言,FourierMoE采用频率自适应路由器,将令牌分配给专门处理不同频率带的专家。每个专家学习一组共轭对称复数系数,保留完整的相位和振幅信息,同时在理论上保证无损IDFT重建为实值空间权重。在28个基准、多种模型架构和规模的广泛评估中,FourierMoE在单任务和多任务设置中均显著优于竞争基线方法,同时使用显著更少的可训练参数。这些结果突显了频域专家适应作为有效且参数高效的LLM微调范式的潜力。

英文摘要:

Parameter-efficient fine-tuning (PEFT) has emerged as a crucial paradigm for adapting large language models (LLMs) under constrained computational budgets. However, standard PEFT methods often struggle in multi-task fine-tuning settings, where diverse optimization objectives induce task interference and limited parameter budgets lead to representational deficiency. While recent approaches incorporate mixture-of-experts (MoE) to alleviate these issues, they predominantly operate in the spatial domain, which may introduce structural redundancy and parameter overhead. To overcome these limitations, we reformulate adaptation in the spectral domain. Our spectral analysis reveals that different tasks exhibit distinct frequency energy distributions, and that LLM layers display heterogeneous frequency sensitivities. Motivated by these insights, we propose FourierMoE, which integrates the MoE architecture with the inverse discrete Fourier transform (IDFT) for frequency-aware adaptation. Specifically, FourierMoE employs a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands. Each expert learns a set of conjugate-symmetric complex coefficients, preserving complete phase and amplitude information while theoretically guaranteeing lossless IDFT reconstruction into real-valued spatial weights. Extensive evaluations across 28 benchmarks, multiple model architectures, and scales demonstrate that FourierMoE consistently outperforms competitive baselines in both single-task and multi-task settings while using significantly fewer trainable parameters. These results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑