Parameter-efficient fine-tuning (PEFT) has emerged as a crucial paradigm for adapting large language models (LLMs) under constrained computational budgets. However, standard PEFT methods often struggle in multi-task fine-tuning settings, where diverse optimization objectives induce task interference and limited parameter budgets lead to representational deficiency. While recent approaches incorporate mixture-of-experts (MoE) to alleviate these issues, they predominantly operate in the spatial domain, which may introduce structural redundancy and parameter overhead. To overcome these limitations, we reformulate adaptation in the spectral domain. Our spectral analysis reveals that different tasks exhibit distinct frequency energy distributions, and that LLM layers display heterogeneous frequency sensitivities. Motivated by these insights, we propose FourierMoE, which integrates the MoE architecture with the inverse discrete Fourier transform (IDFT) for frequency-aware adaptation. Specifically, FourierMoE employs a frequency-adaptive router to dispatch tokens to experts specialized in distinct frequency bands. Each expert learns a set of conjugate-symmetric complex coefficients, preserving complete phase and amplitude information while theoretically guaranteeing lossless IDFT reconstruction into real-valued spatial weights. Extensive evaluations across 28 benchmarks, multiple model architectures, and scales demonstrate that FourierMoE consistently outperforms competitive baselines in both single-task and multi-task settings while using significantly fewer trainable parameters. These results highlight the promise of spectral-domain expert adaptation as an effective and parameter-efficient paradigm for LLM fine-tuning.
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
University of Electronic Science and Technology of China(电子科技大学);The University of Melbourne(墨尔本大学);Nankai University(南开大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Parameter-Efficient Fine-Tuning with Discrete Fourier Transform
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);Sun Yat-sen University(中山大学);International Digital Economy Academy(国际数字经济学院);AI Lab, Tencent(腾讯AI实验室)
Frequency-Aware Continual Learning for Smart Contract Vulnerability Detection with Large Language Models
School of Automation, Guangdong University of Technology(广东工业大学自动化学院);School of Computer Science and Technology, Guangdong University of Technology(广东工业大学计算机科学与技术学院);College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);Department of Computer Science, City University of Hong Kong (Dongguan)(香港城市大学(东莞)计算机科学系);College of Cyber Security, Jinan University(暨南大学网络安全学院);Guangdong Institute of Science and Technology Information(广东省科技情报研究所);Department of Electrical and Computer Engineering, Sungkyunkwan University(成均馆大学电气与计算机工程系)
Un-mixing Test-time Adaptation under Heterogeneous Data Streams
Beijing Academy of Artificial Intelligence(北京人工智能研究院);Alibaba Group(阿里巴巴集团);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);Duke Kunshan University(杜克大学昆山分校)
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院);University of Tübingen(蒂宾根大学);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);The Alan Turing Institute(艾伦·图灵研究所)
Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Sun Yat-sen University(中山大学);Nanyang Technological University(南洋理工大学);Renmin University of China(中国人民大学);Tsinghua University(清华大学)
Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows
School of Science and Engineering University of Missouri-Kansas City(科学与工程学院,密苏里大学-堪萨斯城分校);Department of Mathematics Emory University(数学系,埃默里大学);Center for Applied Scientific Computing Lawrence Livermore National Laboratory(应用科学计算中心,劳伦斯利弗莫尔国家实验室)
F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
School of Computing and Data Science, The University of Hong Kong(计算与数据科学学院,香港大学);Institute for AI Industry Research, Tsinghua University(人工智能产业研究院,清华大学);Beihang University(北航大学)
Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Vision
Dept. of Electrical and Electronics Engineering, Koç University(电子工程系,科奇大学);Codeway AI Research(Codeway人工智能研究);Dept. of Computer Engineering, Middle East Technical University(计算机工程系,中东技术大学)
该研究提出小波微调(WaveFT),利用权重矩阵小波域的稀疏更新实现细粒度参数控制,在视觉任务的参数高效微调方法中达最优,已纳入Hugging Face PEFT库。