Large language model (LLM) is considered a milestone towards achieving Artificial General Intelligence (AGI). With its advanced emergent capabilities, it adapt to a wide range of specific applications. Fine-tuning LLMs for various downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) is well-known for its parameter efficiency. It can reduce the number of parameters needed to fine-tune LLMs by several orders of magnitude. However, LoRA-based approaches encounter a significant limitation due to the bottleneck imposed by rank one decomposition. As the parameters count in LLMs increase, even rank one decomposition might surpass the number of parameters truly necessary for handling more downstream tasks. In this paper, we propose a new method for Parameter-Efficient Fine-Tuning (PEFT) via deconvolution in subspace, dubbed as DCFT. We innovatively use deconvolution to complete details and enhance knowledge in subspace incremental matrices, and dynamically control parameters by adjusting the kernel size, unconstrained by rank-one decomposition. Extensive experiments are conducted to validate the effectiveness of DCFT. Results show that compared to LoRA, DCFT achieve an 8$\times$ reduction in parameters, and still achieves highly impressive performance. Our code is available here: https://github.com/Godz-z/DCFT.
Tsinghua University(清华大学);Zhongguancun Laboratory(中关村实验室);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·本·扎耶德人工智能大学);University of California San Diego(加州大学圣地亚哥分校);Massachusetts Institute of Technology(麻省理工学院)
SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
Shanghai Qi Zhi Institute(上海启智研究院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Xiongan AI Institute(雄安人工智能研究院)
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);Imperial College London(帝国理工学院);Technische Universität Berlin (TU Berlin)(柏林工业大学);Ocean University of China(中国海洋大学);The Hong Kong Polytechnic University(香港理工大学)
Zeroth-Order Fine-Tuning of LLMs in Random Subspaces
Beijing Normal University(北京师范大学);Singapore Management University(新加坡国立大学);TAL Education Group(TAL教育集团);Engineering Research Center of Intelligent Technology and Educational Application (MOE)(智能技术与教育应用工程研究中心(教育部))
Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Georgia Institute of Technology(佐治亚理工学院);Massachusetts Institute of Technology(麻省理工学院)
Mila - Quebec AI Institute(魁北克AI研究所);Polytechnique Montréal(蒙特利尔理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);CIFAR AI Chair(CIFAR人工智能主席);Google DeepMind(谷歌DeepMind)
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning
Tsinghua University(清华大学);Renmin University of China(中国人民大学);International Innovation Center of Tsinghua University(清华大学国际创新中心);Quan Cheng Laboratory(泉城实验室);Tencent Inc.(腾讯公司)
A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);King Abdullah University of Science and Technology(卡塔尔国王大学科学与技术研究院);University of Oxford(牛津大学)
PIXEL: Adaptive Steering Via Position-wise Injection with eXact Estimated Levels under Subspace Calibration
University of Queensland(昆士兰大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学);Provable Responsible AI and Data Analytics (PRADA) Lab(可证责任AI与数据分析师实验室);King Abdullah University of Science and Technology (KAUST)(国王 Abdullah 科学与技术大学)
Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models
University of Electronic Science and Technology of China(电子科技大学);Shenzhen Institute for Advanced Study, UESTC(电子科技大学深圳高等研究院);Mila(米拉研究所);Université de Montréal(蒙特利尔大学)
LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
Department of Statistics, University of Warwick, UK.(威斯敏斯特大学统计学系);Department of Computer Science, University of Warwick, UK.(威斯敏斯特大学计算机科学系);Centre for Discrete Mathematics and its Applications (DIMAP), University of Warwick, UK.(威斯敏斯特大学离散数学及其应用中心(DIMAP))