Adapting models pre-trained on large-scale datasets to a variety of downstream tasks is a common strategy in deep learning. Consequently, parameter-efficient fine-tuning methods have emerged as a promising way to adapt pre-trained models to different tasks while training only a minimal number of parameters. While most of these methods are designed for single-task adaptation, parameter-efficient training in Multi-Task Learning (MTL) architectures is still unexplored. In this paper, we introduce MTLoRA, a novel framework for parameter-efficient training of MTL models. MTLoRA employs Task-Agnostic and Task-Specific Low-Rank Adaptation modules, which effectively disentangle the parameter space in MTL fine-tuning, thereby enabling the model to adeptly handle both task specialization and interaction within MTL contexts. We applied MTLoRA to hierarchical-transformer-based MTL architectures, adapting them to multiple downstream dense prediction tasks. Our extensive experiments on the PASCAL dataset show that MTLoRA achieves higher accuracy on downstream tasks compared to fully fine-tuning the MTL model while reducing the number of trainable parameters by 3.6x. Furthermore, MTLoRA establishes a Pareto-optimal trade-off between the number of trainable parameters and the accuracy of the downstream tasks, outperforming current state-of-the-art parameter-efficient training methods in both accuracy and efficiency. Our code is publicly available.
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Institute of Technology(北京理工大学);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室)
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation
University of Tübingen(图宾根大学);Tübingen AI Center(图宾根人工智能中心);Helmholtz Munich(海德堡-穆恩医疗中心);Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);MDSI(慕尼黑数据科学研究所)
DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks
The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Science, VNU-HCM(胡志明市国家大学科学大学);Vietnam National University, Ho Chi Minh City(胡志明市越南国家大学);Trivita AI
Wuhan University(武汉大学);ByteDance(字节跳动);National University of Defense Technology(国防科技大学);Nanyang Technological University(南洋理工大学);The AGH University of Krakow(克拉科夫AGH大学)
Low-Rank Adaptation for Foundation Models: A Comprehensive Review
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Yale University(耶鲁大学);Nanyang Technological University(南洋理工大学);Central University of Finance and Economics(中央财经大学);The Chinese University of Hong Kong(香港中文大学);Xi’an University of Electronic Science and Technology(西安电子科技大学);University of California, Davis(加州大学戴维斯分校);The University of Cambridge(剑桥大学);LOGS AI
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney(澳大利亚机器人中心,航空航天、机械与机电工程学院,悉尼大学);Google DeepMind(谷歌DeepMind)
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动的人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China(北京大学人工智能研究院);Yuanpei College, Peking University, Beijing, China(北京大学元培学院);State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China(通用人工智能国家重点实验室);Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳))
Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
Georgia Institute of Technology(佐治亚理工学院);National Taiwan University(国立台湾大学);University of Palermo(帕尔米奥大学);NVIDIA Research(NVIDIA研究);Hon-Hai Quantum Computing Research Center(鸿海量子计算研究中心)
University of Science and Technology of China(中国科学技术大学);Institute of Intelligent Machines, HFIPS, Chinese Academy of Sciences(中国科学院智能机械研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Anhui Disaster Warning & Agrometeorological Information Center(安徽省灾害预警与农业气象信息中心)