Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA $A$ matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the $A$ matrix is frozen, and only the $B$ matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.
TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Center for Long-term Artificial Intelligence(长期人工智能中心);Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Chinese Academy of Sciences(中国科学院脑认知与脑智能技术重点实验室)
University of Science and Technology of China(中国科学技术大学);Institute of Intelligent Machines, HFIPS, Chinese Academy of Sciences(中国科学院智能机械研究所);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Anhui Disaster Warning & Agrometeorological Information Center(安徽省灾害预警与农业气象信息中心)
Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation
Institute of Automation, CAS(中国科学院自动化研究所);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究所);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation
University of Tübingen(图宾根大学);Tübingen AI Center(图宾根人工智能中心);Helmholtz Munich(海德堡-穆恩医疗中心);Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);MDSI(慕尼黑数据科学研究所)
Low-Rank Adaptation for Foundation Models: A Comprehensive Review
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Yale University(耶鲁大学);Nanyang Technological University(南洋理工大学);Central University of Finance and Economics(中央财经大学);The Chinese University of Hong Kong(香港中文大学);Xi’an University of Electronic Science and Technology(西安电子科技大学);University of California, Davis(加州大学戴维斯分校);The University of Cambridge(剑桥大学);LOGS AI
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动的人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China(北京大学人工智能研究院);Yuanpei College, Peking University, Beijing, China(北京大学元培学院);State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China(通用人工智能国家重点实验室);Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳))
Bayesian Low-Rank Factorization for Robust Model Adaptation
Interactive Systems Lab, Karlsruhe Institut of Technology (KIT), Germany(交互系统实验室,卡尔斯鲁厄理工学院(KIT),德国);InterACT, Carnegie Mellon University (CMU), USA(InterACT,卡内基梅隆大学(CMU),美国)
Efficient Continual Learning in Neural Machine Translation: A Low-Rank Adaptation Approach
Pattern Recognition and Human Language Technology, Universitat Politècnica de València(模式识别与人类语言技术,瓦伦西亚理工大学);Valencian Graduate School and Research Network of Artificial Intelligence, Universitat Politècnica de València(瓦伦西亚人工智能研究生学院与研究网络,瓦伦西亚理工大学)
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)