LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
East China Normal University(华东师范大学);Sanming University(三明大学);Key Laboratory of Multimedia Trusted Perception(多媒体可信感知关键实验室);Efficient Computing of Ministry of Education of China, Xiamen University, China(中国教育部高效计算实验室,厦门大学,中国)
Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education(交通大数据与人工智能重点实验室(北京交通大学),教育部);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Huazhong University of Science and Technology(华中科技大学);Shenzhen Technology University(深圳技术大学);City University of Hong Kong(香港城市大学);Shenzhen University(深圳大学)
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc.(百度公司)
ChunkWise LoRA: Adaptive Sequence Partitioning for Memory-Efficient Low-Rank Adaptation and Accelerated LLM Inference
Bentley University USA(伯克利大学);Cornell University USA(康奈尔大学);University of California Berkeley USA(加州大学伯克利分校);George Mason University USA(乔治·马歇尔大学);Georgia Institute of Technology USA(佐治亚理工学院)
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
iFLYTEK Co., Ltd(科大讯飞股份有限公司);University of Pennsylvania(宾夕法尼亚大学);Lenovo Connect Co., Ltd(联想智联有限公司);Niuxin Network Technology Co., Ltd(牛信网络科技有限公司);East China Normal University(华东师范大学)
MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
Dynamic Adaptation of LoRA Fine-Tuning for Efficient and Task-Specific Optimization of Large Language Models
New York University(纽约大学);University of Minnesota(明尼苏达大学);Tulane University(杜兰大学);The Chinese University of Hong Kong(香港中文大学);Stevens Institute of Technology(史蒂文斯理工学院)