Parameter-efficient fine-tuning (PEFT) is a popular method for tailoring pre-trained large language models (LLMs), especially as the models' scale and the diversity of tasks increase. Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional, i.e., significant model changes can be represented with relatively few parameters. However, decreasing the rank encounters challenges with generalization errors for specific tasks when compared to full-parameter fine-tuning. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank, thereby offering improved performance potential. The core idea is to freeze original pretrained weights and train a group of mini LoRAs with only a small number of parameters. This can capture a significant degree of diversity among mini LoRAs, thus promoting better generalization ability. We conduct a theoretical analysis and empirical studies on various NLP tasks. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks, which demonstrates the effectiveness of MELoRA.
Indian Institute of Technology, Madras(马德拉斯印度理工学院);AI4Bharat;National Institute of Information and Communications Technology(国立信息与通信技术研究所);Microsoft(微软)
Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean
Department of Big Data, Chungbuk National University(Chungbuk National University大数据系);Institute of Digital Research and Innovation, Cambodia Academy of Digital Technology(柬埔寨数字技术研究院);Department of Management Information Systems, Chungbuk National University(Chungbuk National University管理信息系);BigData Labs Co., Ltd.(BigData Labs公司)
The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints
Biomedical Information Processing Lab, \' E cole de Technologie Sup\' e rieure, Montr\' e al, Qu\' e bec, Canada;The Interdisciplinary Centre for Security, Reliability;Trust (SnT), University of Luxembourg, Luxembourg;CHU Sainte-Justine Research Center, CHU Sainte-Justine Hospital, University of Montreal, Montr\' e al, Qu\' e bec, Canada
Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints
University of Oxford(牛津大学);NLPie Research(NLPie研究院);Infectious Diseases Data Observatory (IDDO)(传染病数据观测站(IDDO));Sharif University of Technology(谢里夫理工大学);ISARIC(国际严重急性呼吸道感染联盟);Pandemic Sciences Institute(大流行科学研究所);Oxford-Suzhou Centre for Advanced Research(牛津-苏州高等研究中心)
Minimal-Edit Instruction Tuning for Low-Resource Indic GEC
Indian Heritage Language Computing Team Special and Strategic Projects (SSP) Group(印度遗产语言计算团队特殊与战略项目组);Centre for Development of Advanced Computing (C DAC)(高级计算发展中心)
PEFTT: Parameter-Efficient Fine-Tuning for low-resource Tibetan pre-trained language models
Tibet University(西藏大学);Collaborative Innovation Center for Tibet Informatization by MOE and Tibet Autonomous Region(教育部与西藏自治区藏文信息化协同创新中心);Engineering Research Center of Tibetan Information Technology, Ministry of Education(教育部藏文信息技术工程研究中心)