With the productive evolution of large language models (LLMs) in the field of natural language processing (NLP), tons of effort has been made to effectively fine-tune common pre-trained LLMs to fulfill a variety of tasks in one or multiple specific domain. In practice, there are two prevailing ways, in which the adaptation can be achieved: (i) Multiple Independent Models: Pre-trained LLMs are fine-tuned a few times independently using the corresponding training samples from each task. (ii) An Integrated Model: Samples from all tasks are employed to fine-tune a pre-trianed LLM unitedly. To address the high computing cost and seesawing issue simultaneously, we propose a unified framework that implements a 1 + N mutli-task fine-tuning pattern in LLMs using a novel Customized Gate Control (CGC) Low-rank Adaptation (LoRA) algorithm. Our work aims to take an advantage of both MTL (i.e., CGC) and PEFT (i.e., LoRA) scheme. For a given cluster of tasks, we design an innovative layer that contains two types of experts as additional trainable parameters to make LoRA be compatible with MTL. To comprehensively evaluate the proposed framework, we conduct well-designed experiments on two public datasets. The experimental results demonstrate that the unified framework with CGC-LoRA modules achieves higher evaluation scores than all benchmarks on both two datasets.
Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
School of Computer Science and Software Engineering, Southwest Petroleum University(西南石油大学计算机科学与软件工程学院);School of Electronic Information and Artificial Intelligence, Leshan Normal University(乐山师范学院电子信息与人工智能学院)
Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
Zhejiang University(浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新区(滨江)区块链与数据安全研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);National University of Singapore(新加坡国立大学);Guangdong OPPO Mobile Telecommunications Corp., Ltd.(广东欧珀移动通信有限公司)
A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
The Hong Kong Polytechnic University(香港理工大学);Tsinghua University(清华大学);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院);University of Trento(特伦托大学);University of Copenhagen(哥本哈根大学);Peking University(北京大学)
Institute of Science Tokyo(东京科学研究院);RIKEN Center for AI Project(RIKEN人工智能项目研究中心);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt(通用知识处理实验室(UKP实验室),计算机科学系,达姆施塔特技术大学);National Research Center for Applied Cybersecurity ATHENE, Germany(应用网络安全国家研究中心ATHENE,德国);Huawei Hong Kong Research Center(华为香港研究中心)
Parameter-Efficient Fine-Tuning with Discrete Fourier Transform
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);Sun Yat-sen University(中山大学);International Digital Economy Academy(国际数字经济学院);AI Lab, Tencent(腾讯AI实验室)
Amity Institute of Information Technology, Amity University Uttar Pradesh(阿米蒂大学北方邦阿米蒂信息技术学院);Amity School of Engineering & Technology, Amity University Uttar Pradesh(阿米蒂大学北方邦阿米蒂工程技术学院)
LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design
Shanghai Jiao Tong University(上海交通大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);National University of Singapore(新加坡国立大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
Academy for Science and Design (ASD)(科学设计学院);Massachusetts Institute of Technology (MIT)(麻省理工学院)
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
School of Data, Mathematical and Statistical Sciences, University of Central Florida, United States(中佛罗里达大学数据、数学与统计科学学院);College of Computing, Mohammed VI Polytechnic University (UM6P), Morocco(穆罕默德六世理工大学计算机学院);Department of Computer Science, University of Central Florida, United States(中佛罗里达大学计算机科学系)
LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University(文化和旅游部可持续旅游智能评估技术重点实验室);Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China(教育部机器智能与先进计算重点实验室)
LoRa Communication for Agriculture 4.0: Opportunities, Challenges, and Future Directions
College of Engineering, United Arab Emirates University(阿联酋大学工程学院);College of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院);King Abdullah University of Science And Technology(阿卜杜拉国王科技大学)
Advancing AI-Powered Medical Image Synthesis: Insights from MedVQA-GI Challenge Using CLIP, Fine-Tuned Stable Diffusion, and Dream-Booth + LoRA
Department of Computer Science, SCMNS School, Morgan State University, Baltimore, Maryland 21251, USA;Electrical \& Computer Engineering Dept., School of Engineering, Morgan State University, Baltimore, Maryland 21251, USA
FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation
Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai(人工智能与未来网络研究院,北京师范大学,珠海);School of Computer Science and Artificial Intelligence, Zhengzhou University(计算机科学与人工智能学院,郑州大学);School of Software, Nanchang University(软件学院,南昌大学)
Institute of Science Tokyo(东京科学研究所);RIKEN Center for AI Project(理化学研究所AI项目中心);Technical University of Darmstadt(达姆施塔特工业大学);Technical University of Munich(慕尼黑工业大学)