The Hong Kong Polytechnic University(香港理工大学);Tsinghua University(清华大学);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院);University of Trento(特伦托大学);University of Copenhagen(哥本哈根大学);Peking University(北京大学)
Yihua Shao, Minxi Yan, Yang Liu, Siyu Chen, Wenjie Chen, Xinwei Long, Ziyang Yan, Lei Li, Chenyu Zhang, Nicu Sebe, Hao Tang, Yan Wang, Hao Zhao, Mengzhu Wang, Jingcai Guo
Low-rank Adaptation (LoRA) has demonstrated remarkable capabilities for task specific fine-tuning. However, in scenarios that involve multiple tasks, training a separate LoRA model for each one results in considerable inefficiency in terms of storage and inference. Moreover, existing parameter generation methods fail to capture the correlations among these tasks, making multi-task LoRA parameter generation challenging. To address these limitations, we propose In-Context Meta LoRA (ICM-LoRA), a novel approach that efficiently achieves task-specific customization of large language models (LLMs). Specifically, we use training data from all tasks to train a tailored generator, Conditional Variational Autoencoder (CVAE). CVAE takes task descriptions as inputs and produces task-aware LoRA weights as outputs. These LoRA weights are then merged with LLMs to create task-specialized models without the need for additional fine-tuning. Furthermore, we utilize in-context meta-learning for knowledge enhancement and task mapping, to capture the relationship between tasks and parameter distributions. As a result, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE. ICM-LoRA enables more accurate LoRA parameter reconstruction than current parameter reconstruction methods and is useful for implementing task-specific enhancements of LoRA parameters. At the same time, our method occupies 283MB, only 1\% storage compared with the original LoRA.
SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Technion, NVIDIA(技术学院与NVIDIA);University of Oxford(牛津大学);School of Electronics Engineering(电子工程学院);Computer Science, Peking University(计算机科学,北京大学)
Chinese Essay Rhetoric Recognition Using LoRA, In-context Learning and Model Ensemble
The Open University of China(中国开放大学);Engineering Research Center of Integration and Application of Digital Learning Technology, Ministry of Education(教育部数字学习技术整合与应用工程研究中心)
LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs
Department of Mathematics, McMaster University, Hamilton, Canada(麦斯特大学数学系);Vector Institute, Toronto, Canada(向量研究所);University of Oxford, Oxford, United Kingdom(牛津大学)
SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
Shanghai Qi Zhi Institute(上海启智研究院);Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Xiongan AI Institute(雄安人工智能研究院)
Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education(交通大数据与人工智能重点实验室(北京交通大学),教育部);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
University of Kurdistan(库尔德斯坦大学);Queen Mary University(女王玛丽大学);Torrens University Australia(澳大利亚托伦斯大学);University of Technology Sydney(悉尼技术大学);University of Kurdistan Sanandaj, Iran(伊朗桑和杰德大学);TU Dortmund University(多特蒙德技术大学);University of Kurdistan Hewler(库尔德斯坦大学海勒);Tehran University(德黑兰大学)
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data