This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks to address the gap between pre-trained and deployed models. In our approach, trainable LoRA (low-rank adaptation) adapters are inserted between the last layer and every other layer to enhance the network expressive power while keeping the backward computation cost low. This architecture is well-suited to cache intermediate computation results of the forward pass and then can skip the forward computation of seen samples as training epochs progress. We implemented the combination of the proposed architecture and cache, denoted as Skip2-LoRA, and tested it on a $15 single board computer. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters while preserving the accuracy, while taking only a few seconds on the microcontroller board.
PRISP: Privacy-Safe Few-Shot Personalization via Lightweight Adaptation
Department of Electrical and Computer Engineering, Seoul National University(电子与计算机工程系,首尔国立大学);ASRI/INMC/IPAI/AIIS, Seoul National University(ASRI/INMC/IPAI/AIIS,首尔国立大学)
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动的人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)
A Lightweight LLM Framework for Disaster Humanitarian Information Classification
Department of Civil, Architectural & Environment Engineering, Sungkyunkwan University(苏州市立大学土木、建筑与环境工程系);School of Geography, University of Leeds(利兹大学地理学院)
Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
School of Computer Science and Software Engineering, Southwest Petroleum University(西南石油大学计算机科学与软件工程学院);School of Electronic Information and Artificial Intelligence, Leshan Normal University(乐山师范学院电子信息与人工智能学院)
Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation
Tsinghua University(清华大学);Bauhaus-Universität Weimar(魏玛包豪斯大学);Leibniz University Hannover(汉诺威莱布尼茨大学);CNPC Engineering Technology RD Company Limited(中国石油集团工程技术研究院有限公司);Queensland University of Technology(昆士兰科技大学)
CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
Cornell University(康奈尔大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Oak Ridge National Laboratory(橡树岭国家实验室);Yale University(耶鲁大学);University of California, Irvine(加州大学欧文分校);Tulane University(路易斯安那大学塔兰帕伊斯分校)
LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta(计算科学系及阿尔伯塔人工智能研究所(Amii),阿尔伯塔大学);RBC Borealis;Canada CIFAR AI Chair(加拿大CIFAR人工智能 chair)
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA;Los Alamos National Laboratory Los Alamos New Mexico USA;Tennessee Tech University Cookeville Tennessee USA;Tennessee Tech University, \ Alamos National Laboratory;Los Alamos National Laboratory;Tennessee Tech University
Quantum-Inspired Fine-Tuning for Few-Shot AIGC Detection via Phase-Structured Reparameterization
Anhui University, Hefei, China(安徽大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China(人工智能研究院);National University of Singapore, Singapore, Singapore(新加坡国立大学);University of Science and Technology of China, Hefei, China(中国科学技术大学)
Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints
Beijing Institute of Technology, Zhuhai(北京理工大学珠海学院);The Hong Kong Polytechnic University(香港理工大学);The University of Osaka(大阪大学);Shenzhen University(深圳大学)
Rapid LoRA Aggregation for Wireless Channel Adaptation in Open-Set Radio Frequency Fingerprinting
School of Internet of Things, Nanjing University of Posts and Telecommunications(南京邮电大学物联网学院);School of Cyber Science and Engineering, Southeast University(东南大学网络空间安全学院);National Mobile Communications Research Lab, Southeast University(东南大学国家移动通信研究中心)
LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs
Department of Mathematics, McMaster University, Hamilton, Canada(麦斯特大学数学系);Vector Institute, Toronto, Canada(向量研究所);University of Oxford, Oxford, United Kingdom(牛津大学)
Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
Zhejiang University(浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新区(滨江)区块链与数据安全研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);National University of Singapore(新加坡国立大学);Guangdong OPPO Mobile Telecommunications Corp., Ltd.(广东欧珀移动通信有限公司)
A Lightweight Transmission Parameter Selection Scheme Using Reinforcement Learning for LoRaWAN
The University of Electro-Communications(电气通信大学);Tokyo University of Science(东京理科大学);National Institute of Information and Communication Technology (NICT)(信息通信研究机构(NICT));The University of Tokyo(东京大学);SOBIN Institute LLC(SOBIN研究所有限责任公司)