arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.21073cs.LGcs.AI
收藏

Skip2-LoRA:一种面向低成本边缘设备的轻量级端侧 DNN 微调方法

Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices

Keio University(庆应义塾大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Hiroki Matsutani, Masaaki Kondo, Kazuki Sunaga, Radu Marculescu

AI总结:

该研究提出 Skip2-LoRA,通过隔层插入 LoRA 适配器并缓存前向中间结果,在低成本边缘设备上实现保持准确率且平均减少 90.0% 微调时间。

摘要:

本文提出 Skip2-LoRA,将其作为一种用于深度神经网络的轻量级微调方法,以解决预训练模型与部署模型之间的差距。在我们的方法中,可训练的 LoRA(低秩适配)适配器被插入到最后一层与每隔一层之间,以增强网络表达能力,同时保持较低的反向传播计算成本。该架构非常适合缓存前向传播的中间计算结果,因此随着训练轮次推进,可以跳过已见过样本的前向计算。我们实现了所提出架构与缓存的结合,记为 Skip2-LoRA,并在一台售价 15 美元的单板计算机上进行了测试。结果表明,与具有相同数量可训练参数的对照方法相比,Skip2-LoRA 在保持准确率的同时,将微调时间平均缩短 90.0%,并且在微控制器板上只需几秒钟。

英文摘要:

This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks to address the gap between pre-trained and deployed models. In our approach, trainable LoRA (low-rank adaptation) adapters are inserted between the last layer and every other layer to enhance the network expressive power while keeping the backward computation cost low. This architecture is well-suited to cache intermediate computation results of the forward pass and then can skip the forward computation of seen samples as training epochs progress. We implemented the combination of the proposed architecture and cache, denoted as Skip2-LoRA, and tested it on a $15 single board computer. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters while preserving the accuracy, while taking only a few seconds on the microcontroller board.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑