GradOT：面向大型语言模型的免训练梯度保持离站调优
GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models
AI总结：
针对集中式微调带来的隐私风险及现有离站调优（OT）方法计算成本高、缺乏理论分析的问题，提出基于梯度保持压缩的免训练方法GradOT，通过优化视角分析并选择性应用压缩技术，在隐私保护和性能上超越现有方法并提供理论基础。
摘要：
随着大型语言模型（LLMs）的快速发展，传统的集中式微调成为将这些模型适配到特定领域挑战的关键技术，但这给模型和数据所有者都带来了隐私风险。一种名为离站调优（OT）的有前景解决方案被提出以应对这些挑战，其中从原始模型中压缩出一个较弱的模拟器，并通过适配器进一步微调以增强隐私。然而，现有的基于OT的方法需要高昂的计算成本且缺乏理论分析。本文介绍了一种基于梯度保持压缩的新型OT方法，名为GradOT。通过从优化角度分析OT问题，我们提出了一种选择性应用秩压缩和通道剪枝等压缩技术的方法，在确保隐私的同时保留微调适配器的梯度。大量实验表明，我们的方法在隐私保护和模型性能方面均优于现有OT方法。我们的方法为OT提供了理论基础，并为大规模LLMs的离站调优提供了一种实用的免训练解决方案。
英文摘要：
The rapid growth of large language models (LLMs) with traditional centralized fine-tuning emerges as a key technique for adapting these models to domain-specific challenges, yielding privacy risks for both model and data owners. One promising solution, called offsite-tuning (OT), is proposed to address these challenges, where a weaker emulator is compressed from the original model and further fine-tuned with adapter to enhance privacy. However, the existing OT-based methods require high computational costs and lack theoretical analysis. This paper introduces a novel OT approach based on gradient-preserving compression, named GradOT. By analyzing the OT problem through the lens of optimization, we propose a method that selectively applies compression techniques such as rank compression and channel pruning, preserving the gradients of fine-tuned adapters while ensuring privacy. Extensive experiments demonstrate that our approach surpasses existing OT methods, both in terms of privacy protection and model performance. Our method provides a theoretical foundation for OT and offers a practical, training-free solution for offsite-tuning of large-scale LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
