arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11087cs.CRcs.AIcs.CLcs.LG
收藏

DP-MemArc:面向内存高效语言模型的差分隐私迁移学习

DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models

Zhejiang University(浙江大学);Southeast University(东南大学);Harvard University(哈佛大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Massachusetts Institute of Technology(麻省理工学院);Renmin University of China(中国人民大学);The University of Tokyo(东京大学);Tongji University(同济大学)

Yanming Liu, Xinyue Peng, Yuwei Zhang, Xiaolan Ke, Songhang Deng, Jiannan Cao, Chen Ma, Mengchen Fu, Tianyu Du, Sheng Cheng, Xun Wang, Jianwei Yin, Xuhong Zhang

AI总结:

本文提出 DP-MemArc 训练框架,通过侧网络或可逆网络支持多种差分隐私内存高效微调,在保障用户数据隐私的同时实现约 2.5 倍内存优化。

摘要:

大型语言模型已在各类应用中反复展现出卓越性能。然而,部署这些模型可能会在无意中危及用户隐私。训练期间巨大的内存需求在资源消耗方面构成了重大挑战。这种庞大的规模给内存资源带来沉重负担,引发了相当多的实际关切。本文提出 DP-MemArc,这是一种新颖的训练框架,旨在降低大型语言模型的内存成本,同时强调保护用户数据隐私。DP-MemArc 结合侧网络或可逆网络设计,以支持多种差分隐私内存高效微调方案。我们的方法不仅实现了约 2.5 倍的内存优化,还确保了强健的隐私保护,使用户数据保持安全和机密。大量实验表明,DP-MemArc 能够在不同任务场景中有效提供差分隐私高效微调。

英文摘要:

Large language models have repeatedly shown outstanding performance across diverse applications. However, deploying these models can inadvertently risk user privacy. The significant memory demands during training pose a major challenge in terms of resource consumption. This substantial size places a heavy load on memory resources, raising considerable practical concerns. In this paper, we introduce DP-MemArc, a novel training framework aimed at reducing the memory costs of large language models while emphasizing the protection of user data privacy. DP-MemArc incorporates side network or reversible network designs to support a variety of differential privacy memory-efficient fine-tuning schemes. Our approach not only achieves about 2.5 times in memory optimization but also ensures robust privacy protection, keeping user data secure and confidential. Extensive experiments have demonstrated that DP-MemArc effectively provides differential privacy-efficient fine-tuning across different task scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑