arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01394cs.CRcs.AI
收藏

PrivacyRestore:通过隐私移除与恢复实现大语言模型中的隐私保护推理

PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration

South China University of Technology(华南理工大学);Pazhou Laboratory(琶洲实验室);Hong Kong University of Science and Technology(香港科学与技术大学)

Ziqian Zeng, Jianwei Wang, Junyao Yang, Zhengdong Lu, Haoran Li, Huiping Zhuang, Cen Chen

AI总结:

针对LLM推理中隐私保护不足、性能下降或开销大的问题,提出即插即用的PrivacyRestore方法,通过服务器训练并发布恢复向量,客户端聚合后发送至服务器,在推理时利用激活引导恢复隐私,有效保护隐私且保持可接受的性能与开销。

摘要:

在线大语言模型(LLM)推理服务的广泛使用引发了重大的隐私担忧,即用户输入中的私人信息可能会暴露给恶意窃听者。现有的LLM隐私保护方法存在隐私保护不足、性能下降或推理时间开销大的问题。为了解决这些局限性,我们提出了PrivacyRestore,这是一种即插即用的方法,用于在LLM推理过程中保护用户输入的隐私。服务器首先为每个隐私片段训练恢复向量,然后将其发布给客户端。隐私片段被定义为文本中包含私人信息的连续标记序列。客户端随后将输入中所有隐私片段的恢复向量聚合为一个单一的元恢复向量,随后将其与移除隐私片段的输入一起发送到服务器端。在推理过程中,通过激活引导恢复私人信息。此外,我们证明了PrivacyRestore本质上防止了隐私预算的线性增长。我们创建了三个数据集,涵盖医疗和法律领域,以评估隐私保护方法的有效性。实验结果表明,PrivacyRestore有效保护了私人信息,并保持了可接受的性能和推理开销水平。

英文摘要:

The widespread usage of online Large Language Models (LLMs) inference services has raised significant privacy concerns about the potential exposure of private information in user inputs to malicious eavesdroppers. Existing privacy protection methods for LLMs suffer from either insufficient privacy protection, performance degradation, or large inference time overhead. To address these limitations, we propose PrivacyRestore, a plug-and-play method to protect the privacy of user inputs during LLM inference. The server first trains restoration vectors for each privacy span and then release to clients. Privacy span is defined as a contiguous sequence of tokens within a text that contain private information. The client then aggregate restoration vectors of all privacy spans in the input into a single meta restoration vector which is later sent to the server side along with the input without privacy spans.The private information is restored via activation steering during inference. Furthermore, we prove that PrivacyRestore inherently prevents the linear growth of the privacy budget.We create three datasets, covering medical and legal domains, to evaluate the effectiveness of privacy preserving methods. The experimental results show that PrivacyRestore effectively protects private information and maintain acceptable levels of performance and inference overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑