arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.09130cs.AIcs.CR
收藏

Split-and-Denoise:利用本地差分隐私保护大型语言模型推理

Split-and-Denoise: Protect large language model inference with local differential privacy

National University of Singapore(新加坡国立大学);North China Electric Power University(华北电力大学);University of South California(南加利福尼亚大学)

Peihua Mai, Ran Yan, Zhe Huang, Youjia Yang, Yan Pang

AI总结:

本文提出Split-N-Denoise(SnD)隐私推理框架,通过在客户端执行嵌入层并引入噪声,在服务器端处理后去噪,有效保护LLM推理隐私,在相同隐私预算下性能平均提升超10%。

摘要:

大型语言模型(LLMs)通过在向量空间中捕获隐藏语义,在自然语言理解方面表现出色。这一过程丰富了文本嵌入在各种下游任务中的价值,从而促进了嵌入即服务(EaaS)商业模式的发展。然而,由于将文本直接传输到服务器而导致的隐私泄露风险仍然是一个关键问题。为了解决这一问题,我们引入了 Split-N-Denoise(SnD),这是一个隐私推理框架,它将模型拆分,以最小的计算成本在客户端执行 token 嵌入层。这使得客户端能够在将嵌入传输到服务器之前引入噪声,并随后接收并对受扰动的输出嵌入进行去噪以用于下游任务。我们的方法专为 LLMs 的推理阶段设计,无需修改模型参数。大量实验表明,SnD 在优化各种 LLM 架构和不同下游任务的隐私-效用权衡方面具有有效性。结果显示,在相同的隐私预算下,与基线相比,性能平均提升了超过 10%,为客户提供了一种用于本地隐私保护的隐私保护解决方案。

英文摘要:

Large Language Models (LLMs) excel in natural language understanding by capturing hidden semantics in vector space. This process enriches the value of text embeddings for various downstream tasks, thereby fostering the Embedding-as-a-Service (EaaS) business model. However, the risk of privacy leakage due to direct text transmission to servers remains a critical concern. To address this, we introduce Split-N-Denoise (SnD), an private inference framework that splits the model to execute the token embedding layer on the client side at minimal computational cost. This allows the client to introduce noise prior to transmitting the embeddings to the server, and subsequently receive and denoise the perturbed output embeddings for downstream tasks. Our approach is designed for the inference stage of LLMs and requires no modifications to the model parameters. Extensive experiments demonstrate SnD's effectiveness in optimizing the privacy-utility tradeoff across various LLM architectures and diverse downstream tasks. The results reveal an improvement in performance under the same privacy budget compared to the baselines by over 10\% on average, offering clients a privacy-preserving solution for local privacy protection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑