面向个性化 LLM 的分割学习中的模型反演：来自信息瓶颈理论的新见解
Model Inversion in Split Learning for Personalized LLMs: New Insights from Information Bottleneck Theory
AI总结：
该研究首次揭示个性化 LLM 分割学习中的模型反演风险，基于互信息熵分析信息传播，并提出两阶段表示到嵌入投影与文本恢复攻击，性能较 SOTA 提升超 60%。
摘要：
个性化大语言模型（LLM）日益普及，展现了 GPT-4 等模型令人瞩目的能力。这一趋势也推动了在移动设备上部署 LLM 的广泛研究。此类边缘-云端部署的可行方法包括使用分割学习。然而，以往研究在很大程度上忽视了从设备传输到服务器的中间表示所带来的隐私泄露。本工作首次在面向 LLM 的分割学习框架中识别出模型反演攻击，并强调了安全防御的必要性。我们首次引入互信息熵来理解基于 Transformer 的 LLM 的信息传播，并评估 LLM 块的隐私攻击性能。为了解决表示比嵌入更稀疏且包含信息更少的问题，我们提出了一种两阶段攻击系统：第一部分将表示投影到嵌入空间，第二部分使用生成模型从这些嵌入中恢复文本。该设计分解了复杂性，并在多种场景下取得了 38%-75% 的攻击分数，较 SOTA 提升超过 60%。本工作全面揭示了在边缘侧部署个性化 LLM 过程中的潜在隐私风险。
英文摘要：
Personalized Large Language Models (LLMs) have become increasingly prevalent, showcasing the impressive capabilities of models like GPT-4. This trend has also catalyzed extensive research on deploying LLMs on mobile devices. Feasible approaches for such edge-cloud deployment include using split learning. However, previous research has largely overlooked the privacy leakage associated with intermediate representations transmitted from devices to servers. This work is the first to identify model inversion attacks in the split learning framework for LLMs, emphasizing the necessity of secure defense. For the first time, we introduce mutual information entropy to understand the information propagation of Transformer-based LLMs and assess privacy attack performance for LLM blocks. To address the issue of representations being sparser and containing less information than embeddings, we propose a two-stage attack system in which the first part projects representations into the embedding space, and the second part uses a generative model to recover text from these embeddings. This design breaks down the complexity and achieves attack scores of 38%-75% in various scenarios, with an over 60% improvement over the SOTA. This work comprehensively highlights the potential privacy risks during the deployment of personalized LLMs on the edge side.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
