arXiv 2501.139042025-02-13cs.LG

面向多模态大型语言模型的隐私保护个性化联邦提示学习

Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Research（IBM研究院）

Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态LLM联邦提示学习中个性化、泛化性与隐私难以平衡的问题，提出DP-FPL方法，通过低秩分解与差异化差分隐私保护策略，在减轻隐私噪声影响的同时平衡权衡，实验效果优于基准方法。

摘要：

多模态大型语言模型（Multimodal LLMs）通过整合文本、图像和音频等多种模态，在革新客户支持与运营方面发挥着关键作用。联邦提示学习（FPL）是一种近期提出的方法，它将视觉语言模型等预训练多模态LLM与联邦学习相结合，以构建个性化、隐私保护的AI系统。然而，平衡个性化、泛化性和隐私这三个相互竞争的目标仍是一项重大挑战。过度个性化可能导致过拟合，降低泛化能力；而严格的隐私措施（如差分隐私）则会阻碍个性化与泛化性。本文提出一种差分隐私联邦提示学习（DP-FPL）方法，通过低秩分解方案捕捉泛化性，同时保留一个残差项以维持个性化的表达能力。为确保隐私，我们引入一种新方法：对本地提示的两个低秩组件应用本地差分隐私，对全局提示应用全局差分隐私。该方法减轻了隐私噪声对模型性能的影响，同时平衡了个性化与泛化性之间的权衡。大量实验表明，我们的方法优于其他基准方法。

英文摘要：

Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。