arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05012cs.LG
收藏

Private PoEtry: 通过专家产品实现隐私保护的上下文学习

Private PoEtry: Private In-Context Learning via Product of Experts

University of Amsterdam(阿姆斯特丹大学);QUvA lab(QUvA实验室);Qualcomm Inc.(高通公司);University of Technology Nuremberg(努尔堡技术大学)

Rob Romijnders, Mohammad Mahdi Derakhshani, Jonathan Petit, Max Welling, Christos Louizos, Yuki M. Asano

AI总结:

本文提出通过专家产品模型实现隐私保护的上下文学习,有效提升准确率并保持隐私保障。

摘要:

上下文学习(ICL)使大语言模型(LLMs)能够在推理时仅使用少量示例来适应新任务,从而避免任务特定的微调。然而,上下文中的示例可能包含敏感隐私信息,不应通过模型输出暴露。现有的差分隐私(DP)方法用于ICL要么计算成本高,要么依赖启发式方法,效果有限,包括上下文过采样、合成数据生成或不必要的阈值处理。我们通过专家产品模型的视角重新表述隐私ICL。这给出了一个理论基础的框架,算法可以简单地并行化。我们在文本分类、数学和视觉语言五个数据集上评估了我们的方法。我们发现,与先前的DP-ICL方法相比,我们的方法在平均上提高了超过30个百分点的准确率,同时保持了强大的隐私保障。

英文摘要:

In-context learning (ICL) enables Large Language Models (LLMs) to adapt to new tasks with only a small set of examples at inference time, thereby avoiding task-specific fine-tuning. However, in-context examples may contain privacy-sensitive information that should not be revealed through model outputs. Existing differential privacy (DP) approaches to ICL are either computationally expensive or rely on heuristics with limited effectiveness, including context oversampling, synthetic data generation, or unnecessary thresholding. We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized. We evaluate our method across five datasets in text classification, math, and vision-language. We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑