arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.05973cs.CLcs.CRcs.IR
收藏

用于隐私保护深度检索系统的差分隐私语言模型合成查询生成

Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models

Stanford University(斯坦福大学);Google Inc.(谷歌公司);Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind)

Aldo Gael Carranza, Rezsa Farahani, Natalia Ponomareva, Alex Kurakin, Matthew Jagielski, Milad Nasr

AI总结:

针对深度检索系统直接差分隐私训练困难的问题,提出先用差分隐私语言模型生成私有合成查询,再用于训练,显著提升检索质量并保持隐私保证。

摘要:

我们解决了在训练深度检索系统中确保差分隐私(DP)保证的挑战。训练这些系统通常涉及使用对比式损失函数,这类损失通常不可按样本分解,因此难以直接进行DP训练,因为常见技术需要逐样本梯度。为了解决这个问题,我们提出了一种方法,优先在训练深度检索系统之前确保查询隐私。我们的方法使用DP语言模型(LMs)生成代表原始数据的私有合成查询。这些合成查询可以在不损害隐私的情况下用于下游检索系统的训练。与直接DP训练相比,我们的方法显著提升了检索质量,同时保持了查询级别的隐私保证。这项工作凸显了利用语言模型克服标准DP训练方法局限性的潜力。

英文摘要:

We address the challenge of ensuring differential privacy (DP) guarantees in training deep retrieval systems. Training these systems often involves the use of contrastive-style losses, which are typically non-per-example decomposable, making them difficult to directly DP-train with since common techniques require per-example gradients. To address this issue, we propose an approach that prioritizes ensuring query privacy prior to training a deep retrieval system. Our method employs DP language models (LMs) to generate private synthetic queries representative of the original data. These synthetic queries can be used in downstream retrieval system training without compromising privacy. Our approach demonstrates a significant enhancement in retrieval quality compared to direct DP-training, all while maintaining query-level privacy guarantees. This work highlights the potential of harnessing LMs to overcome limitations in standard DP-training methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑