基于截断拉普拉斯机制的私有语言模型
Private Language Models via Truncated Laplacian Mechanism
AI总结：
针对NLP模型在高隐私区域下差分隐私保护效用不佳的问题，提出高维截断拉普拉斯机制进行私有词嵌入，理论上降低方差并在多数据集实验中保持高隐私下仅轻微效用下降。
摘要：
用于NLP任务的深度学习模型容易受到各种隐私攻击的变体影响。为了防止隐私泄露，研究人员研究了词级扰动，依赖于嵌入空间中差分隐私（DP）的形式化保证。然而，许多现有方法要么在使用Laplacian或Gaussian机制时在高隐私区域表现不佳，要么采用较弱的DP松弛，在隐私强度上不如标准DP。这引发了一个问题：是否可以设计一种新的私有词嵌入方法来克服这些限制。在本文中，我们提出了一种名为高维截断拉普拉斯机制的新型私有嵌入方法。具体而言，我们引入了截断拉普拉斯机制的非平凡扩展，该机制此前仅在低维空间情形下被研究过。理论上，我们证明了与之前的私有词嵌入方法相比，我们的方法具有更低的方差。为了进一步验证其有效性，我们在三个数据集上进行了私有嵌入和下游任务的全面实验。值得注意的是，即使在高隐私区域，我们的方法与非隐私场景相比也仅导致效用的轻微下降。
英文摘要：
Deep learning models for NLP tasks are prone to variants of privacy attacks. To prevent privacy leakage, researchers have investigated word-level perturbations, relying on the formal guarantees of differential privacy (DP) in the embedding space. However, many existing approaches either achieve unsatisfactory performance in the high privacy regime when using the Laplacian or Gaussian mechanism, or resort to weaker relaxations of DP that are inferior to the canonical DP in terms of privacy strength. This raises the question of whether a new method for private word embedding can be designed to overcome these limitations. In this paper, we propose a novel private embedding method called the high dimensional truncated Laplacian mechanism. Specifically, we introduce a non-trivial extension of the truncated Laplacian mechanism, which was previously only investigated in one-dimensional space cases. Theoretically, we show that our method has a lower variance compared to the previous private word embedding methods. To further validate its effectiveness, we conduct comprehensive experiments on private embedding and downstream tasks using three datasets. Remarkably, even in the high privacy regime, our approach only incurs a slight decrease in utility compared to the non-private scenario.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
