arXiv 1802.082322018-02cs.LGcs.AIcs.CR

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Google Brain（谷歌大脑）；University of California, Berkeley（加州大学伯克利分校）；National University of Singapore（新加坡国立大学）

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song

英文摘要：

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Mitigating Unintended Memorization in Language Models via Alternating Teaching

Meta AI

Recent research has shown that language models have a tendency to memorize rare or unique sequences in the training corpora which can thus leak sensitive attributes of user data. We employ a teacher-student framework and propose a novel approach called alternating teaching to mitigate unintended memorization in sequential modeling. In our method, multiple teachers are trained on disjoint training sets whose privacy one wishes to protect, and teachers' predictions supervise the training of a student model in an alternating manner at each time step. Experiments on LibriSpeech datasets show that …

2022-10-13cs.CLcs.LG

Understanding Unintended Memorization in Federated Learning

Google LLC（谷歌有限责任公司）

Recent works have shown that generative sequence models (e.g., language models) have a tendency to memorize rare or unique sequences in the training data. Since useful models are often trained on sensitive data, to ensure the privacy of the training data it is critical to identify and mitigate such unintended memorization. Federated Learning (FL) has emerged as a novel framework for large-scale distributed learning tasks. However, it differs in many aspects from the well-studied central learning setting where all the data is stored at the central server. In this paper, we initiate a formal stu…

2020-06-12cs.CLcs.LGstat.ML

在微调语言模型中意外记忆敏感信息

Unintended Memorization of Sensitive Information in Fine-Tuned Language Models

TUM University Hospital（慕尼黑工业大学医院）；Technical University of Munich (TUM)（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Computing, Imperial College London（伦敦帝国学院计算系）

研究揭示微调语言模型时意外记忆敏感信息的风险，分析影响因素并评估隐私保护方法的权衡。

2026-01-24cs.AIcs.CLcs.LG

SSL模型存在“似曾相识”现象吗？自监督学习中非预期记忆效应的案例研究

Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning

UCSD（加州大学圣地亚哥分校）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；FAIR, Meta（Meta人工智能研究院）

该研究发现自监督学习（SSL）模型存在“似曾相识记忆”的非预期记忆现象，可通过背景裁剪推断/重建训练图像前景，该现象普遍存在于多种SSL算法中且无法被传统表征评估方法检测，揭示了SSL模型的隐私风险并提出缓解方向。

2023-12-13cs.CRcs.CVcs.LG

Unintended Memorization and Timing Attacks in Named Entity Recognition Models

Macquarie University（麦考瑞大学）

Named entity recognition models (NER), are widely used for identifying named entities (e.g., individuals, locations, and other information) in text documents. Machine learning based NER models are increasingly being applied in privacy-sensitive applications that need automatic and scalable identification of sensitive information to redact text for data sharing. In this paper, we study the setting when NER models are available as a black-box service for identifying sensitive information in user documents and show that these models are vulnerable to membership inference on their training dataset…

2022-11-04cs.AIcs.CRcs.LG

Detecting Unintended Memorization in Language-Model-Fused ASR

Google LLC（谷歌有限责任公司）

End-to-end (E2E) models are often being accompanied by language models (LMs) via shallow fusion for boosting their overall quality as well as recognition of rare words. At the same time, several prior works show that LMs are susceptible to unintentionally memorizing rare or unique sequences in the training data. In this work, we design a framework for detecting memorization of random textual sequences (which we call canaries) in the LM training data when one has only black-box (query) access to LM-fused speech recognizer, as opposed to direct access to the LM. On a production-grade Conformer R…

2022-06-28cs.CLcs.CRcs.LG

更多相关论文收起更多论文

在大语言模型联邦学习中使用LoRA缓解意外记忆

Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs

Tune Insight SA；EPFL（苏黎世联邦理工学院）；Yale University（耶鲁大学）

本文研究了在大语言模型联邦学习中使用LoRA技术有效降低模型对训练数据的记忆能力，并展示了其在不同领域和隐私保护技术中的应用效果。

2026-03-09cs.AIcs.CLcs.LG

大型 ASR 模型中的非预期记忆及其缓解方法

Unintended Memorization in Large ASR Models, and How to Mitigate It

Google（谷歌）

本研究提出一种低成本审计方法，通过加速语音生成检测大型 ASR 模型中的非预期记忆，并采用梯度裁剪技术有效缓解该问题，在保持模型质量的同时提供隐私保护。

2023-10-18cs.LGcs.SDeess.AS

On the Unintended Social Bias of Training Language Generation Models with Data from Local Media

Capital One（第一资本金融公司）

There are concerns that neural language models may preserve some of the stereotypes of the underlying societies that generate the large corpora needed to train these models. For example, gender bias is a significant problem when generating text, and its unintended memorization could impact the user experience of many applications (e.g., the smart-compose feature in Gmail). In this paper, we introduce a novel architecture that decouples the representation learning of a neural model from its memory management role. This architecture allows us to update a memory module with an equal ratio across …

2019-11-01cs.CLcs.LG

Privacy-Preserving Eye-tracking Using Deep Learning

Emory School of Medicine（埃默里医学院）；Georgia Institute of Technology（佐治亚理工学院）

The expanding usage of complex machine learning methods like deep learning has led to an explosion in human activity recognition, particularly applied to health. In particular, as part of a larger body sensor network system, face and full-body analysis is becoming increasingly common for evaluating health status. However, complex models which handle private and sometimes protected data, raise concerns about the potential leak of identifiable data. In this work, we focus on the case of a deep network model trained on images of individual faces. Full-face video recordings taken from 493 individu…

2021-06-22cs.CVcs.LG